Test-time Alignment of Diffusion Models without Reward Over-optimization
Authors: Sunwoo Kim
Minkyu Kim
Dongmin Park
Venue: ICLR 2025 Spotlight (2025)
Keywords: diffusion models
alignment
reward over-optimization
sequential monte carlo samplers
Primary area: generative models

Abstract: Diffusion models excel in generative tasks, but aligning them with specific objectives while maintaining their versatility remains challenging. Existing fine-tuning methods often suffer from reward over-optimization, while approximate guidance approaches fail to optimize target rewards effectively. Addressing these limitations, we propose a training-free, test-time method based on Sequential Monte Carlo (SMC) to sample from the reward-aligned target distribution. Our approach, tailored for diffusion sampling and incorporating tempering techniques, achieves comparable or superior target rewards to fine-tuning methods while preserving diversity and cross-reward generalization. We demonstrate its effectiveness in single-reward optimization, multi-objective scenarios, and online black-box optimization. This work offers a robust solution for aligning diffusion models with diverse downstream objectives without compromising their general capabilities. Code is available at https://github.com/krafton-ai/DAS.

Published as a conference paper at ICLR 2025
TEST-TIME ALIGNMENT OF DIFFUSION MODELS WITHOUT REWARD OVER-OPTIMIZATTON
Sunwoo Kim Minkyu Kim? Dongmin Park
Scoul National University 2 KRAFTON
ksunw02090 ac. (minkyu.king dongmii...arrt/@rraattooo.cmm
ABSTRACT
Diffusion models excel in generative tasks, but aligning them with specific objec- tives while maintaining their versatility remains challenging. Existing fine-tuning methods often suffer from reward over-optimization, while approximate guidance approaches fail to optimize target rewards effectively. Addressing these limita- tions, we propose a training-free, test-time method based on Sequential Monte Carlo (SMC) to sample from the reward-aligen wwward-aligned target distribution. Our approach, tailored for diffusion sampling and incorporating tempering techniques, achieves comparable or superior target rewards to fine-tuning methods while preserving diversity and cross-reward generalization. We demonstrate its effectiveness in single-reward optimization, multi-objective scenarios, and online black-box opti- mization. This work offers robust solution for aligning diffusion models with diverse downstream objectives without compromising their general capabilities. Code is available at https: /gi thub ctt. www./raaaonoaai/aa....
INTRODUCTION
Diffusion models (Sohl-Dickstein et al. 2015) Ho et al. 2020 Song et al. 2021c Park et al. 2024)
have revolutionized generative AI, excelling in tasks from Text-to-Image Image (T21) generation (Rom-) bach et al., 2022) to protein structure design (Watson et al. 2023). However, diffusion models are typically pre-trained on large uncurated datasets that may not accurately represent the desired target distribution. For instance, in the T2I generation, real users want to produce aesthetically pleasing images while faithful to prompt instructions, rather than generating random internet images from the pre-trained dataset. Also, one might want to produce only specific cartoon character images, rather than general styles. These challenges underscore the importance of alignment, process to adapt diffusion models for specific customized rewards.
Existing alignment approaches mainly fall into categories: (1) fine-tuning and (2) guidance
methods. Fine-tuning approaches, including Reinforcement Learning (RL) (Fan et al. 2024: Black
al. 2023) and direct backpropagation (Clark al. 2024 Prabhudesai et al. 2024). ) have shown promising results in optimizing target rewards. However, it often suffers from the reward over- optimization problem, sacrificing general image quality and diversity (Clark al. 2024 Gao al.) 2022). On the other hand, guidance methods (Bansal al., 2023 Yu et al. 20233 Song et al. [023 He et al. 2024) offer a training-free alternative that stays closer to the pre-trained model distribution. Mcanwhile, they suffer from the reward
problem, failing to effectively optimize target rewards due to relying on estimated inference-time conrections of the generation process.
To address these limitations, propose Diffusion Alignment Sampling (DAS), test-time al- gorithm that both achieves effective reward alignment and preserves model generalization. Our approach leverages Sequential Monte Carlo (SMC) sampling (Del Moral et al. 2006) enable high-reward diffusion guidance by evaluating multiple possible solutions in parallel and selecting the best candidates during the generation process. By carefully designing intermediate target dis- tributions with tempering techniques, DAS achieves high sample efficiency in terms of required candidates, as we demonstrate both theoretically and empirically.
"This work was done during an internship at KRAFTON. Corrrsponniig author.

---

Published as a conference paper at ICLR 2025
To validate its effectiveness for optimizing target reward without over-optimization, apply DAS to Stable Diffusion v1.5 (Rombach et al. 2022). targeting aesthetic reward, e.g., LAION aesthetic score (Schuhmann 2022) and human preference, e.g., PickScore (Kirstain al.] 2023). Without the computational burden of training or extensive hyperparameter tuning, DAS outperforms all fine- tuning baselines in target scores, while not sacrificing cross-reward generalization and output diversity. This superiority is consistent with a more recent diffusion backbone, SDXL (Podell et al.) 2024), and more complex reward functions such as CLIPScore (Hessel et al.) 2021) and Compress- ibility (Black et al. 2023). We further demonstrate the efficacy of DAS in multi-objection mization, achieving a new Pareto front when jointly optimizing CLIPScore (Hessel et al. (202T) and aesthetic score. Moreover, the diverse sampling ability is especially beneficial in online settings with limited reward queries. such a difficult scenario, while existing methods drop the scores of unseen rewards due to severe over-optimization, DAS improves the pre-trained T2I model by up to 20% in both target and unseen rewards.
opti-
In summary, our main contributions are:
We propose DAS, test-time method for aligning diffusion models with arbitrary rewards while preserving general capabilities.
We provide theoretical analysis of DAS's asymptotic properties, proving the benefits of tempering in SMC sampling for diffusion models.
We empirically validate DAS's effectiveness across diverse scenarios, including single-reward, multi-objective, and online black-box optimization tasks.
RELATED WORK
2.1 FINE-TUNING DIFFUSION MODELS FOR ALIGNMENT
Aligning pretrained models through fine-tuning has been extensively studied in language models
ZZegler et al.] 2020 Ouyang et al. 2022 Rafailov et al. 2023). For diffusion models, several approaches have emerged. Lee et al.
(2023) and Wu al. 2023b) employ supervised fine-tuning with preference-basee reward models. Black et al. (2023) and Fan et al. (2024) formulate sampling as a Markov decision process and apply reinforcement learning (RL) to maximize rewards. [u
iet al. (2024); Clark al. (2024) and Prabbuuesaai al. (222)) fine-tune by direct backpropagation through differentiable reward models. These approaches, however, face challenges with reward
over-optimization (Gao et al. 2022 Coste et al. 2024), which may distort alignment or reduce sample diversity. KL regularization has been proposed as a mitigation strategy (Fan et al. 2024
Uehara et al.] 2024a). inspired by its success in language models (Stiennon al. 20200 Ouyang et al. [022] Korbak et al. 2022). Section exammies the limitations of this approach, focusing on the mode-seeking behavior observed in the context of variational inference. While diffusion-based samplers (Zhang & |222 Vargas et al. 2023 Berner et al.) | Sanokowski et al. (0224) use similar training objective to sample from multimodal, unnormalized target density, the fine-tuning setup makes training more susceptible to mode collapse (Appendix FF. Alternatively, Zhang et al. (2024) approached over-optimization in RL fine-tuning through inductive and primacy biases.
2.2 GUIDANCE METHODS
Building on the score-based formulation of diffusion models (Song et al. 202 cc), various guidance
methods have been developed. While classifier guidance (Dhariwal & Nichol 102 requires addi-
tional training, recent works approximate guidance to use off-the-sheli classifiers or reward models
directly (Ho et al. 2022, Song et al., 2022, Chung et al. 2023 Bansal al. 2023 Yu et 2023
Song et al.] 2023 Yoon et al. 2023 He et al.) 2024). These methods rely on Tweedie's 's formula [Efron[[00) 1 Chung et al.) (023) for prediction of the original data given noisy data, but our exper- iments indicate that such inaccurate prediction limits effectiveness in maximizing complex rewards.
In contrast, SMC methods have been applied to address inexactness (Trippe et al. 2023 Cardoso et al. 2024 Wu et al. 2023a Dou & Song 2024) or bypass the need of calculating guidance (Li et al. 2024). While SMC methods offer asymptotic exactness, naive applications may fail to sample from complex targets within finite samples due to inefficiency. Our approach incorporates tem- pered SMC sampler to enhance sample efficiency, achieving comparable or superior performance to fine-tuning methods without additional training.
2

---

Published as a conference paper at ICLR 2025
3 DIFFUSION ALIGNMENT AS SAMPLING (DAS)
This section formulates the diffusion alignment problem as sampling from a reward-aligned distri- bution, examines limitations of existing methods, and introduces DAS, Sequential Monte Carlo (SMC) based algorithm with theoretical guarantees for asymptotic exactness and sample efficiency.
3.1 PROBLEM SETUP: ALIGNING DIFFUSION MODELS WITH REWARDS
Aligning diffusion models with rewards can be seen as finding new distribution that maximizes the expectation given reward r. Formally, it can be written as solving:
Per arg max, Max,Eex-/((()))
(1)
However, this approach may lead to reward over-optimization (Gao et al. 2022), disregarding the pre-trained distribution. To mitigate this, employ KL regularization (Uehara et al. 2024a):
Par = arg max, (raxx - a <D. (pllppre
where Ppre is the sample distribution of the pre-trained diffusion model. Following Rafailov et al. 2023). it straightforward to show that the target distribution can be written in an equivalent form:
r(x)
Puar (x) = = Ppre exp
where Z is normalization constant. We frame the diffusson alignment problem sampling from this reward-aligned target distribution Ptar- Note, however, that we only have access to an unnormalized density of Ptar and its evaluation requires running probability flow ODE (Song al. 2021c), even for a single sample, making the sampling problem highly non-trivial.
Before continue, introduce binary optimality variable O E 10, with p(O 1|x) x exp(r(x)/ax) where samples with high reward are interpreted as more likely optimal. Then the pos- terior p(z 1) characterizes the distribution samples that achieve high rewards. Using Bayes' rule with prior p Ppre give p(z|O) = 1) & p(z)p(O = 1|x) = Ppre (x) exp (r(z)/a) ε Pear (I), revealing the equivalence between two perspectives. We omit '=1' hereafter following convention.
3.2 LIMITATIONS OF EXISTING METHODS
Previous approaches to sampling from the target distribution (Fquation 3 primarily fall into two categories: fine-tuning and direct sampling using approximate guidance. Here, we first demonstrate how these approaches struggle to sample from multimodal target distributions, even for simple Gaus- sian mixtures, and explain their limitations leading to potential failures.
Figure T Jillustrates the failure modes of two approaches. agation) fail to fit all modes the multimodal target distribution Par. depicting their mode-seeking behavior. Approximate guidance results in low rewards, failing to effectively optimize target ward, portraying the inexactness of the guidance. In contrast, our SMC-hased method successfully samples from all modes, achieving the lowest Earth Mover's Distance between Ptar with high reward.
methods (RL, direct backprop-
We first investigate the source of the mode-seeking behavior of fine-tuning methods. Fine-tuning methods can be seen as variational inference, with the following objective (Rafailov al. (2023)):
minimize DKL (pelllP (pellPur).
0
This can be optimized using reinforcement learning (RL) Fan al. 2024) or direct backpropaga- tion (Uchara et al.) 2024a). However, the mode-seeking behavior of reverse KL divergence (Chan jt al. 2022 Wang et al.) 2023) may cause the model fit only the modes of the target distribution, especially when Ptar is multimodal (See Figure (T). This connects to low diversity of fine-tuning methods, which we further demonstrate Section 4 for real-world examples.
Next, we turn to approximate guidance methods. If the exact score function of the posteriors
Zz, log V., logp((z)) + vz, log p( (())))
(5)
known, one can use reverse diffusion for generation (Song et al. 2021c), where marginal pr (xx) and conditional distribution (2) is defined by the forward diffusion process. However, to
3

---

Published as a conference paper at ICLR 2025
Pre-trained Distribution
Target Distribution
Direct Backprop
Approx. Guidance
DAS (Ours)
RL
Reward: -10.40
Reward: -0.29
Reward: -0.07 EMD: 1.58
Reward: -0.96
Reward: -138
Reward: -0.22
EMD: 3.69
EMD: 15.49
EMD: 0.82
Reward:
Reward: -1.34
Reward: -1.04 EMD: 1.44
Reward: -2.43 EMD: 1.34
Reward: -407 EMD: 6.73
Reward: -1.44 EMD: 1.4
Figure SMC method excels in sampling from the target distribution compared existing approaches. Left of dashed line: Samples from pre-trained model trained on mixture of Gaussians, reward-aligned target distribution Ptar- Right of dashed line: methods for sampling from Ptar cluding previous methods (RL, direct backpropagation. approximate guidance) and ours using SMC. Top: reward r(X, Y) = -X2/100 /100 - Y bottom: reward r(X,Y) = -x (Y - /10. EMD denotes sample estimation of Earth Mover's 's Distance. Our SMC-based method outperforms existing approaches in capturing multimodal target distributions, as evidenced by lower EMD and successful sampling from all modes. Note that samples may exist outside the grid.
p(O pp)rop dra. line of works (Chung ctt al 2023 Yu et al. 2023 Bansal et al. 2023 Song et al. (203) rely on the approximation
sidestep the intractable integration p(O ()))
p(O)z,) 21) p(aola (a,, [p]
E [e] is given by the Tweedie's Chung et al. 2023 Finally, replacing p p(Olo)) exp (r(zo)/a), approxi-
To xt. ((C) 20
(2t)) where do == E
formula (Efron 2011:
mate guidance i8 given as
logr(O)z.) 1 -V.,, f(xx),
(6)
where f(.) == r())(-)). However, predicting clean data with noise causes errors, especially early in sampling when the noise is large, making difficult to sample exactly from Par (He et 2024).
3.3 SAMPLING FROM REWARD-ALIGNER TARGET DISTRIBUTION VIA TEMPERED SMC
Fine-tuning often leads to over-optimization. while approximate guidance methods struggle with reward optimization. To improve approximate guidance, we can use multiple candidate latents (par- ticles) during sampling, selecting those with high predicted rewards that stay close the pre- prained diffusion model's distribution. This approach leverages Sequential Monte Carlo (SMC) methods, which take incremental guided steps rather than sampling directly from the target distribution. At each diffusion step, the process evaluates and resamples candidates based on both reward scores and alignment with the pre-trained model, ultimately producing samples that satisfy both criteria.
Traditional SMC typically requires thousands of particles, making it computationally expensive for diffusion models. However, by employing techniques like tempering, we achieve high-quality, reward-aligne samples with fewer particles, making the method practical for real-world applica- tions. To formally describe our approach, we first outline the key design choices of SMC samplers that enable this guided sampling process:
Sequence of intermediate target distributions (x() (zz)/ for t = 0 T that bridge between the prior कy and target distribution Tp. where 7t is unnormalized density of π.
Backward kernels (x. which define intermediate joint distributions
(xer)
(x+)
L. (2,,,1,11)) -=11
II
"Backward respect sampling procedure, which is the time direction with forward diffusion process.
4

---

Published as a conference paper at ICLR 2025
Proposals, transition kernels m (2r) for sequential sampling.
Weights
Ht T (F)- &
(0-1(1(11 Te(
Wt (1 == Te (Zerr) m (1(-1
(8)
7e(a)) me-1 (x2-121)))
where proposed particles are resampled from Multinomial(2 L:N w(
A more detailed and theoretical introduction to SMC can be found in Appendix B
3.3.1 BACKWARD KERNEL
To incorporate pre-trained diffusion models, we define the backward kernel using Bayes' rule with general stochastic diffusion samplers. For any stochastic diffusion sampler po (6(z-11(1)) N (H t), I).. we define the backward kernels as:
Le Le(xe((1-1 = Po( 1 (pre
(9)
(1-1(1111)
This formulation also serves as an approximation for general non-Markovian forward processes given pre-trained generation processes (Song et al. 202
3.3.2 INTERMEDIATE TARGETS: APPROXIMATE POSTERIOR WITH TEMPERING
As stated in section 3.2 sampling from the target p(zo| O) requires score functions of the true pos-
teriors pe (xv, (O). Instead, approximate guidance gives a score function of an alternative distribution,
which we refer to as the approximate posterior:
(((1))
(z. x pi(zz) & (z)) exp
(10)
However, we can't sample even from these approximate posteriors since they are not defined by any forward diffusion process anymore. Nevertheless, this approximate posterior becomes exact == O as at = Zp, thus defining sequence of distributions interpolating Pr = (0, I) and Pur which can be incorporated as intermediate targets for SMC sampler. Since prediction dn gets more accurate as goes to O, the approximate posteriors get closer to the true posteriors while the error may be large at the beginning of sampling. Hence, we add tempering for intermediate targets as:
Xe (()) (e()),)
(x(x)) (O( px(x)) exp
(11)
which can interpolate गт PT To Ptar more smoothly where 0 dT AT ... S Ao= =1 sequence of inverse temperature parameters.
While modern SMC samplers often use adaptive tempering (Chopin & Papaspiliopoulos] 2020
Murphy, 2023), we find out simply setting A = (1 + 7) works well in our setting where hyperparameter. In Section 4.1 compare different tempering schemes and explain how to select y. To the best of our knowledge, this adaptation density tempering is novel among works applying SMC methods to diffusion sampling.
3.3.3 PROPOSAL: APPROXIMATING LOCALLY OPTIMAL PROPOSAL
Given the backward kernels and intermediate targets, derive the locally optimal proposal that minimizes the variance of the weights. Minimizing weight variance ensures more uniform impor- tance among particles, thereby enhancing sample efficiency.
Proposition 1 (Locally Optimal Proposal). The locally optimal proposal mi_ (xt-1 that mini- mizes the conditional variance Var(w,- (x,-11 2.)) is given by
(-
p(8-1))
mf (rt) x exp
|| mo t)
(12)
20
a
proof. The full proof can be found in Appendix
C.I.I
5

---

Published as a conference paper at ICLR 2025
Since sampling from m² is non-trivial, we adapt Gaussian approximation of m" our proposal:
(1-1(x)-1 = N [Ho(ze; +
A
(13)
a
where we used first-order Taylor approximation r =( (V ((0g ar)) and io(:, () Rs fo(-:4 ( B
Tempering further improves this approximation by reducing errors from linear approximation, thereby decreasing weight variance. also mitigates off-manifold guidance (He al.) 2024), partic- ularly in early sampling stages. As X increases from to 1, it gradually guides sampling towards the target while minimizing weight degeneracy and manifold deviation. Section 4 provides empirical validation of these effects.
Finally, the unnormalized weights for each particle are calculated as
Po ( exp A1-1 (1
Wt Wr-1 (IT-1,) (0-1(11-1) (x(
(14)
(xre)) mi- (14-1
m m (z (Tt
exp
A f(11)
for f = 1 T and (xrrr) exp ( (AT f(zr)) which are used for resampling. The pseudo-code of the final algorithm with adaptive resampling given in Algorithm A.I
3.3.4 ASYMPTOTIC BEHAVIOR
This section presents asymptotic analysis results for DAS. We first demonstrate asymptotic exact- ness, a key property distinguishing SMC methods from other approximate guidance approaches. Proposition 2 (Asymptotic Exactness). (Informal) Under regularity conditions, sample estimation of Ex-prar [x] given by DAS converge to the true expectation almost surely for test functions ip
Although SMC samplers are asymptotically exact, their sample efficiency depends on design choices. Using Central Limit Theorem analysis, we bound the asymptotic variance of sample estimations. This approach allows us to prove the benefits of tempering for sample efficiency, pro- viding theoretical justification beyond intuitive advantages.
Proposition 3 (Asymptotic Variance and Sample Efficiency). (Informal) Under the same regularity conditions as Proposition 2 the upper bound of asymptotic variances of sample estimations given by DAS when tempering is used, iee. xe 's are not all 1 for f = 0 T, are always smaller or equal to when tempering isn't it used, i.e. X 's are all for 0 T.
These propositions further imply setwise convergence empirical measures to Ptar and quantify the asymptotic error, which is reduced with tempering. Formal statements and proofs are provided in Appendix
EXPERIMENTS
The main benefits of DAS are twofold: (I) it can avoid over-optimization by directly sampling from the target distribution, and (2) it is efficient since there is no need for additional training. We investigate these benefits through various experiments by addressing the following questions:
Can DAS effectively optimize a single reward while avoiding over-optimization? (44.)
Can DAS optimize multiple rewards all at once without training for each combination?
Can DAS effectively search diverse viable solutions an online black-box optimization? (34.3)
Does tempering increase sample efficiency as predicted by the theory? (84.1)
4.1 SINGLE REWARD
4.1.1
EXPERIMENT SETUP
Tasks. For single reward tasks, we use aesthetic scores (Schuhmann et al., 2022) and human prefer- ence evaluated by PickScore (Kirstain al., 2023) as objectives. For fine-tuning methods, we used
to noise prediction of pre-trained diffusion model, in which the output depends time.

---

Published as a conference paper at ICLR 2025
Pureto front (previous)
DDN TDNO
Alignirop
DiflusionDPO
FreeDOM
DAS (Ours)
SDv1.5
AlignPmpp KL
DPS
MPGD
(4) Aesthetic vs, Unseen Reward (b) Aesthetic vs Unseen Reward
(c) Aesthetic vs. Diversity
(d) Aesthetic vs. Diversity
ImugeRteward
LPTt MPD
120
Aesthetic
Aesthetic
Aesthetic
Aesthetic
(e) PickScore vx. Unseen (e PickScare vx. Unseen Reward
(g) PickScore vs. Diversity
(h) PickScore vs. Diversity
ImugeReward rd
8
LIPP MPD
PickSoore
424
PickSoore
PickSoore
PickScore
target is the aesthetic score, bottom: target is PickScore. (a), (e) and (b), (f): evaluation of cross- generalization using HPSv2 and Im- ageReward, respectively. (c), (g) and (d), (h): evaluation of diversity using Truncated CLIP Entropy (TCE) and mean pairwise distance (MPD) calculated with LPIPS, respectively. Our method reach similar or better target reward compared to fine-tuning methods (DDPO, AlignProp) while main- taining reward generalization and diversity like guidance methods (DPS, FreeDoM. MPGD), breaking through the Pareto-front of previous methods.
Figure 2: Target Reward vs. Evaluation Metrics. Top:
animals from Imagenet Dengg et al. (2009) and prompts from Human Preference Dataset v2 (HPDv2) WW al. (203b) when training aesthetic score and PickScore respectively, like previous settings Black et al. [202] Clark et al. 2024). Evaluation uses unseen prompts from the same dataset.
Evaluation metries. We assess three aspects: target rewards, cross-reward generalization, and sample diversity. For cross-reward generalization, we use HPSv2 (Wu et al.] 2023b) and ImageRe- ward (Xu et al. (202)), both alternative rewards that measure human preference. For sample diver- sity, we use Truncated CLIP Entropy (TCE) (Ibarrola & Grace 2024) which measures entropy of CLIP embeddings, and mean pairwise distance (MPD) calculated with LPIPS (Zhang et al. 2018 which quantifies perceptual differences.
Baselines. We employ Stable Diffusion v1.5 (Rombach al. 2022) as the pre-trained model. Other
baselines include fine-tuning methods (DDPO (Black et al. 2023). AlignProp (Prabhudesai et al.)
2024) AlignProp with KI. regularization, TDPO (Zhang et 2004). and DiffusionDPO (Wallace
et al. 2024) 2022) for PickScore) and training-free freee guidance methods (DPS (Chung al. 2023), FreeDoM
YYu et al. 2023), and MPGD HH et al. 2024)).
4.1.2 RESULTS
Quantitative evaluation. Figure 2 shows quantitative results on both the target reward and eval- uation metrics. Fine-tuning methods generally cluster in the bottom right, indicating reward over- optimization with high target rewards but low diversity and poor generalization similar rewards. AlignProp with KI exhibits a similar trend, failing to mitigate over-optimization due to mode- seeking behavior, as demonstrated the mixture of Gaussian example (Section (.2). TDPO, pro- posed alternative to carly stopping and KL regularization, fails to effectively mitigate over- optimization for aesthetic scores and tends to under-optimize for PickScore. Conversely, guidance methods typically occupy the upper left quadrant, failing to optimize target rewards effectively. DAS consistently occupies the upper right quadrant, achieving high target rewards while maintain- ing cross- generalization and diversity, thus effectively mitigating over-optimization.
Preserving diversity while optimizing rewards. Figure 3 showcases samples generated from the prompt "crocodile," aimed at maximizing aesthetic score. Our approach demonstrates superior aes-
7

---

Published as a conference paper at ICLR 2025
Color
Count
Composition
Location
Style
Unusual
Pre-trained
DDPO
AliguProp KI.
DAS (Ours)
Qualitative comparison of T21 alignment. Target reward: PickScore. Unseen prompts: "A green colored rabbit" (color), "Four wolves in the park" (count), "cat and a dog" (composition), "A dog the moon" (location), "A cat in the style of Van Gogh's 's Starry Night" (style), "A door that leads to outer space" (unusual). Samples generated by DAS used only particles.
Figure 4:
thetic appeal while preserving sample diversity and pre-trained features of the animal. In contrast, samples from fine-tuning methods deviate significantly from the pre-traine mraiireeeed model's output and hibit less diversity in colors, backgrounds, and appearances, indicating reward over-optimization.
Improving T2I alignment. Notably, in Figure 2. DAS substantially outperforms fine-tuning methods for the PickScore task across all met- rics. To check whether the quantitative re- sults align with actual human preferences, Fig- ure visualizes samples targeted to maximize PickScore across six categories: color, count, composition, location, style, and generating unusual scenes. DAS successfully generates aligned images with high visual appeal, even compared to fine-tume toning baselines, thus effec- tively aligning the samples with human pref- erences. We provide additional results in Ap- pendix[G,
Pre-trained
DDPO
AlignProp KL
DAS (Ours)
Qualitative comparison of over- conducted ablation study tempering, us- optimization and diversity.
Ablation on design choices. We additionally
Figure 3:
ing the aesthetic score as target reward and ImageReward as evaluation for T2I alignment and cross-
reward generalization. We also tested various tempering strategies, including > 0.008, 7 = 0.024 (where X reaches after 90 or 30 steps, respectively), and adaptive tempering (A.2) In Figure 5 without tempering, SMC suffers from over-optimization, even with particles, resulting in low Im- ageReward. In contrast, with tempering, using only 4 or 8 particles can achieve both high aesthetic scores and ImageReward, greatly improving efficiency regardless of tempering schemes. Temper- ing also reduces deviation from the latent manifold (He et al. 2024), indicating fewer generations of OOD samples. These findings align with the theoretical predictions in Section 3.3.3 and 3.3.4 While the performance of DAS is robust to tempering schemes, recommend low with tuned for the best balance of quality and efficiency general. Test-time scaling experiment in Appendix further demonstrates that DAS effectively leverage additional computation during inference.
8

---

Published as a conference paper at ICLR 2025
y 0.008
00.024
adaptive
No tempering
7.5
Mean Manifold Deviation
0.14
7.00
0.12-
Acshetic
6.75
ImageReward
0:10
6.50
0.08
6.5
-1.0
5.75
0.02-
5.50
-1.5
0.00
Num Particles
Num Particles
Num Particles
Figure 5: Effeet of tempering on sample efficiency and manifold deviation. We compare dif- ferent tempering schemes while changing the numbers of particles. (left) Number of Particles vs. aesthetic score (target reward), (middle) Number of Particles vs. ImageReward (unseen reward), (righ() Number of Particles vs. mean deviation from latent manifold.
4.2
MULTI REWARDS
Aesthetic
AlignProp
DDPO
DAS (Ours)
AlignProp (som)
DDPO (soup)
DDPO (soup)
0.5
Acaebotic weight
0.35
AlignProp (soup)
CLIP
0.30
24
(Ours
Aesthetic Score
(a) Trade-off in multi-objective optimization.
(b) Generated samples according to reward weights
Figure 6: Comparison of multi-objective optimization. (a) DAS achieves a Pareto-optimal trade- off between the two rewards. (b) Prompt: "classic cars city street with people and dog". Interestingly, DAS consistently generates 'dog' in the prompt when the aesthetic score weight gets lower than 0.9, while baselines fail generate the dog for some images.
Experiment setup. Multi-objective optimization is crucial for real-world applications that balance competing goals (Deb et al. (016)) for instance, generating visually appealing images while being faithful prompts. To evaluate DAS in this practical setting, we combine aesthetic score and CLIPScore (Hessel et al.] 202T), which measures image-text alignment. We use weighted sum:
Aesthetic Score (1 w) 20 CLIPScore
(15)
with E {0, 0.1, 0.5, 0.9, 1.0). Baselines include interpolated LoRA weights fine-tuned separately on each objective using DDPO and AlignProp (Ramé et al. 2023; Clark et al. 2024
Prabhudesai al. 2024), and model directly fine-tuned the weighted sum HPDv2 prompts for training and evaluation.
0.5). We use
Pareto-optimality without fine-tuning. Figure 6a shows DAS achieving Pareto-optimal solutions without any fine-tuning or model interpolation, outperforming methods that require extensive train- ing for each objective. While direct fine-tuning on weighted averages fails improve the Pareto- front, DAS obtains optimal solutions for any reward combination by sampling from the reward- aligned target distribution. Figure (bb) demonstrates this capability through superior prompt align- ment and aesthetic quality across different reward weights.
4.3 ONLINE BLACK- BOX OPTIMIZATION

---

Published as a conference paper at ICLR 2025
Target (1)
Unseen Reward
Diversity (1)
Method
Aesthetic
HPSv2
ImageReward
TCE
LPIPS
SEIKO-UCB
6,88
0.25
-0.31
38.5
0.51
BEIKO-Bootstre
6.51
0.25
-1.01
40.4
0.49
DAS-UCB (Ours)
6.77
0.28
1.25
41.1
0.66
DAS- MAAs-Bootttrpp (Ours)
6.73
0.28
1.27
40.6
0.65
Table I:
Comparison of online optimization methods. DAS achieves comparable target aesthetic scores while significantly outperforming in generalization to unseen rewards and output diversity.
Online black-box optimization with diffu- sion models. This approach optimizes an un- known function by receiving iterative feed- back, especially useful when offline data is insufficient or objectives (c.g.. human prefer- ences) change over time. Minimizing feed- back queries is key to reducing costs. SEIKO (Uehara al., 20246) is feedback-efficientt method using an uncertainty-awar optimistic surrogate model built through linear model (UCB) or ensembling (Bootstrap). While SEI- KO guarantees theoretical regret bounds, this result relies sampling from an aligned dis- tribution using the surrogate model, similar to Plar in which they incorporate direct backprop- agation to solve it. Instead, we adapt DAS to directly sample from this distribution (A.3)
SEIKO-
SEIKO- Boutstrup
DAS-UCB (Ours)
DAS-Bootstrap (Ours)
Figure 7: Qualitative comparison of online methods.
Experiment setup. We adopt aesthetic score as black-box reward model, and limit to use only 1024 feedback queries for all methods. Experiment is conducted in batch online setting through an iterative cycle: proposing samples (from fine-tuned model in SEIKO, or using DAS to directly sample from distribution aligned with surrogate model), recieving feedbacks from the black-box reward, and updating the surrogate model.
Efficient exploration of diverse viable solutions. Figure 7 highlights that DAS preserves pre- trained characteristics and generates diverse, high-quality images, while SEIKO, using AlignProp with KL, distorts animal features. Quantitatively, Table [ DAS matches SEIKO in optimizing aesthetic scores but significantly outperforms in unseen rewards and diversity, proving its ability to explore broader solution space. This demonstrates the advantage DAS over direct backpropaga- tion for the online setting: high sample diversity enhances exploration, leading to robust surrogate model and avoiding over-optimization. Furthermore, DAS bypasses fine-tuning the diffusion model every time the surrogate model is updated, enhancing adaptability through frequent updates.
Non-differentiable rewards. Reward maximization often involves non-differentiable or computa- tionally expensive models. While DAS requires differentiable rewards for guidance, it can handle general rewards by posing the reward as black-box reward and learning a differentiable surrogate model with online feedback. We further demonstrate this using JPEG compressibility in G.3
CONCLUSIONS
We introduce DAS, a test-time algorithm using Sequential Monte Carlo sampling align diffusion models with rewards. DAS optimizes rewards while preserving generalization without fine-tuning. In single and multi-reward experiments, DAS achieves comparable or superior target rewards to fine-tuning methods while excelling in diversity and cross-reward generalization. The online op- timization results demonstrate DAS's ability to efficiently explore diverse, high-quality solutions. These findings establish DAS as versatile and efficient approach for aligning diffusion models applicable to a wide range objectives and scenarios while significantly reducing the cost and complexity of the alignment process.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have made several efforts to ensure the reproducibility of our work. We provide complete proofs for all theoretical results in Appendix C
including formal statements and proofs for Propositions 2 and in Appendix C.2.4 and C.2.5 Detailed pseudocode for our full DAS algorithm is included in
Appendix [A] with versions with adaptive resampling (Algorithm [T. adaptive tempering (Algorithm
3 and adaptation to online setting (Algorithm{)) Appendix contains comprehensive implementa-
tion details for our method and basclines, including hyperparameter settings, training and sampling procedures. We will release full codebase upon publication to enable others to replicate our sults, including implementations of DAS. We use publicly available datasets and evaluation metrics, with details of experiment setup provided in Section 4 Appendix [[ontains additional experimen- tal results to supplement those in the main paper. By providing these materials, we aim to enable other researchers reproduce our results and build upon our work. We are committed addressing any questions or requests for additional information to further support reproducibility efforts.
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal Guidance for Diffusion Models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 843-852, Vancouver, BC, Canada, June 2023. IEEE ISBN 9798350302493. doi: 10.
1109/CVPRW59228..8 21238.002.000099 URL httppsss//eeeexllrreeee.ee feee. drg/mncumett/ 10208653/ arXiv:2302.07221 [cs].
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.0155 2024.
Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective diffusion-based generative modeling. Transactions on Machine Learning Research, 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training Diffusion
Models with Reinforcement Learning. October 2023. URL htttps://pperrevieee /ettreeiieeeee///ee/
Forum?id= Corum?id==CR-JaGxJEE
Gabriel Cardoso, Yazid Janati, Sylvain Le Corff, and Eric Moulines. MONTE CARLO GUIDED DENOISING DIFFUSION MODELS FOR BAYESIAN LINEAR INVERSE PROBLEMS.
2024.
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl diver- gences. Journal of Machine Learning Research, 23(253):1-79, 2022.
Nicolas Chopin and Omiros Pupaspiliopoulos. An Introduction to Sequential Monte Carlo. Springer Series in Statistics. Springer International Publishing, Cham, 2020. ISBN 978-3-030-4784455 doi: 10. 11011119777--000077888-08 -000--3-000-88--000 47845-2
978-3-030-44845-2.
URL hhttps:///intt
com ccm//00.107///
Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Mare Louis Klasky, and Jong Chul Ye. Diffusion Posterior Sampling for General Noisy Inverse Problems. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL arXiv:2209. 14687 [cs, stat].
net wewee//oruumli--nD9oooooom
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. DIRECTLY FINE-TUNING DIFFU- SION MODELS ON DIF- FERENTIABLE REWARDS. 2024.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, March 2024. URL http: / arxiv. org/ abs/2310 02743 arXiv:2310.02743 [es].
11

---

Published as a conference paper at ICLR 2025
Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. In Deciston sciences, pp. 161-200. CRC Press, 2016.
Pierre Del Moral, Amaud Doucet, and Ajay Jasra. Sequential Monte Carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3):41 436, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Zehao Dou and Yang Song. DIFFUSION POSTERIOR SAMPLING FOR LINEAR INVERSE PROBLEM SOLVING FILTERING PERSPECTIVE. 2024.
Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Associa- tion, 106(496):1602-1611 2011.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Ieo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, October 2022. URL
wttp://wwwrgg/a//////a// /2210, 10760 arXiv:2210.10760 [cs, stat].
Mathicu Gerber, Nicolas Chopin, and Nick Whiteley. Negative association, ordering and conver- gence of resampling methods. The Annals of Statistics, 47(4):2236-2260, 2019.
Hyojun Go, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Scungtack Choi. Addressing negative transfer in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Tiankai Hang. Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7441-7451, 2023.
Yutong He, Naoki Murata, Chich-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji. J Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. MANI- FOLD PRESERVING GUIDED DIFFUSION. 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. URLpattp:/aaxiii wrw. abs wttp://.wwr..abb/......1. 11299 arXiv:2006.1 1239 [cs, stat].
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models, June 2022. URL httpl/ttpp/aaxv.ooor///////21/3/ arXiv:2204.03455 [cs].
Francisco Ibarrola and Kazjon Grace Measuring Diversity in Co-creative Image Generation, March 2024. URL
/4033 18266 1rXiv:2403.13826 [cs].
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Alignment as reward-guided search. In The Twelfth International Conference Learning Representations, 2024. URL https:
7 /openreeveww net. wwwpee//fooummmeemom http://tttooommommsdooeeeeeeeee
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663 2023.
12

---

Published as a conference paper at ICLR 2025
Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On Reinforcement Learning and Distribution Matching for Fine- Tuning Language Models with no Catastrophic For- getting. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- fion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, arXiv, 2022. URL httLtttppppppperetee// www.ppprrr...pp........e. nips cc/ wwwpppppppppppccppcpc.ccpppe.aee files//aaprr/222 (ol.44//00o..1e66666000000003333 Abstrabbbttaaccc -onstracctooofeeenncee Conference html arXiv:2206.00761 [cs, stat].
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning Image Models using Human Feedback, February 2023. URL http: www.oorwrr.bbb......./ arXiv:2302. 12192 [cs].
Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Bian- calani, Shuiwang Ji, Aviv Regev, Sergey Levine, et al. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding. arXiv preprint arXiv:2408.08252, 2024.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 11461-11471, 2022.
Nanye Ma. Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732 2025.
Kevin P Murphy. Probabilistic machine learning: Advanced topics. MIT press, 2023.
Christian A. Naesseth, Fredrik Lindsten, and Thomas B. Schön. Elements of Sequential Monte Carlo. Found. Trends Mach. Learn., 12(3):307-392, 2019. doi: 00.1561/22000000744
arXiv:1903.0479: [cs, stat].
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. URL http: /ttp::///axvv..../ otg/ abs aww.///........ 2203. C2155 arXiv:2203.022 021155 [cs].
Dongmin Park, Sebin Kim, Tachong Moon, Minkyu Kim, Kangwook Lee, and Jaewoong Cho. Rare- to to-frequent: Unlocking compositional generation power of diffusion models concepts with Ilm guidance. arXiv preprint arXiv:2410.22377 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor,, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning Text-to-
Image Diffusion Models with Reward Backpropagation, June 2024. URL http:://axxi
org/abal ab////b//aat 2310. 03739 arXiv:2310.03739 [cs].
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Er- mon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023. arXiv,
13

---

Published as a conference paper at ICLR 2025
2023. URL
haahh//
085-:0005554771162222222------a- 1EX1::2305.18290 [es].
Alexandre
Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean- Saan-Bppisste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards Pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards, October 2023. URL http:://axiv.oo
abs/2336 04489 arXiv:2306.04488 [cs].
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pullern Recognition (CVPR), pp. 10684-10695, June 2022.
Sebastian Sanokowski, Sepp Hochreiter, and Sebastian Lehner. A diffusion model framework for unsupervised neural combinatorial optimization. In Forty-first International Conference Ma- chine Learning, 2024.
C. Schuhmann.
Laion aesthetic predictor. aestheiccs 2022. Accessed: 2024-09-29.
https://laion.ai/bble
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint wrxi::2408.033144 2024.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Un- supervised Learning using Nonequilibrium Thermodynamics. November 2015. URL http:
// wwwxiiv.rggabs///.oo
03585 arXiv:1503.03585 [cond-mat, q-bio, stat].
Jiaming Song. Chenlin Meng, and Stefano Ermon. DENOISING DIFFUSION IMPLICIT MOD- ELS. 2021a.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pscudoinnerse--Guidee Diffusion Models for Inverse Problems. September 2022. httpstps::/ http::::s///peeeeeeww. net /forum?
id=9_ ________MMKKK]
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning, pp. 32483-32498. PMLR, 2023.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum Likelihood Training of Maore-Based Diffusion Models, October 2021b. URL 09258 arXiv:2101.09255 [cs, stat].
http://aaxxv..rrg//// 01.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-Based Generative Modeling through Stochastic Differential Equations, February 2021c. lttp::///aaxxx org/ abs /tcll//ccllll 13456, arXiv:2011.1345 [cs, stat].
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023. /RLLtttt///ppeeeiiiics http://wweeettoooooodddee-x-oxxo
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M. Tseng, Tommaso Biancalani, and Sergey Levine. Fine-Tuning of Continuous- Time Diffusion Models as Entropy Regulaaiee Control, February 2024a. URL /
arxiv. http://oggaabb////////// 2402. 151.411944 A1N:24002.1194. [cs, stat].
14

---

Published as a conference paper at ICLR 2025
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M. Tseng, Sergey Levine, and Tommaso Biancalani. Feedback Efficient Online Fine-Tuning of Diffusion Models. In Forty-first International Conference Machine Learn- ing, ICML. 2024, Vienna, Austria, July 2 21-27, 2024, OpenReview.net, 2024b. URL https: //openreviews
arXiv:2402.16359 [cs, q-bio, stat].
Francisco Vargas, Will Sussman Grathwohl, and Arnaud Doucet. Denoising diffusion samplers. In The Eleventh International Conference on Learning Representations, 2023.
Bram Wallace, Mcihua Dang, Rafacl Rafailov. Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8228 8238, Seattle, WA, USA, June 2024. IEEE. ISBN 9798350353006. doi: 10.110//CVPP527332224..077
URL https: http://eeeexxporr.e// arXiv:2311. [cs].
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- eralizing direct preference optimization with diverse divergence constraints. arXiv preprint 162422 2023.
Kai Wang, Mingjia Shi, Yukun Zhou, Zekai Li, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. A closer look at time steps is worthy of triple speed-up for diffusion model training. arXiv preprint arXiv:2405. 17403, 2024.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of 2023.
protein structure and function with rfdiffusion. Nature, 120(7976):00991111
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V LLe Denny Zhou, et al. th-ouut prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Luhuan Wu, Brian L. Trippe. Christian A. Naesseth, David M. Blei. and John Cunningham. Practical and Asymptotically Exact Conditional Sampling in Diffusion Models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, nips. cc paper_ weffileepppaae wwt.ff/eeesppppe/223333 Abstt--bbtttttccc Cow.eetteoooeeeee.eee..
2023. arXiv, 2023a. URL
arXiv:2306.17775 [es, bio, stat].
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedingy of the IEEF/CVF International Conference on Computer Vision, pp. 2096-2105, 2023b.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.000 00724, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Taeho Yoon, Kibeom Myoung, Keon Lee, Jaewoong Cho, Albert No. and Ernest K. Ryu. Cen- sored Sampling of Diffusion Models Using 3 Minutes of Human Feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023. arXiv, 2023. URL httt:://ppppeeeee nips cc paper fi www.:/ff---bbataareaeeaaa WwwtracttCoofeeeetteeeeeeeeeeeee www.et-ooooeeeeneecem
arXiv:2307.02770 (es).
15

---

Published as a conference paper at ICLR 2025
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. FreeDoM: Training- Free Energy-Guided Conditional Diffusion Model. In IEEE/CVF International Conference on Computer Viston, ICCV 2023, Paris, France. October 1-6, 2023, pp. 23117-23127. IEEE, 2023. doi: N0.11099CCCV5100022223.00
arXiv:2303.09833 [cs].
Qinsheng Zhang and Yongxin Chen. Path Integral Sampler: a stochastic control approach for sam- pling, March 2022. URL
arXiv:2111.1141 [cs].
Richard Zhang. Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The Unrea- sonable Effectiveness of Deep Features as Perceptual Metric. In 2018 IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pp. 586-595, Salt Lake City, UT, June 2018. IEEE ISBN 978-1-5386-6420-9 doi: 10.1109/CVR..20880000 201.000088.. //ieeexplore. ieee. org/ www.ddetttttnn......eoo
URL hetps:
Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao. Confronting reward overoptimization for diffusion models: perspective of inductive and primacy biases. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 60396-60413, PMLR, 21-27 Jul 2024. URL httph::t///tt..t ptttts::/prrceediinng ngs ml press wttpppsee///////aaaaeeee/
html
Tianyi Zheng, Peng- Tao Jiang, Ben Wan, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Beta-tuned timestep diffusion model. In European Conference Computer Vision, pp. 114-130. Springer, 2025.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Funinn Language Models from Human Preferences, Jan- uary 2020. URL URLhttp: //trx/aaaavvv//a///aa//aaeaaaava org/ abs /1909, 08593 arXiv: 1909.0899 [cs, stat].
16

---

Published as a conference paper at ICLR 2025
A PSEUDOCODES
A.1 PSEUDOCODE FOR FULL ALGORITHM OF DAS
In practice, adaptive resampling (Chopin & Papaspiliopoulos 2020 Murphy 2024 is used instead of resampling every step with more sophisticated resampling schemes instead of multinomial resam- pling for variance reduction. We also used adaptive resampling with Srinivasan Sampling Process (SSP) resampling scheme (Gerber et al. 2019). The full algorithm of DAS is given as follows.
In practice, adaptive resampling (Chopin & Papaspiliopoulos 2020 Murphy 2023
used instead of resampling at every step. This approach helpsmaintainigg particle diversity where resampling every step may lead particle degenarcy. Adaptive resampling uses the Effective Sample Size (ESS) as criterion to determine when resampling is necessary. The ESS is defined as:
ESS
Σ (WT))
(16)
where W are the normalized particle weights. Resampling performed only when the ESS falls below predetermined threshold, indicating significant imbalance among particle weights. Note that weights in Equation now become incremental weight.
Furthermore, more sophisticated resampling schemes are often employed instead of simple multi- nomial resampling to reduce variance. In our implementation of DAS, we use adaptive resampling with the Srinivasan Sampling Process (SSP) resampling scheme (Gerber al. 2019).
The full algorithm of DAS incorporating these techniques given Algorithm
Algorithm Full Algorithm of DAS with Adaptive Resampling
1: Input: Number of time steps T, Number of particles N, Minimum ESS threshold ESSmin Resampling scheme RESAMPLE()), Tempering scheme Xr I 11 S Au
2: Output: Particle approximations of ptar, (Xe. W Jn=1
3: // Initialize particles time T
4: for 1 to N do
5: X7 ~ N (0. (0. 1)
b Sample from prior
6: wT + exp Ar F(X2))
> Initial weights
7: end for
8: WP +- w// 2m=1 wis for N,...,N
Normalize weights
9: // Main loop: reverse time from To
10: for T to do
11:
// Adaptive resampling based on ESS
12:
ESS
(W?)2)
Calculate Effective Sample Size
13:
ESS < ESSmin then A!:NN RRSAMPLEEWWAN Wh < for n = 1, 1,...,N N
14:
D Resample using SSP D Reset weights
15:
else
17:
AD + forn n = I,...,N wh w/ for n 1....,N
N
b. Keep original indices Keep original weights
19:
end if
20:
Importance Sampling
21:
for n. = to N do
22:
X ~ m4-1 m (XxAi
Propose new particles, equation
13
wp_ + up 1 (X. ,
> Update weights, equation 14
24: end for
W t wi_1/ IN w for n = 1....,
Normalize weights
26: end for
27: return (X7, Wo W
1==1
17

---

Published as a conference paper at ICLR 2025
A.2 PSEUDOCODE FOR DAS WITH ADAPTIVE TEMPERING
We also provide pseudocode of DAS with adaptive tempering used in ablation study (Section 4.1
for completeness.
Algorithm Full Algorithm DAS with Adaptive Resampling and Adaptive Tempering
Input: Number of time steps T, Number of particles N, Target ESS ESSget Resampling scheme RESAMPILE(.)
2: Output: Particle approximations of Ptars (X6, W J n==
3: // Initialize particles at time T
4: for n = 1 to N do
5: X2 ~ N 1
Sample from prior b Initial weights
6: WT 4-1
7: end for
8: WT + w/// Em
WT for n = I,...,N
D. Normalize weights
9: ÀT
Initial tempering parameter
10: // Main loop: reverse time from To 1
for ' do
12: // Adaptive tempering
< SOLVEFOR ([X), W Jm=1 ESS Ae)
Solve for Update tempering parameter
14: 1 min(A, + 1)
15:
for = to N do
W - wp² exp A ((())
16:
Update weights
17:
end
for WT < w Σ we for n = I,...,N
18:
D. Normalize weights
Adaptive resampling based on ESS
20:
ESS & E (WP)2
> Calculate Effective Sample Size
if ESS ESSugget then
22:
A7:N +- RESAMPLE(W?.N)
Resample using SSP b Reset weights
23:
t for = I.....N
else
25:
A < for n 1, N W < wp for n 1,...,M N
Keep original indices Keep original weights
27:
end if
28:
Importance Sampling
29:
if 1 then
Skip proposal step for the final iteration
30:
for n = 1 to N do
31:
X m4-1 1 (I:s (32-1 xxAA
Propose particles, equation 13 Update weights using A = X, Eq. equation 14
32:
w._ t- पे W X, x._
33:
end for
34:
W w//1/ Em w for ....,N
Normalize weights
35:
end
36:
end for
37: return {X6, W J
Algorithm SolveForrellaa function with clamping
1: function SOL.VEFFORDELTTA( WP) In=1, ESS A₂)
2: Define f()) ESS((W? exp( ((X")) )) ESS urget
3: Öunclamped NumericallRootFnddnngg
e.g., bisection method
4: ò + max((, 0, min (Ounclamped; 1 X²))
> Clamp 8 between 0 and 1-Ac
5:
return 8
6: end function
18

---

Published as a conference paper at ICLR 2025
A.3 PSEUDOCODE FOR ONLINE BLACK-BOX OPTIMIZATION
We first provide pseudocode of SEIKO (Uehara et al. 2024b) for completeness.
Algorithm SEIKO (OptimiStic finE-tuning of dlffusion with KL cOnstriin))
I: Input: Parameters a, 3, pre-trained diffusion model fpe [0, x X - X, initial distribution
Ppre X - (X)
2: Output: Sequence fine-tuned models ,....... p(K)
3: Initialize f(0) < fpee, 20) V/rre
4: for = 1 to K do
5: Generate new sample (()) ~ (x) following reverse SDE:
dx. = ((i 3t) + (t)dw:; 20 u(l-1)
6:
7: Get feedback y (x(())+
8: Update dataset: D <- D(i D (,())
9:
Train surrogate model p() (x:) and uncertainty oracle (()
(a:) using (
10:
Fine-tune diffusion model to fit the following distribution:
11: p(t) x exp
()+
(p(t-1)()
(Ppre()
a+3
12: and get f(), 2(1).
13:
end for
14:
return p()) p(k)
For every update, SEIKO use direct backpropagation to fit the diffusion model to the target distribu- tion p(9) by solving
(())
arg max E Eopp [r))] aKL(p (0)) SKL (p p(i-
(17)
PEA(X)
similarly to Uchara et al. seeking behavior in Section 3.2 and from
(2024a), However, revealed that this method can easily fall into mode-
Instead, we adapt DAS for the corresponding step to sample
(-) + (-)
5 x P-r exp
(18)
directly. Online algorithm employing DAS is given in Algorithm
Algorithm 5 Online Black-box Optimization using DAS
1: Input: Parameters a, pre-trained diffusion model
2: Output: Samples from final target aligned to black box reward
3: Initialize ())) Ppre
4: for 1 to K do
5: Generate new sample (()) f(i-1)(x) using DAS (Algortihm D
6: Get feedback (()) rx)(()) +e
7: Update dataset: D + D(i u (,,0,))))
8: Train surrogate model p(0) (x) and uncertainty oracle ( (x) using (
9:
end for
return (')...... (K)
For both aesthetic score experiments (Section 4.3 and JPEG compressibility experiments (Sec- G.3), we used MLP on frozen CLIP embeddings (Radford et al. 2021) for surrogate model.
B INTRODUTION TO SMC
In this section, give introdution to SMC and particle filtering using Feyman- Kac formulation following Chopin & Papaspiliopoulos (2020). We additionaly introduce theoretical results in the SMC literature, which will be used in C.2.4 and C.2.5 for asymptotic analysis of DAS.
19

---

Published as a conference paper at ICLR 2025
Please note that in this section, follow similar time notation of general SMC methods which, unlike diffusion sampling, starts at time and ends at time K. We will come back to time notation for diffusion models in appendix [ where analyze DAS.
B.1 FEYNMAN-KE KAC MODELS AND PARTICLE FILTERING
Feynman-Kac Kac models extend Markov process by incorporating potential functions create new sequence of probability measures via change of measure. Starting with a Markov process on state space X. with initial distribution M. (dx)) and transition kernels My (xk-1, dææ) for 1 K, the joint distribution defined on X² is:
M (dZo:)) Mo((d))
II Ma (It=11 drx)
(19)
The Feynman-Kat foramlization introduces potential functions Go(zo) and Cx( ux) for k which are strictly positive. These define a new sequence of probability measures Q2 through change of measure:
Qu(drox)= 1
Go(zo) Lx
II (. Ma(drox))
(20)
refered as sequence of Feynman-Kan Kac models. Here, La is normalizing constant:
LA
EMC
Go(Xo) II G.(XX_1,,X))
(21)
Feynman-Kaa Kac model will also be used as a term that refer to the collection of kernels and potential functions {M,, G& The probability measures can be extended to arbitrary future horizon K > k. allowing Qs to be defined on X for all k:
k II
Qa (dæo::) = (oo)
G. Ga(aa-n)aa)
Mxx (dxo:k)
(22)
Txk
In this extended formulation. Qu (dzo:: is defined on the full time horizon [0, K] for any k S K. Qu (dro:)) becomes the marginal distribution of the first + components of (u(ooo.)) For >> k, the potential functions G, are effectively set to 1, allowing the process to evolve according to the original Markov dynamics M, for the remaining time steps. Note that Q (dzo:.) is marginal of Qu (dzo.k), but not necessarily of Qu (xx(doo. k) for K k. distinction crucial in working with Feynman Kac models and related inference methods.
Sequential Monte Carlo (SMC), also known particle filtering, is generic algorithm that pro- vides recursive approximations a given state-space model. It relies on the Feynman-Kaa model to provide the recursive structure for the probability distributions wish to approximate, and uses im- portance sampling and resampling techniques to achieve these approximations. Algorithm 6 shows the sampling process of generic particle filtering algorithm.
20

---

Published as a conference paper at ICLR 2025
Algorithm 6 Generic Particle Filtering Algorithm
1: Input: Feynman-Kac model {Mr, Gk Number of particles N, Resampling scheme RESAMPLE(.). Number of time steps K
2: Output: Particle approximations (X Wa
Jn=1 for l 0,..., K
3: for n = to N do
4:
X Mo (dro)
5:
w6 + (o(X))
6: end for
7:
Wo f w / Σ m==1 w0 for n = 1, I....,N
8:
for
K do
9:
AL + (W k-1
10:
for n = to N do
11:
X (X k- (
w t G& (Xp Ai Xx)
12:
13:
end for
14:
WE
N w E wi for n.
15:
end for
return {XE, WT) Jn for k OK...,K
Ouput of the particle filtering algorithm can be used for sample approximations of the associated Feyman-Ka -ac models by
N
Q Qu-1(doR)
~
8xg
(23)
N
==1
Qe(dTr)) ~
W
(24)
m=1
Eq. (an.l(((x)) = Qu
(25)
[xx) Qa(p) ~ Σ W (
(26)
where y E (x)) some test function and (x') denotes the set of functions ip X - that are measurable and bounded. We give further asymptotic analysis of these approximations in the following sections when assuming multinomial resampling used. For simplicity, we focus on the approximation 26
Finally, we note that SMC samplers apply particle filtering where the associated Feynman-Ke Kac model targets to approximate intermediate joint distributions in equation 7 but with different time notation.
B.2 CONVERGENCE OF PARTICLE ESTIMATES
We state the following law of large number (LLN) type proposition for approximation 26 without proof. This proposition provides the asymptotic exactness of particle filtering algorithms and SMC samplers, including DAS, when the assumptions are met.
Proposition 4 (Chopin Papaspiliopoulos/ (2020), Proposition 11.4). For algorithm with multi- nomial resampling, if potential functions G 's of the associated Feynman-Kan model are all upper bounded, for and & such that ip G Cx))),
N
Σ WTx((X) 8. (x()).
(27)
n=1
where G.S. denotes almost convergence.
21

---

Published as a conference paper at ICLR 2025
B.3 CENTRAL LIMIT THEOREMS AND STABILITY OF ASYMPTOTIC VARIANCES
Even if approximations given by particle filtering algorithms and SMC samplers are asymptotically exact due to Proposition 4 the aceuracy of approximation with finite number of particle depends on the rate of convergence. Typically, the asymptotic error is characterized by CLT type argument,
where the error of estimation is distributed in Gaussian with scale ONN- +) and the asymptotic variance determines rate of convergence. We state formal version of the argument without proof.
Proposition 5 (Chopin & Papaspillopoulos| (2020), Proposition 11.2). Under the same settings and assumptions as Proposition 2 for and yp such that ip x GR E Cx(2),
N
Σ
Qu(()) -
WT
(28)
where + denotes convergence in distribution and the asymptotic variances Vv are defined cumu- latively as
Va(+)) = Σ (Q. [[G (())(())))))))) =m0
(29)
Here, G.
Le GR. Re(()= = MR vp) and Ret1a(pp)= = Rs+1 Re( Lx--
Due to the cumulative form of the asymptotic variance, it may easily blow up as the sampling errors accumulate. To prevent this, the Markov kernels should be strongly mixing, that is, future states of the Markov process should become increasingly independent of the initial state, making the effect of previous sampling errors vanish. We lay out the desired properties of the Markov kernels and potential functions, then state the stability of the asymptotic variance by providing an upper bound for the asymptotic variances that is uniform over time, based on the assumptions.
We first give a formal definition of strongly mixing Markov kernels.
Definition (Contraction coefficient, Strongly mixing Markov kernel). The contraction coefficient of a Markov kernel M₁ is the quantity PM E [o, 1] defined as sup
pM
dra) Ma(#k-11, (Tv.
(30)
where ||P 1Flllllv=== == SEBAE (P(A) - Q(A)) is total variation distance between to probability measures and Q and B(X) denotes Borel o-algebra of state space X. Furthermore, Markov kernel Mş is said to be strongly mixing if pM S 1.
Next we lay out the assumptions for the associated Feynman-Kac model for the asymptotic variance be stable.
Assumption (1) Markov kernels Mj for k = 1 K admit a probability density my such that
m& (2 my (2x (x)1)
CM
(31)
for any T2. 12. E X. for some CM 2 1.
Assumption (2) Potential functions G 's are uniformly bounded for le 0: K as
0 << q & 1,, S Ca
(32)
where 2k) must be replaced by Go(zo) for l = O. Given these assumptions, both M₁ 's and Markov process defined by Q become strongly mixing as below.
Proposition 6 (Chopin & Papaspiliopoulos (2020), Proposition 11.9). Under Assumptions (1) and (2), Ms is strongly mixing with contraction coefficient contraction coefficient PM S 1 cM Fur- thermore, the Markov process defined by Q2 is also strongly mixing with contraction coefficient
-1
PQ < 11-1/c2 where cg =c/c
Finally, using the assumptions and Proposition 3 one can prove the following proposition bounding the asymptotic variances uniformly in time.
22

---

Published as a conference paper at ICLR 2025
Proposition 7 (Chopin & Papaspiliopoulls/ (2020)
Proposition 11.13). Under Assumptions (1) and define by is bounded uniformly in time by
for any 4 E C,(d)), asymptotic variance (x(
2PMcG
1
Vx(+) S (p) exp
(33)
1-PQ pQ
1- 12
2(1 -1--M CG
$ SG(AA) exp
(1- (chrca)) (1
x
(34)
1 - (1-11((3r (c2rc0)
where App := s=p,,, 1((xx) - p(x')l the variation of yp.
Note that the upper bound is increasing function of both eM and cG. Intuitively, these constants grow, the Markov kernels exhibit stronger mixing properties, which, in turn, accelerates the process of forgetting diminishing the influence of past sampling enrors. We will use this property in C.2.5 to prove that tempering can lower this uniform upper bound.
C PROOFS
C.I LOCALLY OPTIMAL PROPOSAL
In this section, we provide the analytic form of the unnormalized density of the locally optimal pro- posal, which is then approximated by Gaussian distribution for our proposal distribution. Before starting the proof, we note that one alternatively get approximate samples from the locally opti- mal proposal using nested IS (Naesseth et al.] 2019; Li et al.) 2024) with additional use of samples. However, when the reward is differentiable, the Gaussian approximation works well without the need for additional samples, as we demonstrate in our experiments.
C.1.1 PROOF OF PROPOSITION
proof. first show that mi-1 (xt (2x)) such that minimizes (a) satis- fies (x)-(())) x 7--1 (zt-1) L (Iy -1)) Since the minimization problem has constraint I 1 = 1, by introducing Lagrange multiplier v(xz), the dual problem can be written as
min 11-1(F(-1mm) (me-s(errlee) ( (Re [wi (xt-11 =z)))²
+v(x)) (f ( d (x)) -1}}
Here, ()-2(11-1,21)) Re 1:7)
(.(rr)) Jm, (st, 1 Using calculation of variation and that only first and
third term include m m should satisfy
T 12)- 1:T) Fe(FET ) (IC-1
0 1 2m (I)-1)=1) (#1-1,
+ v(xx)
= ( v(xe)
where (2
( ( v..) Since v(zx) is with constant respect to 2T-1,
#t 1 (at, 14)
Te ( x ( La
m m
(35)
(x(r))
Then using the definitions of intermediate targets and backward kernels used in DAS,
( (()-)))))))
mi-1 (I (xx) x pe (e(1r-11zz) exp
(-1 1 po(oe, 1) + d i ((x-1)
(-2o
x exp
23

---

Published as a conference paper at ICLR 2025
C.2 ASYMPTOTIC ANLYSIS OF DAS
C.2.1 FEYNMAN-KAC MODEL FOR DAS
To give asymptotic analysis for DAS, we first clarify the Feynman- Kac model for DAS using the formulations from B
The Feynman Kac model for DAS is given by simply substituting
Mo(dyo) - PT ( ( (deT) NN (drT))
(36)
(1 dy:) my ( (dx) =N -N(uo(te.) + oi Ve (dxx-1)
a
(37)
AT ((F))
Golyo) ur (xT())) exp
(38)
a
po (0 exp
Ae-1 Àr-1 (x)-1)
GR(MN-1,88) = wi 1 (2), xz-1)
(39)
(=1(61(1123)) exp ( (z)(
Then, the associated Feynman-Kan Kac models are
T
(d-i(drr-1:T) WT (xT)
1
II W. w
pT pT(KT) II
m
1 (I
(zs (xs)
L
s=t
T
Te- R (1
(T(TT)
Ms 1005-1(0ss0))) (d)
Re m 1
-e- (
where we used the alternative definition of w in g and telescoping simplify the terms. Thus indeed, Feynman-Kac models become the intermediate joint distributions defined through the inter- mediate targets and backward kernels. Especially, if we marginalize at = O, get
Qo(c)o) (ddo) Ptor(zo))dz))
(40)
C.2.2 ASSUMPTIONS FOR PROPOSITION 2 AND 3
Before we start the main proofs, we lay out the assumptions for the proofs.
Assumption (a) Reward function bounded by 0 S r(-) S R
Assumption (b) Norm of gradient of f is uniformly bounded by |1V,, S
Assumption (c) X defined as the union of support of P. and supports of (-_1(- for all xt. is bounded and d, == (X,
sup(d(z, xz y E X,-13 1:
We go over the viability of these assumptions. Assumption (a) can satisfied lower and upper bounded rewards by adding constant. Real-world rewards are indeed lower and upper bounded in most practical settings, including aesthetic score and PickScore used in our experiments. Even not, we can simply clamp the reward to ensure the condition. Assumption (b) should be ensured for numerical stability of the algorithm. That is, if the gradient explode, generation using the guidance isn't possible. Since r(-) and a(-) are function using neural networks, the assumption can be met unless 2e gets out of support of the training distribution. Experimentally, this is commonly true unless using extremely small a. Note that the assumption of uniform bound in time only for simplicity and the bound may change in time. Assumption (c) is generally not true since my is a Gaussian kernel and Pt is also the marginal distribution over Gaussian noise added to clean data. However, it can 'effectively' be satisfied. We explain what this means in more detail. First, the data manifold Xg can be assumed to be bounded since most real-world data without corruption doesn't 't contain infinitely large or small component. Next, suppose we define the forward, reverse diffusion process and the Markov kernels using Gaussian distribution truncated at tail probability e instead of standard Gaussian. Numerically, when is sufficiently small, the impact of the truncation becomes negligible, hence the diffusion process and the SMC sampler behaves similarly to the original Gaussian case. However, unlike the unbounded Gaussian noise, the bounded support of the truncated noise ensures compactness of X, in Assumption (c). Thus the algorithm can modified to satisfy Assumption (c) without any effect of the practical algorithm.
of
24

---

Published as a conference paper at ICLR 2025
C.2.3 LEMMAS
We first prove lemmas need to prove the propositions. We omit t in po (z(, t) from now on for simplicity.
Lemma 1. Under Assumptions (a) ~ (c),
(4-1(1(-1(21) S exp
di
Ac-1 L +2 2 20 )=11
3d
(41)
m4-1 ( (24)
oi
for any It-1 24-1 and 24 Nt
proof.
M4 (34-1 (xx)
log
M [It-1 [z4)
o2 V f f -1 or a V Va;
|| (o(x))
20
a
#t-1 2 ((o((ze)) mo(x()) o0 - (Va.((b))) + vaif(a()).
1
i
(Ho( mo(x))) + ox - (V V.,f(Er) vot
a
Applying
inequality,
1 ( (mo((x) + po(x!)) 20 A - + ve(f(a())
$ oi
(mo(as)) - (x(2)) + of -e-1 (.,f(ar) - ve(f(a())
Using Assumption (b) to bound
Vay [[+
and Assumption (c) to bound ft-1 2 (ua(se) + no( (110(())) - mo((11)
1
since It- 11-1,,4((;))) (1((x)), (o())) E Xe.
4-1 + oo A1-1 T.)
dd-1 20 de-1 T
$ 07
d +
d-1 +
Ae-1 σ
2
3d--1
of
a
α
Thus Assumption (1) in B.3 holds for Feynman-Kaa model of DAS under Assumptions (a) (c) where the uniform upper bound CM given by
dL + 3d,-1
d1-1 L +2
(o }
ot dt-1
sup CM tE(1.....7)
exp
(42)
ot
a
Lemma 2, Under Assumptions (a) ~ (c).
(- (-(di11 + 20/ L
of d11 d L -
de R
0 < exp
< w w1 21111
(43)
and
W1-1(11, S exp ( +
- 1
2 -
\
2 a
I,
o
1,
+
(44)
a
a
25

---

Published as a conference paper at ICLR 2025
proof.
log (12,
po 124 exp d--1 f(xt-11)
log
14-1 1 (1) exp (x)))
log (o log m m (2)) + 1--1 f(x)-1) 2 f(x1)
a
By Assumption (a),
A R S Ar- f(tz-1
A i(xz) S
A--1 R
(45)
Also,
log po (2c- (2xe)
log my (Iz (20x)
1
||-2-11 (10 | pa(x)))
oi Ae-1 V (xx)))
2
a
1
1 2 Ae-1 V ((z)), #t-1 [x(
2 - x. f(xx)
2 a
oi
of
a
Applying Cauchy-Schhe Mauchy-Schwarz.nng inequality,
d Vn, f(x1)
Tt-1 (0(xz) -
2 -0
V xc f(xz),
of
d
of الاي -e- f220
21-1 mo(x))
2 oi
V ne
f (221)
Using Assumption (b) to bound and Assumption (c) bound
| (xx+)]
| mo(x)) since 2T-1; 21-1; (o(x)) X,
<(de
1 Ac-1 do (d 2 a
L
Combining the two bounds, we conclude the proof.
Thus Assumption (2) in B.3 also holds for Feynman- model of DAS under Assumptions (a) (c). where cG Cu/c q is given
{(d-1 dr--1
2 - of م
Ac-1 R B7
CG exp
sup (E(1.....TT)
+
50
L
a
(dh1 + 1 jo
of ااميي L
de 2 ) (46)
sup IE(I....TT) T)
C.2.4 PROOF OF PROPOSITION
We state formal version of Proposition 2 and prove it. Proposition 8. For DAS with multinomial resampling. under Assumptions (a) (c), for ip such that (Xo) and output of DAS (Xp. Wa J
Σ EwTp(Xy) e." Pear (1p).
(47)
where Ptar is the final target distribution of DAS defined in
proof. By Lemma 2 each potential functions of the Feynman- Kac model are all upper bounded, and thus all conditions for Proposition 4 are met. Using Proposition at = 0 (i.e. k = T respect to SMC for time notation), since (o(d = Ptar (dzo),
N
Σ
W ((X)) a.S. Pear
(48)
26

---

Published as a conference paper at ICLR 2025
Setwise convergence of empirical measure can be derived as direct corollary by substituting (p(X) (X) for all A E B(2)).
C.2.5 PROOF OF PROPOSITION
Finally, we state formal version of Proposition and prove it.
Proposition 9. For DAS with multinomial resampling, under Assumptions (a) ~ (c), for up such that yp (2o) and output DAS {Xo. Wo J==
VN
Σ
Wi (X)( Ptar(s)) NO Vo(s))
(49)
where the asymptotic variance Vol(p) is bounded by
(1 - -x CG
1
Vol+)) S (d(pp) exp
(c (c2)cc)
x
(50)
1-
(c))eG)
using the definitions of Cad and cG in 42 and Furthermore, this upper bound when tempering is used, i.e. dt s are all 1. 0: : T, is always smaller or equal when tempering isn't used, iee. Xe's are all f t T
proof. By Lemma 2 each potential functions of the Feynman- Kac model are upper bounded, and thus all conditions for Proposition 5 are met. Using Proposition 5 at 0 (i.e. k = T respect to SMC for time notation), since (o(c))) = Ptar (dz))
N W xXx)) Pear((o) = NO,, Vo(p))
(51)
Also, by Lemma T and together, the Feynman-Ka model satisfies the Assumption (I) and (2) in B.3 thus using Proposition
we get
(1 c ca
Vol+) - (Ap)) exp
(c (c) (ch)cc)-1)
x
(52)
1 1-
(c))oG)
Looking the definitions of см and eg in 42 46 both values when tempering is used, i.e. A are not all 1 for t = 0 T, is always smaller or equal to when tempering isn't 't used, i.c. A are all for t 0 since the equations in the supremum are all increasing functions of X 2 Finally, since the upper bound is an increasing function of and we conclude that the upper bound when tempering is used is always smaller or equal to when tempering isn't used.
L
Again, similar result can be obtained for setwise convergence of empirical measure by substituting (x)) = (X) and Ap for all E B(%)).
D IMPLEMENTATION DETAILS
In all experiments, we adapted Stable Diffusion (SD) v1.5 Rombach et al. (2022) for pre-trained model.
Fine-tuning methods. We used official PyTorch codebase of DDPO, AlignProp. TDPO with min- imal change of hyperparameters from the settings in the original papers and codebases. We used 200 epoch and effective batch size of 256 using gradient accumulations if need for all methods. For AlignProp, even with KL regularization, severe reward collapse were mostly observed at the end of training, generating unrecognizable images. We used checkpoints before the collapse for compar- used the same coefficient of othe KL regularization
isons. For AlignProp with KL regularization, terms. For DiffusionDPO. we used the official fine-tuned weights SD v1.5 using Pick-a- aiP dataset (Kirstain et al. 2023) released by the authors.
27

---

Published as a conference paper at ICLR 2025
Guidance methods. We adapted the official PyTorch codebase of FreeDoM and MPGD incor- porate with diffusers library. For DPS, which wasn't 't adapted to latent diffusion, we used the same implementation of FreeDoM but withou time-trave travel startegy Yu et al. 2023 Lugmayr et al. (022). As in the official implementations, we scaled the guidance to match the scale of classifier-guidance and multiplied additional constants. These constants 0.2 and 15 for FreeDoM and MPGD re- spectively, following the official implementation.
DAS. Across all experiment results except ablation studies, we used 100 diffusion time steps with = 0.008 for tempering. For single reward experiments, we used KL coefficient a = 0.01 for aesthetic score task and 0.0001 for PickScore task considering the scale of the rewards. For multi-objective experiments and online black- box optimization, we used a = 0.005. We used 16 particles if not explicitly mentioned. Exceptionally, we used 4 particles during online black-box optimization for efficieny.
DAS hyperparameter selection recipe. To ehance easiness of adapting DAS, we propose sys- tematic approach for selecting hyperparameters based empirical performance and convergence behavior. Firstly, tempering parameter 7 can be selected depending on the diffusion time steps T such that (1 + 71+)"²= Rs 1. While more particles are often better, 4 to 16 particles are sufficient to guar- antee good performance as in the ablation study from Section 4. Especially, for rapid prototyping, recommend 4 particles. KL coefficient should be scaled to reward magnitude. Speicificaly, a in the range such that approximate guidance norm 2 classifier guidance norm * [1/5, 5] is appropriate. While a is the main tuning parameter, based on the above criteria, optimal values be efficiently found through few sampling iterations since DAS requires no training. Typically, we fixed T = 100, 7 0.008, N and used grid search for a E {10-1, 1, 10 2, 110-3 10 10= After selecting the best 10-* we again used grid search for E (3 x 10-(++1), 5 x 10- 10-(k+1), 10 10- 3 x 10-* 5 10-*).
E TEST-TIME SCALING
Test-time Scaling. While DAS offers a powerful test-time reward maximization scheme, even out- performing fine-tuning methods, it requires additional computation during inference. Similarly, recent literature on language models suggests that leveraging additional computation during infer- ence can further improve the response's 's quality (Wei et al. 2022, Khanov et al., 2024; Wu et al.] 2024 Snell et al. 2024). However, it's important to verify which method can most effectively utilize additional resources and achieve the highest performance as we scale test-time computation.
Experiment Setup. To demonstrate DAS is indeed an effective way to leverage computation during inference, we compare the test-time scaling of DAS with other possible test-time alignment methods for diffusion models. Baseline methods include Best-of-N (Touvron al. 2023] Beirami et al.) 2024 Ma et al., 2025) which runs N (number of particles) independent generation process and selects the final sample with the highest reward. SMC uses Sequential Monte Carlo sampler as DAS does, but without the approximate locally optimal proposal in Section [3...3 and tempering, using the original generation process the proposal for each step. Including DAS, these methods are naturally scalable by increasing the number of particles used in the algorithm. We use the number of particles in 21, 4, 16, 32, 64, 128) for each method and compare how target reward and unseen reward for cross-reward generalization scale as inference-time computation increase. We use PickScore for target reward and HPSv2 for unseen reward.
Results. Figure&ssowss showss that as the inference compute increases, DAS consistently outperforms both Best-of-N and SMC across both PickScore (target reward) and HPSv2 (unseen reward). Notably, DAS already achieves superior performance over other baselines even when using fewer particles and significantly less compute. This highlights DAS's efficiency in maximizing rewards compared to methods like Best-of-N, which shows limited improvements beyond certain compute threshold, and SMC, which scales better than Best-of-N but still falls short of DAS.
These results emphasize the strength of the proposal and tempering scheme utilized by DAS, which enables it to effectively improve sample efficiency and thus leverage test-time computation without requiring extensive resources. By achieving better performance at lower computational cost and scaling seamlessly with increased resources, DAS proves to be both a compute-efficien and scalable method for inference-time alignment in diffusion models.
28

---

Published as a conference paper at ICLR 2025
Best-of-N
SMC
DAS (Ours)
0.255
0.305
PickScore (Targe) garget
0.250
HPSv2 (Unseen)
0.245
0.240
0.295
0.235
0.290
0.230
0.225
0.285
0.220
10
10*
10a
10'
10
10
Inference Compute (TFlops)
Inference Compute (TFlops)
Figure 8: DAS outperforms with fewer inference-time compute. We compare DAS with other inference-time alignment algorithms while changing the inference-time compute, units in Terafloss. (left) Inference-time Compute PickScore (target reward), (right) Inference-time Compute HPSv2 (unseen reward)
F COMPARISON WITH DIFFUSION-BASER SAMPLERS
Connection to Diffusion-based Samplers. Starting from the objective of fine-tuning methods:
(53)
RL (Fan et al. 2024) or direct backpropagation (Uchara et al. 2024a optimize the variational lower bound this objective given by the data processing inequality, (Po(::))) (lP:r)) which is the KL divergence between joint distribution along the diffusion process (path measure for continu- ous processes). This objective can alternatively written as
Plar (zo)
DEL (poo) (Par( Dr (po (ForT) ((perr))) + Ezz
log Pret (aro)
(54)
where Pret is defined by the same forward diffusion starting from different reference distribution, since
(aa(To):) = Ptur (Fu:r (oourr/ao))ee Ptar (a (no) = Prer [2o) = Prer Puur (
(55)
Pre((Fo)
In problem setting for reward maximization, the reference diffusion is given by the pre-trained model by Pref = Ppre and log pur(z))
(an) = r(xo), and thus the objective becomes reward with KL
regularization term enforcing the diffusion process to stay close to the pre-trained diffusion process.
This formulation reveals the connection to diffusion-baane based samplers (Zhang & Chen 2022 Vargas
et al. 2023; Berner et al. 2024) which also use similar variational objective for training diffusion model sample from given unnormalized target density. While standard diffusion training can be interpreted as optimizing similar variational objective (Ho et al. (2020) for discrete time framework, Song et al. (2021b) for continuous time framework), they use conditional score matching using the samples from the target distribution. However, when sampling from unnormalized density, due to the lack of samples from the target, the methods use direct backpropagation for optimization. Sim- ilarly, reward alignment tasks also have no samples from the target distribution, thus previously proposed works are also based on direct backpropagation or RL. Compared to RL and direct back- propgation methods for reward maximization, diffusion-based samplers use different reference dif- fusions. Specifically, PIS (Zhang & Chen 2022) use pinned Brownian motion running backwards in time, DDS (Vargas et al., 2023) use Variance Preserving (VP) SDE from Song et al. (2021c) (where DDPM is its discretization) as forward diffusion starting from a Normal distribution, and DIS (Berner al. 2024) allows general reference diffusions. Note that given the same final target
29

---

Published as a conference paper at ICLR 2025
Pre-traincd Target
FT-NN (Backprop)
Train-Grad (DDS)
DAS (Ours)
Train-NN
FT-Grad
Distribution Distribution
Reward: -10.40
Rewurd: 0.29
Reward: -0.96
EMD: 3.69
Reward: -0.15
EMD: 2.58
Reward:-1. EMD: 6.09
Reward: -0.55
EMD: 0.91
Reward: 40.22
EMD: 0.82
Reward: -9.15
Reward: -1.34
Reward: -2,43 EMD: 1.34
Reward: -0.30 EMD: 3.55
Reward: -2.14 EMD: 1.47
Reward: -2.36
EMD: 2.88
Reward: EMD: 1.04
Figure 9: Comparison with diffusion-based samplers. Without either the Grad parameterization (Train-NN) or random initialization (FT-Grad), the methods fail to capture all modes in multimodal target distributions unlike DDS (Train-Grad), explaining the mode collapse direct backpropaga- tion (FT-NN). Furthermore, DAS still outperforms diffusion-based sampler for sampling from the target distribution
distribution, only the forward diffusion process matters for the training objective in equation 54 Thus when using DDPM sampling, i.e. VP SDE as forward diffusion, training of direct backpropa- gation methods, DDS, and DIS coincide.
Key Differences. In fact, the key distinction comes from the initialization of the models where diffusion- samplers typically train from scratch, i.e. random initialization, while fine-tuning methods start from pre-trained models. Fine-tuning enables incorporate prior knowledge from pre-trained model, offering practical solution, especially for complex and high-dimensional data like images. However, may make models more susceptible to mode collapse since it can only generate from the pre-trained distribution initially, as we demonstrate in our experiment below.
Additionally, diffusion- based samplers use model parameterization that incorporates the target score function. For example, PIS and DDS use sp t) = NN, (Zz; t) + NN₂(t) V log Ptar (x), which we will refer as 'Grad parameterization' Without Grad parameterization, diffusion-
samplers
failed to fit multimodal target distribution, even for simple GMM (for example, Figure 2 in Zhang & Chen (2022) and Figure 13 in [Berner et al.] (2024). However, we demonstrate that even we incorporate Grad parameterization during fine-tuning, the model fails to capture all modes of the target, indicating the fundamental difference between fine-tuning and training from scratch.
Experiment Setup. We conducted additional experiments using GMM examples from Figure 1. To check our hypothesis and demystify the effect each element, we conducted experiment using fine-tuning NN parameterization (FT-NN, equivalent direct backpropagation with KL regular- ization), fine-tuning + Grad parameterization (FT-Grad), training from scratch + NN parameteriza- tion (Train-NN), training from scratch Grad parameterization (Train-Grad, equivalent DDS), We excluded PIS from our comparison since DDS already outperforms PIS, where PIS uses pinned Brownian motion running backward in time as reference diffusion, which proven incur insta- bility both theoretically (Appendix A.2 of Vargas et al. (2023)) and empirically (Appendix C.4 of Vargas et (023))..
We used a two-layer architecture with 64 hidden units for all neural networks used in Figure 1 and (for Grad parameterization, both NN, and N N2), following /argas et al. (2023). We used 100 diffusion time steps for DDPM sampling with linear beta schedule from 0.0001 to 0.02 as com- monly used. For pre- tra-training via conditional score matching, we used learning rate 0.001 with 1000 epochs. For DDS, used learning rate 3e with 300 epochs. We used Adam optimizer for all training or one-tuning. For other methods, we only changed epochs since each methods had differ- convegence rate when optimizing the objective. For example, the simple NN parameterization
30

---

Published as a conference paper at ICLR 2025
Pre-trained Distribution
Target
Direct Backprop
Approx. Guidance
Distribution
RL
DAS (Ours)
Reward: -2.59
Reward: -2.49
Reward: -927
Reward: -0.50
Reward: -3,26
Reward: -0.72
EMD: 2.99
EMD: 5.20
EMD: 26.60
EMD: 0.86
Aheiorne
Figure 10: SMC method excels in sampling from the target distribution compared to existing approaches. Left dashed line: Samples from pre-trained model trained on 3d Swiss roll, reward- aligned target distribution Ptar using reward r(X, Y,
-x2/100 y2/100 Z Right of dashed line: methods for sampling from Ptar including previous methods (RL, direct backpropaga- tion, approximate guidance) and ours using SMC. Top: Projection of samples XY plane, reward, bottom: 3d plot of samples. EMD denotes sample estimation of Earth Mover's Distance, also known as Wasserstein distance between the sample distribution using each method and the target distribu- tion. DAS outperforms existing approaches in capturing complex target distributions, as evidenced by lower EMD and the similarity with the target samples. Note that samples may exist outside the grid.
typically requires more iterations to converge compared to Grad parameterization, and training from scratch generally needs more iterations than fine-tuning.
Empirical Validation. The result presented in Figure M shows that the methods fail capture all modes in multimodal target distributions without either the Grad parameterization from diffusion- based samplers or random initialization. Furthermore, DAS still outperforms DDS EMD (earth mover's distance) between the target distribution, indicating that the samples from DAS are closer to the target distribution.
In conclusion, claim that constraints in the reward alignment setting pose additional difficulty in training a diffusion model that can sample from unnormalized target distribution. DAS can overcome this problem by offering training- free solution.
G ADDITIONAL EXPERIMENT RESULTS
G. 3D SWISS ROLL
To further visualize the effectiveness of DAS for sampling from unnormlized target density using pre-trained diffusion model, we conducted additional experiment using 3d Swiss roll. As in Figure DAS again demonstrates superior performance in sampling from the target distribution.
10
G.2 DAS WITH SDXL
We conducted experiment using SDXL (Podell al.) 2024) pre-trained base model to demon- strate the generality of our approach. We compare with the pre-trained SDXL (base + refiner), DPO-SDXL which fine-tuned SDXL using DiffusionDPO (Wallace et al. 2024 and our DAS with SDXL as base model. The results summarized in Figure 12 show that DAS's ' effectiveness gen- eralizes beyond SD v1.5, achieving superior performance both target optimization (PickScore) and rross-reward generalization (HPSv2, ImageReward) while maintaining competitive diversity metrics (TCE, I.IPS MPD), even with just 4 particles.
31

---

Published as a conference paper at ICLR 2025
SDXL
DPO-SDXL
DAS-SDXI. (Ours)
SDXL
DPO
DAS
PickScore
0.23
0.23
0.25
HPSv2
0.29
0.29
0.31
ImageReward
0.82
1.24
1.40
ТCE
46.1
46.7
46.1
LPIPS MPD
0.57
0.61
0.61
Targett reward
Figure 11: Qualitative Comparison.
Table 2:
Quantitative Comparison.
Experiment using SDXL. Target reward: PickScore. Unseen prompts for qualitative comparison: "A close up of handpalm with leaves growing from it.", "A photo-realistic image of flying lion with blue butterfly wings". Unseen prompts for quantitative comparison: HPDv2 evaluation prompts. Samples generated by DAS used only 4 particles.
Figure 12:
SD1.5
DAS-SD1.5
SDXL
DAS-SIXL
PickScore (Target)
0.220
0.256 (+0.036)
0.225
0.255 (+0.031)
HPSv2
0.284
0.303 (+0.019)
0.285
0.306 (+0.021)
ImageReward
0.41
1.06 (+0.65)
0.82
1.40 (+0.58)
TCE
46.0
46.0
46.1
46.1
LPIPS MPD
0.662
0.647
0.568
0.608
Table 3: Comparison of different backbones. We compare DAS combined with SD1.5 and SDXL.
To check if the effectiveness of DAS depends the pre-trained model's quality, compared the performance of DAS when combined with SD1.5 and SDXL. As shown in Table DAS improves the target reward score (PickScore) and cross reward scores (HPSv2 and ImageReward) while main- taining the diversity metrics (TCE and LPIPS MPD) regardless of the backbone's 's quality. Specif- ically, looking the target reward (PickScore), DAS-SD1.5 achieves the most performance im- provements, outpeforming the vanilla SDXL and performing nearly identical to DAS-SDXL This indicates that the effectiveness of DAS does not depend on the quality of pre-trained models.
G.3 NON-DIFFERENTIBLLE REWARDS
As stated in Section 4.3 DAS effectively optimize non-differentiable rewards by incorporating them as black-box rewards in our online black-box optimization framework. This allows us to han- dle con-differentiable objectives without modifying the core DAS algorithm. To further demonstrate this capability, we conducted experiments using JPEG compressibility- strictly non-differentiable reward measured the negative file size (in KB) after JPEG compression at quality factor 95. Our experimental setup included 4096 reward feedback queries, ImageNet animal prompts for evalua- tion, and comparison with DDPO (Black et al. 2023), which naturally handles non-differentiable rewards via RL. We combined DAS with UCB as described in Section 4.3
The results in Figure 14 shows that DAS-UCB achieves the best compressibility score, outperform- ing both pre- treierd model and DDPO. Also, it maintains CLIPScore and diversity metrics compared to the pre- traitre model, mitigating over-optimization as intended. The qualitative results show how method effectively minimizes background complexity while preserving key semantic features of the subjects.
In conclusion, while DAS is designed for differentiable rewards, approach provides practical and effective solution for non-din con-differentiable objectives through online black-bor box optimization. The empirical results demonstrate that this approach outperforms methods specifically designed for
32

---

Published as a conference paper at ICLR 2025
differentiable rewards while maintaining the key benefits of DAS such as diversity preservation and avoiding over-optimization.
SD1.5
DDPO
DAS
DDPO
Compressibility"
-116
-84
-71
CLIPScore
0.26
0.26
0.26
DAS-UCE
TCE
39.8
41.1
43.4
(Ours)
LPIPS MPD
0.66
0.62
0.62
Figure 13: Qualitative Comparison.
Table 4: Quantitative Comparison.
Figure 14: JPEG Compressibility Online black-box optimization framework enable DAS to
optimize non-differentiable rewards, outperforming DDPO which can naturally incorporate non- differentiable rewards using RL.
G.4 AESTHETIC SCORE ADDITIONAL RESULTS
Figure 15 provides additional samples generated from each methods target Aesthetic Score.
G.5 PICKSCORE ADDITIONAL RESULTS
Figure 16 provides additional samples generated from each methods to target PickScore.
H FUTURE WORKS
Extended Analysis of Tempering. While our theoretical and empirical analysis in Section and 4.1 builds on both SMC literature for sampling efficiency and latent manifold deviation for understanding off-manifold behavior, recent work has uncovered additional interesting properties about non-uniform behavior (Wang et al., 2024] Zheng et al., 2025) and gradient contradictions (Hang et al. 2023 Go al., 2024) in diffusion models. Our tempering approach already shows connections to these findings the gradual adjustment influence across timesteps aligns with
3.3
the non-uniform importance discovered in recent works, and our careful tempering and resampling may naturally help mitigate gradient contradictions. Analyzing these connections more formally could provide additional theoretical insights complementing our existing analyses and further our understanding of why tempering proves particularly effective in the diffusion model context.
Extensions to Other Modalities. While demonstrated DAS's effectiveness image generation, our method could naturally extend to other modalities where diffusion models have shown promise. The core advantages of DAS its training-free approach enabling quick adaptation to changing re- wards, strong croas-reward generalization capabilities, and maintenance of sample diversity make it particularly valuable for complex domains. For instance, protein structure design, DAS could help optimize properties like binding affinity or stability while maintaining general protein functionality and exploring diverse structural variants. audio generation, could align generated speech or mu- sic with desired acoustic qualities while preserving broader sound characteristics and generalization across different quality metrics. For video generation, DAS could help optimize temporal consis- tency and visual quality across frames while maintaining robust performance across various video quality measures and diverse motion patterns. As these domains often involve complex, domain- specific rewards and constraints that may evolve during development, DAS's ability to generalize across rewards while preserving diversity-all without requiring any training-coule significantly re- duce development cycles while ensuring robust and versatile generation.
33

---

Published as a conference paper at ICLR 2025
DONO
Alightop KL
DPS
PradDeM
Figure 15: Images generated to target aesthetic score using prompts: "bat", "cheetah", "crocodile", "gorilla", "hedgehog". "hippopotamus", "kangaroo", "lobster", "octopus", "snail".
34

---

Published as a conference paper at ICLR 2025
DDPO
Aliphhp KL
IDP
DPS
FreDDM
MIGD
(Cum)
Images generated to target PickScore using prompts: "A toy elephant is sitting inside wooden car toy.", "A white toilet in a generic public bathroom stall.", "An eye level counter-view shows blue tile, faucet, dish scrubbers, bowls, squirt bottle and similar kitchen items.". ". "People getting on a bus the city.", "Street merchant with bowls of grains and other products.". " "The black motorcycle is parked on the sidewalk.", "Three people preparing meal in small kitchen.", "a black motorcycle is parked by the side of the road.", "a dog with plate of food on the ground.", "there is red bus that has mans face it."
Figure 16:
35


Reviews:
[{"review_id": "agxbkXElWg", "reviewer": "Reviewer_Kkg4", "summary": "The paper proposes DAS, a training-free approach for aligning diffusion models with specific objectives. It uses Sequential Monte Carlo (SMC) with tempering for reward alignment. This method is demonstrated across generative tasks, including single-reward and multi-objective cases, with performance comparable to fine-tuning methods in terms of target reward optimization and diversity.", "questions": "1. The method relies on specific tempering schemes and parameters, and the practical guidelines for selecting these could be more detailed.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. DAS does not require additional training, which reduces computational cost.\n2.The use of SMC with tempering is justified through asymptotic properties.\n3. DAS balances reward optimization and diversity, and is demonstrated across single-reward, multi-objective, and online settings.", "weaknesses": "1. While DAS is compared with fine-tuning and guidance methods, comparisons to baselines like STEGANODE or controlled diffusion could have strengthened the evaluation.\n2. DAS assumes differentiable reward functions, which may limit applicability in scenarios involving non-differentiable objectives.\n3. Most experiments use Stable Diffusion v1.5, and additional models would have enhanced the generality of the findings.\n4. The paper can do more image tasks. Currently it emphasizes findings on aesthetic score, which might not generalize well to other tasks.\n5. Limitations: the setup of SMC with tempering, intermediate targets, and backward kernels can be technically demanding. And the effectiveness of DAS depends on the pre-trained model's quality, limiting performance on models with low initial diversity or reward alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "orhPLAf4lA", "reviewer": "Reviewer_J1vs", "summary": "This paper proposes a training-free diffusion sampling method based on Sequential Monte Carlo (SMC) to sample from the reward-aligned target distribution. By incorporating tempering techniques, it offers a robust solution for aligning diffusion models with arbitrary rewards\nwhile preserving general capabilities", "questions": "Please see the weaknesses part above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is overall well-written and the motivation is clear. It aims to address the trade-off in diffusion models that align them with specific objectives while maintaining their versatility, which is a critical problem in generative modeling.\n\n2. DAS’s effectiveness is comprehensively validated across diverse scenarios, including toy distribution simulation, single-reward, multi-objective, and online black-box optimization tasks.", "weaknesses": "1. More intuitive explanations of SMC are suggested to add between the motivation and method to make it more consistent and intuitive since the introduction of SMC in supplementary material is a bit abstruse to understand, making the superiority of adopting SMC to address the training problem unclear.\n\n2. How to choose hyperparameters such as $\\gamma, \\alpha$ and particles should be discussed across different scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fNlh9DXFK3", "reviewer": "Reviewer_QHss", "summary": "The paper introduces a novel, training-free sampling method aimed at generating samples from a target distribution, with a specific focus on applications in Reinforcement Learning from Human Feedback (RLHF). The authors compare their approach against existing fine-tuning baselines and guidance techniques.", "questions": "1. Baselines for Comparison: The authors correctly state that RLHF can be formulated as learning to sample from an unnormalized target distribution (Section 3.2). They show that current fine-tuning approaches in RLHF struggle to sample from multimodal target distributions, which highlights the limitations of these methods. However, this comparison may not be sufficient. There is significant research on using diffusion methods for sampling from multimodal distributions which would not fail at the examples presented in Figure 1 (e.g., [1], [2], [3] for continuous-time models, and [4] for discrete-time models). Including these approaches would provide a more convincing set of baselines. If this is not possible within the rebuttal phase's timeline, I believe it is necessary to at least mention this line of work in the paper. \n\n2. Clarification on Calculations: The calculation presented on line 153 and the following lines is unclear. Can you provide a detailed derivation?\n\n3. Explanation of Evaluation Metrics: The evaluation metrics mentioned in line 355 and onward lack a clear explanation. Currently, understanding them requires consulting multiple references. Could you include a brief explanation in the paper for clarity?\n\n4. Inference Time Comparison: How does the inference time of your method compare with fine-tuning techniques? It seems plausible that fine-tuning methods might produce samples more quickly. Is this the case?\n\n5. Novelty of the Method: Is the proposed method entirely new, or is it simply novel in the context of RLHF? How does it compare to other Sequential Monte Carlo methods?\n\nI will initially give a score of 3, but I am willing to update my score if my questions are properly addressed.\n\n# References\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Path Integral Sampler: A Stochastic Control Approach For Sampling.\" International Conference on Learning Representations.\n\n[2] Berner, Julius, Lorenz Richter, and Karen Ullrich. \"An Optimal Control Perspective on Diffusion-Based Generative Modeling.\" Transactions on Machine Learning Research.\n\n[3] Vargas, Francisco, Will Sussman Grathwohl, and Arnaud Doucet. \"Denoising Diffusion Samplers.\" Eleventh International Conference on Learning Representations.\n\n[4] Sanokowski, Sebastian, Sepp Hochreiter, and Sebastian Lehner. \"A Diffusion Model Framework for Unsupervised Neural Combinatorial Optimization.\" Forty-First International Conference on Machine Learning.\n\n[5] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, Il-Chul Moon Proceedings of the 40th International Conference on Machine Learning, PMLR 202:16567-16598, 2023.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction provides a clear overview of the problem.\n\n- The proposed method appears promising and might be innovative (see Question 5.)", "weaknesses": "- The choice of finetuning-based RLHF baselines may not be appropriate (see Question 1).\n\n- The paper is sometimes hard to follow due to the delayed definition of new notations. For instance, the symbol $\\gamma$ is used on line 208 but is not defined until line 250.\n\n- The evaluation metrics used in the paper (line 355 and onward) are not explained, making it difficult to assess their relevance and meaning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A4bARUT2Mr", "reviewer": "Reviewer_31k3", "summary": "This paper propose a training-free sampling method based on Sequential Monte Carlo (SMC)  to align the diffusion models with specific objectives. Specially, Diffusion Alignment as Sampling (DAS) is designed to address the  limitations of the previous alignment approaches include fine-tuning and guidance methods. The author also provide theoretical analysis of DAS’s asymptotic properties and empirically validate DAS’s effectiveness across different tasks. Meanwhile, the authors conducte sufficient experiments to verify the validity of the DAS methodology.", "questions": "Please refer to Weaknesses. I will also refer to other reviewers' comments", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and there is a good discussion of the work involved. Based on the fact that the existing fine-tuning methods lead to the reward overoptimization problem while the guidance methods lead to the under-optimization problem, the authors propose the DAS method to alleviate these deficiencies. In addition, the authors provide a theoretical analysis of the method and give the relevant code, making the work very solid. Figure 1 illustrates the shortcomings of the existing methods as well as the advantages of the proposed method, and the experimental results are visualized by using an example of a mixed Gaussian distribution.", "weaknesses": "+ The models underlying the experiments in this paper have some weaknesses, and the Stable Diffusion (SD) v1.5 model is somewhat outdated now. The Consistency model [1] and Flow model (SD3) [2]  are widely used nowadays, so I suggest the authors to conduct some experiments on the newer model so as to further illustrate the validity of the proposed method.\n\n[1] Consistency models ICML-2024\n\n[2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis ICML-2024\n\n+ In addition to mixing Gaussian distributions, **Swiss rolls** are also commonly used to visualize whether a distribution has been learned or not, and due to their structural features, which can further reflect the model's ability to fit the distribution, the authors can give some visualizations that further illustrate the strengths of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)