Training-Free Diffusion Model Alignment with Sampling Demons
Authors: Po-Hung Yeh
Kuang-Huei Lee
Jun-cheng Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Model
Consistency Model
Score-Based Method
EDM
Stochastic Process
Itô's Lemma
RLHF
Value Function
Preference Alignment
Primary area: generative models
TL;DR: First inference-time, backpropagation-free preference alignment method for diffusion models

Abstract: Aligning diffusion models with user preferences has been a key challenge.
Existing methods for aligning diffusion models either require retraining or are limited to differentiable reward functions.
To address these limitations, we propose a stochastic optimization approach, dubbed *Demon*, to guide the denoising process at inference time without backpropagation through reward functions or model retraining.
Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through stochastic optimization.
We provide comprehensive theoretical and empirical evidence to support and validate our approach, including experiments that use non-differentiable sources of rewards such as Visual-Language Model (VLM) APIs and human judgements.
To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models.
Our method can be easily integrated with existing diffusion models without further training.
Our experiments show that the proposed approach significantly improves the average aesthetics scores for text-to-image generation. Implementation is available at [this URL](https://github.com/aiiu-lab/DemonSampling).

Published as a conference paper at ICLR 2025
TRAINING-FREE DIFFUSION MODEL ALIGNMENT WITH SAMPLING DEMONS
Po-Hung Yeh', Kuang-Huei Lee", Jun- Chen'
Academia Sinica, Google DeepMind
ppohungyeh, pullpul1/0cin (capnl)@@cii... sinica edu. tw, Lee.h@googee..com
ABSTRACT
Aligning diffusion models with user preferences has been a key challenge. Existing methods for aligning diffusion models either require retraining or are limited dif- ferentiable reward functions. To address these limitations, we propose a stochastic optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density regions corresponding to high rewards through stochastic optimization. We provide comprehensive theoretical and empirical evidence to support and val- idate approach, including experiments that use non-differentiable sources of rewards such as Visual- Visual-Language Model (VLM) APIs and human judgements. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-ffie
preference alignment method for diffusion models. Our method can be easily integrated with existing diffusion models without further training. Our experiments show that the proposed approach significantly improves the average aesthetics scores for text-to-image generation. Implementation is available at //gitt/////ttbb com ai iu- lab htt///Dmoonnaaaaa ing
INTRODUCTION
Diffusion models have been the ifte-oofthhe-rd for image generation (Sohl-Dicksteil et al.) 2015 Ho et al. 2020 Song al. 2021 Karras et al.) 2022 Saharia et al. [022]] Rombach et al., 2022), but, commonly, the end users' preferences and intention diverge from the data distribution on which the model was trained. Aligning diffusion models with diverse user preferences is ongoing and critical area of research.
One approach to aligning diffusion models with user preferences is to fine-tune using reinforcement learning (RL) to optimize the models based on rewards signals that reflect the user preferences (Black) et al. 2023 Fan et al. 2023). However, retraining the model every time when the preference changes is computationally expensive and time-consuming.
An alternative approach is guide the denoising process using differentiable reward function. This can be done through classifier guidance inference time (Dhariwal & Nichol 2021: Wallace et al. 2023b Bansal al. 2024 Yoon et (2023) or backpropagation training time (Prabhudesai et al. 2024 Clark et al., 2024 Xu et al. 2023). These methods are generally less resource-demanding and more efficient. While these methods are generally more efficient, they require the reward function to be differentiable. This limits the types of reward sources that can be used, as it excludes the conddifferentiable sources like third party Visual Langul-L Model (VLM) APIs and human judgements.
To address these limitations, we propose Demon, a novel stochastic optimization approach for preference optimization of diffusion models at inference time. Demon is metaphor from Maxwell's Demon, an imaginary manipulator of natural thermodynamic processes, The core ideas are: (I) Quality of noises that seed different possible backward steps in discretized reverse-time Stochastic Differential Equation (SDE) can be evaluated given a reward source; (2) Such evaluation enables us synthesize "optimal" noises that theoretically and empirically improve the final reward of the generated image through stochastic optimization. Specifically, we leverage Probability Flow Ordinary
Differential Equation (PF-ODE) (Song al. 2021) or Consistency Model (CM) (Song al. 2023

---

Published as a conference paper at ICLR 2025
z (2)
A
fp
(2)
Noomiited
x(K
PK-1
x(K)
bx
Figure I: Illustration of Demon. Given a reverse-time SDE for denoising and an interval [mmax bmin]: we first discretize into T steps, fmax denoising step, from t to A. we synthesize an "optimal" noise z' from K i.i.d. noises w.r.t given reward source and use generating images that are more aligned with the reward source and the preference that the reward source represents. More details are presented Section
-
mmin- At every reverse-time
seed the step. This enables guiding the denoising process towards
Luo et al. 2023 to help us efficiently evaluate the possible backward steps, seeded with different Gaussian noises.
Our key contributions are summarized as follows:
Our approach enables the use of reward signals in the denoising process regardless of whether the reward function is differentiable. This allows for the incorporation of previously inaccessible reward sources, such as VL.M APIs. To the best of our knowledge, this the first inference-tings backpropagation-free preference alignment method.
Our method can be easily integrated with existing diffusion models in a plug-and-play fashion without retraining fine-tuning.
We provide theoretical explanation for why our approach can improve the given reward function for image generation, which can be exploited for tuning hyperparameters.
We demonstrate that our approach significantly improves the average aesthetics score (LAION 2023) of Stable Diffusion models, achieving averages well above 8.0 com- pared to the Best-of-N random sampling upper bounds of 6.5 for SD v1.4 and for SDXL. This improvement achieved across various text-to-image generation tasks using prompts from prior work (Black et al., (2023), without relying on backpropagation-biin preference alignment or model retraining.
2 RELATED WORK
Diffusion Model. Diffusion models for data generation were first proposed by Dicl--Dccktten et al.
(2013). further developed for high-fidelin image generation by Ho et al. (2020), and generalized by
Song et al. (2021) through the lens of SDEs. Karras et al. (2022) comprehensively studied the design
space of Diffusion SDEs. this work, we base many of the derivations on theirs. Furthermore, we
focus on evaluating our method in the text-to-image generation setting (Rombach al. 2022 Ho &
Salimans
2021 Podell et al. 2024)
Human Preference Alignment. Aligning models with human preferences has been studied with
several wwwroacche.rrinnooreemiie
learning-based policy optimization (Fan et al. 2023 Yang et al.
2024: Black et al. 2023); y training with reward backpropagation (Clark et al. 2004 4 Xu al. 2023
backpropagation through the reward model and the diffusion chain (Prabhudesai et al. 2024 Wa lace
et al. 2203bb Bansal et al. 2024 Yoon et al. 2023). Many metrics and benchmarks for evaluating (irssainn alignment has also been proposed, including those by Xu et al. (2023): Kirstain et al. (2023) LAION
(2023); Wu et al. (2023). and these either optimization objectives evaluation of the
2

---

Published as a conference paper at ICLR 2025
generated image. In Table we further provide detailed comparisons of the proposed Demon approach with relevant existing methods the literature from different aspects.
Table A detailed comparison of different methods along various dimensions, including the ability to generalize to open vocabulary, the necessity of backpropagation signal for optimization, the method's is capacity to avoid mode collapse and ensure distributional guarantees (Divergence Control). Our proposed method stands out for its zero-shot learning capabilities.
Open Vocab
Non-Backprop Objective
Divergence Control
Type
Methods
Training
DPOK Fan et al. 2023
Training
DDPO (Black al. 2023
Inference
DOODL Wallace et al. 2023b
X
Training
DPO (Wallace et 2023a
Training
DRaFT (Clark et al.
2024
X
X
Inference
Demon
PRELIMINARY
Score-Based Diffusion Model. We base our derivation on EDM (Karras et al. 2022). With sampling schedule og = t, we can write the reverse-time SDE sampling towards the diffusion marginal distribution follows.
dx, [ (x,, Bt log (x,,t) dt + 2,3t dui,
(1)
(x(
(a)
where t(x,,t) = p(xo, 0) N (0,tI,) and @ denotes the convolution operation. X is a clean sample, Xa ~ Pdatas and x, is noisy sample time I. 3 expresses the relative at which existing noise injected with new noise. In EDM, 3 function of I, but in our study, we set 3 to constant for all for simplicity. Essentially, fg(x, L) corresponds to drift and ga corresponds to diffusion. As common in diffusion models, since p(x,, t) xs (0, IIx) for large enough sample Xmma ~ NO, P In) ( as the initial sample.
A comprehensive list of the notations and conventions used in this paper is provided Appendix A
REWARD-GUIDED DENOISING WITH DEMONS
In this section, we describe how Demon works in two steps: Section 4.1 explains the process of scoring Gaussian noises in reverse-time SDE with a reward function; Section [4.2 further explains how the noise scoring allows to guide the denoising process to align with the reward function. which is what we refer to as Demon.
4.1 SCORING NOISES IN REVERSE- TIME SDE
Let Xo be the clean image corresponds a x, time step I, say:
Xo x, J. (x,, u) du + g dw
(2)
where Equation 22 denoted as Xo la x4, shorthanded as Xo X₁- For an arbitrary reward function r e.g. aesthetics score, we define the reward estimate of x, time step as
r (xxx,, t) := Exolx, [r(x))]-
(3)
This be estimated with Monte Carlo estimator by averaging over the reward of several SDE samples, but requires many sample evaluations for high accuracy. To address this weakness, we introduce alternative estimator for (x,,,,,)) based on PF-ODE (Song et al. 2021).
3

---

Published as a conference paper at ICLR 2025
r(xp²)
r(x!"))
Reward
(ro cx(,)
r
rp(x:)
Ex, [x(r(
r(xb=))
14)
X
(1)
c(X+)
X
(2) X
Figure 2: The illustration of the proximity between the rag and r.c. this figure, the 3 is nonzero and r near harmonic (i.e., V2r 0.). The red points indicate i.i.d. SDE samples and the purple ODE approximation of X,. The green line indicates the expectation of the rewards of the SDE samples (e.g., an approximate estimation, E r(x)))) )).
As shown in Song et al. (2021) Karras et al. (2022), the reversed-time SDE reduces to PF-ODE when
3 = 0. For each t, a diffeomorphic relationship exists between noisy sample x, and clean sample Xo generated by PF-ODE.
Similar to consistency models, with x denoting an ODE trajectory instead of xt, we can denote this deterministic mapping from the domain of x, to the domain of xo c(x,, ()
c( c(x;, = xo = x f dx
=xx+
where dx = -u V log (X u) du.
(4)
Then, we can write (r (x,, = r(c(x,, 1)) as the reward of the generated clean sample. This approximates r3(x,, using only evaluated sample. In fact, between the approximate reward using ODE (r (x,, t) and the exact reward estimate using SDE (x(x, in Lemma D
characterize the difference
the approximation becomes exact: lim + ra(x,, f) = (r o (x,, t). Intuitively, this result aligns with SDEs reducing ODEs when 33 approaches zero in image domains (Song et al.) (021].
The right hand side of Equation (5) shows that,
Lemma (Itō Integral Representation of Reward Proximity Error. Proof is in Appendix D.I the reward estimate function, h(x,, () = (r c)(x,, !), be shorthanded as h. We have:
Let
To(x;, t) (r c) = Exo Io S." Vx.h dJ dd Bu Bur 2
(5)
where Xa is sampled from Equation
4hh is the Laplacian of h and
dJ g (X u) Bu log (Xu; du V 2/3u dws.
(6)
As demonstrated in Appendix D.I Lemma limplies that when the Laplacian of the reward function is approximately zero (V2r = 0), r3 =s r c. We also illustrated the idea in Figure 2 For better presentation, we conveniently abbreviate r (xe, () (xt),,, c(xe, () c(x.) and o (x,, () as (r o (x(,) in this paper.
4.2 DEMONS FOR LEWARD-GUIDED DENOISING
In the section, we first outline the general pipeline the proposed algorithm. Then, we introduce two approaches, Tanh Demon and Boltymann Demon, synthesize optimal noises for guiding reverse-time SDE solution; we show that the proposed methods optimize the final reward value with theoretical guarantee, essentially achieving alignment.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 A Numerical Step with Demon
1: Input: x,, f, A, K
2: Output: X,-A
3: for k == to K do
4: Draw (8) I,)
5: & heun(&,, (h)) ,, A)
6: Ra t ( fimplementing ra ( (-)))
7: end for
8: [bx] + (Rx)
9: z" + VN normalized EE bzz2(k)
10: x + houn(&, z", ,,, A)
Return X.-A
Following Karras et al. (2022) SDE numerical evaluation of & sampled from can be seeded
by noise z via a step of Heun's s 2nd order method (Ascher & Petzold 1998) as follows:
z ~ N,)
(7)
A heehheu (X,,
(8)
Xx
2 fo (x). + ( fa(X(,,, A)] A (ga(t) + ga(t A)]
(9)
where z is Gaussian noise, and heun is the stochastic backward step from X to & The intermediate approximation A is given by X = x, - fa(x, t) gt)t)v While we use Heun's method here, other solvers can work too.
For image generation, Gaussian noise z is usually high-dimensional. For high- high-dimensional z, we can assume that it's likely on a / sphere (Lemma Appendix). This allows us to weighted-combine various noises into a new noise z":
z V normalized
Σ by (2))(a)
(10)
where (*)) are i.i.d. unit Gaussian noises, and by are the search space. We outline the pseudocode of a numerical step with our proposed method in Algorithm [ the following, we describe the details of the proposed Tanh Demon and the Boltzmann Demon to determine the weights by-
Intuitively, we may consider up-weighting the good noises that improve the reward and down-weiggs weighttig the bad noises that harm the reward, compared to the average reward j. As shown in Figure 3 Tanh Demon assigns positive weights to the good noises and negative weights to the bad noises with the tanh function, based the reward estimates of the noises (Equation (5)
Tanh Demon.
relative to the average of (r o (xx(x)
K Σ
((roc)(2())) (11)
x" N normalized
tanh (k)
where
gaanh tanh
where is the temperature parameter to tanh, which can be adaptively tuned (as shown in Table B).
The average ğ computed & Σ ()))
Under the assumption of our reward estimate proximity r3 = rro c, the Tanh Demon method is guaranteed improve the final results, formalized in the following lemma:
Lemma (Improvement Guarantee of Tanh Demon. Proof in Appendix D.3). Assume the truncation error terms in Equation (36) are negligible and ra = r c. Let x² be derived from Equation 11]
With probability 1, r r(Stamb) r rx(x)), where stanh is derived by applying z every step.
5

---

Published as a conference paper at ICLR 2025
Z 4)
2 (3)
(
(2)
Z (2)
(3)
Z
(4)
(a) Before
(b) After
(c) Final
Figure 3: An illustration of the Tanh Demon sampling method where K 4. (a) A SDE step generates several samples, each determined by sampled noise Z₃. We use Tanh Demon to classify each noise sample "low-reward" "high-reward" w.r.t r5(1,) based their respective reward estimates. (b) We penalize low-reward noise with tanh to multiply a negative weight which is equivalent flipping the noise, (c) It shows how the post-processed noises are averaged and projected onto the high-dimensional sphere, resulting in a feasible noise representation z with high-reward estimate.
Boltzmann Demon. Another intuitive approach, equivalent to the single-step entropy ap- proach (De Boer et al. 2005), is to estimate the candidate with maximum reward. We propose the Boltzmann Demon, which assigns noise weights as follows:
exp (roc(x)
bboltz
(12)
Ek exp (roc(s(-)))
where the Boltzmann distribution (i.c., softmax function) approximates the behavior of the maxi- The theoretical guarantee of improvement
mum function as the temperature approaches in ra in expectation is provided in Lemma assuming r = r o c. Although empirically, we find that Tanh Demon outperforms Boltzmann Demon, adjusting in Boltzmann Demon provides control over deviation from the original SDE distribution, as demonstrated in Lemma (Appendix).
4.3 COMPUTATIONAL CONSIDERATIONS
Let's first consider a Demon sampling trajectory Xt, > Xt, > ... Xty R 0 for a fixed number T. Each Demon's trajectory requires O(K T)) evaluations of c, and each evaluation comes with one reward estimation. The compute time is mainly influenced by the implementation of r We discuss two aspects of time complexity and reward performance, respectively.
the temporal cost and the fidclity-wiihh are vital to the algorithm's
Note that Tanh Boltzmann Demon itself does strictly specify the implementation of r c; our default option uses Heun's ' ODE solver, but using a Consistency Model (CM) distilled from the original diffusion model significantly accelerates computation. An alternative, which we refer as Tanh-C, C. is to combine our Tanh Demon algorithm with an off- off-the-shelf CM to implement r coc. While using Tanh-C may slightly degrade the results due the fidelity loss from using CM (see Table ( this approach is particularly effective when faster results are required since the computation of much quicker. For larger T, however, the default Tanh Demon using Heun's method outperforms Tanh-C in terms of reward performance.
As shown in Table 10 using the text-to-image generation task settings from Black et al. (202)), the Demon algorithm achieves aesthetics score of 6.72 + 0.26 on SD vl.4, requiring 5 minutes (i.e., K = 16, = 16) on an NVIDIA RTX 3090 GPU. Within the same 5-minute computation window, the Tanh-Co C variant achieves improved score of 7.27 + 0.33 (i.e., K 16. T 64). Notably, the upper bound for randomly sampled SD v1.4 is approximately 6.5, obtained after more than 10 minutes and 800 reward function queries. See Appendix for parameter guidelines and settings.

---

Published as a conference paper at ICLR 2025
Croakling Natr PUCO 1o Baak
Aesthetics Score
Aesthetics Score "]
5
2
2
2
2
2
2
21
2
Numer Reward Function Queries
Execution (seconds)
(a) Performance w.r.t Reward Query Number
(b) Performance w.r.t Execution Time
Figure 4: Performance comparison of the proposed algorithm and other baseline methods in terms of the number of reward queries and execution time; the dependent variable is T', which is suggested to be larger for SDE solver reduce truncation error. Although DOODL can achieve similar results to ours, it relies on reward backpropagation, whereas our backpropagation- methods do not require this. The shaded areas and the radii of solid circles represent the standard deviation the evaluation results.
EXPERIMENTS
In this section, we present both quantitative and qualitative evaluations of our methods. Due to the page limit, we include the details of the implementation and experimental settings in Appendix [ subjective results Appendix G.2
Baseline Comparison. For the performance comparisons between our method and other baselines, use the LAION (2023) aesthetics scores (Aes) as the evaluation metric, and the scores are evaluated on a set of various prompts for generating animal images, which were from the full set of 45 common animals ImageNet- 1K (Deng et 2009), created by Black al. (2023.. We use 20-step Heun's 's ODE for reward estimate for our methods and Best-of-N (SD v1.5).
In terms of reward queries, our Boltzmann method and Best-of-N, Our methods are even comparable to the backpropagation- based DOODL. (Wallace et al. [0236)) the state-of-the-art method optimized over the reward function. In terms of execution time, Tanh/Tanh-C consistently outperforms DOODL due to the exclusion of backpropagation; Tanh-C further benefits from its effective computational cost, given limited time. Moreover, our method's nackpropagation-free nature makes it more resistant to reward hacking (Table 3). For further comparison on PickScore (Kirstain et al. 2023), please refer to Appendix E.
outperforms other baseline methods in most cases, including
Comparison of Reward Estimation Approaches. Figure demonstrates a comparison of the proposed methods with different r c implementations, including 20-step Heun's ODE (Tanh) and I-step CM (Tanh-C). Tanh, which uses 20-step ODE for accurate (r o c). consistently outperforms the Best-of-N baseline given an equivalent number of reward queries. Tanh-C, which employs 1-step CM for fast reward evaluation, outperforms Tanh when considering limited execution time. These observations suggest that the quality of indeed plays a significant role the effectiveness of our method.
To further validate the importance of r c, we conduct a comparative analysis based on Lemma (r o R ra). In this analysis, evaluate the accuracy and computational cost across three methods: 20-step Heun's ODE, 4-step Heun's 's ODE, and 1tstep CM; both diffusion and consistency models based on the SD v1.5 and distilled by Luo et al. (2023). Experiments were performed with t = 0.5, 3.0, 1.0, 7.0, 14.0 ranging from 0.002 to 14.648, Accuracy was quantified using the standard of (xx,)) (r (x;). Here, is sampled from (O,, 12 12 I,) and integrated from f f using a 200-step diffusion model ODE, performed on the full set of Black et al. (2023).
The results, presented in Table support that the quality of r influences both the algorithm's speed and reward performance. For the ODE methods, the trend follows our expectation: As t approaches 0, the standard deviation decreases, which be attributed the diminishing noise
7

---

Published as a conference paper at ICLR 2025
Table Comparison of accuracy and time cost across different r implementations using the full set of animal prompis Black et al. (2023).
Standard Deviation r, roc
Implementation
Time (s)
1.0
1 3.0
t 7.0
[ 14.0
20-step ODE
1.44
8.1 x 10-
1.53 10-1
3.02 10
3,25 x 10
3.97 10
4-step ODE
0.41
8.3 x 10
1.82 10-1
3.39 x 10"1
3.52 x 10"1
4.15 x 10"
step Consistency
0.18
1.71 x 10
2.64 10
4.03 x 10
3.85 x 10
4.85 x 10
as the posterior p(x, xo) becomes more sharply peaked; the number of ODE steps is crucial to the quality of the generated outputs; more steps generally lead to higher fidelity results, although this comes at the cost of increased computational time; using 1-step CM leads to inferior results compared to using ODE, supposedly as the distillation gap and the limited model capacity result in lower-fidelity reconstructions.
Table 3: Results using various reward functions and different generation methods. Each column represents a specific reward objective, with the best performance highlighted in bold.
Generation method
Aes
IR
Pick
IIPSv2
Time
SD v1.4
5.34 0.56
0.202
5
DPO
5.36 + 0.72
0.03 + 0.84
0.203 = 0.007
0.229 = 0.027
5
Uni (CLIP-guided)
4.11 0.74
-1.11 0.50
0.191 0.014
0.173 0.022
55 min
Tanh Acs
7.35 0.40
-0.03 = 1.24
0.211 = 0.010
0.257 0.041
Tanh
5.96
Tanh Pick
6.14 0.48
1.39 1. 0.57
0.245 1 0.010
0.312 0.033
18 min
0.45
0.111
Tanh Ensemble
6.53 0.50
1.81 + 0.15
0.236 = 0.014
0.356 0.030
Best-of-N
6.32 0.34
0,218
0.291
18
DOODL- Aes
5.59
0.29
-0.68 :: 1.06
0.197 + 0.008
0.221 + 0.028
18 min
DOODL+
Table 4: Using Tanh Demons with various reward functions. The baseline, Stable Diffusion v1.4, refers to the standard model without our proposed enhancements.
Baseline
Best-of-N
Uni
DOODL
Aes
Ensemble
DPO
A demon exiting through portal
A painting of girl encountering giant sunflower blocking her path in a hallway
Image Generation with Various Reward Functions. While our method optimizes given reward function, shown in Figure 4 we also present qualitative results in Table 4 and cross-validation results in 3 These results demonstrate perceptual preferences by averaging rewards derived from prompts provided in Tables and 15
employ our Tanh Demon with various reward functions, such as Aes (LAION 2023), ImageRe- et al. (202),, HPSv2 (Wu al.) 2023), and
ward (XR) et al.] 2023). PickScore scaled sum (Ensemble) of Aes, IR, Pick, and HPSv2. For comparison, we include the best reward during sampling under the same time condition (best-of-N) and DOODL (Wallace al.) 2023b), which is optimized on Aes and Pick to modify results generated by PF-ODE using their recommended settings. For reference, we also provide the performance of training- -aased method, DPO Wallace
8

---

Published as a conference paper at ICLR 2025
et al. 2023a), and a backpropagation sraeroppaggatoonbbeeedddd method, Universal Guidance (Uni) by Bansal al. (2024). guided by the CLIP condition.
Starting from the baseline SDv 1.4 under similar computational conditions, Tanh exhibits improvement across the four metrics, demonstrating robustness even when acknowledging slight over-optimization of the objective-in contrast to DOODL. By comparing best-of-N and Tanh with the Ensemble reward, our method achieves superior performance on each objective using the Ensemble, demonstrating only the ability integrate a mixture of rewards but also generating a superior samples that outperforms all individual best samples selected by best-of-N methods.
Alignment with preferences of VLMs (Non-differrettaable) In Table we present qualitative results of aligning diffusion models SDXL to preferences of VLMs from API, as a demonstration of using non-di difffereretiabblee reward sources. this experiment, we use Google Gemini Pro v1.0 (Gemini Team Google, (202)) and GPT4 Turbo (OpenAl 2024). In each step, the VLM receives a fixed prompt, e.g. "You are a journalist who wants to add a visual teaser for your article to grab attention on social media your news website", and is asked to select the best-matching intermediate sample
from generated images. VLMs are presented with c(x;) and c(x(")) produced by PF-ODE. The
(k) reward 6VLM is 0.5 the VLM selects c(x(") A) and -0.5 otherwise. We also use PickScore (Kirstain et al. 2023) to evaluate the results and find that 14 out of 16 images generated with VLMs show
improvements compared directly generating with PF-ODE.
Table 5: Using VLMs generate images. PF-ODE (baseline) refers a baseline without using our method for alignment. Columns 3-6 indicate the role that the agent plays in the given prompt.
Model
Baseline
Teacher
Artist
Researcher
Journalist
Gemini-SD v1.4
AURSST OHO STRTTIN
Gemini-Si SDXL
OPr-sD v1.4
GPT-SDXL
Manual Selection. We also explore using online interactive human judgements to guide diffusion. That means, the users themselves would be cion-differentiable) reward functions. We users directly interact with our method to generate desired images. Figure 5a shows an example interface created by us for an image resembling a given reference cat image. At each iterative step from é to t - A, we sample 16 i.i.d. copies of æt-A and compute c(Ø,-A) a) with PF-ODE. The user then manually select their preferred image, assigning a reward of +1 to it and - to the others. We continue this process until there is obvious preferred ones among the generated images. As shown in Figure Sbb the image generated by our method more closely matches the target than the one produced by PF-ODE. We also measure the improvement with DINOv2 (Oquab al. 2023) embedding cosine similarity

---

Published as a conference paper at ICLR 2025
(b) (Top Left) Image generated by PF-ODE (0.622 cosine similarity). (Bottom Left) Image generated by our method (0.708 cosine similarity). (Right) Reference image.
(a) Our user interface for interacting with our algorithm (0.594 cosine similarity).
Figure 5: We design an application for manual interaction with our algorithm. Our author selects the images, and the criteria are based on the author's preference (non-preferred images are kept unselected), where the author tries to align the reference image. We evaluate performance by measuring the cosine similarity of DINOv2 features between the targeted and reference images.
between the reference image and the generated image, and observe that the similarity improves from 0.594 to 0.708 through online user interactions.
CONCLUSION
This work addresses the challenge better aligning pre-trained diffusion models without training backpropagation. We first demonstrate how estimate noisy samples' rewards based on clean samples using PF-ODE. Additionally, we introduce a novel inference-time sampling method, based on stochastic optimization, guide the denoising process with any reward sources, including non- differentiable reward sources that includes VLMs and interactive human judgements. Theoretical analysis and extensive experimental results validate the effectiveness our proposed method for improved image generation without requiring additional training. Through comprehensive empirical and theoretical analysis, we observe that the quality and efficiency of reward estimation r are essential for our algorithm, especially in balancing computational speed and reward performance.
ACKNOWLEDGGEMENT
This research supported by National Science and Technology Council, Taiwan (R.O.C), under grant number of
11TTC112-2222--2 1--00--01--YY- 001-002-MBK and Academia Sinica under the grant number of AASCCDA-11--MA We specially thank Sirui Xie, I-Sheng Fang, and Wei Liao for the insightful discussions. We also extend gratitude to Prof. Lam Wai-Kit, National Taiwan University, for his courses MATH7509, MATH7510, and MA7H5269, which have significantly influenced this paper.
REFERENCES
Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and differential-algetee equations. SIAM, 1998.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. International Conference Learning Representations, 2024.
10

---

Published as a conference paper at ICLR 2025
Patrick Billingsley. Probability and measure. John Wiley & Sons, 2017.
Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Machine Learning. 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations, 2024.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Amnals of operations research, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
Prafulla Dhariwal and Alexander Quimn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Ying Fan, Olivia Watkins, Yuging Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning eext-to-image diffusion models. In Advances in Neural Information Processing Systems, 2023.
Gemini Team Google. Gemini: A family of highly capable multimodal models, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS Workshop on Deep Generutive Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Charles AR Hoare. Quicksort. The computer journal, 1962.
Kiyosi Ito, Kiyosi Itô, Kiyosi Itô, Japon Mathématicien, Kiyosi Itô, and Japan Mathematician. On stochastic disferential equations. American Mathematical Society New York, 1951.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.
Andrei Kolmogoroff. Über die analytischen methoden in der walnschenniichkeeisrccnnngg Springer, 1931.
LAION. Laion aesthetic score predictor, http://s/aaaooaa///////aaaaaaat. 2023.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
OpenAl. Gpt-4 technical report, 2024.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal. Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
11

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation, 2024. URL https:///penrreveew... net/
Forum?id=Vaf4s
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
Alvin C Rencher. A review of "methods of multivariate analysis, ", 2005.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recogni-
tion, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour; Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, 2023.
Roman Vershynin. High-dimensional probability. University of California, Irvine, 2020.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023a.
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimiza- tion improves classifier guidance. In International Conference on Computer Vision, 2023b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model, 2024.
ТаeHo Yoon, Kibeom Myoung, Keon Lee, Jaewoong Cho, Albert No, and Emest K. Ryu. Cen- sored sampling of diffusion models using minutes of human feedback. In Advances in Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2025
A NOTATIONS AND CONVENTIONS
Although keep the main paper self-consistent, provide this section establish consistent notation and convention for this paper as an aid.
A.1 NOTATIONS
Table Notations
Notation
Description
State dimension
K
Noise sample number
Emins fmax
Upper bound and Lower bound the noise level in numerical integration Number of time steps to solve SDE/ODE
T
Noise parameter
State variable
z
Noise from Gaussian
Time step
bk
Unnormalized weight of noise SDE policy drift
fg
ga
SDE policy diffusion coefficient
PF-ODE policy drift
J3
Langevin diffusion SDE reversed time Brownian motion Reward
W1
Reward estimates of SDE policy
Function get expected ODE result Heuns's 's method, SDE solver for Karras SDE
heun
A.2 CONVENTIONS
Table 7: Conventions
Convention
Details
TOC
ODE reward estimate approximation, r(c(x,, t)) (r (xt,,)) For all of our interest, f fxx) g(x)
f= 9
&
Numerical approximation with SDE solver Intermediate value of Heun's 's method
x
x'
An ODE trajectory
Uniformly sampled from the sphere of radius VN Optimal noise generated by our algorithm
z²
Mean of next state ODE reward estimates, K E (r c) (roc)))) Shorthand for r(x,, when the context clear
r(x:)
c(xi)
Shorthand for c c(x;, t) when the context is clear
(x)))(
Shorthand for (r c) (x,, t) when the context clear
Xo X1
Shorthand for Xo |e Xt, where Xo = x, + 5 fx(Xu, u) du + gæ(u) dw Standard Brownian motion
W
Instead of ODE, we sometimes use PF-ODE to highlight Song et al. (2021 s contribution or when the context unclear. They equivalent in this paper.
13

---

Published as a conference paper at ICLR 2025
B GUIDELINE ON PARAMETER SETTING
We explore the optimal setting for parameter T with respect to the Boltzmann Demon and the Tanh Demon. For the Tanh Demon, the most effective is neither nor We recommend setting 7 to the standard deviation of the estimations {(r ((=)))) rendering it an adaptive parameter that is robust to scaling. For the Boltzmann Demon, optimal performance is achieved by setting 7 to 0, as demonstrated in Table g
Table 8: Comparison of performance for different settings of r in the setting of Figure
7 1
-= 0.01
Adaptive 7
Tanh
7.40 + 0.30
7.24 + 0.31
7.45 = 0.33
Boltzmann 6.30 + 0.35
7.28 + 0.30
6.85 + 0.37
We also conduct an ablation study on the remaining parameters K and B. The base configuration K 16, 3 0.1, with adaptive temperature for the Tanh Demon. We set T 32 for the ablation study of B and 7 64 for K.
Acs vs
Aes vs Beta
3.0
75
33
7.0
6.0
5.5
10
Dato 181g scales
Figure 6: Comparison of our algorithm with respect to K and
We found large 3 makes the sampling unstable, given the number of steps 7' is fixed. Predictably, sampling with a close to 0 is reduced to ODE. From our theoretical result Lemma [ the design methodology, and empirical results, the guidelines Table can assist users in setting parameters. We provide a sparse parameter search in Table
Parameter
Description
Controls the noise distribution bias, positively affecting final quality and linearly increasing computational time.
K
Adjusts the distribution's proximity the original PF-ODE. Set em- pirically based on r's characteristics. Lemma] suggests smaller / for reward functions with Laplacian deviations.
8
Inherit the properties of time steps T from diffusion models, scaling computational time linearly. Karras's 's EDM recommends T > 17. Recommended values vary for Boltzmann and Tanh Demons, detailed
T
in Table g
roc
Accurate reward estimates are critical for ensuring high final quality.
Table 9: Guidelines for Setting Hyperparameters
14

---

Published as a conference paper at ICLR 2025
Table 10: This table presents the experimental configurations used to measure aesthetics score under various animal prompts, presenting a sparse search of parameters. The time column represents the duration required to generate each image. We alias adaptive temperature as Adaptive.
Demon
Checkpoint
B
Aes
Time (min)
Adaptive
6.408 = 0.36
17.6
SD vl.4
0.1
16
64
le-10
7.111 = 0.32
16.6
Boltzmann
Adaptive
6.853 = 0.37
45.8
0.05
32
le-02
7.276 = 0.30
45,4
SDXL
6.300 0.35
46.1
Adaptive
6.990 0.38
94.2
0.1
16
64
le-10
7.501 0.31
93.1
16
Adaptive
6.723 + 0.26
5.0
0.05
16
32
Adaptive
7.073 = 0.22
9.7
SD v1.4
64
Adaptive
7.394 = 0.29
18.7
16
64
Adaptive
7.549 = 0.43
18.7
0.1
64
64
Adaptive
8.566 0.33
79.1
Tanh
Diffusion-DPO
0.1
Adaptive
7.564 = 0.34
94.5
0.01
16
Adaptive
6.876 = 0.40
22.0
16
Adaptive
6.866 = 0.35
21.9
0.05
16
Adaptive
7.459 = 0.33
46.0
SDXL
le-02
7.244 = 0.31
46.0
7.398 + 0.30
46.2
Adaptive
7.446 0.37
47.0
0.1
Adaptive
7,841 = 0.32
94.4
32
64
Adaptive
8.179 + 0.35
188.8
0.5
16
32
Adaptive
6.370 + 0.35
46.0
0.5
16
64
Adaptive
7.269 = 0.33
5.0
Tanh-C
SD v1.4
0.1
16
64
Adaptive
6.710 = 0.34
5.0
SDXL
0.5
Adaptive
7.301 0.24
17.9
15

---

Published as a conference paper at ICLR 2025
C
PSEUDOCODES
As an aid, we provide pseudocodes for the design of Demons Algorithm
2
Algorithm
Algorithm 2 Tanh Demon with Adaptive Temperature
1: Input: A list of ODE reward estimate [RR]
2: Output: Noise Weights [bg]]
3: K + length(
4: j R +E
K
5: T + V E (Rk - 12)2
6: for k = to K do
7:
tanh RE-A
8: end for
9: Return [bg]
Algorithm 3 Boltzmann Demon with Fixed Temperature
1: Input: A list of ODE reward estimate [Rs]
2: Output: Noise Weights [bx]
3: K + length([/R.)
4: + # +
K k= exp
R
5: for k = to K do
6:
exp
7: end for
8: Return [bx]
16

---

Published as a conference paper at ICLR 2025
D MATHEMATICS
D.1 ERROR COMPRBIENSION FOR REWARD ESTIMATE APPROXIMATION
In this section, present the theoretical analysis and proof to better understand the error in reward estimate approximation.
D.1.1 ERROR TERM AS AN ITÔ INTEGRAL.
Lemma 1. Let the reward estimate function, h(xt,!)= (ro 1), be shorthanded as h. We have:
rx(x,, (r (x1, = Ex bxi S: Vxuhi h cJ u) Bu Bu2
[[V
(13)
where Xo is sampled from Equation
2 is the Laplacian of h and
dJ (J u) Bu Vx., log (Xu: u) du / 2Bu dwas
(14)
Proof. We aim to prove:
r(xo) - (r (roc)(x(,t) = Vxch- dJ u) 2
(15)
Recall that
Xo + L( fø(Xu, u) cu + gu(u) dua.
(16)
(x,,t) = x + L fo(x u) du.
(17)
For an ODE trajectory x' x'()), notice that:
0h
dx' Oh
0 = dt
1(x(, 1)
+ vxh
Vxh fo-
(18)
Øt.
dt
Ot
We can write:
r(xo) - (r ()x,, = h(xu, 0) - t) [- dh,
(19)
where Xt, which is not an ODE trajectory (noted by x)), follows the SDE trajectory. Using Itô's lemma Ito al. [95T), we find:
dh
0h
+
Vzh
fg
1
-93 v2h
di + 9 vx dui
(20)
01
2
1h
1hh
- dt + g0 Vxh dw
+ Vxh+ fs
vxh. Vxh.fo) fo)
(21)
Øt
Øt
1 5 v2h dt + 98 v dus
Vzh (fg fo)
(22)
V.h. (-BIV logp t) dt + 23t 2t
(23)
The sign the Itô correction term is flipped due to reverse time Brownian Motion and the other is followed by expansion. We thus derived Equation (5)
17

---

Published as a conference paper at ICLR 2025
D.1.2 DISCUSSION
We interpret the enror terms of the reward estimates approximation as follows:
The estimate becomes more accurate 3 decreases, satisfying the intuition that SDE trajectories will reduce to the ODE trajectory as |3 +
If Vx, h 1 xx logp (Xa: u), the term Vx, dJ (Xu, u) cancels out in expectation.
If h 0 and the previous condition holds, then rg-
For estimation purposes, we make the following assumptions to facilitate understanding and derivation of Equation (5)
Vx, log p(x,() ~
X 1 c(x,, 2 C.X, xx 1x
(24)
(25)
(26)
where C₁ is time-dependent constant and r is scale-invariant.
Equation 24 is derived from the assumption that p(x:) Rs (0),
Equation 25 stems from image preprocessing algorithms, such those used in Stable Diffusion, which normalize the image distribution. This normalization implies that images in the dataset are often scaled lie on a sphere. Therefore, we can reasonably assume that randomly generated X, is close to image in the dataset in direction.
Equation 266 is based on the intuition that minor changes in brightness do not significantly affect the semantic interpretation of an image. Besides, many training algorithms incorporate scaling as part of data augmentation, which aligns with the assumption that the gradient of Vxr is orthogonal to x.
Under these assumptions, we obtain:
dh vxh (-BIVV,, logg((x..)) di ( 223 2/2
(27)
Vvr (-3x, di + 2,8 /23) /
(28)
= v23tC, VxT dui - -Bt
(29)
If r harmonic, i.e. v2r = 0, then dh becomes martingale (Billingsley 2017) and:
rx(x,, - (r o (xx, Ex Xx [ V23IC. duy = 0.
(30)
The mean value property, an equivalent statement of harmonic function, states that the value of harmonic function at any point is the average of its values any sphere centered at that point. This property provides an intuitive explanation of our method: if is harmonic, the reward of the ODE-generated image is the mean value of the reward of SDE-generated ones, while empirically, observe that the ODE generation resembles the SDE variants.
1.1.3 ILLUSTRATION OF MISMATCH
For better understanding, we provide an example that r' is far from r o c. We adopt assumptions in Appendix [D.122 illustrate the intuition, and suppose x, is noisy sample time such that c(x;) is sharp local maxima of r, where 2 < 0 near c(x;). Further, suppose that 3 is small enough such that the generated Xo is near c(x,). ). In this case, ra(x)) (r c)(x)) as (xx)) = Ex, [r(xo)] < (r (x,, by intuition.
We can also verify rx(x,) - (r o (x,, < 0 using Equation Appendix D.1.2 write:
15
Under the assumptions in
18

---

Published as a conference paper at ICLR 2025
[. V28tC, VxT dui - Bu v
(x())) (r o (x,) x Exxlxx
(31)
Exol U." -Bu -F
(32)
< 0.
(33)
Note that the value of 2e is taken c(x,), fluctuating with SDE.
D.2 MARTINGALE PROPERTY OF REWARD ESTIMATES.
A martingale is sequence of random variables that maintains certain property over time Billingsley (2017): the expected future value, given all past values, is equal to the current value; for fixed SDE, the current reward estimate is the expected value of the reward estimates the next time step:
Fact 1. For any time step A < 0 such that t t A > O:
(x()))
Ex a/x (rp (X+-
(34)
Intuitively speaking, this idea stems from the principles of conditional probability, which tell us that our current prediction of the final score should be the same as the average of all possible future predictions.
Proof. This result follows directly from the foundational definition of expectation. For variable re (xt), we have:
(x(x))= = Exo xe, [r(x))] = Ex, alx; [Exo/x a [r(xx)]] Ex, ale (x)(x
(35)
D.3 TANH DEMON
We provide the theoretical idea behind the development of the algorithm. To start with, there exists a linear relationship between the reward estimate increment from x; to x and the injected noise (*k), which can be derived from Itô's lemma Ito et al. (1951 and Kolmogorov backward equations/Kolmognnne Kolmogorofi 1931). as follows:
( rx(x)) g(t) 2
(k)
(1) +oA)), where
X heun(x;, z (8) (t, A), (36)
(k)
which can be interpreted from SDE with the following Lemma.
Claim 1. Let (X1, t) Ex ix, [xxx]] be the expected future reward at time 0, given the current
state x, time Then, under the SDE:
dx, fg dl go dui,
(37)
the differential rg is:
dr 98 Vx, dwy.
(38)
Proof. We begin by introducing change of variables. Let = faax - ft so that as decreases from Emax 0, increases from fmax. This allows us consider forward-ting process with standard Brownian motion W..
Given the original SDE, we can write:
dx, fos cs + ga di.,,
(39)
where W, is the standard Brownian motion.
19

---

Published as a conference paper at ICLR 2025
Now, applying Itö's lemma r (x( drg
[rs fg Vx, + 2 2
ds 99 Vx, rB dāg..
(40)
Øs
We aim to prove the Kolmogorov backward equation:
Or
fs Vx, 5 5 0.
(41)
Ōs
To do so, we integrate Itô's lemma from to mmx
Jmas
T (xtm)
rg (x,)
dr3
(42)
(Urg
Os fg Vx, 3
5 /258 ds
Amn
+
93 Vx rg diy..
(43)
Since (x( martingale, by taking the expectation (conditioned on x,) on both sides, obtain:
0 Exiuu (xc, -
(44)
[Samee (
(8rg
1 fg Vx 3/2 do'
Exema (xc
Os
+ Exinul
Ia Vx rg diy.
(45)
The expectation of the stochastic integral is zero, Itô integrals have mean of zero:
[.,00
E Exim (xc
98 V TB TB
dwa 0.
(46)
Thus, we are left with:
frnex (Ōrg
Exiga Xx,
fg Vx, T3 + 3 do'
(47)
Os'
Since the expectation is zero for any interval [s, /mmx) the integrand itself must be zero:
Or
fo Vx, ,
V 0.
(48)
Øs
Thus, the differential of rg is given by:
dr = g3 Vx, dw.s
(49)
Returning to the original time variable substitute tmax yielding:
drg = Vx. dwe.
(50)
completing the proof.
Although 93 Vx, ra inaccessible without distillation and thus an intractable static vector, we can still leverage the linear relationship to derive applications. Using our standard approach of interpreting r c as Ty and recognizing that r, (x(xx)) a) is an unbiased estimator of r (x;) (from Appendix D.2).
practically interpret Equation (36)
20

---

Published as a conference paper at ICLR 2025
K K (roc))( (51)
(r (x))((((-))))) - ga(t)Vx. 20 VA,
where i
From Equation (36). flipping the sign (k) reverses its contribution to r.. Therefore, based on the observation (r c) ( - jr, flip z (*) accordingly. We show the theoretical analysis and proof for the eπTor of the reward estimate of our Tanh Demon in the following.
Lemma 2. Assume the truncation error terms in Equation (36) are negligible and r3 = Froc c. Let z
be derived from Equation (1
With probability 1, r ((80ma)) > rx((x), where Shanh is derived by
applying z² on every step.
Let l 96 V xx rg. Recall that assume
ra(x(->)
(k)
(x()) (2) VA
(52)
ra (x(())))
(x((x)) = 2.2'VS
(53)
x 1-A
= heun(x, (x,, (8) ,,,,t, A)
(54)
tanh --A
heun(x,, z² tt, A)
(55)
z' VNN normalized
Σ
b
tanh z
(56)
ra(xx=2) (x))))
baan tanh
(57)
We aim to prove the sufficient condition: Th (x((a)))) > (x,) for each numerical step. Under rotation basis, without loss of generality, we assume l only has value in the first component, i.e.,
L (f1, O,..,,....0 and & O. We have:
re ( > rg(xi)
- & 0
(58)
Claim 2. With probability 1, the first component 27 of n" is positive.
Proof. Since
rx(x(()) (x(
gtanh
tanh
(59)
l (A)
tanh
(60)
(12
(k)
VA
tanh
(61)
T
where (()) is the first component of (R))
Almost surely, 21 (k) # so btanh will have the same sign as (x)) This implies Manh (M)
> 0.
Since the first component of 2" will have the same sign as the first component Σ M Maan (k) i.e.
ΣK
K branh (k) 0. We conclude that 2 >> O.
In addition, we provide proof of the linear relationship presented in Equation 36
21

---

Published as a conference paper at ICLR 2025
D.4 BOLTZMANN DEMON
Recall that
A = Xt + /
1-4 fz(Xu, u) du + ga(u) dw
(62)
x A X 1x(xx,t) + (t(tvv
(63)
& = X [xx(xe 1) - A)I + (9s(())+++ ga 2)
(64)
We first present the theoretical analysis and proof for the reward estimate error of the proposed Boltzmann Demon as follows.
Lemma 3. Assume is bounded by mmax and rg is L-Lipschitz. Given It. if the truncation error per
Heun's 's SDE step in Equation 64 is X1 8 o(A) as A - 0t, then we have:
E [r(abolte) 2 rr(())) - o( tmax),
(65)
where the expectation denotes that each step of the numerical approximation from every it + A taken with the maximum value of (-)))) among ii.d. SDE samples L+A representing the Boltzmann Demon with 0.
(k)
Lemma 3 establishes lower bound based on the sample maximum and reward estimate accuracy. providing an improvement guarantee of expected reward in expectation.
We first claim the following statement. Claim 3.
E [ (((xbbll))] 2 rx((x) olI. A..
(66)
The rest is the induction of SDE time steps to 1 >> > 1T- > 1- > by = 0, i.e.
E |r(80 bo
E Is (xbolte)
(67)
E rs rx(xboltz)
(T..T--11)
(68)
2 E rs (bboltz)
o(L (IT-1 1
(69)
(70)
2 E (x ] - -(L
(71)
2 r (2 boltz) o(L lmax)
(72)
Proof. We list the premise as the following:
x = heun (xe, (+) A)
(73)
x 1-1 = =x(-1 -((A)
(74)
iig (n holtay = max(ro (x (1)
A)) ;T (x((x())))))))))))))))))))))))))
(75)
We can deduce that:
E [ra (oobb)))]]] E
max(ra( ( -
(76)
[ra(x(2a)]
(77)
E (xx - L. o(
(78)
= rx() o(L
(79)
The last equation is followed by Equation underlying SDE value (x)
334.. Here, a (Xt, A) is the numerical estimation of the
22

---

Published as a conference paper at ICLR 2025
Lemma 4. When and the time step is small enough, the Boltamann Demon sampling is identically distributed as the SDE sampling.
By adjusting т, we can smoothly transition from prioritizing high-reward noise samples to the standard SDE sampling method, balancing Demon and SDE strategies; note that when oc, the weights are by cxp(0) = 1. Thus, Ek=1 by results in Gaussian distribution ((0, KIv). This distribution is identical distributed to drawing a Gaussian after both are projected onto a sphere of radius VN.
We justify replacing Gaussian sampling with uniform sampling from sphere of radius VN could result in the same effect of SDE during the Euler-Maruyama discretization of SDEs. Assuming constant drift f and diffusion g for Euler-Maruyama step, the SDE is dx- d + g dW. We aim to demonstrate that this replacement yields an identical distribution under small step sizes. Define:
Y, = - FA +
2
FA
OV
(80)
where 2, are i..d. vectors uniformly sampled from the surface of a sphere with radius VN i.c. 2, ~ Unif(V/g* Also, define:
Y= + gvAz
(81)
Claim 4. Y, converges to Y in distribution as n - DC.
Proof. To justify replacing Gaussian sampling with uniform sampling from the sphere, it is sufficient to show that the normalized converges in distribution to Gaussian vector 2, i.e.
4
(82)
Due the symmetry of the uniform distribution, the expectation of each vector is zero, i.c., E(2,] Moreover, the distribution satisfies E. =
By applying the Central Limit Theorem for vector-valued random variables (see, e.g.. Rencher (2005)) ) we conclude that as + oo, the normalized sum converges in distribution to Gaussian vector z with mean O and covariance matrix IN. It justified, in the limit of n - oc, the uniform sampling from the sphere replicates the statistical properties of Gaussian sampling in the diffusion term of the original SDE.
D.5 HIGH DIMENSIONAL GAUSSIAN SPHERE
The original statement is more general in the textbook, but we provide specific proof for Gaussian.
Lemma 5. (Vershynin 2020 Chap. 3) Let z be independent and identically distributed (i.i.d.) instances of a standard isotropic Gaussian (00, In) in a high--dieensionll space N. With a high probubility (e.g., 0.9999), it holds that
||||| = VN + 001)
(83)
Proof. Consider the norm ||z|2, where z an instance of standard isotropic Gaussian (0 IN) in N dimensions. The distribution of | follows Chi-squared distribution with N degrees of freedom. The mean and variance of this distribution are and 2NV, respectively.
Applying the central limit theorem argument, we approximate the distribution of [ by normal distribution when N large, giving:
1z| 2 C
=
(84)
for some constant C, where C (11) represents fluctuations around the mean which are typically on the order of the standard deviation of | which is V2N.
23

---

Published as a conference paper at ICLR 2025
To connect this with the norm of consider:
C
V + N
lim
VN= lim
VN
1
1
-1
(85)
N-000
VN
C
lim
VN
(86)
N-000
2/N
C
(87)
2
Here, we use the Taylor series expansion for vI+ I, approximated as 1 1+ x for small z, to find the limit. This expansion leads to the conclusion that ||||| vN + O(). 2
E COMPARISON ON PICKSCORE
E. PICKSCORE COMPARISONS.
Since PickScore Kirstain et al. 2023) is trained specifically generated images, we believe it is a more reliable measure and objective than the aesthetics score. To emphasize the strength of our method, show how the median PickScore reward function improves across 20 different prompts using our Tanh Demon, as shown in Figure 7a
Our approach utilizes 1,440 reward queries per sample and achieves PickScore of 0.253, outper- forming other methods alongside reduced computation time (180 minutes for our method vs. 240 minutes for resampling methods due to shortened ODE trajectories). Specifically, compare method to:
SDXL/SDXL-DPO Wallace et al. (2023a): A state-of-the-ar method for direct preference optimization diffusion models, which achieves a PickScore of 0.226, while the baseline SDXL reaches 0.222.
DiffusionnDDO(14400)) A variant that selects the highest quality median PickScore from 1440 samples among 20 prompts, achieving a PickScore of 0.246.
(144x1440x): Similar to the above, but without preference optimization, achieving PickScore of 0.243.
Additionally, resampling ODE from X is erucial in applications where the distribution Xt,.ox Xo plays a key role, such as in SDEdit Meng et al. (2022). Resampling methods fail to address such applications, highlighting the advantage of our approach.
0.26
100minn
0.25
10
0.23
(a) Trajectory of Tanh Demon. plot (r o
(x,) for different
(b) The performance of each method on PickScore.
Figure Quantitative results for Tanh Demon.
E.2 QUALITATIVE RESULTS
In this section, we demonstrate the quantitative and qualitative results of PickScore in SDXL with Tanh Demon.
24

---

Published as a conference paper at ICLR 2025
Figure 8: Fach row in the figure presents two pairs of images where the image of each pair on the left illustrates results generated using the original PF-ODE method. The image on the right in each pair showcases enhancements achieved by applying our Tanh Demon based on the PickScore metric and SDXL. This figure demonstrates the improvements in visual fidelity and adherence to targeted characteristics achieved through our proposed method.
25

---

Published as a conference paper at ICLR 2025
F COMPARISON ON HPSv2
0.40
Fextal ol 2110
0.40
Fou dil ST.
0.38
0.36
0.36
HPSv2
0.34
0.34
0.30
0.30
0.28
0.28
0.26
0.267
2
Numer of Reward Function Queries
Execution Time (seconds)
(a) Performance w.r.t Reward Query Number
(b) Performance w.r.t Exccution Time
Figure 9: Comparison in HPSv2 and HPDv2. The performance comparison of the proposed algorithm and the best-of-N baseline methods presented in terms of the number of reward queries and execution time, with the dependent variable being T. The shaded areas and the solid circle radii represent the evaluation results' standard deviations. the computational bottleneck is the number of reward queries, we recommend Tanh; if it is computational time, we recommend Tanh-C.
Table 11: We present qualitative results for various methods. For our method, we set T 128. 3 = 0.5, and K = 16. The Best-of-N samples are generated using 2,336 (5,440 for CM) reward queries and 3.8k seconds, which significantly more than our method's 's 1,424 reward queries and 1.8k seconds. Moreover, the presented image from Best-of-N possesses an inferior HPSv2 score compared to ours.
Best-of-N (CM)
Best-of-N
Tanh-C
Tanh
Boltzmann
a castle is in the middle of a eurpean city
motorcycle that sitting the dirt.
We present quantitative and qualitative results in Figure 9 and table I1 using 10 prompts sampled from HPDv2 (Wu al. 2023 Both the diffusion model and CM are implemented and distilled with SD v1.5.
We observe similar results in Figure Regarding reward queries, the Tanh Demon method outper- forms Tanh-C, followed by the Boltzmann Demon method. Regarding execution time, however, Tanh-C is recommended over the Tanh Demon if computational time is limited.
26

---

Published as a conference paper at ICLR 2025
G ADDITIONAL RESULTS WITH VARIOUS REWARD FUNCTIONS.
G.1 IMAGE GENERATION RESULTS WITH DIFFERENT REWARD FUNCTIONS
We show more image generation results SDv1.4/SDX with our Tanh Demon and other reward functions in Tables 12 13 using the four reward as objective.
G.2 SUBJECTIVE TEST OVERVIEW
We surveyed with 101 participants via Google Forms, as shown in Figure 10 Participants evaluated different image generation methods based on:
Subjective Preference: Visual aesthetics and image quality.
Semantic Alignment: Correspondence between generated images and text prompts.
Each participant ranked images across four sections, with rankings aggregated using the following formula:
M
exp( (rank;j 1))
(88)
ML
where:
M = (number of sections),
L 101 (participants),
rank; is the ranking by participant for method i.
Preferance across Methods
Semantic Alignment across Methods
20-
15-
(a) Comparison across methods.
Preference Different Objectives
Semanti Alignment Different Objectives
Mean
Basoline Ensemble
Pick
HP5v2
Aes
Baseiie Ensemble
IS
Pick
HP5v2
Ads
Dbjertivw
abjective
Comparison objectives.
Figure 10: Subjective test results: Preferences and prompt alignment across methods and objectives.
G.2.1 SURVEY STRUCTURE
The subjective test comprised four sections: two comparing methods (DOODL, Baseline (SD SDXL.,, Ensemble) based subjective preference and prompt alignment, each with sets
27

---

Published as a conference paper at ICLR 2025
Table 12: Generative Results using SDXL
SDXL
Aes
IR
Pick
HPSv2
Ensemble
An Octopus Playing Chess with Robot Underwater
A Samurai Gardening Floating Island in the Sky
Insanely detailed portrait, wise man
A painting of girl encountering a giant sunflower blocking her path in a hallway
demon exiting through portal
A butterfly flying above
28

---

Published as a conference paper at ICLR 2025
Table 13: Generative Results using SDXL (Cont)
SDXL
Aes
IR
Pick
IPPSv2
Ensemble
Two-faced biomechanical cyborg
Highway to hell
A Jazz Band Different Alien Species Performing on an Exoplanet
Victorian Inventor Testing Her Flying Bicycle Above Steampunk City
Time Traveler's Picnic the Edge of Volcano During the Mesozoic
jedi duck holding lightsaber
29

---

Published as a conference paper at ICLR 2025
Table 14: More Qualitative Results of SD1.4
Baseline
DOODL
Pick
HPSvv
Ensemble
An Octopus Playing Chess with Robot Underwater
Two-faced biomechanical cyborg
CIO O
Highway to hell
jedi duck holding lightsaber
exiting through portal
A Samurai Gardening Floating Island in the Sky
30

---

Published as a conference paper at ICLR 2025
Table 15: More Qualitative Results of SD1.4 (cont)
Baseline
Aes
IR
Pick
HPSv2
Ensemble
DOODL
Victorian Inventor Testing Her Flying Bicycle Above Steampunk City
Time Traveler's Picnie the Edge of Volcano During the Mesozoic Era
Insanely detailed portrait, wise man
A butterfly flying above an ocean
A Jazz Band of Different Alien Species Performing on an Exoplanct
painting a girl encountering a giant sunflower blocking her path a hallway
31

---

Published as a conference paper at ICLR 2025
containing image per method; and two comparing methods applied to different objectives (Baseline, Ensemble, IR, Pick, HPSv2, Aes) also based on preference and prompt alignment, each with sets containing six images per set.
G.2.2 EVALUATION RESULT OVERVIEW
Methods Comparison Figure 10a shows that DOODL slightly outperforms the Baseline in aesthetic preference and prompt alignment. The Ensemble method significantly surpasses both, indicating superior visual quality and semantic accuracy.
Objectives Comparison As seen in Figure 10b all objectives outperform the Baseline in prompt alignment, with the HPSv2 method leading. In subjective preference, methods applied to different objectives show varied improvements, with some achieving substantial gains over the Baseline.
G.2.3 ANALYSIS
We compared DOODL, Baseline, and Ensemble based on aesthetics and prompt alignment. DOODL marginally improves over the Baseline in both criteria, while the Ensemble method consistently outperforms both DOODL and Baseline, excelling in image quality and semantic accuracy. The Ensemble method demonstrates significant enhancements, particularly in tasks requiring visual refinement.
Evaluating different objectives (IR, Pick, HPSv2, Aes) against Baseline and Ensemble revealed that almost all objectives surpass the Baseline in both preference and prompt alignment. However, Aes, an objective without explicit text guidance, shows weaker prompt alignment. Among the objectives, HPSy2 achieves the best performance both criteria.
The Ensemble method provides the most substantial improvements in visual aesthetics and seman- tic alignment among method comparisons. Among the factors of the Ensemble method, HPSv2 outperforms other objectives, even the Ensemble method, highlighting its effectiveness aligning preference for real human.
H MORE DETAILS OF VLM AS DEMON
In this section, we provide more details of experiments and quantitative results of utilizing VLM during generation.
H.1 GENERATION WITH THE LATEST MODEL
In Table 16 we present an experiment aligning diffusion models to preferences of VLMs from APIs Google Gemini Flash v1.5 (Gemini Team Google, 2024) and GPT-4o (OpenAI 2024
demonstration of using non-differentiable reward sources. In each setting, the VLM receives a given scenario, e.g., "You are journalist who wants add visual teaser for your article to grab attention on social media or your news website," " and is asked to select the best-matching state vector x based generated images. Each scenario is assigned prompt. In generation, apply Tanh Demon with K 16 on SDXL. For scenarios and quantitative results, please refer to Appendix H
(k)
Empirically, VLMs achieve better accuracy when selecting out of 2 options rather than 8 out of 16. We, hence, utilize binary comparisons by applying the Quicksort partition (Hoare) 1962) twice on the array of c(x;-A) derived from x,. The first application partitions the entire array, and the second partitions the larger subset resulting from the first step. This process allows us to identify roughly the top 8 images within 2K comparison.
We assign a +1 reward on the roughly top images and -1 on the rest for each Demon step. Using PickScore (Kirstain et al. (202)), trained based on image comparisons, convert logits to probabilities and assess the effectiveness of our method. The results indicate improvements with this metric on every image in any degree.
32

---

Published as a conference paper at ICLR 2025
Table 16: Using VLMs to generate images. PF-ODE (baseline) refers baseline without using our method for alignment. The top row of each set indicates the agent's 's role in the given prompt. The hyperparameters are set to 0.1, 0.0001, K 16, and T 128 using Tanh Demon. We apply ODE after applying 15 Demon steps.
Teacher GPT
Artist GPT
PF-ODE
Gemini
PF-ODE
Gemini
13.7%
17.7%
68.6%
24.3%
27.2%
48.5%
Rescarcher GPT
Journalist GPT
PF-ODE
Gemini
PF-ODE
Gemini
9.3%
69.4%
21.3%
32.7%
33.1 1%
34.2%
Role Teacher
Prompts
colorful, labeled educational illustration with simple, engaging visuals.
Artist Rescarcher
detailed, cinematic concept of unique characters or scenes for games or movies. realistic, vivid visualization of scientific data with professional style.
Journalist
bold, vibrant illustration with dynamic design for digital platforms.
H.2 EXPERIMENTS SETTINGS
We provide the prompt template we used in Table 5 to VLMs. The following are the full system prompts for the scenarios:
Table 17: Roles and System Prompts
Role
System Prompt
Teacher
You are teacher looking create custom illustrations for your educational materials to make learning more engaging for your students.
Artist
You are a game or movie concept artist tasked with creating concept art for characters, settings, and scenes speed up the pre-production process.
Researcher
You are a researcher needing to visualize complex data, such as molecular struc- tures in chemistry or weather pullerns in meteorology, for better understanding or presentation.
Journalist
You are journalist who wants to add a visual teaser for your article to grab attention social media your news website.
Listing 1: Prompt template used in the Gemini image selection task. This template guides the decision-making process for choosing between two images generated from a fixed prompt. For the prompt used in GPT selection, replace the output format by asking return JSON.
Scenario: (scenario)
You are presented with two images generated from the prompt "[ prompt). " Examine both images carefully to decide which one best matches the given scenario. Consider how each image
33

---

Published as a conference paper at ICLR 2025
relates to the scenaric, including its relevance and how well it captures the intended theme and concept.
5 Choose the image that you believe is the most appropriate ate for the scenario. Each image has an id:
1.
Please share your thought process or any observations you made
while making your decision. This reflection helps in understanding your choice.
Respond with the id of the image you' ve chosen in a JSON format. For example, if you choose the first image, your response should look like this:
11 vijson
[0]
13
15
Or, if you choose the second image:
17 jrjson
18 [1]
20
22
Please response in the following format:
24
Here write down your argument.
Here write down your decision, either [0] or [1], the [] is 26
necessary >
Take a deep breath and work on this problem step-by-step. Ensure your choice truly aligns with the intended scenario.
Listing An example of GPT generated output
2 "justification"; "Considering the scenario described, the first image is better choice. This image effectively showcases a mysterious, illuminated object that instantly attracts the viewer's curiosity. The background hints at an academic or historical setting, with architectural elements and various artifacts that seem like elements from different times and places. This diverse and enigmatic setup aligns well with the theme trans and curiosity across different realms like education, history, literature, and science. The glowing object in seemingly ancient, cluttered environment truly sparks wonder, making it ideal for grabbing attention on social media news website.", "chosen_image":
34

---

Published as a conference paper at ICLR 2025
H.3 QUANTITATIVE MEASUREMENT OF EFFECTIVENESS
Pickscore Kirstain al. 20233, which is trained from [LIP|Radford et al. (002T) to evaluate the effectiveness of VLM in a aligning designed scenarios during image generation. For each scenario, we create a corresponding prompt that partially describes the scenario: "For education" for Teacher, "For entertainment" for Artist, "For research" for Researcher, and "For Journalism" for Journalist. Then, we assess the PickScore between the prompt and the scenario. The results are presented in Table 18] Table 19] Table (20 and Table 21] where the highest score for each prompt highlighted in bold. Our observations indicate that 14 out of our VLM- generated 16 images demonstrate better PickScore alignment with the corresponding prompt than PF-ODE. Given that all images are generated using the same prompt and initial noisy sample same table, these results demonstrate the effectiveness of our approach employing VLM in aligning the scenarios.
For VLMs reward functions,
Table 18: GPT-SDXL generation, validated by PickScore on related prompts
Prompt
Teacher
Artist
Researcher
Journalist
PF-ODE
For education
0.2050
0.2069
0.2080
0.2071
0.2073
For entertainment
0.2042
0.2073
0.2061
0.2058
0.2032
For research
0.1980
0.1989
0.1996
0.1985
0.1971
For journalism
0.1994
0.1957
0.1978
0.1946
0.1970
Table 19: Gemini-SDXI. generation, validated by PickScore on related prompts
Prompt
Teacher
Artist
Researcher
Journalist
PF-ODE
For education
0.2111
0.2042
0.2102
0.2072
0.2073
For entertainment
0.2057
0.2058
0.2062
0.2013
0,2032
For research
0.2018
0.1979
0.2035
0.1986
0.1971
For journalism
0.2011
0.1991
0.1978
0.2049
0.1970
Table 20: GPT-SD v1.4 generation, validated by PickScore on related prompts
Prompt
Teacher
Artist
Researcher
Journalist
PF-ODE
For education
0.1978
0.2008
0.1996
0.1988
0.1941
For entertainment
0.2026
0.2018
0.1991
0.2004
0.1966
For research
0.1896
0.1936
0.1912
0.1935
0.1878
For journalism
0.1930
0.1951
0.1918
0.1942
0.1901
Table 21: Gemini-SD v1.4 generation, validated by PickScore related prompts
Prompt
Teacher
Artist
Researcher
Journalist
PF-ODE
For education
0.1997
0.1945
0.1961
0.1961
0.1941
For entertainment
0.1954
0.1973
0.1982
0.1989
0.1966
For research
0.1910
0.1910
0.1897
0.1895
0.1878
For journalism
0.1935
0.1936
0.1927
0.1914
0.1901
I GENERAL IMPLEMENTATION DETAILS
In this section, we show the details of the implementation and experimental settings of the proposed approach follows.
35

---

Published as a conference paper at ICLR 2025
1.1 ADAPTING STABLE DIFFUSION TO EDM FRAMEWORK
In this paper, we tailor the existing text-to-image Stable Diffusion v1.4/v1.5/XL v1.0 (SDXL) (i.e., we use fp16 SD v1.4/SDXL v1.0 for generation.) to the SDE formulation proposed in EDM Karras et al. (2022) by Karras et al. for image generation since its reparameterized time domain, t E [imi (aax). improves numerical stability and sample quality during image generation. We realize the modification through the equation, Va log p(æ,t) = (D, t) x) /t", where the function D(x, t) x iF (t) u(t) derived from the original model In addition, s(t) and u(t) represent the scaling schedule and the original temporal domain of the reparameterized temporal domain i, respectively.
1.2 NUMERICAL METHODS FOR IMAGE GENERATION
adopting setting p= 7,7 20 and in fmax ~ 2.7, In min -.2. The classifier-free guidance parameter set to throughout this paper. Across all temporal steps of image generation, we keep K and [ constant. We have found that when less than 0. 0.11, i.i.d. samples from SDE all appear similar human perception. For the remaining evaluations, we will directly use ODE. As a result, the actual number of samples will be slightly smaller than K
Moreover, for image sampling with ODE/SDE, our approach follows Karras et al. Heun's method and time intervals determined by t, = (1// +
(2022)
i-1 1/p T- min
tmax
1.3 SIMPLIFICATION IN DIFFUSION PROCESS MODIFICATION
is worth noting that in our work, since our main focus is on the modification of the diffusion process, without loss of generality, we omit the VAEs (Kingma & Welling (2014)) of Stable Diffusson models, the prompt and of classifier-free guidance (CFG) & Salimans (202T) in our formulation for simplicity (i.e., using p(æ) to denote the unnormalized p(æ)p(( x)" conciseness).
1.4 BATCH SIZE AND MEMORY CONSTRAINTS
When we generate many SDE samples, the batch size for solving ODE/SDE is 8 for both Stable Diffusion v1.4, v1.5, and SDXL models. However, due to memory limitations on the RTX 3090, the batch size for evaluating the VAE in SDXL is restricted to 1. This memory bottleneck prevents any further acceleration from using larger batch sizes, as it limits the parallelization during VAE evaluation.
Due to memory limitations, DOODL was run on an Nvidia RTX A6000, which is slightly slower (0.92x) than the RTX 3090 used for the other experiments.
1.5 EXPERIMENTAL SETUP AND HYPERPARAMETERS
Among all the discussed methods, DOODL is the only one that utilizes the differentiation of the reward function. Due to memory limitations, DOODL was run on an Nvidia RTX A6000, which is slightly slower (0.92x) than the RTX 3090 used for the other experiments. We present the detailed hyperparameter settings different experiments as follows:
Baseline Comparison. The hyperparameters for generation set 0.5, K 16, n7 and 7 adaptive for Tanh, 10-5 for Boltzmann.
Reward Estimate Approximation Comparison. We set 8 0.5 and use SD v1.5 and its distilled CM. The CFG parameter is ignored in CM(set to 1). The reward estimate r's is obtained by averaging over 200 Monte Carlo i.i.d. SDE samples- each with 200 SDE steps.
Generation with Various Reward Functions. We use Tanh Demon for sampling with adaptive temperature. The hyperparameters for generation are set to 3 = 0.05, K 16, T = 64 as shown in Tables 4 to 14 15 on SD v 1..//DXXI.
36

---

Published as a conference paper at ICLR 2025
For reward scaling in the ensemble setting, the PickScore multiplied by 98.86, and HPSv2 was multiplied by 40.
The interaction step of DOODL is used as suggested by their implementation, 25 iteration for Aes and 100 iteration for Pick.
Non-differentiable Reward. In Table 5 the hyperparameters
0.05, 0.0001,
K 16, and T = 32 using Tanh Demon.
Manual Selection. Figure 5 the parameters are / 0.1, K 16, 128 but terminate manually, using Tanh Demon with adaptive temperature. We terminate the iteration after ten rounds of operating the UI.
J LIMITATIONS
We present the theoretical result in Equation (5). which demonstrates that r C Rs rg. This result relies the assumption that the reward function is near harmonic near the ODE sample ourput, as detailed Appendix D.1.2
In practice, implementing faces challenges related to time complexity and accuracy bottlenecks, thoroughly discussed in Section
K FUTURE WORKS
The only difference between Tanh-C and Tanh Demon lies in how r o c is implemented. Analysis of the data in Table 2 and Figure indicates that Tanh-C's reward performance be enhanced by mitigating the fidelity in r o without compromising Tanh-C's speed performance. Potential strategies for improvement include increasing the fidelity of CM distillation training dedicated distilled model for r o c. We propose these enhancements as future work.
L CODE OF ETHICS
The experiments involving human judgment are fully compliant with established ethical standards. Approval is obtained from the Institutional Review Board (IRB) of Academia Sinica under IRB number AS-IRB-HS 02-24031 to ensure that the research meets all necessary guidelines for the ethical treatment of human subjects.
M SOCIETAL IMPACT
Our method has the potential both discourage and encourage harmful content. Users can generate images through manual selections with malicious intentions (Figure This increases accessibility but also raises concerns about misuse. We implement safeguards provided by Stable Diffusion; end-users are responsible for employing them, as recommended in prior works OpenAl (2024);
Gemini Team Google (2024); Rombach al. (2022): Podell et (2024), to mitigate potential risks.
37


Reviews:
[{"review_id": "D6ze4D2r3N", "reviewer": "Reviewer_UYoG", "summary": "This paper studies the problem of aligning diffusion models without additional training or backpropagation. The main idea is to optimize the sampling at inference time, by optimizing the noise used during generation based on a (possibly differentiable) reward signal. To compute the (expected) reward on a given noise, the method leverages a reduction from SDE to ODE dynamics.", "questions": "- Given that the \"noise\" is now optimized over, is there a good intuitive way to think about it from a different perspective (diffusion, physics, or bayesian inference, etc.)? (other than the \"noise\" obviously corresponding to the latent vector)\n- My intuition for the proposed sampling strategies improving over \"best-of-N\" sampling is that now you additionally get to optimize over the simplex spanned by the sampled noise samples (rather than just querying the \"vertices\" of the simplex). \nCould there be more intelligent ways to choose the \"basis\" vectors for the simplex other than random sample?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- very clearly written and polished writing\n- clear motivation\n- compares to prior methods systematically, and shows consistent improvements", "weaknesses": "- It's hard to gauge the significance/improvements in some of the results. For example, Appendix G.2 the numbers are all tightly concentrated around 0.2, so being unfamiliar with the metric, it's hard to know whether the differences are statistically significant or interesting.\n- It would be nice to see more experiments with more realistic/extreme use cases of alignment. Most of the variations explored in the paper seem sort of minor. For example, make the model (not) generate NSFW material (or something else distributionally more obvious). Or is there a reason why such examples were note explored?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTotR1diah", "reviewer": "Reviewer_fD8w", "summary": "The authors present a backpropagation-free, inference-time method for preference alignment in diffusion models. Specifically, the method synthesizes an optimal noise distribution via stochastic optimization thus enhancing the final reward of the generated image.", "questions": "The paper primarily compares the results with DOODL in terms of computational cost and performance. I would be curious to see how this method compares with other methods mentioned in Table 1 and how it performs with different base models (e.g., SDXL).\n\nAdditionally, I would like to know where the performance and computational cost of CM without your method are positioned in Figure 4.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, this paper is thoroughly written, with comprehensive theoretical and empirical analysis. The implementation details are carefully listed, and various qualitative and quantitative metrics are displayed.", "weaknesses": "In the “Alignment with Preferences of VLMs” experiment, the role of the VLM doesn’t seem to exhibit a clear preference for a specific style, etc. I understand that the goal is to demonstrate that your method can optimize for a particular non-differentiable preference, such as a VLM API. However, based on the results presented, I’m not convinced that this experiment design effectively shows that your method truly optimizes for a specific preference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbzECoNBes", "reviewer": "Reviewer_RS1X", "summary": "This paper introduces Demon, an inference-time method for aligning diffusion models with user preferences without retraining or backpropagation. Using stochastic optimization, it guides denoising to focus on high-reward regions, even with non-differentiable rewards like VLM APIs and human judgments. Experiments across three diverse tasks demonstrate improved improvements.", "questions": "- The authors used the Consistency Model [1] to accelerate computation. Could this model be applied to other diffusion models with different learned data distributions? A more general acceleration method would be beneficial.\n\n[1] Song, Yang, et al. \"Consistency models.\" arXiv preprint arXiv:2303.01469 (2023).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The concept of analyzing noise quality is interesting\n2. The proposed method is a backpropagation-free preference alignment approach that operates during inference, allowing it to be applied flexibly across various conditions.", "weaknesses": "1. The comparison includes only a single baseline method, and quantitative results for BoN are missing in Table 3.\n2. The evaluation is limited to a small subset of the dataset from DDPO [1]. Expanding the evaluation to include larger datasets, such as the full set from DDPO [1] and the Human Preference Dataset (HPDv2) [2], is recommended.\n3. In Table 2, there remains a large performance gap between the 1-step CM and the 6-step ODE. It would be beneficial to consider additional state-of-the-art solvers of diffusion sampling for comparison.\n\n[1] Black, Kevin, et al. \"Training diffusion models with reinforcement learning.\" arXiv preprint arXiv:2305.13301 (2023).\n\n[2] Wu, Xiaoshi, et al. \"Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.\" arXiv preprint arXiv:2306.09341 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZAYKNOe4YR", "reviewer": "Reviewer_m9Nn", "summary": "In this study, the authors develop a method to align the outputs of diffusion models with human preference at inference time. Specifically, the authors address the alignment from the perspective of noise generation. They first propose to measure the quality of noises in each denoising step. Then, they combined different noises to search for an ''optimal'' noise towards a high reward of outputs. Experimental results show that the proposed method effectively improves the aesthetic score of generated images.", "questions": "Please refer to the above weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The authors first develop a method to align the generation of diffusion models without training the model or computing the gradient of the reward model.\n+ They conducted experiments on various reward models and preference types.\n+ The paper is well-organized and easy to follow.", "weaknesses": "- Is the proof in this study based on the linear assumption of the reward model? Given that the human preference for images is extremely complicated and certainly nonlinear, does it apply to the scenario of complex human preference labels?\n- The noise is flipped in Figure 3, but this operation is neither included in Eq. (11) nor discussed in the main text. Is it a standard operation in your method? On the other hand, I'm not at all convinced that flipping it can ``transform it into high-reward noise’’ because the reward space is not linear and symmetric.\n- The comparison between the proposed method and best-of-N is interesting. However, the experiment settings of best-of-N is unclear. I’m not sure whether they selected the final generation with the highest reward or selected the noise at each step. Second, it would be better if the authors could discuss why is the proposed method better than best-of-N based on their theoretical analysis.\n- The baseline methods for comparison in experiments are not sufficient. First, they should compare the method with a baseline which always selects the best noise $z^{(k)}$ with the highest reward $(r\\odot c)(\\cdot)$ at each denoising step. Second, I expect a comparison between the proposed method with more methods based on gradient guidance (Bansal et al. 2024).\n- The quantitative results in Section 5 are based on 22 simple prompts of animals, which are not convincing enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqtI6q6jKi", "reviewer": "Reviewer_tuY1", "summary": "This paper introduces an approach for aligning diffusion models with user preferences at inference time without needing model retraining or backpropagation. The method, named Demon, leverages stochastic optimization to control the noise distribution during the denoising steps of diffusion models, steering the generation process towards user-specified rewards. This paper provides theoretical backing and empirical evidence, including experiments with non-differentiable reward sources such as Visual-Language Model (VLM) APIs and human judgments.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper presents an interesting approach that eliminates the need for model retraining or backpropagation, which is an advancement in the field of diffusion models and user preference alignment.\n2.\tThe proposed method can be easily integrated with existing diffusion models without further training, making it accessible for immediate application.", "weaknesses": "1.\tComputational complexity of the proposed method needs to be discussed. While the authors claim ease of integration, the computational complexity of implementing the stochastic optimization approach might be high. I suggest the authors provide the required inference time and GPU memory in Table 3.\n2.\tMany strong baseline methods have been neglected. I suggest the authors compare with DDPO, DPO, and DPOK in Tables 3 and 4.\n3.\tThe effectiveness of the Demon may be heavily dependent on the quality and fidelity of the reward signal, which may not always be reliable or accurate. Therefore, I suggest the authors conduct experiments to quantitatively compare the results between the human reward signal and a pre-trained reward model.\n4.\tThe writing quality could be further improved. The overall intuition of the proposed method is easy to understand, but the description in Section 4.2 is a bit redundant and complex.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)