Training-Free Diffusion Model Alignment with Sampling Demons

## Abstract

Abstract Aligning diffusion models with user preferences has been a key challenge.
Existing methods for aligning diffusion models either require retraining or are limited to differentiable reward functions.
To address these limitations, we propose a stochastic optimization approach, dubbed Demon , to guide the denoising process at inference time without backpropagation through reward functions or model retraining.
Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through stochastic optimization.
We provide comprehensive theoretical and empirical evidence to support and validate our approach, including experiments that use non-differentiable sources of rewards such as Visual-Language Model (VLM) APIs and human judgements.
To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models.
Our method can be easily integrated with existing diffusion models without further training.
Our experiments show that the proposed approach significantly improves the average aesthetics scores for text-to-image generation.

## 1 Introduction

Diffusion models have been the state-of-the-art for image generation (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ; Karras et al., ; Saharia et al., ; Rombach et al., ), but, commonly, the end users’ preferences and intention diverge from the data distribution on which the model was trained. Aligning diffusion models with diverse user preferences is an ongoing and critical area of research.

One approach to aligning diffusion models with user preferences is to fine-tune using reinforcement learning (RL) to optimize the models based on rewards signals that reflect the user preferences (Black et al., ; Fan et al., ). However, retraining the model every time when the preference changes is computationally expensive and time-consuming.

An alternative approach is to guide the denoising process using a differentiable reward function.
This can be done through classifier guidance at inference time  (Dhariwal & Nichol, ; Wallace et al., ; Bansal et al., ) or backpropagation at training time (Prabhudesai et al., ; Clark et al., ; Xu et al., ).
These methods are generally less resource-demanding and more efficient.
While these methods are generally more efficient, they require the reward function to be differentiable.
This limits the types of reward sources that can be used, as it excludes the non-differentiable sources like third-party Visual-Language Model (VLM) APIs and human judgements.

To address these limitations, we propose *Demon*, a novel stochastic optimization approach for preference optimization of diffusion models at inference time.
Demon is a metaphor from Maxwell’s Demon, an imaginary manipulator of natural thermodynamic processes.
The core ideas are: (1) Quality of noises that seed different possible backward steps in a discretized reverse-time Stochastic Differential Equation (SDE) can be evaluated given a reward source; (2) Such evaluation enables us to synthesize “optimal” noises that theoretically and empirically improve the final reward of the generated image through stochastic optimization.
Specifically, we leverage Probability Flow Ordinary Differential Equation (PF-ODE) (Song et al., ) or Consistency Model (CM) (Song et al., ; Luo et al., ) to help us efficiently evaluate the possible backward steps, seeded with different Gaussian noises.

Our key contributions are summarized as follows:

- •
Our approach enables the use of reward signals in the denoising process regardless of whether the reward function is differentiable. This allows for the incorporation of previously inaccessible reward sources, such as VLM APIs. To the best of our knowledge, this is the first inference-time, backpropagation-free preference alignment method.
- •
Our method can be easily integrated with existing diffusion models in a plug-and-play fashion without retraining or fine-tuning.
- •
We provide a theoretical explanation for why our approach can improve the given reward function for image generation, which can be exploited for tuning hyperparameters.
- •
We demonstrate that our approach significantly improves the average aesthetics score (LAION, ) of Stable Diffusion models, achieving averages well above 8.0 compared to the Best-of-N random sampling upper bounds of 6.5 for SD v1.4 and 7 for SDXL. This improvement is achieved across various text-to-image generation tasks using prompts from prior work (Black et al., ), without relying on backpropagation-based preference alignment or model retraining.

Figure: Figure 1: Illustration of Demon. Given a reverse-time SDE for denoising and an interval $[t_{\text{max}},t_{\text{min}}]$, we first discretize it into $T$ steps, $t_{\mathrm{\max}}>\cdots>t>t-\Delta>\cdots>t_{\mathrm{\min}}$. At every reverse-time denoising step, from $t$ to $t-\Delta$, we synthesize an “optimal” noise $\mathbf{z}^{*}$ from $K$ i.i.d. noises w.r.t a given reward source and use $\mathbf{z}^{*}$ to seed the step. This enables guiding the denoising process towards generating images that are more aligned with the reward source and the preference that the reward source represents. More details are presented in Section 4.
Refer to caption: x1.png

## 2 Related Work

##### Diffusion Model.

Diffusion models for data generation were first proposed by Sohl-Dickstein et al. (), further developed for high-fidelity image generation by Ho et al. (), and generalized by Song et al. () through the lens of SDEs.
Karras et al. () comprehensively studied the design space of Diffusion SDEs.
In this work, we base many of the derivations on theirs.
Furthermore, we focus on evaluating our method in the text-to-image generation setting (Rombach et al., ; Ho & Salimans, ; Podell et al., )

##### Human Preference Alignment.

Aligning models with human preferences has been studied with several approaches:reinforcement learning-based policy optimization (Fan et al., ; Yang et al., ; Black et al., );
training with reward backpropagation (Clark et al., ; Xu et al., );
backpropagation through the reward model and the diffusion chain (Prabhudesai et al., ; Wallace et al., ; Bansal et al., ).
Many metrics and benchmarks for evaluating alignment has also been proposed, including those by Xu et al. (); Kirstain et al. (); LAION (); Wu et al. (), and we use these either as optimization objectives or evaluation of the generated image.
In Table 1, we further provide detailed comparisons of the proposed Demon approach with relevant existing methods in the literature from different aspects.

**Table 1: A detailed comparison of different methods along various dimensions, including the ability to generalize to an open vocabulary, the necessity of a backpropagation signal for optimization, the method’s capacity to avoid mode collapse and ensure distributional guarantees (Divergence Control). Our proposed method stands out for its zero-shot learning capabilities.**
|  |  | Open | Non-Backprop | Divergence |
| --- | --- | --- | --- | --- |
| Type | Methods | Vocab | Objective | Control |
| Training | DPOK (Fan et al., ) | $\times$ | ✓ | ✓ |
| Training | DDPO (Black et al., ) | $\times$ | ✓ | $\times$ |
| Inference | DOODL (Wallace et al., ) | ✓ | $\times$ | $\times$ |
| Training | DPO (Wallace et al., ) | ✓ | ✓ | ✓ |
| Training | DRaFT (Clark et al., ) | ✓ | $\times$ | $\times$ |
| Inference | Demon | ✓ | ✓ | ✓ |

## 3 Preliminary

##### Score-Based Diffusion Model.

We base our derivation on EDM (Karras et al., ).
With a sampling schedule $\sigma_{t}=t$, we can write the reverse-time SDE sampling towards the diffusion marginal distribution as follows.

$$ $\mathrm{d}{\mathbf{x}}_{t}=\underbrace{\left[-t\nabla_{{\mathbf{x}}_{t}}\log p \left({\mathbf{x}}_{t},t\right)-\beta t^{2}\nabla_{{\mathbf{x}}_{t}}\log p \left({\mathbf{x}}_{t},t\right)\right]}_{{\mathbf{f}}_{\beta}({\mathbf{x}}_{t} ,t)}\,\mathrm{d}t+\underbrace{\sqrt{2\beta}t}_{g_{\beta}(t)}\,\mathrm{d}\omega _{t},$ (1) $$

where $p({\mathbf{x}}_{t},t)=p({\mathbf{x}}_{0},0)\otimes{\mathcal{N}}\left({\bm{0}},
t^{2}{\bm{I}}_{n}\right)$ and $\otimes$ denotes the convolution operation.
${\mathbf{x}}_{0}$ is a clean sample, ${\mathbf{x}}_{0}\sim p_{\mathrm{data}}$, and ${\mathbf{x}}_{t}$ is a noisy sample at time $t$.
$\beta$ expresses the relative rate at which existing noise is injected with new noise.
In EDM, $\beta$ is a function of $t$, but in our study, we set $\beta$ to a constant for all $t$ for simplicity.
Essentially, $\mathbf{f}_{\beta}({\mathbf{x}},t)$ corresponds to drift and $g_{\beta}(t)$ corresponds to diffusion.
As common in diffusion models, since $p({\mathbf{x}}_{t},t)\approx\mathcal{N}(\mathbf{0},t^{2}\mathbf{I}_{N})$ for a large enough $t$, we sample ${\mathbf{x}}_{t_{\mathrm{max}}}\sim{\mathcal{N}}({\bm{0}},t_{\mathrm{max}}^{2}
{\bm{I}}_{N})$ as the initial sample.

A comprehensive list of the notations and conventions used in this paper is provided at Appendix A.

## 4 Reward-Guided Denoising with Demons

In this section, we describe how Demon works in two steps:
Section 4.1 explains the process of scoring Gaussian noises in reverse-time SDE with a reward function;
Section 4.2 further explains how the noise scoring allows us to guide the denoising process to align with the reward function, which is what we refer to as *Demon*.

### 4.1 Scoring Noises in Reverse-Time SDE

Let ${\mathbf{x}}_{0}$ be the clean image corresponds to a ${\mathbf{x}}_{t}$ at time step $t$, say:

$$ ${\mathbf{x}}_{0}={\mathbf{x}}_{t}+\int_{t}^{0}\mathbf{f}_{\beta}({\mathbf{x}}_ {u},u)\,\mathrm{d}u+g_{\beta}(u)\,\mathrm{d}\omega_{u}\,,$ (2) $$

where Equation 2 is denoted as ${\mathbf{x}}_{0}\mid_{\beta}{\mathbf{x}}_{t},$ shorthanded as ${\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}.$ For an arbitrary reward function $r$ e.g. aesthetics score, we define the reward estimate of $x_{t}$ at time step $t$ as

$$ $r_{\beta}({\mathbf{x}}_{t},t)\coloneqq\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf {x}}_{t}}\left[r({\mathbf{x}}_{0})\right].$ (3) $$

This can be estimated with a Monte Carlo estimator by averaging over the reward of several SDE samples, but it requires many sample evaluations for high accuracy.
To address this weakness, we introduce an alternative estimator for $r_{\beta}(\mathbf{x}_{t},t)$ based on PF-ODE Song et al. ().

As shown in Song et al. (); Karras et al. (), the reversed-time SDE reduces to PF-ODE when $\beta\equiv 0.$
For each $t$, a diffeomorphic relationship exists between a noisy sample ${\mathbf{x}}_{t}$ and a clean sample ${\mathbf{x}}_{0}$ generated by PF-ODE.

Similar to consistency models, with ${\mathbf{x}}_{(t)}^{\prime}$ denoting an ODE trajectory instead of ${\mathbf{x}}_{(t)}$, we can denote this deterministic mapping from the domain of ${\mathbf{x}}_{t}$ to the domain of ${\mathbf{x}}_{0}$ as ${\mathbf{c}}({\mathbf{x}}_{t},t)$ as

$$ ${\mathbf{c}}({\mathbf{x}}_{t}^{\prime},t)\coloneq{\mathbf{x}}_{0}^{\prime}={ \mathbf{x}}_{t}^{\prime}+\int_{t}^{0}\,\mathrm{d}{\mathbf{x}}_{u}^{\prime}, \quad\text{where}\quad\mathrm{d}{\mathbf{x}}_{u}^{\prime}=-u\nabla_{{\mathbf{x }}_{u}^{\prime}}\log p\left({\mathbf{x}}_{u}^{\prime},u\right)\,\mathrm{d}u.$ (4) $$

Then, we can write $(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)=r({\mathbf{c}}({\mathbf{x}}_{t},t))$ as the reward of the generated clean sample.
This approximates $r_{\beta}({\mathbf{x}}_{t},t)$ using only one evaluated sample.
In fact, we can characterize the difference between the approximate reward using ODE $(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)$ and the exact reward estimate using SDE $r_{\beta}({\mathbf{x}}_{t},t)$ as in Lemma 1. The right hand side of Equation 5 shows that, as $\beta\to 0$, the approximation becomes exact: $\lim_{\beta\to 0^{+}}r_{\beta}({\mathbf{x}}_{t},t)=(r\circ{\mathbf{c}})({
\mathbf{x}}_{t},t).$
Intuitively, this result aligns with SDEs reducing to ODEs when $\beta$ approaches zero in image domains (Song et al., ).

###### Lemma 1 (Itô Integral Representation of Reward Proximity Error. Proof is in Section D.1 ) .

Let the reward estimate function, $h({\mathbf{x}}_{t},t)=(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)$, be shorthanded as $h$. We have:

$$ $r_{\beta}({\mathbf{x}}_{t},t)-(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)=\mathbb {E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[\int_{t}^{0}\nabla_{{\mathbf{x }}_{u}}h\cdot\,\mathrm{d}\mathbf{J}_{\beta}({\mathbf{x}}_{u},u)-\beta u^{2} \nabla^{2}h\,\mathrm{d}u\right].$ (5) $$

where ${\mathbf{x}}_{0}$ is sampled from Equation 2 and

$$ $\mathrm{d}\mathbf{J}_{\beta}({\mathbf{x}}_{u},u)=-\beta u^{2}\nabla_{{\mathbf{ x}}_{u}}\log p\left({\mathbf{x}}_{u},u\right)\,\mathrm{d}u+\sqrt{2\beta}u\, \mathrm{d}\omega_{u},$ (6) $$

is the Langevin diffusion SDE term, and $\nabla^{2}h$ is the Laplacian of $h$.

As demonstrated in Section D.1, Lemma 1 implies that when the Laplacian of the reward function is approximately zero ($\nabla^{2}r\approx 0$), $r_{\beta}\approx r\circ{\mathbf{c}}$. We also illustrated the idea in Figure 2.
For better presentation, we conveniently abbreviate $r_{\beta}({\mathbf{x}}_{t},t)$ as $r_{\beta}({\mathbf{x}}_{t})$, ${\mathbf{c}}({\mathbf{x}}_{t},t)$ as ${\mathbf{c}}({\mathbf{x}}_{t})$ and $(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)$ as $(r\circ{\mathbf{c}})({\mathbf{x}}_{t})$ in this paper.

Figure: Figure 2: The illustration of the proximity between the $r_{\beta}$ and $r\circ{\mathbf{c}}$. In this figure, the $\beta$ is nonzero and $r$ is near harmonic (i.e., $\nabla^{2}r\approx 0.$). The red points indicate i.i.d. SDE samples and the purple ODE approximation of ${\mathbf{x}}_{t}$. The green line indicates the expectation of the rewards of the SDE samples (e.g., an approximate estimation, $\frac{1}{4}\sum_{i=1}^{4}r({\mathbf{x}}_{0}^{(i)})$).
Refer to caption: x2.png

###### Lemma 1 (Itô Integral Representation of Reward Proximity Error. Proof is in Section D.1 ) .

### 4.2 Demons for Reward-Guided Denoising

Let’s first revisit reverse-time SDE.
Following Karras et al. (), an SDE numerical evaluation of $\hat{{\mathbf{x}}}_{t-\Delta}$ sampled from ${\mathbf{x}}_{t}$ can be seeded by noise ${\mathbf{z}}$ via a step of Heun’s $2^{nd}$ order method (Ascher & Petzold, ) as follows:

$$ $\displaystyle{\mathbf{z}}$ $\displaystyle\sim{\mathcal{N}}({\bm{0}},{\bm{I}}_{n})$ (7) $\displaystyle\hat{{\mathbf{x}}}_{t-\Delta}$ $\displaystyle=\mathrm{heun}({\mathbf{x}}_{t},{\mathbf{z}},t,\Delta)$ (8) $\displaystyle\coloneqq{\mathbf{x}}_{t}-\frac{1}{2}\left[{\mathbf{f}}_{\beta}({ \mathbf{x}}_{t},t)+{\mathbf{f}}_{\beta}(\tilde{{\mathbf{x}}}_{t-\Delta},t- \Delta)\right]\Delta+\frac{1}{2}\left[g_{\beta}(t)+g_{\beta}(t-\Delta)\right]{ \mathbf{z}}\sqrt{\Delta}\,,$ (9) $$

where ${\mathbf{z}}$ is a Gaussian noise, and $\mathrm{heun}$ is the stochastic backward step from ${\mathbf{x}}_{t}$ to $\hat{{\mathbf{x}}}_{t-\Delta}$.
The intermediate approximation $\tilde{{\mathbf{x}}}_{t-\Delta}$ is given by $\tilde{{\mathbf{x}}}_{t-\Delta}\coloneqq{\mathbf{x}}_{t}-{\mathbf{f}}_{\beta}(
{\mathbf{x}}_{t},t)\Delta+g_{\beta}(t){\mathbf{z}}\sqrt{\Delta}\,$.
Note that, while we use Heun’s method here, other solvers can work too.

For image generation, Gaussian noise ${\mathbf{z}}$ is usually high-dimensional.
For a high-dimensional $z$, we can assume that it’s likely on a $\sqrt{N}$ sphere (Lemma 5, Appendix).
This allows us to weighted-combine various noises into a new noise ${\mathbf{z}}^{*}$:

$$ ${\mathbf{z}}^{*}=\sqrt{N}\,\mathrm{normalized}\left(\sum_{k=1}^{K}{\color[rgb] {.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b_{k}}{\mathbf{z}}^{ (k)}\right),$ (10) $$

where ${\mathbf{z}}^{(k)}$ are i.i.d. unit Gaussian noises, and ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b_{k}}$ are the search space. The pseudocode of a numerical step with our proposed method is outlined in Algorithm 1.

In the following, we introduce two stochastic optimization approaches, the *Tanh Demon* and the *Boltzmann Demon*, to determine the weights ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b_{k}}$ and synthesize an optimal ${\mathbf{z}}^{*}$, with a goal of optimizing the final reward value.
And we show that solving the reverse-time SDE with such optimal ${\mathbf{z}}^{*}$ theoretically and empirically improve $r_{\beta}$, essentially achieving alignment.

Figure: Algorithm 1 A Numerical Step with Demon

#### 4.2.1 Tanh Demon

Intuitively, we may consider up-weighting the good noises that improve the reward and down-weighting the bad noises that harm the reward, compared to the average reward $\hat{\mu}$.
Tanh Demon assigns positive weights to the good noises and negative weights to the bad noises with the $tanh$ function, based on the reward estimates of the noises (Section 4.1):

$$ ${\mathbf{z}}^{*}=\sqrt{N}\,\mathrm{normalized}\left(\sum_{k=1}^{K}{\color[rgb] {.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b^{\tanh}_{k}}\,{ \mathbf{z}}^{(k)}\right),\quad\text{where}\quad{\color[rgb]{.5,0,.5} \definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b^{\tanh}_{k}}\leftarrow\tanh \left(\frac{(r\circ\mathbf{c})(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-\hat{\mu}} {\tau}\right)$ (11) $$

Here we can estimate $\hat{\mu}$ with $\hat{\mu}=\frac{1}{K}\sum_{k=1}^{K}(r\circ\mathbf{c})(\hat{{\mathbf{x}}}_{t-
\Delta}^{(k)})$.
$\tau$ is the temperature parameter to $\tanh$, which can be adaptively tuned (as shown in Table 8).

In the following, we demonstrate that synthesizing ${\mathbf{z}}^{*}$ with Equation 11 in every backward step, which nudges the sample towards the data distribution, leads to reward improvement of the final clean sample ${\mathbf{x}}_{0}$ with theoretical guarantee.

Figure: (a) Before
Refer to caption: x3.png

###### Lemma 2 .

Assume the truncation error terms in Equation 39 is ignorable. Let ${\mathbf{z}}^{*}$ be derived from ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b^{\tanh
}_{k}}=\tanh\left(\frac{r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-r_{
\beta}({\mathbf{x}}_{t})}{\tau}\right)$ using Equation 10 for every numerical step from $t$ to $t-\Delta$ and $\tau<\infty$. Then, almost surely, $r(\hat{{\mathbf{x}}}_{0}^{\tanh})>r_{\beta}({\mathbf{x}}_{t}),$ where $\hat{{\mathbf{x}}}_{0}^{\tanh}$ is derived by applying ${\mathbf{z}}^{*}$ on every step.

The formal proof of Lemma 2 is in Section D.3.
Lemma 2 states that the improvement of $r_{\beta}$ in reversed-time order almost surely, assuming that $r_{\beta}\equiv r\circ\mathbf{c}$ and that, based on 1 (Appendix), $\frac{1}{K}\sum_{k=1}^{K}r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})$ is an unbiased estimator of $r_{\beta}({\mathbf{x}}_{t})$.
Figure 3 provides an illustration of the Tanh Demon.

Since $(r\circ{\mathbf{c}})$ is not a perfect estimate of $r_{\beta}$, selecting an appropriate $\tau$ is crucial to manage errors. A small $\tau$ makes $\tanh(\cdot/\tau)$ nearly binary and can weight good noise negatively and vice versa. Conversely, a large $\tau$ diminishes contribution of noises.

###### Lemma 2 .

#### 4.2.2 Boltzmann Demon

Another intuitive approach is to estimate the candidate with maximum reward.
We propose the Boltzmann demon, which assign noise weights as follows.

$$ ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b^{ \mathrm{boltz}}_{k}}\leftarrow\frac{\exp\left(r\circ{\mathbf{c}}(\hat{{\mathbf {x}}}_{t-\Delta}^{(k)})/\tau\right)}{\sum_{k=1}^{K}\exp\left(r\circ{\mathbf{c} }(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})/\tau\right)}.$ (12) $$

The theoretical guarantee of improvement in $r_{\beta}$ in expectation is provided in Lemma 3 of Appendix, assuming $r_{\beta}\equiv r\circ{\mathbf{c}}.$ This method is equivalent to the single-step cross entropy approach (De Boer et al., ).
Although, empirically we find that Tanh Demon outperforms Boltzmann demon, adjusting $\tau$ in Boltzmann demon provides control over deviation from the original SDE distribution, as demonstrated in Lemma 4 (Appendix).

#### 4.2.3 Computational Considerations

Let’s first consider a Demon sampling trajectory ${\mathbf{x}}_{t_{1}}>{\mathbf{x}}_{t_{2}}>\cdots>{\mathbf{x}}_{t_{T}}\approx 0$ for a fixed number $T$. Each Demon’s trajectory requires $\mathcal{O}(K\cdot T)$ evaluations of ${\mathbf{c}}$, and each evaluation comes with one reward estimation. The compute time is mainly influenced by the implementation of $r\circ\mathbf{c}$.
We discuss two aspects of $r\circ\mathbf{c}$—the temporal cost and the fidelity—which are vital to the algorithm’s time complexity and reward performance, respectively.

Note that Tanh or Boltzmann Demon itself does not strictly specify the implementation of $r\circ{\mathbf{c}}$; our default option uses Heun’s ODE solver, but using a Consistency Model (CM) distilled from the original diffusion model significantly accelerates computation. An alternative, which we refer as Tanh-C, is to combine our Tanh Demon algorithm with an off-the-shelf CM to implement $r\circ{\mathbf{c}}$. While using Tanh-C may slightly degrade the results due to the fidelity loss from using a CM (see Table 2), this approach is particularly effective when faster results are required since the computation of
${\mathbf{c}}$ is much quicker. For a larger $T$, however, the default Tanh Demon using Heun’s method outperforms Tanh-C in terms of reward performance.

As shown in Table 10, using the text-to-image generation task settings from Black et al. (),
the Demon algorithm achieves an aesthetics score of $6.72\pm 0.26$ on SD v1.4, requiring 5 minutes (i.e., $K=16,T=16$) on an NVIDIA RTX 3090 GPU. Within the same 5-minute computation window, the Tanh-C variant achieves an improved score of $7.27\pm 0.33$ (i.e., $K=16,T=64$). Notably, the upper bound for randomly sampled SD v1.4 is approximately 6.5, obtained after more than 10 minutes and 800 reward function queries. See Appendix B for parameter guidelines and settings.

## 5 Experiments

In this section, we present both quantitative and qualitative evaluations of our methods. Due to the page limit, we include the details of the implementation and experimental settings in Appendix H and the subjective results in Section F.2.

##### Baseline Comparison.

For the performance comparisons between our method and other baselines, we use the LAION () aesthetics scores (Aes) as the evaluation metric, and the scores are evaluated on a set of various prompts for generating animal images, which were selected from a subset of 22 common animals in ImageNet-1K (Deng et al., ), created by Black et al. ()(^1^11Obtained from the official repository [github.com/jannerm/ddpo/blob/main/assets/very_simple_animals.txt](https://github.com/jannerm/ddpo/blob/main/assets/very_simple_animals.txt)).
We use 20-step Heun’s ODE for reward estimate for our methods and Best-of-N (SD v1.4).
In Figure 4, we can observe that the proposed Tanh Demon sampling method, in most cases, outperforms other baseline methods, including our Boltzmann Demon sampling method, Best-of-N, and DOODL (Wallace et al., ), the state-of-the-art inference-time method.
It is worth noting that, although given more number of reward queries, the performance of DOODL eventually surpasses Tanh, if we consider the same amount of execution time, Tanh is still consistently better.
This is because of the computational cost associated with DOODL’s backpropagation through the diffusion model chain.
In addition, we empirically observe more reward hacking with DOODL (based on backpropagation) compared to our method (see Table 3), though the underlying reason isn’t as clear.
For further comparison on PickScore (Kirstain et al., ), please refer to Section E.1.

Figure: (a) Performance w.r.t Reward Query Number
Refer to caption: x6.png

##### Comparison of Reward Estimation Approaches.

Figure 4 also shows a comparison of the proposed method with different $r\circ\mathbf{c}$ implementations, including 20-step Heun’s ODE (Tanh) and 1-step CM (Tanh-C).
Tanh-C, which uses 1-step CM for fast reward evaluation, consistently outperforms the Best-of-N baseline.
Tanh’s performance has been overall better given the equivalent number of reward queries. With these observations, we posit that the quality of $r\circ{\mathbf{c}}$ indeed matters to our method.

To further validate the importance of $r\circ\mathbf{c}$, we conduct a comparative analysis based on Lemma 1 ($r\circ\mathbf{c}\approx r_{\beta}$).
In this analysis, we evaluate accuracy and computational cost across three methods: 20-step Heun’s ODE, 6-step Heun’s ODE, and 1-step CM; both diffusion and consistency models are based on the SD v1.5 distilled by Luo et al. ().
Experiments were performed with $t=1,7,14$ ranging from 0.002 to 14.648 and $\beta=0.1$.
Accuracy was quantified using the RMSE, $\sqrt{\mathbb{E}_{{\mathbf{x}}_{t}}\left[((r\circ{\mathbf{c}})({\mathbf{x}}_{t
})-r_{\beta}({\mathbf{x}}_{t}))^{2}\right]}$, where $r_{\beta}$ is estimated by averaging over 100 Monte Carlo i.i.d. SDE samples.
Here, $\mathbf{x}_{t}$ is sampled from $\mathcal{N}(\mathbf{0},t_{\max}^{2}\mathbf{I}_{n})$ and integrated from $t_{\max}$ to $t$ using a 40-step diffusion model ODE.

**Table 2: Comparison of accuracy and time cost across different $r\circ\mathbf{c}$ implementations.**
| Implementation | Time (s) | RMSE |  |  |
| --- | --- | --- | --- | --- |
| $t=1.0$ | $t=7.0$ | $t=14.0$ |  |  |
| 20-step ODE | 1.94 | 0.044 | 0.258 | 0.333 |
| 6-step ODE | 0.62 | 0.056 | 0.341 | 0.368 |
| 1-step CM | 0.18 | 0.306 | 0.527 | 0.632 |

The results, presented in Table 2, support that the quality of $r\circ\mathbf{c}$ influences both the algorithm’s speed and reward performance. For the ODE methods, the trend follows our expectation: As $t$ approaches 0 0, RMSE decreases, which can be attributed to the diminishing noise as the posterior $p({\mathbf{x}}_{t}\mid{\mathbf{x}}_{0})$ becomes more sharply peaked;
the number of ODE steps is crucial to the quality of the generated outputs;
more steps generally lead to higher fidelity results, although this comes at the cost of increased computational time;
using 1-step CM leads to inferior results compared to using ODE, supposedly as the distillation gap and the limited model capacity results in lower-fidelity reconstructions.

**Table 3: Results using various reward functions and different generation methods. Each column represents a specific reward objective, with the best performance highlighted in bold.**
| Generation method | Aes $\uparrow$ | IR $\uparrow$ | Pick $\uparrow$ | HPSv2 $\uparrow$ |
| --- | --- | --- | --- | --- |
| SD v1.4 | 5.34 $\pm$ 0.56 | -0.00 $\pm$ 0.95 | 0.202 $\pm$ 0.008 | 0.216 $\pm$ 0.036 |
| Tanh + Aes | 7.35 $\pm$ 0.40 | -0.03 $\pm$ 1.24 | 0.211 $\pm$ 0.010 | 0.257 $\pm$ 0.041 |
| Tanh + IR | 5.96 $\pm$ 0.28 | 1.95 $\pm$ 0.07 | 0.216 $\pm$ 0.012 | 0.286 $\pm$ 0.033 |
| Tanh + Pick | 6.14 $\pm$ 0.48 | 1.39 $\pm$ 0.57 | 0.245 $\pm$ 0.010 | 0.312 $\pm$ 0.033 |
| Tanh + HPSv2 | 5.98 $\pm$ 0.45 | 1.51 $\pm$ 0.63 | 0.228 $\pm$ 0.011 | 0.367 $\pm$ 0.027 |
| Tanh + Ensemble | 6.53 $\pm$ 0.50 | 1.81 $\pm$ 0.15 | 0.236 $\pm$ 0.014 | 0.356 $\pm$ 0.030 |
| DOODL + Aes | 5.59 $\pm$ 0.29 | -0.68 $\pm$ 1.06 | 0.197 $\pm$ 0.008 | 0.221 $\pm$ 0.028 |
| DOODL + Pick | 5.21 $\pm$ 0.46 | -0.12 $\pm$ 0.84 | 0.204 $\pm$ 0.010 | 0.220 $\pm$ 0.035 |

**Table 4: Using Tanh Demons with various reward functions. The baseline, Stable Diffusion v1.4, refers to the standard model without our proposed enhancements.**
| Baseline | DOODL | Aes | IR | Pick | HPSv2 | Ensemble |
| --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |

##### Image Generation with Various Reward Functions.

While our method optimizes a given reward function, as shown in Figure 4,
it doesn’t conclusively demonstrate user perceptual preferences.
To address this, we present qualitative in Section 5. Moreover, in Table 3, we also show more quantitative results using our Tanh Demon with various reward functions followed by performance evaluation in metrics like Aes (LAION, ), ImageReward (IR) (Xu et al., ), PickScore (Pick) (Kirstain et al., ), and HPSv2 (Wu et al., ). As shown in Table 3, our Tanh Demon with various reward functions consistently outperforms the inference-time state-of-the-art backpropagation-based method, DOODL (Wallace et al., ), which employs Aes and Pick as the objective to modify the results generated by PF-ODE using their recommended setting.
Though slightly over-optimized (hacking) on the objective, our method shows improvement across various metrics. In comparison, DOODL, a backpropagation-based method, demonstrates a decreased score on other objectives.
Additionally, our method achieves the runner-up generation results on *each* objective using an Ensemble reward function, a scaled sum of Aes, IR, Pick, and HPSv2 scores, demonstrating the ability to integrate a mixture of rewards.

##### Alignment with preferences of VLMs (Non-differentiable).

In Table 5, we present qualitative results of aligning diffusion models to preferences of VLMs from API, as a demonstration of using non-differentiable reward sources.
In this experiment, we use Google Gemini Pro v1.0 (Gemini Team Google, ) and GPT4 Turbo (OpenAI, ).
In each step, the VLM receives a fixed prompt, e.g. “You are a journalist who wants to add a visual teaser for your article to grab attention on social media or your news website”, and is asked to select the best-matching intermediate sample from generated images.
VLMs are presented with $\mathbf{c}({\mathbf{x}}_{t})$ and $\mathbf{c}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})$ produced by PF-ODE.
The reward ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b^{
\mathrm{VLM}}_{k}}$ is $0.5$ if the VLM selects $\mathbf{c}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})$ and $-0.5$ otherwise.
We also use PickScore (Kirstain et al., ) to evaluate the results, and find that 14 out of 16 images generated with VLMs show improvements compared to directly generating with PF-ODE.
For full prompts, scenarios, and quantitative results, please refer to Appendix G.

**Table 5: Using VLMs to generate images. PF-ODE (baseline) refers to a baseline without using our method for alignment. Columns 3-6 indicate the role that the agent plays in the given prompt.**
| Model | Baseline | Teacher | Artist | Researcher | Journalist |
| --- | --- | --- | --- | --- | --- |
| Gemini-SD v1.4 |  |  |  |  |  |
| Gemini-SDXL |  |  |  |  |  |
| GPT-SD v1.4 |  |  |  |  |  |
| GPT-SDXL |  |  |  |  |  |

##### Manual Selection.

We also explore using online interactive human judgements to guide diffusion.
That means, the users themselves would be (non-differentiable) reward functions.
We let users directly interact with our method to generate desired images.
Figure 4(c) shows an example interface created by us for an image resembling a given reference cat image.
At each iterative step from $t$ to $t-\Delta$, we sample 16 i.i.d. copies of $\bm{x}_{t-\Delta}$ and compute $\mathbf{c}(\bm{x}_{t-\Delta})$ with PF-ODE.
The user then manually select their preferred image, assigning a reward of $+1$ to it and $-1$ to the others.
We continue this process until there is no obvious preferred ones among the generated images.
As shown in Figure 4(d), the image generated by our method more closely matches the target than the one produced by PF-ODE.
We also measure the improvement with DINOv2 (Oquab et al., ) embedding cosine similarity between the reference image and the generated image, and observe that the similarity improves from $0.594$ to $0.708$ through online user interactions.

Figure: (c) Our user interface for interacting with our algorithm (0.594 cosine similarity).
Refer to caption: extracted/5907739/Images/manual/ui.jpg

## 6 Conclusion

This work addresses the challenge of better aligning pre-trained diffusion models without training or backpropagation. We first demonstrate how to estimate noisy samples’ rewards based on clean samples using PF-ODE.
Additionally, we introduce a novel inference-time sampling method, based on stochastic optimization, to guide the denoising process with any reward sources, including non-differentiable reward sources
that includes VLMs and interactive human judgements.
Theoretical analysis and extensive experimental results validate the effectiveness of our proposed method for improved image generation without requiring additional training.
Through comprehensive empirical and theoretical analysis, we observe that the quality and efficiency of reward estimation $r\circ{\mathbf{c}}$ are essential for our algorithm, especially in balancing computational speed and reward performance.

## Acknowledgement

This research is supported by National Science and Technology Council, Taiwan (R.O.C), under the grant number of NSTC-112-2634-F-002-006, NSTC-112-2222-E-001-001-MY2, NSTC-113-2634-F-001-002-MBK and Academia Sinica under the grant number of AS-CDA-110-M09. We specially thank Sirui Xie, I-Sheng Fang, and Jia-Wei Liao for the insightful discussions.

## Appendix A Notations and Conventions

Although we keep the main paper self-consistent, we provide this section to establish a consistent notation and convention for this paper as an aid.

### A.1 Notations

**Table 6: Notations**
| Notation | Description |
| --- | --- |
| $N$ | State dimension |
| $K$ | Noise sample number |
| $t_{\mathrm{min}},t_{\mathrm{max}}$ | Upper bound and Lower bound of the noise level in numerical integration |
| $T$ | Number of time steps to solve SDE/ODE |
| $\beta$ | Noise parameter |
| ${\mathbf{x}}$ | State variable |
| ${\mathbf{z}}$ | Noise from Gaussian |
| $\Delta$ | Time step |
| $b_{k}$ | Unnormalized weight of noise |
| ${\mathbf{f}}_{\beta}$ | SDE policy drift |
| $g_{\beta}$ | SDE policy diffusion coefficient |
| ${\mathbf{f}}_{0}$ | PF-ODE policy drift |
| $\mathbf{J}_{\beta}$ | Langevin diffusion SDE |
| $\omega_{t}$ | reversed time Brownian motion |
| $r$ | Reward |
| $r_{\beta}$ | Reward estimates of SDE policy |
| ${\mathbf{c}}$ | Function to get expected ODE result |
| $\mathrm{heun}$ | Heuns’s method, SDE solver for Karras SDE |

### A.2 Conventions

**Table 7: Conventions**
| Convention | Details |
| --- | --- |
| $r\circ{\mathbf{c}}$ | ODE reward estimate approximation, $r({\mathbf{c}}({\mathbf{x}}_{t},t))=(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)$ |
| $f\equiv g$ | For all $x$ of our interest, $f(x)=g(x)$ |
| $\hat{{\mathbf{x}}}$ | Numerical approximation with SDE solver |
| $\tilde{{\mathbf{x}}}$ | Intermediate value of Heun’s method |
| ${\mathbf{x}}^{\prime}$ | An ODE trajectory |
| $\tilde{\mathbf{z}}$ | Uniformly sampled from the sphere of radius $\sqrt{N}$ |
| $\mathbf{z}^{*}$ | Optimal noise generated by our algorithm |
| $\hat{\mu}$ | Mean of next state ODE reward estimates, $\frac{1}{K}\sum_{k=1}^{K}(r\circ{\mathbf{c}})(\hat{{\mathbf{x}}}_{t-\Delta}^{(<br>k)})$ |
| $r({\mathbf{x}}_{t})$ | Shorthand for $r({\mathbf{x}}_{t},t)$ when the context is clear |
| ${\mathbf{c}}({\mathbf{x}}_{t})$ | Shorthand for ${\mathbf{c}}({\mathbf{x}}_{t},t)$ when the context is clear |
| $(r\circ{\mathbf{c}})({\mathbf{x}}_{t})$ | Shorthand for $(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)$ when the context is clear |
| ${\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}$ | Shorthand for ${\mathbf{x}}_{0}\mid_{\beta}{\mathbf{x}}_{t},$ where ${\mathbf{x}}_{0}={\mathbf{x}}_{t}+\int_{t}^{0}{\mathbf{f}}_{\beta}({\mathbf{x}<br>}_{u},u)\,\mathrm{d}u+g_{\beta}(u)\,\mathrm{d}\omega_{u}$ |
| $\tilde{\omega}_{t}$ | standard Brownian motion |

Instead of just ODE, we use PF-ODE to highlight Song et al. ()’s contribution or when the context is unclear. They are equivalent here.

## Appendix B Guideline on Parameter Setting

We explore the optimal setting for parameter $\tau$ with respect to the Boltzmann Demon and the Tanh Demon. For the Tanh Demon, the most effective $\tau$ is neither $\infty$ nor 0 0. We recommend setting $\tau$ to the standard deviation of the estimations $\{(r\circ\mathbf{c})(\bm{x}_{t-\Delta}^{(k)})\}_{k=1}^{K}$, rendering it an adaptive parameter that is robust to scaling. For the Boltzmann Demon, optimal performance is achieved by setting $\tau$ to 0 0, as demonstrated in Table 8.

**Table 8: Comparison of performance for different settings of $\tau$ in the setting of Figure 4.**
|  | $\tau=1$ | $\tau=0.01$ | Adaptive $\tau$ |
| --- | --- | --- | --- |
| Tanh | $7.40\pm 0.30$ | $7.24\pm 0.31$ | $\mathbf{7.45}\pm 0.33$ |
| Boltzmann | $6.30\pm 0.3$5 | $\mathbf{7.28}\pm 0.30$ | $6.85\pm 0.37$ |

We also conduct an ablation study on the remaining parameters $K$ and $\beta$. The base configuration is $K=16,\beta=0.1$, with an adaptive temperature $\tau$ for the Tanh Demon. We set $T=32$ for the ablation study of $\beta$ and $T=64$ for $K$.

Figure: Figure 6: Comparison of our algorithm with respect to $K$ and $\beta$
Refer to caption: extracted/5907739/Images/ablation/vs_k.jpg

We found a large $\beta$ makes the sampling unstable, given the number of steps $T$ is fixed. Predictably, sampling with a $\beta$ close to 0 0 is reduced to ODE. From our theoretical result Lemma 1, the design methodology, and empirical results, the guidelines Table 9 can assist users in setting parameters.

**Table 9: Guidelines for Setting Hyperparameters**
| Parameter | Description |
| --- | --- |
| $K$ | Controls the noise distribution bias, positively affecting final quality and linearly increasing computational time. |
| $\beta$ | Adjusts the distribution’s proximity to the original PF-ODE. Set empirically based on $r$’s characteristics. Lemma 1 suggests smaller $\beta$ for reward functions with Laplacian deviations. |
| $T$ | Inherit the properties of time steps $T$ from diffusion models, scaling computational time linearly. Karras’s EDM recommends $T>17$. |
| $\tau$ | Recommended values vary for Boltzmann and Tanh Demons, as detailed in Table 8. |
| $r\circ\mathbf{c}$ | Accurate reward estimates are critical for ensuring high final quality. |

We provide a sparse parameter search in Table 10.

**Table 10: This table presents the experimental configurations used to measure aesthetics score under various animal prompts, presenting a sparse search of parameters. The time column represents the duration required to generate each image. We alias adaptive temperature as Adaptive.**
| Demon | Checkpoint | $\beta$ | $K$ | $T$ | $\tau$ | Aes | Time (min) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Boltzmann | SD v1.4 | 0.1 | 16 | 64 | Adaptive | 6.408 $\pm$ 0.36 | 17.6 |
| 1e-10 | 7.111 $\pm$ 0.32 | 16.6 |  |  |  |  |  |
| SDXL | 0.05 | 16 | 32 | Adaptive | 6.853 $\pm$ 0.37 | 45.8 |  |
| 1e-02 | 7.276 $\pm$ 0.30 | 45.4 |  |  |  |  |  |
| 1 | 6.300 $\pm$ 0.35 | 46.1 |  |  |  |  |  |
| 0.1 | 16 | 64 | Adaptive | 6.990 $\pm$ 0.38 | 94.2 |  |  |
| 1e-10 | 7.501 $\pm$ 0.31 | 93.1 |  |  |  |  |  |
| Tanh | SD v1.4 | 0.05 | 16 | 16 | Adaptive | 6.723 $\pm$ 0.26 | 5.0 |
| 32 | Adaptive | 7.073 $\pm$ 0.22 | 9.7 |  |  |  |  |
| 64 | Adaptive | 7.394 $\pm$ 0.29 | 18.7 |  |  |  |  |
| 0.1 | 16 | 64 | Adaptive | 7.549 $\pm$ 0.43 | 18.7 |  |  |
| 64 | 64 | Adaptive | 8.566 $\pm$ 0.33 | 79.1 |  |  |  |
| Diffusion-DPO | 0.1 | 16 | 64 | Adaptive | 7.564 $\pm$ 0.34 | 94.5 |  |
| SDXL | 0.01 | 16 | 16 | Adaptive | 6.876 $\pm$ 0.40 | 22.0 |  |
| 0.05 | 16 | 16 | Adaptive | 6.866 $\pm$ 0.35 | 21.9 |  |  |
| 32 | Adaptive | 7.459 $\pm$ 0.33 | 46.0 |  |  |  |  |
| 1e-02 | 7.244 $\pm$ 0.31 | 46.0 |  |  |  |  |  |
| 1 | 7.398 $\pm$ 0.30 | 46.2 |  |  |  |  |  |
| 0.1 | 8 | 64 | Adaptive | 7.446 $\pm$ 0.37 | 47.0 |  |  |
| 16 | 64 | Adaptive | 7.841 $\pm$ 0.32 | 94.4 |  |  |  |
| 32 | 64 | Adaptive | 8.179 $\pm$ 0.35 | 188.8 |  |  |  |
| 0.5 | 16 | 32 | Adaptive | 6.370 $\pm$ 0.35 | 46.0 |  |  |
| Tanh-C | SD v1.4 | 0.5 | 16 | 64 | Adaptive | 7.269 $\pm$ 0.33 | 5.0 |
| 0.1 | 16 | 64 | Adaptive | 6.710 $\pm$ 0.34 | 5.0 |  |  |
| SDXL | 0.5 | 16 | 64 | Adaptive | 7.301 $\pm$ 0.24 | 17.9 |  |

## Appendix C Pseudocodes

As an aid, we provide pseudocodes for the design of Demons Algorithm 2, Algorithm 3:

Figure: Algorithm 2 Tanh Demon with Adaptive Temperature

Figure: Algorithm 3 Boltzmann Demon with Fixed Temperature $\tau$

## Appendix D Mathematics

### D.1 Error Comprehension for Reward Estimate Approximation

In this section, we present the theoretical analysis and proof better to understand the error in our reward estimate approximation.

#### D.1.1 Error Term as an Itô Integral

###### Lemma 1 .

Let the reward estimate function, $h({\mathbf{x}}_{t},t)=(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)$, be shorthanded as $h$. We have:

$$ $r_{\beta}({\mathbf{x}}_{t},t)-(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)=\mathbb {E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[\int_{t}^{0}\nabla_{{\mathbf{x }}_{u}}h\cdot\,\mathrm{d}\mathbf{J}_{\beta}({\mathbf{x}}_{u},u)-\beta u^{2} \nabla^{2}h\,\mathrm{d}u\right].$ (13) $$

where ${\mathbf{x}}_{0}$ is sampled from Equation 2 and

$$ $\mathrm{d}\mathbf{J}_{\beta}({\mathbf{x}}_{u},u)=-\beta u^{2}\nabla_{{\mathbf{ x}}_{u}}\log p\left({\mathbf{x}}_{u},u\right)\,\mathrm{d}u+\sqrt{2\beta}u\, \mathrm{d}\omega_{u},$ (14) $$

is the Langevin diffusion SDE term, and $\nabla^{2}h$ is the Laplacian of $h$.

###### Proof.

We aim to prove:

$$ $r({\mathbf{x}}_{0})-(r\circ{\mathbf{c}})({\mathbf{x}}_{t},t)=\int_{t}^{0} \nabla_{{\mathbf{x}}_{u}}h\cdot\,\mathrm{d}\mathbf{J}_{\beta}({\mathbf{x}}_{u} ,u)-\beta u^{2}\nabla^{2}h\,\mathrm{d}u,$ (15) $$

Recall that

$$ $\displaystyle{\mathbf{x}}_{0}$ $\displaystyle={\mathbf{x}}_{t}+\int_{t}^{0}{\mathbf{f}}_{\beta}({\mathbf{x}}_{ u},u)\,\mathrm{d}u+g_{\beta}(u)\,\mathrm{d}\omega_{u},$ (16) $\displaystyle\mathbf{c}\left({\mathbf{x}}_{t}^{\prime},t\right)$ $\displaystyle={\mathbf{x}}_{t}^{\prime}+\int_{t}^{0}{\mathbf{f}}_{0}({\mathbf{ x}}_{u}^{\prime},u)\,\mathrm{d}u.$ (17) $$

For an ODE trajectory ${\mathbf{x}}^{\prime}(t)$, notice that:

$$ $0=\frac{\mathrm{d}}{\mathrm{d}t}h({\mathbf{x}}^{\prime}(t),t)=\frac{\partial h }{\partial t}+\nabla_{{\mathbf{x}}}h\cdot\frac{\mathrm{d}{\mathbf{x}}^{\prime} }{\mathrm{d}t}=\frac{\partial h}{\partial t}+\nabla_{{\mathbf{x}}}h\cdot{ \mathbf{f}}_{0}.$ (18) $$

We can write:

$$ $r({\mathbf{x}}_{0})-(r\circ\mathbf{c})({\mathbf{x}}_{t},t)=h({\mathbf{x}}_{0}, 0)-h({\mathbf{x}}_{t},t)=\int_{t}^{0}\mathrm{d}h,$ (19) $$

where ${\mathbf{x}}_{t}$, which is not an ODE trajectory (noted by ${\mathbf{x}}_{t}^{\prime}$), follows the SDE trajectory. Using Itô’s lemma Ito et al. (), we find:

$$ $\displaystyle\mathrm{d}h$ $\displaystyle=\left(\frac{\partial h}{\partial t}+\nabla_{{\mathbf{x}}}h\cdot{ \mathbf{f}}_{\beta}-\frac{1}{2}\cdot 2g_{\beta}^{2}\nabla^{2}h\right)\,\mathrm {d}t+g_{\beta}\nabla_{{\mathbf{x}}}h\cdot\,\mathrm{d}\omega_{t}$ (20) $\displaystyle=\left(\frac{\partial h}{\partial t}+\nabla_{{\mathbf{x}}}h\cdot{ \mathbf{f}}_{\beta}-\left(\frac{\partial h}{\partial t}+\nabla_{{\mathbf{x}}}h \cdot{\mathbf{f}}_{0}\right)-g_{\beta}^{2}\nabla^{2}h\right)\,\mathrm{d}t+g_{ \beta}\nabla_{{\mathbf{x}}}h\cdot\,\mathrm{d}\omega_{t}$ (21) $\displaystyle=\left(\nabla_{{\mathbf{x}}}h\cdot({\mathbf{f}}_{\beta}-{\mathbf{ f}}_{0})-g_{\beta}^{2}\nabla^{2}h\right)\,\mathrm{d}t+g_{\beta}\nabla_{{ \mathbf{x}}}h\cdot\,\mathrm{d}\omega_{t}$ (22) $\displaystyle=\nabla_{{\mathbf{x}}}h\cdot\left(-\beta t^{2}\nabla_{{\mathbf{x} }_{t}}\log p({\mathbf{x}}_{t},t)\,\mathrm{d}t+\sqrt{2\beta}t\,\mathrm{d}\omega _{t}\right)-\beta t^{2}\nabla^{2}h\,\mathrm{d}t.$ (23) $$

The sign of the Itô correction term is flipped due to reverse time Brownian Motion-—and the other is followed by expansion. We thus derived Equation 15; by taking the expectation $\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[\cdot\right]$ on both sides, we can derive Equation 5.
∎

###### Lemma 1 .

###### Proof.

#### D.1.2 Discussion

We interpret the error terms of the reward estimates approximation as follows:

- •
The estimate becomes more accurate as $\beta$ decreases, satisfying the intuition that SDE trajectories will reduce to the ODE trajectory as $\beta\to 0$.
- •
If $\nabla_{{\mathbf{x}}_{u}}h\perp\nabla_{{\mathbf{x}}_{u}}\log p\left({\mathbf{x
}}_{u},u\right)$, the term $\nabla_{{\mathbf{x}}_{u}}h\cdot\mathrm{d}\mathbf{J}_{\beta}({\mathbf{x}}_{u},u)$ cancels out in expectation.
- •
If $\nabla^{2}h\equiv 0$ and the previous condition holds, then $r\circ\mathbf{c}\equiv r_{\beta}$.

For estimation purposes, we make the following assumptions to facilitate understanding and derivation of Equation 5:

$$ $\displaystyle\nabla_{{\mathbf{x}}_{t}}\log p({\mathbf{x}}_{t},t)$ $\displaystyle\approx-\frac{{\mathbf{x}}_{t}}{t^{2}}$ (24) $\displaystyle\mathbf{c}({\mathbf{x}}_{t},t)$ $\displaystyle\approx C_{t}{\mathbf{x}}_{t}$ (25) $\displaystyle\nabla_{{\mathbf{x}}}r$ $\displaystyle\perp{\mathbf{x}}$ (26) $$

where $C_{t}$ is a time-dependent constant and $r$ is scale-invariant.

- •
Equation 24 is derived from the assumption that $p({\mathbf{x}}_{t})\approx\mathcal{N}(\mathbf{0},t^{2}\mathbf{I})$.
- •
Equation 25 stems from image preprocessing algorithms, such as those used in Stable Diffusion, which normalize the image distribution. This normalization implies that images in the dataset are often scaled to lie on a sphere. Therefore, we can reasonably assume that a randomly generated ${\mathbf{x}}_{t}$ is close to an image in the dataset in direction.
- •
Equation 26 is based on the intuition that minor changes in brightness do not significantly affect the semantic interpretation of an image. Besides, many training algorithms incorporate scaling as part of data augmentation, which aligns with the assumption that the gradient of $\nabla_{{\mathbf{x}}}r$ is orthogonal to ${\mathbf{x}}$.

Under these assumptions, we obtain:

$$ $\displaystyle\mathrm{d}h$ $\displaystyle=\nabla_{{\mathbf{x}}}h\cdot\left(-\beta t^{2}\nabla_{{\mathbf{x} }_{t}}\log p\left({\mathbf{x}}_{t},t\right)\,\mathrm{d}t+\sqrt{2\beta}t\, \mathrm{d}\omega_{t}\right)-\beta t^{2}\nabla^{2}h\,\mathrm{d}t$ (27) $\displaystyle\approx C_{t}\nabla_{{\mathbf{x}}}r\cdot\left(-\beta{\mathbf{x}}_ {t}\,\mathrm{d}t+\sqrt{2\beta}t\,\mathrm{d}\omega_{t}\right)-\beta t^{2}\nabla ^{2}h\,\mathrm{d}t$ (28) $\displaystyle\approx\sqrt{2\beta}tC_{t}\nabla_{{\mathbf{x}}}r\cdot\,\mathrm{d} \omega_{t}-\beta t^{2}C_{t}^{2}\nabla^{2}r\,\mathrm{d}t$ (29) $$

If $r$ is harmonic, i.e., $\nabla^{2}r\equiv 0$, then $\mathrm{d}h$ becomes a martingale (Billingsley, ) and:

$$ $r_{\beta}({\mathbf{x}}_{t},t)-(r\circ\mathbf{c})({\mathbf{x}}_{t},t)\approx \mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[\int_{t}^{0}\sqrt{2 \beta}tC_{t}\nabla_{{\mathbf{x}}}r\cdot\,\mathrm{d}\omega_{t}\right]=0.$ (30) $$

The mean value property, an equivalent statement of a harmonic function, states that the value of a harmonic function at any point is the average of its values on any sphere centered at that point. This property provides an intuitive explanation of our method: if $r$ is harmonic, the reward of the ODE-generated image is the mean value of the reward of SDE-generated ones, while empirically, we observe that the ODE generation resembles the SDE variants.

#### D.1.3 Illustration of Mismatch

For better understanding, we provide an example that $r_{\beta}$ and $r\circ\mathbf{c}$ don’t meet. We adopt assumptions in Section D.1.2 to illustrate the intuition, and suppose ${\mathbf{x}}_{t}$ is a noisy sample at time $t$ such that $\mathbf{c}({\mathbf{x}}_{t})$ is a sharp local maxima of $r$, where $\nabla^{2}r\ll 0$ near $\mathbf{c}({\mathbf{x}}_{t})$. Suppose further that $\beta$ is small enough such that the generated ${\mathbf{x}}_{0}$ is near $\mathbf{c}({\mathbf{x}}_{t})$. In this case, $r_{\beta}({\mathbf{x}}_{t})-(r\circ\mathbf{c})({\mathbf{x}}_{t})<0$ as $r_{\beta}({\mathbf{x}}_{t})=\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}
\left[r({\mathbf{x}}_{0})\right]<(r\circ\mathbf{c})({\mathbf{x}}_{t})$ by intuition.

We can also verify $r_{\beta}({\mathbf{x}}_{t})-(r\circ\mathbf{c})({\mathbf{x}}_{t})<0$ using Equation 15. Under the assumptions in Section D.1.2, we can write:

$$ $\displaystyle r_{\beta}({\mathbf{x}}_{t})-(r\circ\mathbf{c})({\mathbf{x}}_{t})$ $\displaystyle\approx\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[ \int_{t}^{0}\sqrt{2\beta}tC_{t}\nabla_{{\mathbf{x}}}r\cdot\,\mathrm{d}\omega_{ t}-\beta u^{2}\nabla^{2}h\,\mathrm{d}u\right]$ (31) $\displaystyle=\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[\int_{t}^ {0}-\beta u^{2}C_{t}^{2}\nabla^{2}r\,\mathrm{d}u\right]$ (32) $\displaystyle<0.$ (33) $$

Note that the value of $\nabla^{2}r$ is taken at $\mathbf{c}({\mathbf{x}}_{t})$, fluctuating with SDE.

### D.2 Martingale Property of Reward Estimates.

A martingale is a sequence of random variables that maintains a certain property over time Billingsley (): the expected future value, given all past values, is equal to the current value; for a fixed SDE, the current reward estimate is the expected value of the reward estimates at the next time step:

###### Fact 1 .

For any time step $\Delta<0$ such that $t>t-\Delta>0$:

$$ $r_{\beta}({\mathbf{x}}_{t})=\mathbb{E}_{{\mathbf{x}}_{t-\Delta}\mid{\mathbf{x} }_{t}}\left[r_{\beta}({\mathbf{x}}_{t-\Delta})\right].$ (34) $$

Intuitively speaking, this idea stems from the principles of conditional probability, which tell us that our current prediction of the final score should be the same as the average of all possible future predictions.

###### Proof.

This result follows directly from the foundational theorem of expectation. Let $\mathcal{G}$ be the $\sigma$-algebra generated by ${\mathbf{x}}_{t-\Delta}$ and $\mathcal{F}$ be the $\sigma$-algebra generated by ${\mathbf{x}}_{t}$. Note that $\mathcal{G}$ is a refinement of $\mathcal{F}$.

For an integrable random variable $r({\mathbf{x}}_{0})$, we have:

$$ $\mathbb{E}\left[r({\mathbf{x}}_{0})\mid\mathcal{F}\right]=\mathbb{E}\left[ \mathbb{E}\left[r({\mathbf{x}}_{0})\mid\mathcal{G}\right]\mid\mathcal{F}\right]$ $$

Applying this to our context:

$$ $\displaystyle\mathbb{E}_{{\mathbf{x}}_{t-\Delta}\mid{\mathbf{x}}_{t}}\left[r_{ \beta}({\mathbf{x}}_{t-\Delta})\right]$ $\displaystyle=\mathbb{E}\left[\mathbb{E}\left[r({\mathbf{x}}_{0})\mid\mathcal{ G}\right]\mid\mathcal{F}\right]$ (35) $\displaystyle=\mathbb{E}\left[r({\mathbf{x}}_{0})\mid\mathcal{F}\right]$ (36) $\displaystyle=\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}}\left[r({ \mathbf{x}}_{0})\right]$ (37) $\displaystyle=r_{\beta}({\mathbf{x}}_{t})$ (38) $$

Therefore, we conclude that:

$$ $r_{\beta}({\mathbf{x}}_{t})=\mathbb{E}_{{\mathbf{x}}_{t-\Delta}\mid{\mathbf{x} }_{t}}\left[r_{\beta}({\mathbf{x}}_{t-\Delta})\right]$ $$

∎

###### Fact 1 .

###### Proof.

### D.3 Tanh Demon

We provide the theoretical idea behind the development of the algorithm. To start with, there exists a linear relationship between the reward estimate increment from ${\mathbf{x}}_{t}$ to $\hat{{\mathbf{x}}}_{t-\Delta}^{(k)}$ and the injected noise ${\mathbf{z}}^{(k)}$, which can be derived from Itô’s lemma Ito et al. () and Kolmogorov backward equations Kolmogoroff (), as follows:

$$ $r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-r_{\beta}({\mathbf{x}}_{t})=g(t )\nabla_{{\mathbf{x}}_{t}}r_{\beta}\cdot{\mathbf{z}}^{(k)}\sqrt{\Delta}+o( \Delta),\quad\text{where}\quad\hat{{\mathbf{x}}}_{t-\Delta}^{(k)}=\mathrm{heun }({\mathbf{x}}_{t},{\mathbf{z}}^{(k)},t,\Delta),$ (39) $$

which can be interpreted from an SDE with the following Lemma.

###### Claim 1 .

Let $r_{\beta}({\mathbf{x}}_{t},t)=\mathbb{E}_{{\mathbf{x}}_{0}\mid{\mathbf{x}}_{t}
}[r({\mathbf{x}}_{0})\mid{\mathbf{x}}_{t}]$ be the expected future value of a function $r$ at time 0 0, given the current state ${\mathbf{x}}_{t}$ at time $t$. Then, under the SDE:

$$ $\mathrm{d}{\mathbf{x}}_{t}={\mathbf{f}}_{\beta}\,\mathrm{d}t+g_{\beta}\, \mathrm{d}\omega_{t},$ (40) $$

the differential of $r_{\beta}$ is:

$$ $\mathrm{d}r_{\beta}=g_{\beta}\,\nabla_{{\mathbf{x}}_{t}}r_{\beta}\cdot\mathrm{ d}\omega_{t}.$ (41) $$

###### Proof.

We begin by introducing a change of variables. Let $s=t_{\text{max}}-t$, so that as $t$ decreases from $t_{\text{max}}$ to 0 0, $s$ increases from 0 0 to $t_{\text{max}}$. This allows us to consider a forward-time process with standard Brownian motion $\tilde{\omega}_{s}$.

Given the original SDE, we can write:

$$ $\mathrm{d}{\mathbf{x}}_{s}=-{\mathbf{f}}_{\beta}\,\mathrm{d}s+g_{\beta}\, \mathrm{d}\tilde{\omega}_{s},$ (42) $$

where $\tilde{\omega}_{s}$ is the standard Brownian motion.

Now, applying Itô’s lemma to $r_{\beta}({\mathbf{x}}_{s},s)$:

$$ $\mathrm{d}r_{\beta}=\left(\frac{\partial r_{\beta}}{\partial s}-{\mathbf{f}}_{ \beta}\cdot\nabla_{{\mathbf{x}}_{s}}r_{\beta}+\frac{1}{2}g_{\beta}^{2}\nabla^{ 2}r_{\beta}\right)\mathrm{d}s+g_{\beta}\nabla_{{\mathbf{x}}_{s}}r_{\beta}\cdot \mathrm{d}\tilde{\omega}_{s}.$ (43) $$

We aim to prove the Kolmogorov backward equation:

$$ $\frac{\partial r_{\beta}}{\partial s}-{\mathbf{f}}_{\beta}\cdot\nabla_{{ \mathbf{x}}_{s}}r_{\beta}+\frac{1}{2}g_{\beta}^{2}\nabla^{2}r_{\beta}=0.$ (44) $$

To do so, we integrate Itô’s lemma from $s$ to $t_{\text{max}}$:

$$ $\displaystyle r_{\beta}({\mathbf{x}}_{t_{\text{max}}})-r_{\beta}({\mathbf{x}}_ {s})$ $\displaystyle=\int_{s}^{t_{\text{max}}}\mathrm{d}r_{\beta}$ (45) $\displaystyle=\int_{s}^{t_{\text{max}}}\left(\frac{\partial r_{\beta}}{ \partial s^{\prime}}-{\mathbf{f}}_{\beta}\cdot\nabla_{{\mathbf{x}}_{s^{\prime} }}r_{\beta}+\frac{1}{2}g_{\beta}^{2}\nabla^{2}r_{\beta}\right)\mathrm{d}s^{\prime}$ $\displaystyle+\int_{s}^{t_{\text{max}}}g_{\beta}\nabla_{{\mathbf{x}}_{s^{ \prime}}}r_{\beta}\cdot\mathrm{d}\tilde{\omega}_{s^{\prime}}.$ (46) $$

Since $r_{\beta}({\mathbf{x}}_{t_{\text{max}}})$ is a martingale, by taking the expectation (conditioned on ${\mathbf{x}}_{s}$) on both sides, we obtain:

$$ $\displaystyle 0$ $\displaystyle=\mathbb{E}_{{\mathbf{x}}_{t_{\text{max}}}\mid{\mathbf{x}}_{s}} \left[r_{\beta}({\mathbf{x}}_{t_{\text{max}}})-r_{\beta}({\mathbf{x}}_{s})\right]$ (47) $\displaystyle=\mathbb{E}_{{\mathbf{x}}_{t_{\text{max}}}\mid{\mathbf{x}}_{s}} \left[\int_{s}^{t_{\text{max}}}\left(\frac{\partial r_{\beta}}{\partial s^{ \prime}}-{\mathbf{f}}_{\beta}\cdot\nabla_{{\mathbf{x}}_{s^{\prime}}}r_{\beta}+ \frac{1}{2}g_{\beta}^{2}\nabla^{2}r_{\beta}\right)\mathrm{d}s^{\prime}\right]$ $\displaystyle+\mathbb{E}_{{\mathbf{x}}_{t_{\text{max}}}\mid{\mathbf{x}}_{s}} \left[\int_{s}^{t_{\text{max}}}g_{\beta}\nabla_{{\mathbf{x}}_{s^{\prime}}}r_{ \beta}\cdot\mathrm{d}\tilde{\omega}_{s^{\prime}}\right].$ (48) $$

The expectation of the stochastic integral is zero, as Itô integrals have a mean of zero:

$$ $\mathbb{E}_{{\mathbf{x}}_{t_{\text{max}}}\mid{\mathbf{x}}_{s}}\left[\int_{s}^{ t_{\text{max}}}g_{\beta}\nabla_{{\mathbf{x}}_{s^{\prime}}}r_{\beta}\cdot \mathrm{d}\tilde{\omega}_{s^{\prime}}\right]=0.$ (49) $$

Thus, we are left with:

$$ $\mathbb{E}_{{\mathbf{x}}_{t_{\text{max}}}\mid{\mathbf{x}}_{s}}\left[\int_{s}^{ t_{\text{max}}}\left(\frac{\partial r_{\beta}}{\partial s^{\prime}}-{\mathbf{f }}_{\beta}\cdot\nabla_{{\mathbf{x}}_{s^{\prime}}}r_{\beta}+\frac{1}{2}g_{\beta }^{2}\nabla^{2}r_{\beta}\right)\mathrm{d}s^{\prime}\right]=0.$ (50) $$

Since the expectation is zero for any interval $[s,t_{\text{max}}]$, the integrand itself must be zero:

$$ $\frac{\partial r_{\beta}}{\partial s}-{\mathbf{f}}_{\beta}\cdot\nabla_{{ \mathbf{x}}_{s}}r_{\beta}+\frac{1}{2}g_{\beta}^{2}\nabla^{2}r_{\beta}=0.$ (51) $$

Thus, the differential of $r_{\beta}$ is given by:

$$ $\mathrm{d}r_{\beta}=g_{\beta}\nabla_{{\mathbf{x}}_{s}}r_{\beta}\cdot\mathrm{d} \tilde{\omega}_{s},$ (52) $$

Returning to the original time variable $t$, we substitute $s=t_{\text{max}}-t$ yielding:

$$ $\mathrm{d}r_{\beta}=g_{\beta}\nabla_{{\mathbf{x}}_{t}}r_{\beta}\cdot\mathrm{d} \omega_{t},$ (53) $$

completing the proof.

∎

Although $g(t)\nabla_{{\mathbf{x}}_{t}}r_{\beta}$ is inaccessible without distillation and thus an intractable static vector, we can still leverage the linear relationship to derive applications. Using our standard approach of interpreting $r\circ\mathbf{c}$ as $r_{\beta}$ and recognizing that $r_{\beta}({\mathbf{x}}_{t-\Delta})$ is an unbiased estimator of $r_{\beta}({\mathbf{x}}_{t})$ (from Section D.2), we practically interpret Equation 39 as:

$$ $(r\circ\mathbf{c})(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-\hat{\mu}\approx g(t) \nabla_{{\mathbf{x}}_{t}}r_{\beta}\cdot{\mathbf{z}}^{(k)}\sqrt{\Delta},\quad \text{where}\quad\hat{\mu}=\frac{1}{K}\sum_{k=1}^{K}(r\circ\mathbf{c})(\hat{{ \mathbf{x}}}_{t-\Delta}^{(k)}).$ (54) $$

From Equation 39, flipping the sign of ${\mathbf{z}}^{(k)}$ reverses its contribution to $r_{\beta}$. Therefore, based on the observation $(r\circ\mathbf{c})(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-\hat{\mu}$, we flip ${\mathbf{z}}^{(k)}$ accordingly.
We show the theoretical analysis and proof for the error of the reward estimate of our Tanh Demon in the following.

###### Lemma 2 .

Assume the truncation error terms in Equation 39 is ignorable. Let ${\mathbf{z}}^{*}$ be derived from ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b^{\tanh
}_{k}}=\tanh\left(\frac{r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-r_{
\beta}({\mathbf{x}}_{t})}{\tau}\right)$ using Equation 10 for every numerical step from $t$ to $t-\Delta$ and $\tau<\infty$. Then, almost surely, $r(\hat{{\mathbf{x}}}_{0}^{\tanh})>r_{\beta}({\mathbf{x}}_{t}),$ where $\hat{{\mathbf{x}}}_{0}^{\tanh}$ is derived by applying ${\mathbf{z}}^{*}$ on every step.

Let $\bm{\ell}=g(t)\nabla_{{\mathbf{x}}_{t}}r_{\beta}$. Recall that we assume

$$ $\displaystyle r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})-r_{\beta}({ \mathbf{x}}_{t})$ $\displaystyle=\bm{\ell}\cdot\mathbf{z}^{(k)}\sqrt{\Delta}$ (55) $\displaystyle r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{\tanh})-r_{\beta}({ \mathbf{x}}_{t})$ $\displaystyle=\bm{\ell}\cdot\mathbf{z}^{*}\sqrt{\Delta}$ (56) $\displaystyle\hat{{\mathbf{x}}}_{t-\Delta}^{(k)}$ $\displaystyle=\mathrm{heun}({\mathbf{x}}_{t},\mathbf{z}^{(k)},t,\Delta)$ (57) $\displaystyle\hat{{\mathbf{x}}}_{t-\Delta}^{\mathrm{\tanh}}$ $\displaystyle=\mathrm{heun}({\mathbf{x}}_{t},\mathbf{z}^{*},t,\Delta)$ (58) $\displaystyle\mathbf{z}^{*}$ $\displaystyle=\sqrt{N}\,\mathrm{normalized}\left(\sum_{k=1}^{K}{\color[rgb]{ .5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b_{k}^{\tanh}}\mathbf {z}^{(k)}\right)$ (59) $\displaystyle{\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{ .5,0,.5}b_{k}^{\tanh}}$ $\displaystyle=\tanh\left(\frac{r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})- r_{\beta}({\mathbf{x}}_{t})}{\tau}\right).$ (60) $$

We aim to prove the sufficient condition: $r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{\mathrm{\tanh}})>r_{\beta}({\mathbf{x
}}_{t})$ for each numerical step.
Under a rotation of basis, without loss of generality, we assume $\bm{\ell}$ only has value in the first component, i.e., $\bm{\ell}=(\ell,0,\dots,0)$ and $\ell>0$. We have:

$$ $\displaystyle r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{\mathrm{\tanh}})>r_{ \beta}({\mathbf{x}}_{t})\iff\ell z_{1}^{*}\sqrt{\Delta}>0$ (61) $$

###### Claim 2 .

Almost surely, the first component $z_{1}^{*}$ of $\mathbf{z}^{*}$ is positive.

###### Proof.

Since

$$ $\displaystyle{\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{ .5,0,.5}b_{k}^{\tanh}}$ $\displaystyle=\tanh\left(\frac{r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(k)})- r_{\beta}({\mathbf{x}}_{t})}{\tau}\right)$ (62) $\displaystyle=\tanh\left(\frac{\bm{\ell}\cdot\mathbf{z}^{(k)}\sqrt{\Delta}}{ \tau}\right)$ (63) $\displaystyle=\tanh\left(\frac{\ell z_{1}^{(k)}\sqrt{\Delta}}{\tau}\right),$ (64) $$

where $z_{1}^{(k)}$ is the first component of $\mathbf{z}^{(k)}$.

Almost surely, $z_{1}^{(k)}\neq 0$, so ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b_{k}^{
\tanh}}$ will have the same sign as $z_{1}^{(k)}$. This implies ${\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}b_{k}^{
\tanh}}z_{1}^{(k)}>0.$

Since the first component of $\mathbf{z}^{*}$ will have the same sign as the first component of $\sum_{k=1}^{K}{\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{
.5,0,.5}b_{k}^{\tanh}}\mathbf{z}^{(k)}$ i.e. $\sum_{k=1}^{K}{\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{
.5,0,.5}b_{k}^{\tanh}}z_{1}^{(k)}>0.$ We conclude that $z_{1}^{*}>0.$

∎

In addition, we provide proof of the linear relationship presented in Equation 39.

###### Claim 1 .

###### Proof.

###### Lemma 2 .

###### Claim 2 .

###### Proof.

### D.4 Boltzmann Demon

Recall that

$$ $\displaystyle{\mathbf{x}}_{t-\Delta}$ $\displaystyle\coloneqq{\mathbf{x}}_{t}+\int_{t}^{t-\Delta}{\mathbf{f}}_{\beta} ({\mathbf{x}}_{u},u)\,\mathrm{d}u+g_{\beta}(u)\,\mathrm{d}\omega_{u}$ (65) $\displaystyle\tilde{{\mathbf{x}}}_{t-\Delta}$ $\displaystyle\coloneqq{\mathbf{x}}_{t}-{\mathbf{f}}_{\beta}({\mathbf{x}}_{t},t )\Delta+g_{\beta}(t){\mathbf{z}}\sqrt{\Delta}$ (66) $\displaystyle\hat{{\mathbf{x}}}_{t-\Delta}$ $\displaystyle\coloneqq{\mathbf{x}}_{t}-\frac{1}{2}\left[{\mathbf{f}}_{\beta}({ \mathbf{x}}_{t},t)+{\mathbf{f}}_{\beta}(\tilde{{\mathbf{x}}}_{t-\Delta},t- \Delta)\right]\Delta+\frac{1}{2}\left[g_{\beta}(t)+g_{\beta}(t-\Delta)\right]{ \mathbf{z}}\sqrt{\Delta}$ (67) $$

We first present the theoretical analysis and proof for the reward estimate error of the proposed Boltzmann Demon as follows.

###### Lemma 3 .

Assume $t$ is bounded by $t_{\max}$, $\lvert r_{\beta}\rvert$ is bounded by $L$. Given $\bm{x}_{t}$, if the truncation error per Heun’s SDE step in Equation 67 is ${\mathbf{x}}_{t-\Delta}=\hat{{\mathbf{x}}}_{t-\Delta}+o(\Delta)$ as $\Delta\to 0^{+}$, then we have:

$$ $\mathbb{E}\left[r(\hat{\bm{x}}^{\mathrm{boltz}}_{0})\right]\geq r_{\beta}(\bm{ x}_{t})-o(L\cdot t_{\max}),$ (68) $$

where the expectation denotes that each step of the numerical approximation from every $t$ to $t+\Delta$ is taken with the maximum value of $r_{\beta}(\cdot)$ among i.i.d. SDE samples $\hat{\bm{x}}_{t+\Delta}^{(k)}$, representing the Boltzmann Demon with $\tau=0$.

Lemma 3 establishes a lower bound based on the sample maximum and reward estimate accuracy, providing an improvement guarantee of expected reward in expectation.

We first claim the following statement.

###### Claim 3 .

$$ $\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}}_{t-\Delta}) \right]\geq r_{\beta}({\mathbf{x}}_{t})-o(L\cdot\Delta).$ (69) $$

The rest is the induction of SDE time steps $t_{0}=t>\cdots>t_{T-2}>t_{T-1}>t_{T}=0$, i.e.,

$$ $\displaystyle\mathbb{E}\left[r(\hat{{\mathbf{x}}}^{\mathrm{boltz}}_{0})\right]$ $\displaystyle=\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}}_{0 })\right]$ (70) $\displaystyle\geq\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}} _{t_{T-1}})\right]-o(L\cdot t_{T-1})$ (71) $\displaystyle\geq\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}} _{t_{T-2}})\right]-o(L\cdot(t_{T-1}+(t_{T-2}-t_{T-1})))$ (72) $\displaystyle\vdots$ (73) $\displaystyle\geq\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}} _{t})\right]-o(L\cdot t)$ (74) $\displaystyle\geq r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}}_{t})-o(L\cdot t _{\mathrm{max}})$ (75) $$

###### Proof.

We list the premise as the following:

$$ $\displaystyle\mathbf{z}^{(k)}$ $\displaystyle=\omega^{(k)}_{t-\Delta}-\omega^{(k)}_{t}$ (76) $\displaystyle\hat{{\mathbf{x}}}_{t-\Delta}^{(k)}$ $\displaystyle=\mathrm{heun}({\mathbf{x}}_{t},\mathbf{z}^{(k)},t,\Delta)$ (77) $\displaystyle\hat{{\mathbf{x}}}_{t-\Delta}^{(k)}$ $\displaystyle={\mathbf{x}}_{t-\Delta}^{(k)}-o(\Delta)$ (78) $\displaystyle r_{\beta}(\mathbf{z}^{\mathrm{boltz}})$ $\displaystyle=\max\{r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(1)}),\cdots,r_{ \beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(K)})\}.$ (79) $$

We can deduce that:

$$ $\displaystyle\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}^{\mathrm{boltz}}_{t- \Delta})\right]$ $\displaystyle=\mathbb{E}\left[\max\{r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{( 1)}),\cdots,r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(K)})\}\right]$ (80) $\displaystyle\geq\mathbb{E}\left[r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta}^{(1)} )\right]$ (81) $\displaystyle=\mathbb{E}\left[r_{\beta}({\mathbf{x}}_{t-\Delta}^{(1)})-L\cdot o (\Delta)\right]$ (82) $\displaystyle=r_{\beta}({\mathbf{x}}_{t})-o(L\cdot\Delta)$ (83) $$

The last equation is followed by Equation 34. Here, $r_{\beta}(\hat{{\mathbf{x}}}_{t-\Delta})$ is the numerical estimation of the underlying SDE value $r_{\beta}({\mathbf{x}}_{t-\Delta})$.
∎

###### Lemma 4 .

When $\tau=\infty$ and the time step is small enough, the Boltzmann Demon sampling is identically distributed as the SDE sampling.

By adjusting $\tau$, we can smoothly transition from prioritizing high-reward noise samples to the standard SDE sampling method, balancing Demon and SDE strategies; note that when $\tau=\infty$, the weights are $b_{k}=\exp(0)=1$. Thus, $\sum_{k=1}^{K}b_{k}\mathbf{z}_{k}$ results in a Gaussian distribution $\mathcal{N}(\mathbf{0},K\mathbf{I}_{N})$. This distribution is identical distributed to drawing a Gaussian after both are projected onto a sphere of radius $\sqrt{N}$.

We justify replacing Gaussian sampling with uniform sampling from a sphere of radius $\sqrt{N}$ could result in the same effect of SDE during the Euler-Maruyama discretization of SDEs. Assuming constant drift ${\mathbf{f}}$ and diffusion $g$ for Euler-Maruyama step, the SDE is $d{\mathbf{x}}={\mathbf{f}}\,dt+g\,d\mathbf{W}$. We aim to demonstrate that this replacement yields an identical distribution under small step sizes. Define:

$$ $\mathbf{Y}_{n}=-{\mathbf{f}}\Delta+\sum_{k=1}^{n}g\sqrt{\frac{\Delta}{n}} \tilde{\mathbf{z}}_{k}=-{\mathbf{f}}\Delta+g\sqrt{\Delta}\frac{1}{\sqrt{n}} \sum_{k=1}^{n}\tilde{\mathbf{z}}_{k}$ (84) $$

where $\tilde{\mathbf{z}}_{k}$ are i.i.d. vectors uniformly sampled from the surface of a sphere with radius $\sqrt{N}$. Also, define:

$$ $\mathbf{Y}=-{\mathbf{f}}\Delta+g\sqrt{\Delta}\mathbf{z}$ (85) $$

###### Claim 4 .

$\mathbf{Y}_{n}$ converges to $\mathbf{Y}$ in distribution as $n\to\infty$.

###### Proof.

It’s sufficient to determine whether the normalized sum $\frac{1}{\sqrt{n}}\sum_{k=1}^{n}\tilde{\mathbf{z}}_{k}$ converges to $\mathbf{z}$ in distribution. Consider each $\tilde{\mathbf{z}}_{k}$ as an i.i.d. vector uniformly distributed on the surface of a sphere with radius $\sqrt{N}$. The expectation of each vector is $\mathbb{E}[\tilde{\mathbf{z}}_{k}]=\mathbf{0}$ by symmetry of the uniform distribution on the sphere.

We need to establish the covariance matrix of $\tilde{\mathbf{z}}_{k}$. Let $\tilde{\mathbf{z}}$ represent any of these i.i.d vectors without the subscript to generalize the analysis. We investigate the covariance between any two components $\tilde{\mathbf{z}}^{(i)}$ and $\tilde{\mathbf{z}}^{(j)}$ of the vector $\tilde{\mathbf{z}}$:

If $i\neq j$, by the symmetric condition of the sphere, we have:

$$ $\mathbb{E}\left[\tilde{\mathbf{z}}^{(i)}\tilde{\mathbf{z}}^{(j)}\right]= \mathbb{E}\left[\tilde{\mathbf{z}}^{(i)}\mathbb{E}\left[\tilde{\mathbf{z}}^{(j )}\mid\tilde{\mathbf{z}}^{(i)}\right]\right]=\mathbb{E}\left[\tilde{\mathbf{z} }^{(i)}\cdot 0\right]=0,$ (86) $$

reflecting the orthogonality of different components.

For the diagonal entries, where $i=j$, we know that the sum of the squares of the components of $\tilde{\mathbf{z}}$ equals the square of the radius of the sphere:

$$ $\sum_{i=1}^{N}\mathbb{E}\left[(\tilde{\mathbf{z}}^{(i)})^{2}\right]=\mathbb{E} \left[\sum_{i=1}^{N}(\tilde{\mathbf{z}}^{(i)})^{2}\right]=N,$ (87) $$

since $\|\tilde{\mathbf{z}}\|^{2}=N$. This implies that each component $\tilde{\mathbf{z}}^{(i)}$ has an expected squared value of 1, because all components contribute equally due to the symmetry of the sphere. Therefore, $\mathrm{Var}[\tilde{\mathbf{z}}^{(i)}]=\mathbb{E}\left[(\tilde{\mathbf{z}}^{(i
)})^{2}\right]=1$ for all $i$.

Combining these results, the covariance matrix $\mathrm{Var}(\tilde{\mathbf{z}})=\mathbf{I}_{N}$.
The Central Limit Theorem for vector-valued random variables (as described in Rencher ()) then asserts that $\frac{1}{\sqrt{n}}\sum_{k=1}^{n}\tilde{\mathbf{z}}_{k}$ converges in distribution to a Gaussian vector $\mathbf{z}$ with mean $\mathbf{0}$ and covariance matrix $\mathbf{I}_{N}$ as $n\to\infty$. Hence, the normalized sum approximates the Gaussian vector $\mathbf{z}$ used in the diffusion term of the original SDE, which justifies the replacement of Gaussian sampling with uniform sampling from the sphere.
∎

###### Lemma 3 .

###### Claim 3 .

###### Proof.

###### Lemma 4 .

###### Claim 4 .

###### Proof.

### D.5 High Dimensional Gaussian on Sphere

The original statement is more general in the textbook, but we provide specific proof for Gaussian.

###### Lemma 5 .

(Vershynin, , Chap. 3)
Let $\mathbf{z}$ be independent and identically distributed (i.i.d.) instances of a standard isotropic Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I}_{N})$ in a high-dimensional space $N$. With a high probability (e.g., 0.9999), it holds that

$$ $\displaystyle\lVert\mathbf{z}\rVert$ $\displaystyle=\sqrt{N}+\mathcal{O}(1)$ (88) $$

###### Proof.

Consider the norm $\lVert\mathbf{z}\rVert^{2}$, where $\mathbf{z}$ is an instance of a standard isotropic Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I}_{N})$ in $N$ dimensions. The distribution of $\lVert\mathbf{z}\rVert^{2}$ follows a Chi-squared distribution with $N$ degrees of freedom. The mean and variance of this distribution are $N$ and $2N$, respectively.

Applying a central limit theorem argument, we approximate the distribution of $\lVert\mathbf{z}\rVert^{2}$ by a normal distribution when $N$ is large, giving:

$$ $\lVert\mathbf{z}\rVert^{2}=N+C\sqrt{N}$ (89) $$

for some constant $C$, where $C\in\mathcal{O}(1)$ represents fluctuations around the mean which are typically on the order of the standard deviation of $\lVert\mathbf{z}\rVert^{2}$, which is $\sqrt{2N}$.

To connect this with the norm of $\mathbf{z}$, we consider:

$$ $\displaystyle\lim_{N\to\infty}\sqrt{N+C\sqrt{N}}-\sqrt{N}$ $\displaystyle=\lim_{N\to\infty}\sqrt{N}\left(\sqrt{1+\frac{C}{\sqrt{N}}}-1\right)$ (90) $\displaystyle=\lim_{N\to\infty}\sqrt{N}\left(\frac{C}{2\sqrt{N}}\right)$ (91) $\displaystyle=\frac{C}{2}$ (92) $$

Here, we use the Taylor series expansion for $\sqrt{1+x}$, approximated as $1+\frac{x}{2}$ for small $x$, to find the limit. This expansion leads to the conclusion that $\lVert\mathbf{z}\rVert=\sqrt{N}+\mathcal{O}(1)$.
∎

###### Lemma 5 .

###### Proof.

## Appendix E Compare on PickScore

### E.1 PickScore Comparisons.

Since PickScore Kirstain et al. () is trained specifically on generated images, we believe it is a more reliable measure and objective than the aesthetics score. To emphasize the strength of our method, we show how the median PickScore reward function improves across 20 different prompts using our Tanh Demon, as shown in Figure 6(a).

Our approach utilizes 1440 reward queries per sample and achieves a PickScore of 0.253, outperforming other methods alongside reduced computation time (180 minutes for our method vs. 240 minutes for resampling methods due to shortened ODE trajectories). Specifically, we compare our method to:

- •
SDXL/SDXL-DPO Wallace et al. (): A state-of-the-art method for direct preference optimization in diffusion models, which achieves a PickScore of 0.226, while the baseline SDXL reaches 0.222.
- •
Diffusion-DPO(1440x): A variant that selects the highest quality median PickScore from 1440 samples among 20 prompts, achieving a PickScore of 0.246.
- •
SDXL(1440x): Similar to the above, but without preference optimization, achieving a PickScore of 0.243.

Additionally, resampling an ODE from ${\mathbf{x}}_{t_{\mathrm{max}}}$ is crucial in applications where the distribution ${\mathbf{x}}_{t_{\mathrm{max}}}\mid{\mathbf{x}}_{0}$ plays a key role, such as in SDEdit Meng et al. (). Resampling methods fail to address such applications, highlighting the advantage of our approach.

Figure: (a) A Trajectory of Tanh Demon. We plot $(r\circ{\mathbf{c}})({\mathbf{x}}_{t})$ for different $t$.
Refer to caption: extracted/5907739/Images/pickscore/trajectory.jpg

### E.2 Qualitative Results

In this section, we demonstrate the quantitative and qualitative results of PickScore in SDXL with our Tanh Demon.

Figure: Figure 8: Each row in the figure presents two pairs of images where the image of each pair on the left illustrates results generated using the original PF-ODE method. The image on the right in each pair showcases enhancements achieved by applying our Tanh Demon based on the PickScore metric and SDXL. This figure demonstrates the improvements in visual fidelity and adherence to targeted characteristics achieved through our proposed method.
Refer to caption: extracted/5907739/Images/pickscore/preview.jpg

## Appendix F More Results with Various Reward Functions.

### F.1 More Image Generation Results with Different Reward Functions

Here, we show more image generaion results in SDXL with our Tanh Demon and other reward functions.

#### F.1.1 Generation on Stable Diffusion XL

Table: Table 11: Generative Results using SDXL

A Samurai Gardening on a Floating Island in the Sky
Insanely detailed portrait, wise man
A painting of a girl encountering a giant sunflower blocking her path in a hallway
A demon exiting through a portal
A butterfly flying above an ocean

Table: Table 12: Generative Results using SDXL (Cont)

Highway to hell
A Jazz Band of Different Alien Species Performing on an Exoplanet
A Victorian Inventor Testing Her Flying Bicycle Above a Steampunk City
A Time Traveler’s Picnic at the Edge of a Volcano During the Mesozoic Era
jedi duck holding a lightsaber

#### F.1.2 Generation on Stable Diffusion v1.4

Here, we provide more qualitative results as a continuation of Section 5.

Table: Table 13: More Qualitative Results of SD1.4 (Baseline)

Two-faced biomechanical cyborg
Highway to hell
jedi duck holding a lightsaber
A Samurai Gardening on a Floating Island in the Sky

Table: Table 14: More Qualitative Results of SD1.4 (cont)

A Time Traveler’s Picnic at the Edge of a Volcano During the Mesozoic Era
Insanely detailed portrait, wise man
A butterfly flying above an ocean
A Jazz Band of Different Alien Species Performing on an Exoplanet

### F.2 Subjective Test Overview

We surveyed with 101 participants via Google Forms, as shown in Figure 9. Participants evaluated different image generation methods based on:

- •
Subjective Preference: Visual aesthetics and image quality.
- •
Semantic Alignment: Correspondence between generated images and text prompts.

Each participant ranked images across four sections, with rankings aggregated using the following formula:

$$ $\frac{1}{ML}\sum_{i=1}^{M}\sum_{j=1}^{L}\exp(-(\text{rank}_{ij}-1))$ (93) $$

where:

- •
$M=4$ (number of sections),
- •
$L=101$ (participants),
- •
$\text{rank}_{ij}$ is the ranking by participant $j$ for method $i$.

Figure: (a) Comparison across methods.
Refer to caption: x9.png

#### F.2.1 Survey Structure

The subjective test comprised four sections: two comparing methods (DOODL, Baseline (SD or SDXL), Ensemble) based on subjective preference and prompt alignment, each with 3 sets containing one image per method; and two comparing methods applied to different objectives (Baseline, Ensemble, IR, Pick, HPSv2, Aes) also based on preference and prompt alignment, each with 3 sets containing six images per set.

#### F.2.2 Results Overview

##### Methods Comparison

Figure 8(a) shows that DOODL slightly outperforms the Baseline in aesthetic preference and prompt alignment. The Ensemble method significantly surpasses both, indicating superior visual quality and semantic accuracy.

##### Objectives Comparison

As seen in Figure 8(b), all objectives outperform the Baseline in prompt alignment, with the HPSv2 method leading. In subjective preference, methods applied to different objectives show varied improvements, with some achieving substantial gains over the Baseline.

#### F.2.3 Analysis

We compared DOODL, Baseline, and Ensemble based on aesthetics and prompt alignment. DOODL marginally improves over the Baseline in both criteria, while the Ensemble method consistently outperforms both DOODL and Baseline, excelling in image quality and semantic accuracy. The Ensemble method demonstrates significant enhancements, particularly in tasks requiring visual refinement.

Evaluating different objectives (IR, Pick, HPSv2, Aes) against Baseline and Ensemble revealed that almost all objectives surpass the Baseline in both preference and prompt alignment. However, Aes, an objective without explicit text guidance, shows weaker prompt alignment. Among the objectives, HPSv2 achieves the best performance on both criteria.

The Ensemble method provides the most substantial improvements in visual aesthetics and semantic alignment among method comparisons. Among the factors of the Ensemble method, HPSv2 outperforms other objectives, even the Ensemble method, highlighting its effectiveness in aligning preference for a real human.

## Appendix G More Details of VLM as Demon

In this section, we provide more details of experiments and quantitative results of utilizing VLM during generation.

### G.1 Experiments Settings

We provide the prompt template we used in Table 5 to VLMs. The prompt is fixed as “A mysterious, glowing object discovered in an unexpected place, sparking curiosity and wonder. The setting changes based on the viewer’s background, transforming the object’s significance and the surrounding environment to match the realms of education, history, literature, design, science, and imagination.” for all experiments in the VLM generation. At each step, the VLM is given a fixed prompt with different scenarios and asked to choose one of the images from $\mathbf{c}(\bm{x}_{t})$ and $\mathbf{c}(\hat{\bm{x}}_{t+\Delta}^{(k)})$ that best matches the given scenario.

The following are the full prompts for the scenarios:

- 1.
Teacher: You are a teacher looking to create custom illustrations for your educational materials to make learning more engaging for your students.
- 2.
Artist: You are a game or movie concept artist tasked with creating concept art for characters, settings, and scenes to speed up the pre-production process.
- 3.
Researcher: You are a researcher needing to visualize complex data, such as molecular structures in chemistry or weather patterns in meteorology, for better understanding or presentation.
- 4.
Journalist: You are a journalist who wants to add a visual teaser for your article to grab attention on social media or your news website.

Figure: Listing 1: Prompt template used in the Gemini image selection task. This template guides the decision-making process for choosing between two images generated from a fixed prompt. For the prompt used in GPT selection, we replace the output format by asking it to return JSON.

Figure: Listing 2: An example of GPT generated output

### G.2 Quantitative Measurement of Effectiveness

For VLMs as reward functions, we use Pickscore Kirstain et al. (), which is trained from CLIP Radford et al. (), to evaluate the effectiveness of VLM in aligning designed scenarios during image generation. For each scenario, we create a corresponding prompt that partially describes the scenario: “For education” for Teacher, “For entertainment” for Artist, “For research” for Researcher, and “For Journalism” for Journalist. Then, we assess the PickScore between the prompt and the scenario. The results are presented in Table 15, Table 16, Table 17 and Table 18, where the highest score for each prompt is highlighted in bold. Our observations indicate that 14 out of our VLM-generated 16 images demonstrate better PickScore alignment with the corresponding prompt than PF-ODE. Given that all images are generated using the same prompt and initial noisy sample in the same table, these results demonstrate the effectiveness of our approach employing VLM in aligning the scenarios.

**Table 15: GPT-SDXL generation, validated by PickScore on related prompts**
| Prompt | Teacher | Artist | Researcher | Journalist | PF-ODE |
| --- | --- | --- | --- | --- | --- |
| For education | 0.2050 | 0.2069 | 0.2080 | 0.2071 | 0.2073 |
| For entertainment | 0.2042 | 0.2073 | 0.2061 | 0.2058 | 0.2032 |
| For research | 0.1980 | 0.1989 | 0.1996 | 0.1985 | 0.1971 |
| For journalism | 0.1994 | 0.1957 | 0.1978 | 0.1946 | 0.1970 |

**Table 16: Gemini-SDXL generation, validated by PickScore on related prompts**
| Prompt | Teacher | Artist | Researcher | Journalist | PF-ODE |
| --- | --- | --- | --- | --- | --- |
| For education | 0.2111 | 0.2042 | 0.2102 | 0.2072 | 0.2073 |
| For entertainment | 0.2057 | 0.2058 | 0.2062 | 0.2013 | 0.2032 |
| For research | 0.2018 | 0.1979 | 0.2035 | 0.1986 | 0.1971 |
| For journalism | 0.2011 | 0.1991 | 0.1978 | 0.2049 | 0.1970 |

**Table 17: GPT-SD v1.4 generation, validated by PickScore on related prompts**
| Prompt | Teacher | Artist | Researcher | Journalist | PF-ODE |
| --- | --- | --- | --- | --- | --- |
| For education | 0.1978 | 0.2008 | 0.1996 | 0.1988 | 0.1941 |
| For entertainment | 0.2026 | 0.2018 | 0.1991 | 0.2004 | 0.1966 |
| For research | 0.1896 | 0.1936 | 0.1912 | 0.1935 | 0.1878 |
| For journalism | 0.1930 | 0.1951 | 0.1918 | 0.1942 | 0.1901 |

**Table 18: Gemini-SD v1.4 generation, validated by PickScore on related prompts**
| Prompt | Teacher | Artist | Researcher | Journalist | PF-ODE |
| --- | --- | --- | --- | --- | --- |
| For education | 0.1997 | 0.1945 | 0.1961 | 0.1961 | 0.1941 |
| For entertainment | 0.1954 | 0.1973 | 0.1982 | 0.1989 | 0.1966 |
| For research | 0.1910 | 0.1910 | 0.1897 | 0.1895 | 0.1878 |
| For journalism | 0.1935 | 0.1936 | 0.1927 | 0.1914 | 0.1901 |

## Appendix H General Implementation Details

In this section, we show the details of the implementation and experimental settings of the proposed approach as follows.

### H.1 Adapting Stable Diffusion to EDM Framework

In this paper, we tailor the existing text-to-image Stable Diffusion v1.4/v1.5/XL v1.0 (SDXL) (i.e., we use fp16 SD v1.4/SDXL v1.0 for generation.) to the SDE formulation proposed in EDM Karras et al. () by Karras et al. for image generation since its reparameterized time domain, $t\in[t_{\mathrm{min}},t_{\mathrm{max}}]$, improves numerical stability and sample quality during image generation. We realize the modification through the equation, $\nabla_{\bm{x}}\log p(\bm{x},t)=\left(D(\bm{x},t)-\bm{x}\right)/t^{2}$, where the function $D(\bm{x},t)=\bm{x}-t\mathbf{F}(s(t)\bm{x},u(t))$ derived from the original model $\mathbf{F}$. In addition, $s(t)$ and $u(t)$ represent the scaling schedule and the original temporal domain of the reparameterized temporal domain $t$, respectively.

### H.2 Numerical Methods for Image Generation

Moreover, for image sampling with ODE/SDE, our approach follows Karras et al. (), adopting Heun’s method and time intervals determined by $t_{i}=\left(t_{\mathrm{max}}^{1/\rho}+\frac{i-1}{T-1}(t_{\mathrm{min}}^{1/\rho
}-t_{\mathrm{max}}^{1/\rho})\right)^{\rho}$, setting $\rho=7,T\geq 20$ and $\ln t_{\mathrm{max}}\approx 2.7,\ln t_{\mathrm{min}}\approx-6.2$. The classifier-free guidance parameter is set to $2$ throughout this paper. Across all temporal steps $t$ of image generation, we keep $K$ and $\beta$ constant. We have found that when t is less than 0.11, i.i.d. samples from SDE all appear similar to human perception. For the remaining evaluations, we will directly use ODE. As a result, the actual number of samples will be slightly smaller than $K\cdot T$.

### H.3 Simplifications in Diffusion Process Modification

It is worth noting that in our work, since our main focus is on the modification of the diffusion process, without loss of generality, we omit the VAEs (Kingma & Welling ()) of Stable Diffusion models, the prompt $c$, and $\eta$ of classifier-free guidance (CFG) Ho & Salimans () in our formulation for simplicity (i.e., using $p(\bm{x})$ to denote the unnormalized $p(\bm{x})p(c\mid\bm{x})^{\eta}$ for conciseness).

### H.4 Batch Size and Memory Constraints

When we generate many SDE samples, the batch size for solving ODE/SDE is 8 for both Stable Diffusion v1.4, v1.5, and SDXL models. However, due to memory limitations on the RTX 3090, the batch size for evaluating the VAE in SDXL is restricted to 1. This memory bottleneck prevents any further acceleration from using larger batch sizes, as it limits the parallelization during VAE evaluation.

### H.5 Experimental Setup and Hyperparameters

We present the detailed hyperparameter settings of different experiments as follows:

##### Baseline Comparison.

The hyperparameters for generation are set to $\beta=0.1$, $K=16$, $\eta=2$ and $\tau$ adaptive for Tanh, $10^{-5}$ for Boltzmann.

The parameter of DOODL optimized on aesthetics score is set as their demo provided.

##### Reward Estimate Approximation Comparison.

We use SD v1.5 and its distilled CM. The CFG parameter is ignored in CM(set to $1$).

##### Generation with Various Reward Functions.

We use Tanh Demon for sampling with adaptive temperature. The hyperparameters for generation are set to $\beta=0.05$, $K=16$, $T=64$ as shown in Section 5, Section F.1.2, Section F.1.1, and Section F.1.1.

For reward scaling in the ensemble setting, the PickScore was multiplied by $98.86$, and HPSv2 was multiplied by $40$.

The interaction step of DOODL is used as suggested by their implementation, $50$ iteration for Aes and $100$ iteration for Pick.

##### Non-differentiable Reward.

In Table 5, the hyperparameters are set to $\beta=0.05$, $\tau=0.0001$, $K=16$, and $T=32$ using Tanh Demon.

##### Manual Selection.

In Figure 5, the parameters are $\beta=0.1,K=16,T=128$ but terminate manually, using Tanh Demon with adaptive temperature. We terminate the iteration after ten rounds of operating the UI.

## Appendix I Limitations

We present the theoretical result in  Equation 5, which demonstrates that $r\circ\mathbf{c}\approx r_{\beta}$. This result relies on the assumption that the reward function $r$ is near harmonic near the ODE sample ourput, as detailed in Section D.1.2.

In practice, implementing $r\circ\mathbf{c}$ faces challenges related to time complexity and accuracy bottlenecks, thoroughly discussed in Section 4.

## Appendix J Future Work

The only difference between Tanh-C and Tanh Demon lies in how $r\circ\mathbf{c}$ is implemented. Analysis of the data in Table 2 and Figure 4 indicates that Tanh-C’s reward performance can be enhanced by mitigating the RMSE in $r\circ\mathbf{c}$ without compromising Tanh-C’s speed performance. Potential strategies for improvement include increasing the fidelity of CM distillation or training a dedicated distilled model for $r\circ\mathbf{c}$. We propose these enhancements as future work.

## Appendix K Code of Ethics

The experiments involving human judgment are fully compliant with established ethical standards. Approval is obtained from the Institutional Review Board (IRB) of Academia Sinica under IRB number AS-IRB-HS 02-24031 to ensure that the research meets all necessary guidelines for the ethical treatment of human subjects.

## Appendix L Societal Impact

Our method has the potential to both discourage and encourage harmful content. Users can generate images through manual selections with malicious intentions (Figure 5). This increases accessibility but also raises concerns about misuse.
We implement safeguards provided by Stable Diffusion;
end-users are responsible for employing them, as recommended in prior works OpenAI (); Gemini Team Google (); Rombach et al. (); Podell et al. (), to mitigate potential risks.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - Given that the "noise" is now optimized over, is there a good intuitive way to think about it from a different perspective (diffusion, physics, or bayesian inference, etc.)? (other than the "noise" obviously corresponding to the latent vector)
- My intuition for the proposed sampling strategies improving over "best-of-N" sampling is that now you additionally get to optimize over the simplex spanned by the sampled noise samples (rather than just querying the "vertices" of the simplex). 
Could there be more intelligent ways to choose the "basis" vectors for the simplex other than random sample?
soundness: 3
strengths: - very clearly written and polished writing
- clear motivation
- compares to prior methods systematically, and shows consistent improvements
summary: This paper studies the problem of aligning diffusion models without additional training or backpropagation. The main idea is to optimize the sampling at inference time, by optimizing the noise used during generation based on a (possibly differentiable) reward signal. To compute the (expected) reward on a given noise, the method leverages a reduction from SDE to ODE dynamics.
weaknesses: - It's hard to gauge the significance/improvements in some of the results. For example, Appendix G.2 the numbers are all tightly concentrated around 0.2, so being unfamiliar with the metric, it's hard to know whether the differences are statistically significant or interesting.
- It would be nice to see more experiments with more realistic/extreme use cases of alignment. Most of the variations explored in the paper seem sort of minor. For example, make the model (not) generate NSFW material (or something else distributionally more obvious). Or is there a reason why such examples were note explored?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The paper primarily compares the results with DOODL in terms of computational cost and performance. I would be curious to see how this method compares with other methods mentioned in Table 1 and how it performs with different base models (e.g., SDXL).

Additionally, I would like to know where the performance and computational cost of CM without your method are positioned in Figure 4.
soundness: 2
strengths: Overall, this paper is thoroughly written, with comprehensive theoretical and empirical analysis. The implementation details are carefully listed, and various qualitative and quantitative metrics are displayed.
summary: The authors present a backpropagation-free, inference-time method for preference alignment in diffusion models. Specifically, the method synthesizes an optimal noise distribution via stochastic optimization thus enhancing the final reward of the generated image.
weaknesses: In the “Alignment with Preferences of VLMs” experiment, the role of the VLM doesn’t seem to exhibit a clear preference for a specific style, etc. I understand that the goal is to demonstrate that your method can optimize for a particular non-differentiable preference, such as a VLM API. However, based on the results presented, I’m not convinced that this experiment design effectively shows that your method truly optimizes for a specific preference.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - The authors used the Consistency Model [1] to accelerate computation. Could this model be applied to other diffusion models with different learned data distributions? A more general acceleration method would be beneficial.

[1] Song, Yang, et al. "Consistency models." arXiv preprint arXiv:2303.01469 (2023).
soundness: 3
strengths: 1. The concept of analyzing noise quality is interesting
2. The proposed method is a backpropagation-free preference alignment approach that operates during inference, allowing it to be applied flexibly across various conditions.
summary: This paper introduces Demon, an inference-time method for aligning diffusion models with user preferences without retraining or backpropagation. Using stochastic optimization, it guides denoising to focus on high-reward regions, even with non-differentiable rewards like VLM APIs and human judgments. Experiments across three diverse tasks demonstrate improved improvements.
weaknesses: 1. The comparison includes only a single baseline method, and quantitative results for BoN are missing in Table 3.
2. The evaluation is limited to a small subset of the dataset from DDPO [1]. Expanding the evaluation to include larger datasets, such as the full set from DDPO [1] and the Human Preference Dataset (HPDv2) [2], is recommended.
3. In Table 2, there remains a large performance gap between the 1-step CM and the 6-step ODE. It would be beneficial to consider additional state-of-the-art solvers of diffusion sampling for comparison.

[1] Black, Kevin, et al. "Training diffusion models with reinforcement learning." arXiv preprint arXiv:2305.13301 (2023).

[2] Wu, Xiaoshi, et al. "Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis." arXiv preprint arXiv:2306.09341 (2023).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the above weakness part.
soundness: 3
strengths: + The authors first develop a method to align the generation of diffusion models without training the model or computing the gradient of the reward model.
+ They conducted experiments on various reward models and preference types.
+ The paper is well-organized and easy to follow.
summary: In this study, the authors develop a method to align the outputs of diffusion models with human preference at inference time. Specifically, the authors address the alignment from the perspective of noise generation. They first propose to measure the quality of noises in each denoising step. Then, they combined different noises to search for an ''optimal'' noise towards a high reward of outputs. Experimental results show that the proposed method effectively improves the aesthetic score of generated images.
weaknesses: - Is the proof in this study based on the linear assumption of the reward model? Given that the human preference for images is extremely complicated and certainly nonlinear, does it apply to the scenario of complex human preference labels?
- The noise is flipped in Figure 3, but this operation is neither included in Eq. (11) nor discussed in the main text. Is it a standard operation in your method? On the other hand, I'm not at all convinced that flipping it can ``transform it into high-reward noise’’ because the reward space is not linear and symmetric.
- The comparison between the proposed method and best-of-N is interesting. However, the experiment settings of best-of-N is unclear. I’m not sure whether they selected the final generation with the highest reward or selected the noise at each step. Second, it would be better if the authors could discuss why is the proposed method better than best-of-N based on their theoretical analysis.
- The baseline methods for comparison in experiments are not sufficient. First, they should compare the method with a baseline which always selects the best noise $z^{(k)}$ with the highest reward $(r\odot c)(\cdot)$ at each denoising step. Second, I expect a comparison between the proposed method with more methods based on gradient guidance (Bansal et al. 2024).
- The quantitative results in Section 5 are based on 22 simple prompts of animals, which are not convincing enough.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Please see the Weaknesses.
soundness: 3
strengths: 1.	This paper presents an interesting approach that eliminates the need for model retraining or backpropagation, which is an advancement in the field of diffusion models and user preference alignment.
2.	The proposed method can be easily integrated with existing diffusion models without further training, making it accessible for immediate application.
summary: This paper introduces an approach for aligning diffusion models with user preferences at inference time without needing model retraining or backpropagation. The method, named Demon, leverages stochastic optimization to control the noise distribution during the denoising steps of diffusion models, steering the generation process towards user-specified rewards. This paper provides theoretical backing and empirical evidence, including experiments with non-differentiable reward sources such as Visual-Language Model (VLM) APIs and human judgments.
weaknesses: 1.	Computational complexity of the proposed method needs to be discussed. While the authors claim ease of integration, the computational complexity of implementing the stochastic optimization approach might be high. I suggest the authors provide the required inference time and GPU memory in Table 3.
2.	Many strong baseline methods have been neglected. I suggest the authors compare with DDPO, DPO, and DPOK in Tables 3 and 4.
3.	The effectiveness of the Demon may be heavily dependent on the quality and fidelity of the reward signal, which may not always be reliable or accurate. Therefore, I suggest the authors conduct experiments to quantitatively compare the results between the human reward signal and a pre-trained reward model.
4.	The writing quality could be further improved. The overall intuition of the proposed method is easy to understand, but the description in Section 4.2 is a bit redundant and complex.

Meta-review: This paper proposes a backpropagation-free, inference-time method for preference alignment in diffusion models. Specifically, the authors optimize for the noise distribution to search for an ''optimal'' noise that yields high rewards. Since this method is training-free, it is quite interesting and can be applied to several off the shelf models.

Decision: Accept (Poster)