Rectified Diffusion Guidance for Conditional Generation

## Abstract

Abstract Classifier-Free Guidance (CFG), which combines the conditional and unconditional score functions with two coefficients summing to one, serves as a practical technique for diffusion model sampling.
Theoretically, however, denoising with CFG cannot be expressed as a reciprocal diffusion process, which may consequently leave some hidden risks during use.
In this work, we revisit the theory behind CFG and rigorously confirm that the improper configuration of the combination coefficients ( i.e. , the widely used summing-to-one version) brings about expectation shift of the generative distribution.
To rectify this issue, we propose ReCFG 1 1 1 ReCFG, pronounced as “reconfigure”, is the abbreviation for “rectified Classifier-Free Guidance”. with a relaxation on the guidance coefficients such that denoising with ReCFG strictly aligns with the diffusion theory.
We further show that our approach enjoys a closed-form solution given the guidance strength.
That way, the rectified coefficients can be readily pre-computed via traversing the observed data, leaving the sampling speed barely affected.
Empirical evidence on real-world data demonstrate the compatibility of our post-hoc design with existing state-of-the-art diffusion models, including both class-conditioned ones ( e.g. , EDM2 on ImageNet) and text-conditioned ones ( e.g. , SD3 on CC12M), without any retraining.
We will open-source the code to facilitate further research.

## 1 Introduction

Diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ), known simply as diffusion models, have achieved unprecedented capability improvement of high-resolution image generation.
It is well recognized that, DPMs are the most prominent generative paradigm for a broad distribution (i.e., text-to-image generation) (Podell et al., ; Chen et al., ; Esser et al., ).
Among DPM literature, Classifier-Free Guidance (CFG) (Ho & Salimans, ) serves as an essential factor, enabling better conditional sampling.
Vanilla conditional sampling via DPMs introduces the conditional score function $s_{t}(\mathbf{x},c)=\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)$, resulting in poor performance in which synthesized samples appear to be visually incoherent and not faithful to the condition, even for large-scale models (Rombach et al., ).
By drawing lessons from Bayesian theory, CFG employs an interpolation between conditional and unconditional score functions with a preset weight $\gamma$, i.e.,

$$ $\displaystyle s_{t,\gamma}(\mathbf{x},c)=\gamma\nabla_{\mathbf{x}_{t}}\log q_{ t}(\mathbf{x}_{t}|c)+(1-\gamma)\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t }),$ (1) $$

in which $\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t})$ is the unconditional score function by annihilating the condition effect.
By doing so, DPMs turn out to formulate the underlying distribution with a gamma-powered distribution (Bradley & Nakkiran, ), i.e.,

$$ $\displaystyle q_{t,\gamma}(\mathbf{x}|c)=q_{t}(\mathbf{x}|c)^{\gamma}q_{t}( \mathbf{x})^{1-\gamma},$ (2) $$

which is proportional to $q_{t}(\mathbf{x})q_{t}(c|\mathbf{x})^{\gamma}$.
Enlarging $\gamma>1$ focuses more on the classifier effect $q_{t}(c|\mathbf{x})$, concentrating on better exemplars of given condition and thereby sharpening the gamma-powered distribution.
In other words, CFG is designed to promote the influence of the condition.

However, inspired by seminal works (Bradley & Nakkiran, ), we argue theoretically that denoising with CFG cannot be expressed as a reciprocal of vanilla diffusion process by adding Gaussian noises, since the expectation of score function of gamma-powered $q_{t,\gamma}(\mathbf{x}|c)$ is normally nonzero, violating the underlying theory of DPMs.
Theoretically, score functions with zero expectation at all timesteps guarantee that the denoised $\tilde{\mathbf{x}}_{0}$ has expectation $\mathbb{E}[\tilde{\mathbf{x}}_{0}]=\frac{\alpha_{0}}{\alpha_{T}}\mathbb{E}[
\mathbf{x}_{T}]$, thus $\mathbb{E}[\tilde{\mathbf{x}}_{0}]=\mathbb{E}[\mathbf{x}_{0}]$ and no bias on the conditional fidelity.
Therefore, this theoretical flaw leaves some hidden risks during use, manifesting as a severe expectation shift phenomenon, i.e., the expectation of the gamma-powered distribution will be shifted away from the ground-truth of the conditional distribution $q_{t}(\mathbf{x}|c)$.
This is more conspicuous when applying larger $\gamma$.
Fig. 2 clearly clarifies the expectation shift, in which the peak of induced distribution via CFG in red fails to coincide with that of ground-truth $q_{0}(\mathbf{x}_{0}|c)$.
This theoretical flaw is known in theory (Du et al., ; Karras et al., ; Bradley & Nakkiran, ), while being largely ignored in practice.

Figure: Figure 1: Visualization of the lookup table on LDM (Rombach et al., ), EDM2 (Karras et al., ), and SD3 (Esser et al., ), each of which consists of the expectation ratio $\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)]/ \mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]$. Each pixel represents the scale of the pixel-wise ratio, i.e., color red implies that ratio is greater than one, while color blue stands for ratio smaller than one. The darker the color is, the farther the ratio appears away from one. The two images in one cell report the ratio of the first and the last denoising step under different NFEs.
Refer to caption: x1.png

In this work, we first revisit the formulation of native CFG, theoretically confirming its flaw that we concluded above and summarizing as Theorem 1.
Then, to quantitatively reveal the consequent expectation shift phenomenon by CFG, we employ a toy distribution, enjoying closed-form description of the behavior on the gamma-powered distribution.
Under the toy settings, we analytically calculate the function of the precise value of expectation shift in correspondence with $\gamma$, as summarized in Theorem 2.
Motivated by theoretical compatibility and canceling the expectation shift, we apply relaxation on the guidance coefficients in native CFG by circumventing the constraint that two coefficients sum to one, enabling a more flexible control on the induced distributions.
To be more concrete, we propose to formulate the underlying distribution with two coefficients, i.e.,

$$ $\displaystyle q_{t,\gamma_{1},\gamma_{0}}(\mathbf{x}|c)=q_{t}(\mathbf{x}|c)^{ \gamma_{1}}q_{t}(\mathbf{x})^{\gamma_{0}}.$ (3) $$

Aiming at consistency with the diffusion theory and thus better guidance efficacy, we specially design the constraints on $\gamma_{1}$ and $\gamma_{0}$, and theoretically confirm the feasibility.
We further provide a closed-form solution to the constraints, and propose an algorithm to analytically determine $\gamma_{0}$ from a pre-computed lookup table in a post-hoc fashion.
Thanks to the neat formulation, we can employ pixel-wise $\gamma_{0}$ according to the lookup table involving guidance strength $\gamma_{1}$, condition $c$ and timestep $t$, as demonstrated in Fig. 1.
We name the above process ReCFG.
Experiments with state-of-the-art DPMs, including both class-conditioned ones (e.g., EDM2 (Karras et al., )) and text-conditioned ones (e.g., SD3 (Esser et al., )) under different NFEs and guidance strengths show that our ReCFG can achieve better guidance efficacy without retraining or extra time cost during inference stage.
Hence, our work offers a new perspective on guided sampling of DPMs, encouraging more studies in the field of guided generation.

## 2 Related Work

DPMs and conditional generation.
Diffusion probabilistic model (DPM) introduces a new scheme of generative modeling, formulated by forward diffusing and reverse denoising processes (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ).
It is trained by optimizing the variational lower bound.
Benefiting from this breakthrough, DPM achieves high generation fidelity, and even beat GANs on image generation.
Conditional generation (Choi et al., ; Huang et al., ) takes better advantage of intrinsic intricate knowledge of data distribution, making DPM easier to scale up and the most promising option for generative modeling.
Among the literature, text-to-image generation injects the embedding of text prompts to DPM, faithfully demonstrating the text content (Podell et al., ; Chen et al., ; Esser et al., ).

Classifier-Free Guidance.
Classifier-Free Guidance (CFG) serves as the successor of Classifier Guidance (CG) (Dhariwal & Nichol, ), circumventing the usage of a classifier for noisy images.
Both CFG and CG attempt to formulate the underlying distribution by concentrating more on condition influence, achieving better conditional fidelity.
Despite great success in large-scale conditional generation, CFG faces a technical flaw that the guided distribution is not theoretically guaranteed to recover the ground-truth conditional distribution (Du et al., ; Karras et al., ; Bradley & Nakkiran, ).
There exists a shifting issue that the expectation of guided distribution is drifted away from the correct one.
This phenomenon may harm the condition faithfulness, especially for extremely broad distribution (e.g., open-vocabulary synthesis).

## 3 Method

### 3.1 Background on conditional DPMs and CFG

Let $\mathbf{x}_{0}\in\mathbb{R}^{D}$ be a $D$-dimensional random variable with an unknown distribution $q_{0}(\mathbf{x}_{0}|c)$, where $c\sim q(c)$ is the given condition.
DPM (Sohl-Dickstein et al., ; Song et al., ; Ho et al., ) introduces a forward process $\{\mathbf{x}_{t}\}_{t\in(0,T]}$ by gradually corrupting data signal of $\mathbf{x}_{0}$ with Gaussian noise, i.e., the following transition distribution holds for any $t\in(0,T]$:

$$ $\displaystyle q_{0t}(\mathbf{x}_{t}|\mathbf{x}_{0},c)=q_{0t}(\mathbf{x}_{t}| \mathbf{x}_{0})=\mathcal{N}(\alpha_{t}\mathbf{x}_{0},\sigma_{t}^{2}\mathbf{I}),$ (4) $$

in which $\alpha_{t},\sigma_{t}\in\mathbb{R}^{+}$ are differentiable functions of $t$ with bounded derivatives, referred to as the noise schedule.
Let $q_{t}(\mathbf{x}_{t}|c)$ be the marginal distribution of $\mathbf{x}_{t}$ conditioned on $c$, DPM ensures that $q_{T}(\mathbf{x}_{T}|c)\approx\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$ for some $\sigma>0$, and the signal-to-noise-ratio (SNR) $\alpha_{t}^{2}/\sigma_{t}^{2}$ is strictly decreasing with respect to timestep $t$ (Kingma et al., ).

Seminal works (Kingma et al., ; Song et al., ) studied the underlying stochastic differential equation (SDE) and ordinary differential equation (ODE) theory of DPM.
The forward and reverse processes are as below for any $t\in[0,T]$:

$$ $\displaystyle\mathrm{d}\mathbf{x}_{t}$ $\displaystyle=f(t)\mathbf{x}_{t}\mathrm{d}t+g(t)\mathrm{d}\mathbf{w}_{t},\quad \mathbf{x}_{0}\sim q_{0}(\mathbf{x}_{0}|c),$ (5) $\displaystyle\mathrm{d}\mathbf{x}_{t}$ $\displaystyle=[f(t)\mathbf{x}_{t}-g^{2}(t)\nabla_{\mathbf{x}_{t}}\log q_{t}( \mathbf{x}_{t}|c)]\mathrm{d}t+g(t)\mathrm{d}\bar{\mathbf{w}}_{t},$ (6) $$

where $\mathbf{w}_{t},\bar{\mathbf{w}}_{t}$ are standard Wiener processes in forward and reverse time, respectively, and $f,g$ have closed-form expressions with respect to $\alpha_{t},\sigma_{t}$.
The unknown $\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)$ is referred to as the conditional score function.
Probability flow ODE (PF-ODE) from Fokker-Planck equation enjoys the identical marginal distribution at each $t$ as that of the SDE in Eq. 6, i.e.,

$$ $\displaystyle\frac{\mathrm{d}\mathbf{x}_{t}}{\mathrm{d}t}=f(t)\mathbf{x}_{t}- \frac{1}{2}g^{2}(t)\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c).$ (7) $$

Technically, DPM implements sampling by solving the reverse SDE or ODE from $T$ to 0.
To this end, it introduces a neural network $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)$, namely the noise prediction model, to approximate the conditional score function from the given $\mathbf{x}_{t}$ and $c$ at timestep $t$, i.e., $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)=-\sigma_{t}\nabla_{\mathbf{x}_{t}}
\log q_{t}(\mathbf{x}_{t}|c)$, where the parameter $\theta$ can be optimized by the objective below:

$$ $\displaystyle\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon},c,t}[\omega_{t}\|\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},c,t)-\bm{\epsilon}\|_{2}^{2}],$ (8) $$

where $\omega_{t}$ is the weighting function, $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, $c\sim q(c)$, $\mathbf{x}_{t}=\alpha_{t}\mathbf{x}_{0}+\sigma_{t}\bm{\epsilon}$, and $t\sim\mathcal{U}[0,T]$.

For better condition fidelity, during denoising stage, CFG (Ho & Salimans, ) turns to use a linear interpolation between conditional and unconditional score functions, i.e.,

$$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma}(\mathbf{x}_{t}|c)=\gamma \nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)+(1-\gamma)\nabla_{\mathbf{ x}_{t}}\log q_{t}(\mathbf{x}_{t}).$ (9) $$

Then PF-ODE can be rewrote as

$$ $\displaystyle\frac{\mathrm{d}\mathbf{x}_{t}}{\mathrm{d}t}=f(t)\mathbf{x}_{t}- \frac{1}{2}g^{2}(t)\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma}(\mathbf{x}_{t}|c).$ (10) $$

We further describe the CFG under the original DDIM theory.
Recall that DDIM turns out to formulate non-Markovian forward diffusing process such that the reverse denoising process obeys the distribution with parameters $\{\delta_{t}\}_{t}$ (Song et al., ):

$$ $\displaystyle q_{\delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)=q_{ \delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})\sim\mathcal{N}\left( \alpha_{t-1}\mathbf{x}_{0}+\sqrt{\sigma_{t-1}^{2}-\delta_{t}^{2}}\cdot\frac{ \mathbf{x}_{t}-\alpha_{t}\mathbf{x}_{0}}{\sigma_{t}},\delta_{t}^{2}\mathbf{I} \right).$ (11) $$

Trainable generative process $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},c)$ is designed to leverage $q_{\delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)$ with a further designed denoised observation $\mathbf{f}_{\theta}^{t}$ with noise prediction model $\bm{\epsilon}_{\theta}$, i.e.,

$$ $\displaystyle\mathbf{f}_{\theta}^{t}(\mathbf{x}_{t},c)$ $\displaystyle=\frac{1}{\alpha_{t}}(\mathbf{x}_{t}-\sigma_{t}\bm{\epsilon}_{ \theta}(\mathbf{x}_{t},c,t)),$ (12) $\displaystyle p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},c)$ $\displaystyle=\begin{cases}{q_{\delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf {f}_{\theta}^{t}(\mathbf{x}_{t},c),c)},&t>1,\[5.0pt] {\mathcal{N}(\mathbf{f}_{\theta}^{t}(\mathbf{x}_{1}),\sigma_{1}^{2}\mathbf{I}) },&t=1.\end{cases}$ (13) $$

DDIM proves that for any $\{\delta_{t}\}_{t}$, score matching of non-Markovian process above is equivalent to native DPM up to a constant.
Under CFG setting with weight $\gamma$, we generalize the theory as below:

$$ $\displaystyle\mathbf{f}_{\theta,\gamma}^{t}(\mathbf{x}_{t},c)$ $\displaystyle=\frac{1}{\alpha_{t}}(\mathbf{x}_{t}-\sigma_{t}(\gamma\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},c,t)+(1-\gamma)\bm{\epsilon}_{\theta}( \mathbf{x}_{t},t))),$ (14) $\displaystyle p_{\theta,\gamma}(\mathbf{x}_{t-1}|\mathbf{x}_{t},c)$ $\displaystyle=\begin{cases}{q_{\delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf {f}_{\theta,\gamma}^{t}(\mathbf{x}_{t},c),c)},&t>1,\[5.0pt] {\mathcal{N}(\mathbf{f}_{\theta,\gamma}^{t}(\mathbf{x}_{1},c),\sigma_{1}^{2} \mathbf{I})},&t=1.\end{cases}$ (15) $$

Native DDIM theory still holds since $q_{\delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)=q_{\delta}(
\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})$, i.e., with the definition

$$ $\displaystyle J_{\delta,\gamma}(\bm{\epsilon}_{\theta})=\mathbb{E}_{q_{\delta} (\mathbf{x}_{0:T}|c)}[\log q_{\delta}(\mathbf{x}_{1:T}|\mathbf{x}_{0},c)-\log p _{\theta,\gamma}(\mathbf{x}_{0:T}|c)],$ (16) $$

we have the following theorem.
Proof is in Section A.1.

###### Theorem 1 .

For any $\{\delta_{t}\}_{t}$ and $\gamma>1$, $J_{\delta,\gamma}$ is equivalent to native DPM under CFG up to a constant.
However, denoising with CFG is not a reciprocal of the original diffusion process with Gaussian noise due to nonzero expectation of unconditional score function $\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf
{x}_{t})]$.

###### Remark 1 .

$\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)$ and $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ are proportional to $\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)$ and $\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t})$ with coefficients being each minus standard deviation respectively, and empirically we use the same fixed variance for both $q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)$ and $q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})$.
Therefore, Theorem 1 is consistent with the original CFG using score functions in Eq. 9.

Figure: Figure 2: Visualization of expectation shift. The demonstrated toy data is simulated by $q_{0}(\mathbf{x}_{0}|c)\sim\mathcal{N}(c,1)$, $q(c)\sim\mathcal{N}(0,1)$, $q_{0}(\mathbf{x}_{0})\sim\mathcal{N}(0,2)$. Gamma-powered distribution $q_{0,\gamma}(\mathbf{x}_{0}|c)$ from CFG (Ho & Salimans, ) fails to recover the same conditional expectation as ground-truth due to expectation shift (i.e., probability density function and histogram by DDIM (Song et al., ) sampler in red). To make a further step, larger $\gamma$ suggests more severe expectation shift, i.e., the peak of $q_{0,\gamma}(\mathbf{x}_{0}|c)$ tends further away from $q_{0}(\mathbf{x}_{0}|c)$ (i.e., probability density function in blue) as $\gamma$ goes from $1.5$ to $2.5$. As a comparison, our ReCFG successfully recovers the ground-truth expectation and smaller variance (i.e., probability density function and histogram by DDIM (Song et al., ) sampler in green), consistent with the motivation of guided sampling.
Refer to caption: x2.png

###### Theorem 1 .

###### Remark 1 .

### 3.2 Misconceptions about CFG on Expectation Shift

CFG is designed to concentrate on better exemplars for each denoising step by sharpening the gamma-powered distribution as below (Bradley & Nakkiran, )

$$ $\displaystyle q_{t,\gamma}(\mathbf{x}|c)=q_{t}(\mathbf{x}|c)^{\gamma}q_{t}( \mathbf{x})^{1-\gamma}.$ (17) $$

We first generalize the counterexample in Bradley & Nakkiran () to confirm the expectation shift phenomenon.
For VE-SDE with deterministic sampling recipe, we consider the 1-dimensional distribution with $q_{0}(\mathbf{x}_{0}|c)\sim\mathcal{N}(c,1)$, $q(c)\sim\mathcal{N}(0,1)$, $q_{0}(\mathbf{x}_{0})\sim\mathcal{N}(0,2)$.
Then we can formulate the forward process and score functions as below:

$$ $\displaystyle q_{t}(\mathbf{x}_{t}|c)$ $\displaystyle\sim\mathcal{N}(c,1+t),$ (18) $\displaystyle q_{t}(\mathbf{x}_{t})$ $\displaystyle\sim\mathcal{N}(0,2+t),$ (19) $\displaystyle\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)$ $\displaystyle=-\frac{\mathbf{x}_{t}-c}{1+t},$ (20) $\displaystyle\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t})$ $\displaystyle=-\frac{\mathbf{x}_{t}}{2+t}.$ (21) $$

We state the theorem below describing the expectation shift.
Proof is addressed in Section A.2.

###### Theorem 2 .

Denote by $q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ the conditional distribution by solving PF-ODE in Eq. 10 with CFG weight $\gamma>1$.
Then $q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ follows the closed-form expression as below.

$$ $\displaystyle q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)\sim\mathcal{N} \left(c\phi(\gamma,T),2^{1-\gamma}\frac{T+1}{(T+1)^{\gamma}(T+2)^{1-\gamma}} \right),$ (22) $$

in which

$$ $\displaystyle\phi(\gamma,T)=2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{ \gamma}{2}}(T+2)^{\frac{1-\gamma}{2}}}+\frac{\gamma}{2}\int_{0}^{T}(s+1)^{- \frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s\right).$ (23) $$

Specifically, when $T\rightarrow+\infty$, denote by $\phi(\gamma)$ with

$$ $\displaystyle\phi(\gamma)=\lim_{T\rightarrow+\infty}\phi(\gamma,T),$ (24) $$

we have $\phi(\gamma)\geqslant\gamma\frac{7}{15}\left(\frac{10}{7}\right)^{\frac{5-
\gamma}{2}}$ for $\gamma\in[1,3]$, $\phi(1)=1$, $\phi(3)=2$, $\phi(\gamma)\geqslant 2$ for all $\gamma>3$, and

$$ $\displaystyle q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)\sim\mathcal{N}(c \phi(\gamma),2^{1-\gamma}).$ (25) $$

Furthermore, we have closed-form expression for $\phi(\gamma)$ when $\gamma\in\mathbb{N}$ and $\gamma>1$, i.e.,

$$ $\displaystyle\phi(\gamma)=\begin{cases}{2^{-n}\left(\sum_{k=0}^{n}C_{n}^{k} \frac{2n+1}{2n-2k+1}\right)},&\gamma=2n+1,\[20.0pt] {2^{-\frac{1}{2}}\left(\sqrt{2}-\frac{1}{2}\log\frac{\sqrt{2}-1}{\sqrt{2}+1} \right)},&\gamma=2,\[20.0pt] {\frac{(2n-1)!!\sqrt{2}n}{(2n)!!2^{n}}\left(\left(\sum_{k=2}^{n}\frac{1}{k} \frac{(2k)!!2^{k-\frac{1}{2}}}{(2k-1)!!}\right)+2\sqrt{2}-\log\frac{\sqrt{2}-1 }{\sqrt{2}+1}\right)},&\gamma=2n\geqslant 4.\end{cases}$ (26) $$

However, note that the ground-truth conditional distribution $q_{0}(\mathbf{x}_{0}|c)\sim\mathcal{N}(c,1)$, indicating that the ground-truth expectation is equal to $c$.
That is to say, denoising with CFG achieves at least twice as large expectation as the ground-truth one.
Fig. 2 clearly describes the phenomenon.

###### Theorem 2 .

### 3.3 Rectified Classifier-Free Guidance

Recall that the constraint of the two coefficients with summation one disables the compatibility with diffusion theory and indicates expectation shift.
Theorem 2 quantitatively describes the expectation shift, claiming that the two coefficients of conditional and unconditional score functions in Eq. 9 dominate both the expectation and variance of $q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$.
To this end, we propose to rectify CFG with relaxation on the guidance coefficients, i.e.,

$$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma_{1},\gamma_{0}}(\mathbf{x }_{t}|c)=\gamma_{1}\otimes\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)+ \gamma_{0}\otimes\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}),$ (27) $$

in which $\gamma_{1}$, $\gamma_{0}\in\mathbb{R}^{D}$ are functions with respect to condition $c$ and timestep $t$, and $\otimes$ indicates element-wise product.
Denote by $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ the attached conditional distribution following PF-ODE in Eq. 10 with $\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma_{1},\gamma_{0}}(\mathbf{x}_{t}|c)$.

To make guided sampling compatible with the diffusion theory and annihilate expectation shift, it suffices to choose more appropriate $\gamma_{1}$ and $\gamma_{0}$ according to input condition $c$ and timestep $t$.
Intuitively, we need the constraint such that:

- •
Each component $\gamma_{1,i}>1$ for strengthened conditional fidelity,
- •
Denoising with PF-ODE and Eq. 27 is theoretically the reciprocal of forward process, thus $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ enjoys the same expectation as ground-truth $q_{0}(\mathbf{x}_{0}|c)$,
- •
$q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ enjoys smaller or the same variance as ground-truth $q_{0}(\mathbf{x}_{0}|c)$ for sharper distribution and better exemplars.

In the sequel, we omit $\otimes$ for simplicity.
We first focus on the compatibility with the diffusion theory.
We have claimed in Theorem 1 that CFG cannot satisfy the diffusion theory due to nonzero $\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf
{x}_{t})]$.
To this end, it suffices to annihilate the expectation shift as below:

$$ $\displaystyle\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\nabla_{\mathbf{x}_{t}}\log q _{t,\gamma_{1},\gamma_{0}}(\mathbf{x}_{t}|c)]=-\frac{1}{\sigma_{t}}\mathbb{E}_ {q_{t}(\mathbf{x}_{t}|c)}[\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t) +\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]=0.$ (28) $$

To confirm the feasibility and precisely describe the expectation of $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$, resembling Eqs. 14 and 15 we can write:

$$ $\displaystyle\mathbf{f}_{\theta,\gamma_{1},\gamma_{0}}^{t}(\mathbf{x}_{t},c)$ $\displaystyle=\frac{1}{\alpha_{t}}(\mathbf{x}_{t}-\sigma_{t}(\gamma_{1}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},t))),$ (29) $\displaystyle p_{\theta,\gamma_{1},\gamma_{0}}(\mathbf{x}_{t-1}|\mathbf{x}_{t} ,c)$ $\displaystyle=\begin{cases}{q_{\delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf {f}_{\theta,\gamma_{1},\gamma_{0}}^{t}(\mathbf{x}_{t},c),c)},&t>1,\[5.0pt] {\mathcal{N}(\mathbf{f}_{\theta,\gamma_{1},\gamma_{0}}^{t}(\mathbf{x}_{1},c), \sigma_{1}^{2}\mathbf{I})},&t=1.\end{cases}$ (30) $$

And we have the following theorem, in which proof is addressed in Section A.3.

###### Theorem 3 .

Let $\mathbf{x}_{t}\sim q_{t}(\mathbf{x}_{t}|c)$, $\tilde{\mathbf{x}}_{t}\sim p_{\theta,\gamma_{1},\gamma_{0}}(\tilde{\mathbf{x}}
_{t}|c)$ induced from DDIM sampler in Eq. 30.
Assume that all $\delta_{t}=0$, denote by $\Delta_{t}$ the difference between expectation of $\mathbf{x}_{t}$ and $\tilde{\mathbf{x}}_{t}$, i.e.,

$$ $\displaystyle\Delta_{t}=\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\mathbf{x}_{t}]- \mathbb{E}_{p_{\theta,\gamma_{1},\gamma_{0}}(\tilde{\mathbf{x}}_{t}|c)}[\tilde {\mathbf{x}}_{t}].$ (31) $$

Then we have the following recursive equality:

$$ $\displaystyle\Delta_{t-1}=\frac{\sigma_{t-1}}{\sigma_{t}}\Delta_{t}-(\sigma_{t -1}-\frac{\alpha_{t-1}}{\alpha_{t}}\sigma_{t})\mathbb{E}_{\tilde{\mathbf{x}}_{ t}}[(\gamma_{1}-1)\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},c,t)+\gamma_{0 }\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},t)]).$ (32) $$

Specifically, when $\Delta_{t}=0$, we have:

$$ $\displaystyle\Delta_{t-1}$ $\displaystyle=-(\sigma_{t-1}-\frac{\alpha_{t-1}}{\alpha_{t}}\sigma_{t})\mathbb {E}_{\mathbf{x}_{t}}[(\gamma_{1}-1)\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+ \gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]).$ (33) $$

Theorem 3 studies the difference between expectation of denoising with Eq. 27 and the ground-truth.
Note that $\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)]=
\mathbb{E}_{\mathbf{x}_{t}}[\mathbb{E}_{q_{t}(\bm{\epsilon}|\mathbf{x}_{t})}[
\bm{\epsilon}|\mathbf{x}_{t}]]=\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}]=0$, we have

$$ $\displaystyle\mathbb{E}_{\mathbf{x}_{t}}[(\gamma_{1}-1)\bm{\epsilon}_{\theta}( \mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]= \mathbb{E}_{\mathbf{x}_{t}}[\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c, t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)],$ (34) $$

which coincides with Eq. 28, indicating the feasibility and a closed-form solution given $c$ and $t$.

As for variance, however, normally we cannot calculate the variance of $p_{\theta,\gamma_{1},\gamma_{0}}(\mathbf{x}_{t}|c)$.
Instead, we study the variance in Sec. 3.2 as an empirical evidence, where proof is in Section A.4.

###### Theorem 4 .

Under settings in Theorem 2, denote by $q_{0,\gamma_{1},\gamma_{0}}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ the conditional distribution by PF-ODE and deterministic sampling with $\gamma_{1}$ and $\gamma_{0}$ as in Eq. 27. Then we have

$$ $\displaystyle\mathrm{var}_{q_{0,\gamma_{1},\gamma_{0}}^{\mathrm{deter}}( \mathbf{x}_{0}|c)}[\mathbf{x}_{0}]=2^{\gamma_{0}}(T+1)^{1-\gamma_{1}}(T+2)^{- \gamma_{0}}.$ (35) $$

According to Theorem 4, variance of $q_{0,\gamma_{1},\gamma_{0}}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ is guaranteed to be smaller than the ground-truth $\mathrm{var}_{q_{0}(\mathbf{x}_{0}|c)}[\mathbf{x}_{0}]=1$ when each $\gamma_{0,i}\leqslant 0$ and $\gamma_{1,i}+\gamma_{0,i}\geqslant 1$, especially when $T\rightarrow+\infty$.

Now we formally propose the constraints.
First, we need $\gamma_{1,i}>1$ for strengthened conditional fidelity.
Then for expectation, it is noteworthy that $\Delta_{T}=0$ satisfies the assumption in Theorem 3.
Therefore by induction, it is feasible to annihilate $\Delta_{0}$ by annihilation of Eq. 28.
Finally as for variance, we empirically set $\gamma_{0,i}\leqslant 0$ and $\gamma_{1,i}+\gamma_{0,i}\geqslant 0$.

Practically, we can determine $\gamma_{0}$ according to the guidance strength $\gamma_{1}$, condition $c$, and timestep $t$, according to the closed-form solution of Eq. 33.
Concretely, given condition $c$, it is feasible to pre-compute a collection of $\{(\epsilon_{\theta}(\mathbf{x}_{t},c,t),\bm{\epsilon}_{\theta}(\mathbf{x}_{t}
,t))\}_{t}$ by traversing $q_{0}(\mathbf{x}_{0}|c)$, and maintain a lookup table consisting of $\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)]/
\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]$.
Then given any $\gamma_{1}$, we can directly achieve $\gamma_{0}$ by multiplying $-(\gamma_{1}-1)$ with the expectation ratio.
Pseudo-code is addressed in Appendix B.

We make further discussion about ReCFG.
By Cauchy-Schwarz inequality and Eq. 33 we have:

$$ $\displaystyle\|\Delta_{t-1}\|_{2}^{2}$ $\displaystyle\leqslant(\sigma_{t-1}-\frac{\alpha_{t-1}}{\alpha_{t}}\sigma_{t}) ^{2}\mathbb{E}_{\mathbf{x}_{t}}[\|(\gamma_{1}-1)\bm{\epsilon}_{\theta}(\mathbf {x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}].$ (36) $$

Then we can define the objective resembling DPMs as below, optimizing reversely from $t=T$ to 0 0.

$$ $\displaystyle\mathcal{L}_{\gamma_{1},\gamma_{0}}=\mathbb{E}_{\mathbf{x}_{t},t} [\|(\gamma_{1}-1)\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}].$ (37) $$

Resembling Theorem 1, with Eq. 37, we can also show the compatibility of ReCFG with DDIM, which is summarized as the theorem below.
Proof is addressed in Section A.5

###### Theorem 5 .

For any $\{\delta_{t}\}_{t}$, ReCFG with $\mathcal{L}_{\gamma_{1},\gamma_{0}}$ is compatible with native DPM up to a constant.

###### Theorem 3 .

###### Theorem 4 .

###### Theorem 5 .

## 4 Experiments

### 4.1 Experimental Setups

Datasets and baselines.
We apply ReCFG to previous seminal DPMs, including LDM (Rombach et al., ) on ImageNet 256 (Deng et al., ), EDM2 (Karras et al., ) on ImageNet 512, and SD3 (Esser et al., ) on CC12M (Changpinyo et al., ), respectively.

Evaluation metrics.
As for LDM and EDM2, we draw 50,000 samples for Fréchet Inception Distance (FID) (Heusel et al., ) to evaluate the fidelity of the synthesized images.
We further use Improved Precision (Prec.) and Recall (Rec.) (Kynkäänniemi et al., ) to separately measure sample fidelity (Precision) and diversity (Recall).
As for SD3, following the official implementation, we use CLIP Score (CLIP-S) (Radford et al., ; Hessel et al., ) and FID on CLIP features (Sauer et al., ) on 1,000 samples to evaluate conditional faithfulness and fidelity of the synthesized images, respectively.
Both two metrics are evaluated on the MS-COCO validation split (Lin et al., ).

Implementation details.
We implement ReCFG with NVIDIA A100 GPUs, and use pre-trained LDM(^2^22https://github.com/CompVis/latent-diffusion), EDM2(^3^33https://github.com/NVlabs/edm2), and SD3(^4^44https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) provided in official implementation.
We reproduce all the experiments with official and more other configurations including NFEs and guidance strengths.

### 4.2 Results on Toy Example in Section 3.2

We first confirm the effectiveness of our method on toy data, as presented in Sec. 3.2.
Given the closed-form expressions of score functions, we are able to precisely describe the distribution of both gamma-powered distribution $q_{0,\gamma}(\mathbf{x}_{0}|c)$ by native CFG and $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ by ReCFG.
The theoretical and numerical DDIM-based simulation value of probability density functions of both $q_{0,\gamma}(\mathbf{x}_{0}|c)$ and $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ are shown in Fig. 2.
It is noteworthy that native CFG drifts the expectation of $q_{0,\gamma}(\mathbf{x}_{0}|c)$ further away from the peak of the ground-truth $q_{0}(\mathbf{x}_{0}|c)$ as $\gamma$ becomes larger, consistent with Theorem 2.
As a comparison, the peaks of $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ and $q_{0}(\mathbf{x}_{0}|c)$ coincide, while $q_{0,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|c)$ is sharpened with smaller variance.
Therefore, by adopting relaxation on coefficients $\gamma_{1}$ and $\gamma_{0}$ with specially proposed constraints, our ReCFG manages to annihilate expectation shift, enabling better guidance and thus better conditional fidelity.

Table: Table 1: Sample quality on ImageNet (Deng et al., ). For clearer demonstration, settings of $\gamma_{1}+\gamma_{0}=1$ (i.e. falling into native CFG) are highlighted in gray.

### 4.3 Results on Real Datasets

We conduct extensive experiments on state-of-the-art DPMs to quantitatively convey the efficacy of ReCFG.
Results are reported in Tabs. 2 and 2.
We can tell that ReCFG is capable of better performance on both class-conditioned and text-conditioned DPMs under various guidance strengths and NFEs.
Additionally, ReCFG achieves better CLIP-S especially on small NFEs and large guidance strength, indicating better conditional fidelity on open-vocabulary synthesis.

### 4.4 Analyses

Table: Table 3: Variance of lookup table over condition $c$. Note that we employ pixel-wise lookup table involving timestep $t$. We report the the mean and variance of lookup table over $c$, which is computed by averaging on all timesteps $t$ and pixels.

Table: Table 4: Ablation study of the number of traversals (the number after ReCFG) for lookup table on ImageNet (Deng et al., ). For clearer demonstration, baselines of native CFG are highlighted in gray.

Variance of lookup table over condition $c$.
Note that we need to pre-compute the lookup table consisting of expectation ratios for all conditions $c$, which is time-consuming and impractical for open-vocabulary distributions (e.g., text-conditioned DPMs).
In Tab. 3 we report the mean and variance of expectation ratios over condition $c$, which is averaged on all timesteps and pixels.
It is noteworthy that the variance of text-conditioned DPMs is larger than that of class-conditioned ones, while both of which is insignificant compared to the mean.
Therefore, it is feasible to prepare the lookup table for only part of all potential conditions and use the mean for all conditions, serving as a practical strategy to improve time efficiency.

Ablation studies.
Recall that we pre-compute the lookup table by traversing the dataset $q_{0}(\mathbf{x}_{0}|c)$.
We conduct comprehensive ablation studies to convey a direct and clear picture of the efficacy
of ReCFG under different numbers of traversals, as reported in Tabs. 5 and 5.
We can conclude that larger number of traversals suggests better guidance performance, yet improvements from 100 to 500 traversals are relatively inconspicuous.
In other words, employment of 500 samples per condition is adequate to serve as an empirical setting.

Pixel-wise lookup table.
ReCFG enables pixel-specific guidance coefficients $\gamma_{1}$ and $\gamma_{0}$, thanks to the closed-form solution to Eq. 33, i.e., we can assign $\gamma_{0}$ for each pixel by maintaining the lookup table of pixel-wise expectation ratios.
Fig. 1 demonstrates the ratios on LDM, EDM2, and SD3 under different NFEs.
It is noteworthy that there appears no general rules on the relation between $\gamma_{1}$ and $\gamma_{0}$, indicating that trivially setting $\gamma_{1}$ and $\gamma_{0}$ to be scalars is less reasonable.
As a comparison, our method makes it possible to employ more precise control on guided sampling in a simple and post-hoc fashion without further fine-tuning, enabling better performance.

### 4.5 Discussions

Classifier-Free Guidance is designed from Bayesian theory to facilitate conditional sampling, yet appears incompatible with original diffusion theory.
Therefore, we believe ReCFG is attached to great importance on guided sampling by fixing the theoretical flaw of CFG.
Despite the success on better conditional fidelity, our algorithm has several potential limitations.
We need to pre-compute the lookup table by traversing the dataset to achieve rectified coefficients for each condition.
Although we conduct extensive ablation studies on the number of traversals and variance over condition $c$, providing an adequate strategy especially for open-vocabulary datasets on text-conditioned synthesis, the optimal strategy is unexplored.
Besides, we at present cannot provide precise control on variance of ReCFG, and turn to employ empirical values.
Therefore, how to further conquer these problems (e.g., employing a predictor network $\bm{\omega}(c,t)$ for better $\gamma_{0}$ on open-vocabulary datasets according to Eq. 37) will be an interesting avenue for future research.
Although leaving the variance behavior unexplored, we hope that ReCFG will encourage the community to close the gap in the future.

## 5 Conclusion

In this paper, we analyze the theoretical flaws of native Classifier-Free Guidance technique and the induced expectation shift phenomenon.
We theoretically claim the exact value of expectation shift on a toy distribution.
Introducing a relaxation on coefficients of CFG and novel constraints, we manage to complete the theory of guided sampling by fixing the incompatibility between CFG and diffusion theory.
Accordingly, thanks to the closed-form solution to the constraints, we propose ReCFG, a post-hoc algorithm aiming at more faithful guided sampling by determining the coefficients from a pre-computed lookup table.
We further study the behavior of the lookup table, proposing an adequate strategy for better time efficiency in practice.
Comprehensive experiments demonstrate the efficacy of our method on various state-of-the-art DPMs under different NFEs and guidance strengths.

## Appendix

## Appendix A Proofs and derivations

In this section, we will prove the theorems stated in the main manuscript.

### A.1 Proof of Theorem 1

We first claim two lemmas which are crucial for the proof.

###### Lemma 1 .

Let $g(\mathbf{x}_{t})$ and $h(\mathbf{x}_{t},\bm{\epsilon})$ be integrable functions, then the following equality holds.

$$ $\displaystyle\mathbb{E}_{q(\mathbf{x})}[\langle g(\mathbf{x}),\mathbb{E}_{q( \bm{\epsilon}|\mathbf{x})}[h(\mathbf{x},\bm{\epsilon})|\mathbf{x}]\rangle]= \mathbb{E}_{q(\mathbf{x},\bm{\epsilon})}[\langle g(\mathbf{x}),h(\mathbf{x}, \bm{\epsilon})\rangle],$ (S1) $$

in which $\langle\cdot,\cdot\rangle$ is inner product.

###### Proof of Lemma 1 .

Note that

$$ $\displaystyle\mathbb{E}_{q(\mathbf{x})}[\langle g(\mathbf{x}),\mathbb{E}_{q( \bm{\epsilon}|\mathbf{x})}[h(\mathbf{x},\bm{\epsilon})|\mathbf{x}]\rangle]$ $\displaystyle=\int\langle g(\mathbf{x}),\mathbb{E}_{q(\bm{\epsilon}|\mathbf{x} )}[h(\mathbf{x},\bm{\epsilon})|\mathbf{x}]\rangle q(\mathbf{x})\mathrm{d} \mathbf{x}$ (S2) $\displaystyle=\int\langle g(\mathbf{x}),\int h(\mathbf{x},\bm{\epsilon})q(\bm{ \epsilon}|\mathbf{x})\mathrm{d}\bm{\epsilon}\rangle q(\mathbf{x})\mathrm{d} \mathbf{x}$ (S3) $\displaystyle=\iint\langle g(\mathbf{x}),h(\mathbf{x},\bm{\epsilon})\rangle q( \mathbf{x})q(\bm{\epsilon}|\mathbf{x})\mathrm{d}\bm{\epsilon}\mathrm{d}\mathbf {x}$ (S4) $\displaystyle=\mathbb{E}_{q(\mathbf{x},\bm{\epsilon})}[\langle g(\mathbf{x}),h (\mathbf{x},\bm{\epsilon})\rangle],$ (S5) $$

in which Eq. S4 is by linearity of integral.
∎

###### Lemma 2 .

The following equality of expectation holds:

$$ $\displaystyle\mathbb{E}_{\mathbf{x}}[\bm{\epsilon}_{\theta}(\mathbf{x},t)]= \frac{1}{\sigma_{t}}\mathbb{E}_{\mathbf{x}}[\mathbf{x}]-\frac{\alpha_{t}}{ \sigma_{t}}\mathbb{E}_{c,\mathbf{x}_{0},\mathbf{x}}[\mathbf{x}_{0}].$ (S6) $$

###### Proof of Lemma 2 .

Note that

$$ $\displaystyle\nabla_{\mathbf{x}}\log q_{t}(\mathbf{x})$ $\displaystyle=\frac{\nabla_{\mathbf{x}}q_{t}(\mathbf{x})}{q_{t}(\mathbf{x})}$ (S7) $\displaystyle=\frac{\nabla_{\mathbf{x}}\int q_{t}(\mathbf{x}|c)q(c)\mathrm{d}c }{q_{t}(\mathbf{x})}$ (S8) $\displaystyle=\frac{\int\nabla_{\mathbf{x}}q_{t}(\mathbf{x}|c)q(c)\mathrm{d}c} {q_{t}(\mathbf{x})}$ (S9) $\displaystyle=\frac{\int q_{t}(\mathbf{x}|c)q(c)\nabla_{\mathbf{x}}\log q_{t}( \mathbf{x}|c)\mathrm{d}c}{q_{t}(\mathbf{x})}$ (S10) $\displaystyle=\int\frac{q_{t}(\mathbf{x}|c)q(c)}{q_{t}(\mathbf{x})}\nabla_{ \mathbf{x}}\log q_{t}(\mathbf{x}|c)\mathrm{d}c$ (S11) $\displaystyle=\mathbb{E}_{q_{t}(c|\mathbf{x})}[\nabla_{\mathbf{x}}\log q_{t}( \mathbf{x}|c)|\mathbf{x}].$ (S12) $$

Therefore, we have

$$ $\displaystyle\bm{\epsilon}_{\theta}(\mathbf{x},t)$ $\displaystyle=\mathbb{E}_{q_{t}(c|\mathbf{x})}[\bm{\epsilon}_{\theta}(\mathbf{ x},c,t)|\mathbf{x}]$ (S13) $\displaystyle=\mathbb{E}_{q_{t}(c|\mathbf{x})}\left[\mathbb{E}_{q(\mathbf{x}_{ 0}|\mathbf{x},c)}\left[\frac{\mathbf{x}-\alpha_{t}\mathbf{x}_{0}}{\sigma_{t}} \right]|\mathbf{x}\right]$ (S14) $\displaystyle=\mathbb{E}_{q_{t}(c,\mathbf{x}_{0}|\mathbf{x})}\left[\frac{ \mathbf{x}-\alpha_{t}\mathbf{x}_{0}}{\sigma_{t}}|\mathbf{x}\right]$ (S15) $\displaystyle=\frac{1}{\sigma_{t}}\mathbf{x}-\frac{\alpha_{t}}{\sigma_{t}} \mathbb{E}_{q_{t}(c,\mathbf{x}_{0}|\mathbf{x})}[\mathbf{x}_{0}|\mathbf{x}],$ (S16) $$

and

$$ $\displaystyle\mathbb{E}_{\mathbf{x}}[\bm{\epsilon}_{\theta}(\mathbf{x},t)]$ $\displaystyle=\frac{1}{\sigma_{t}}\mathbb{E}_{\mathbf{x}}[\mathbf{x}]-\frac{ \alpha_{t}}{\sigma_{t}}\mathbb{E}_{\mathbf{x}}[\mathbb{E}_{q_{t}(c,\mathbf{x}_ {0}|\mathbf{x})}[\mathbf{x}_{0}|\mathbf{x}]]$ (S17) $\displaystyle=\frac{1}{\sigma_{t}}\mathbb{E}_{\mathbf{x}}[\mathbf{x}]-\frac{ \alpha_{t}}{\sigma_{t}}\mathbb{E}_{c,\mathbf{x}_{0},\mathbf{x}}[\mathbf{x}_{0}].$ (S18) $$

∎

Then we start to prove Theorem 1.

###### Proof of Theorem 1 .

Similar to derivation in DDIM (Song et al., ), first rewrite $J_{\delta,\gamma}$ as below:

$$ $\displaystyle J_{\delta,\gamma}=\mathbb{E}\left[-\log p_{\theta,\gamma}( \mathbf{x}_{0}|\mathbf{x}_{1},c)+\sum_{t=2}^{T}D_{KL}(q_{\delta}(\mathbf{x}_{t -1}|\mathbf{x}_{t},\mathbf{x}_{0},c)\|p_{\theta,\gamma}(\mathbf{x}_{t-1}| \mathbf{x}_{t},c))\right]+C_{1},$ (S19) $$

in which $C_{1}$ is a constant not involving $\gamma$ and $\theta$.

Note that $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)=\mathbb{E}_{q(\bm{\epsilon}|\mathbf
{x}_{t},c)}[\bm{\epsilon}|\mathbf{x}_{t}]$.
Hence, for $t>1$:

$$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{t},\mathbf{x}_{0}|c)}[D_{KL}(q_{ \delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)\|p_{\theta,\gamma}( \mathbf{x}_{t-1}|\mathbf{x}_{t},c))]$ (S20) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{t},\mathbf{x}_{0}|c)}[D_{KL}(q_{ \delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)\|q_{\delta}(\mathbf{ x}_{t-1}|\mathbf{x}_{t},\mathbf{f}_{\theta,\gamma}^{t}(\mathbf{x}_{t},c),c))]$ (S21) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{t},\mathbf{x}_{0}|c)}[\|\mathbf{x}_{0 }-\mathbf{f}_{\theta,\gamma}^{t}(\mathbf{x}_{t},c)\|_{2}^{2}]$ (S22) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{\begin{subarray}{c}\mathbf{x}_{0}\sim q(\mathbf{x}_ {0}|c)\\ \bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ \mathbf{x}_{t}=\alpha_{t}\mathbf{x}_{0}+\sigma_{t}\bm{\epsilon}\end{subarray}} [\|\bm{\epsilon}-(\gamma\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+(1-\gamma) \bm{\epsilon}_{\theta}(\mathbf{x}_{t},t))\|_{2}^{2}]$ (S23) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\gamma(\bm{\epsilon }-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t))+(1-\gamma)(\bm{\epsilon}-\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},t))\|_{2}^{2}]$ (S24) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\gamma^{2}\|\bm{ \epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}+(1-\gamma)^{2} \|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}]$ $\displaystyle\;\qquad+2\gamma(1-\gamma)\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon }}[\langle\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t),\bm{ \epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\rangle]$ (S25) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\gamma^{2}\|\bm{ \epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}+(1-\gamma)^{2} \|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}]$ $\displaystyle\;\qquad+2\gamma(1-\gamma)\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon }}[\langle\bm{\epsilon}-\mathbb{E}_{q(\bm{\epsilon}|\mathbf{x}_{t},c)}[\bm{ \epsilon}|\mathbf{x}_{t}],\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t}, t)\rangle]$ (S26) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\gamma^{2}\|\bm{ \epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}+(1-\gamma)^{2} \|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}]$ $\displaystyle\;\qquad+2\gamma(1-\gamma)\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon }}[\langle\bm{\epsilon}-\bm{\epsilon},\bm{\epsilon}-\bm{\epsilon}_{\theta}( \mathbf{x}_{t},t)\rangle]$ (S27) $\displaystyle=$ $\displaystyle\;\gamma^{2}\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{ \epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}]+(1-\gamma)^{2} \mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}-\bm{\epsilon}_{ \theta}(\mathbf{x}_{t},t)\|_{2}^{2}],$ (S28) $$

in which Eq. S27 is from Lemma 1.
As for $t=1$ we have similar derivation:

$$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{1},\mathbf{x}_{0}|c)}[-\log p_{\theta ,\gamma}(\mathbf{x}_{0}|\mathbf{x}_{1},c))]$ (S29) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{1},\mathbf{x}_{0}|c)}[\|\mathbf{x}_{0 }-\mathbf{f}_{\theta,\gamma}^{t}(\mathbf{x}_{1},c)\|_{2}^{2}]+C_{2}$ (S30) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{\begin{subarray}{c}\mathbf{x}_{0}\sim q(\mathbf{x}_ {0}|c)\\ \bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ \mathbf{x}_{1}=\alpha_{1}\mathbf{x}_{0}+\sigma_{1}\bm{\epsilon}\end{subarray}} [\|\bm{\epsilon}-(\gamma\bm{\epsilon}_{\theta}(\mathbf{x}_{1},c,1)+(1-\gamma) \bm{\epsilon}_{\theta}(\mathbf{x}_{1},1))\|_{2}^{2}]+C_{3}$ (S31) $\displaystyle=$ $\displaystyle\;\gamma^{2}\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{ \epsilon}-\bm{\epsilon}_{\theta}(\mathbf{x}_{1},c,1)\|_{2}^{2}]+(1-\gamma)^{2} \mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}-\bm{\epsilon}_{ \theta}(\mathbf{x}_{1},1)\|_{2}^{2}]+C_{3},$ (S32) $$

in which $C_{2}$ and $C_{3}$ are constants not involving $\gamma$ and $\theta$.
Given that CFG involves score matching using both conditional and unconditional distributions, and that $J_{\delta,\gamma}$ is proportional to the score matching objective up to a constant, we confirm the equivalence between $J_{\delta,\gamma}$ and objective of native DPM under CFG.

Note that in native PF-ODE, we have

$$ $\displaystyle\frac{\mathrm{d}\mathbf{x}_{t}}{\mathrm{d}t}$ $\displaystyle=f(t)\mathbf{x}_{t}-\frac{1}{2}g^{2}(t)\nabla_{\mathbf{x}_{t}} \log q_{t}(\mathbf{x}_{t}|c),$ (S33) $\displaystyle\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\nabla_{\mathbf{x}_{t}}\log q _{t}(\mathbf{x}_{t}|c)]$ $\displaystyle=\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\mathbb{E}_{q_{t}(\mathbf{x }_{0}|\mathbf{x}_{t},c)}[\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}| \mathbf{x}_{0},c)]]$ (S34) $\displaystyle=\mathbb{E}_{q_{t}(\mathbf{x}_{0},\mathbf{x}_{t}|c)}[\nabla_{ \mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|\mathbf{x}_{0},c)]$ (S35) $\displaystyle=0,$ (S36) $$

in which Eq. S36 holds since forward diffusion process $q_{t}(\mathbf{x}_{t}|\mathbf{x}_{0},c)$ is implemented by adding Gaussian noise.
However, according to Eq. 10 and Lemma 2, we have

$$ $\displaystyle\mathbb{E}_{\mathbf{x}_{t}}[\nabla_{\mathbf{x}_{t}}\log q_{t, \gamma}(\mathbf{x}_{t}|c)]$ $\displaystyle=\mathbb{E}_{\mathbf{x}_{t}}[\gamma\nabla_{\mathbf{x}_{t}}\log q_ {t}(\mathbf{x}_{t}|c)+(1-\gamma)\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{ t})]$ (S37) $\displaystyle=(1-\gamma)\mathbb{E}_{\mathbf{x}_{t}}[\nabla_{\mathbf{x}_{t}} \log q_{t}(\mathbf{x}_{t})]$ (S38) $\displaystyle=\frac{\gamma-1}{\sigma_{t}^{2}}(\mathbb{E}_{\mathbf{x}_{t}}[ \mathbf{x}_{t}]-\alpha_{t}\mathbb{E}_{c,\mathbf{x}_{0},\mathbf{x}_{t}}[\mathbf {x}_{0}])$ (S39) $\displaystyle=\frac{\gamma-1}{\sigma_{t}^{2}}(\mathbb{E}_{q_{t}(\mathbf{x}_{t} |c)}[\mathbf{x}_{t}]-\alpha_{t}\mathbb{E}_{q_{0}(\mathbf{x}_{0},c)}[\mathbf{x} _{0}]).$ (S40) $$

Note that $\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\mathbf{x}_{t}]=\alpha_{t}\mathbb{E}_{q_{
0}(\mathbf{x}_{0}|c)}[\mathbf{x}_{0}]$, and that $\mathbb{E}_{\mathbf{x}_{0},c}[\mathbf{x}_{0}]=\int\mathbb{E}_{q_{0}(\mathbf{x}
_{0}|c)}[\mathbf{x}_{0}]\mathrm{d}c$.
Therefore when $\gamma\neq 1$, $\mathbb{E}_{\mathbf{x}_{t}}[\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma}(\mathbf{x
}_{t}|c)]$ is not guaranteed to be identical with 0.
In other words, denoising with CFG cannot be expressed as a reciprocal of diffusion process with Gaussian noise.
∎

###### Lemma 1 .

###### Proof of Lemma 1 .

###### Lemma 2 .

###### Proof of Lemma 2 .

###### Proof of Theorem 1 .

### A.2 Proof of Theorem 2

###### Proof.

Given Eq. 9, for $\gamma>1$, we have

$$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma}(\mathbf{x}_{t}|c)$ $\displaystyle=\gamma\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)+(1- \gamma)\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t})$ (S41) $\displaystyle=-\gamma\frac{\mathbf{x}_{t}-c}{t+1}-(1-\gamma)\frac{\mathbf{x}_{ t}}{t+2},$ (S42) $\displaystyle\frac{\mathrm{d}\mathbf{x}_{t}}{\mathrm{d}t}$ $\displaystyle=-\frac{1}{2}\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma}(\mathbf{x}_ {t}|c)$ (S43) $\displaystyle=\mathbf{x}_{t}\left(\frac{\gamma}{2(t+1)}+\frac{1-\gamma}{2(t+2) }\right)-c\frac{\gamma}{2(t+1)}.$ (S44) $$

By variation of constants formula, we can analytically solve $q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ in Eq. S44.

$$ $\displaystyle\mathbf{x}_{t}$ $\displaystyle=e^{\int_{T}^{t}\frac{\gamma}{2(s+1)}+\frac{1-\gamma}{2(s+2)} \mathrm{d}s}\left(C-\int_{T}^{t}c\frac{\gamma}{2(s+1)}e^{-\int_{s}^{t}\frac{ \gamma}{2(r+1)}+\frac{1-\gamma}{2(r+2)}\mathrm{d}r}\mathrm{d}s\right)$ (S45) $\displaystyle=(t+1)^{\frac{\gamma}{2}}(t+2)^{\frac{1-\gamma}{2}}\left(C-c\frac {\gamma}{2}\int_{T}^{t}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}} \mathrm{d}s\right),$ (S46) $$

in which $C$ is a constant to determine.
Let $t=T$, we can see that

$$ $\displaystyle C=\frac{\mathbf{x}_{T}}{(T+1)^{\frac{\gamma}{2}}(T+2)^{\frac{1- \gamma}{2}}}.$ (S47) $$

Therefore, we achieve the closed-form formula for $q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ as below:

$$ $\displaystyle\mathbf{x}_{0}=2^{\frac{1-\gamma}{2}}\left(\frac{\mathbf{x}_{T}}{ (T+1)^{\frac{\gamma}{2}}(T+2)^{\frac{1-\gamma}{2}}}+c\frac{\gamma}{2}\int_{0}^ {T}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s\right).$ (S48) $$

Since $q_{T}(\mathbf{x}_{T}|c)\sim\mathcal{N}(c,T+1)$, we can deduce that

$$ $\displaystyle q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)\sim\mathcal{N} \left(c\phi(\gamma,T),2^{1-\gamma}\frac{T+1}{(T+1)^{\gamma}(T+2)^{1-\gamma}} \right),$ (S49) $$

in which

$$ $\displaystyle\phi(\gamma,T)=2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{ \gamma}{2}}(T+2)^{\frac{1-\gamma}{2}}}+\frac{\gamma}{2}\int_{0}^{T}(s+1)^{- \frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s\right).$ (S50) $$

It is obvious that

$$ $\displaystyle\phi(\gamma)$ $\displaystyle=2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\int_{0}^{+\infty}(s+1)^{- \frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s,$ (S51) $\displaystyle\lim_{T\rightarrow+\infty}\frac{T+1}{(T+1)^{\gamma}(T+2)^{1- \gamma}}=1.$ (S52) $$

Then it is suffices to calculate $\phi(\gamma)$ for all $\gamma>1$.
First note that

$$ $\displaystyle\phi(1)$ $\displaystyle=\frac{1}{2}\int_{0}^{+\infty}(s+1)^{-\frac{3}{2}}\mathrm{d}s=1,$ (S53) $\displaystyle\phi(3)$ $\displaystyle=2^{-1}\frac{3}{2}\int_{0}^{+\infty}(s+1)^{-\frac{5}{2}}(s+2) \mathrm{d}s=2,$ (S54) $\displaystyle\phi(5)$ $\displaystyle=2^{-2}\frac{5}{2}\int_{0}^{+\infty}(s+1)^{-\frac{7}{2}}(s+2)^{2} \mathrm{d}s=\frac{7}{3}.$ (S55) $$

For $\gamma>1$, denote by $I(\gamma)$ with

$$ $\displaystyle I(\gamma)=\int_{0}^{+\infty}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{- \frac{1-\gamma}{2}}\mathrm{d}s.$ (S56) $$

Note that for $\gamma>1$ we have

$$ $\displaystyle I(\gamma)$ $\displaystyle=\int_{0}^{+\infty}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1- \gamma}{2}}\mathrm{d}s$ (S57) $\displaystyle=\frac{2}{\gamma+1}\int_{0}^{+\infty}(s+1)^{-\frac{\gamma+2}{2}} \mathrm{d}(s+2)^{\frac{\gamma+1}{2}}$ (S58) $\displaystyle=\frac{2}{\gamma+1}\left((s+1)^{-\frac{\gamma+1}{2}}(s+2)^{\frac{ \gamma+1}{2}}\Big{|}_{0}^{+\infty}+\frac{\gamma+2}{2}\int_{0}^{+\infty}(s+1)^{ -\frac{\gamma+4}{2}}(s+2)^{\frac{\gamma+1}{2}}\mathrm{d}s\right)$ (S59) $\displaystyle=\frac{2}{\gamma+1}\left(\frac{\gamma+2}{2}I(\gamma+2)-2^{\frac{ \gamma+1}{2}}\right).$ (S60) $$

Therefore, for $\gamma>1$ we have

$$ $\displaystyle\phi(\gamma)$ $\displaystyle=\frac{2\gamma}{\gamma+1}(\phi(\gamma+2)-1),$ (S61) $\displaystyle\phi(\gamma+2)$ $\displaystyle=1+\frac{\gamma+1}{2\gamma}\phi(\gamma).$ (S62) $$

From Eqs. S53, S54 and S55 we have

$$ $\displaystyle I(1)=2,\quad I(3)=\frac{8}{3},\quad I(5)=\frac{56}{15}.$ (S63) $$

For $\gamma\in[1,3]$, by Cauchy-Schwarz inequality with $p\in[0,1]$, we have

$$ $\displaystyle\Big{(}I(\gamma)\Big{)}^{p}\Big{(}I(5)\Big{)}^{1-p}$ (S64) $\displaystyle=$ $\displaystyle\left(\int_{0}^{+\infty}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{ 1-\gamma}{2}}\mathrm{d}s\right)^{p}\left(\int_{0}^{+\infty}(s+1)^{-\frac{7}{2} }(s+2)^{2}\mathrm{d}s\right)^{1-p}$ (S65) $\displaystyle\geqslant$ $\displaystyle\int_{0}^{+\infty}\Big{(}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac {1-\gamma}{2}}\Big{)}^{p}\Big{(}(s+1)^{-\frac{7}{2}}(s+2)^{2}\Big{)}^{1-p} \mathrm{d}s$ (S66) $\displaystyle=$ $\displaystyle\int_{0}^{+\infty}(s+1)^{-\frac{\gamma p-5p+7}{2}}(s+2)^{-\frac{5 p-\gamma p-4}{2}}\mathrm{d}s.$ (S67) $$

Let $p=\frac{2}{5-\gamma}\in[0,1]$ for $\gamma\in[1,3]$, from Eq. S67 we have

$$ $\displaystyle I(\gamma)\geqslant\Big{(}I(3)\Big{)}^{\frac{5-\gamma}{2}}\Big{(} I(5)\Big{)}^{\frac{\gamma-3}{2}}=\left(\frac{8}{3}\right)^{\frac{5-\gamma}{2}} \left(\frac{56}{15}\right)^{\frac{\gamma-3}{2}}.$ (S68) $$

Therefore for $\gamma\in[1,3]$, we have

$$ $\displaystyle\phi(\gamma)\geqslant 2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\left (\frac{8}{3}\right)^{\frac{5-\gamma}{2}}\left(\frac{56}{15}\right)^{\frac{ \gamma-3}{2}}=\gamma\frac{7}{15}\left(\frac{10}{7}\right)^{\frac{5-\gamma}{2}} =:h_{1}(\gamma)$ (S69) $$

Since $\frac{1}{\gamma}-\frac{1}{2}\log\frac{10}{7}>0$ for $\gamma\in[1,3]$, $h_{1}(\gamma)$ increases monotonically on $\in[1,3]$ and $h_{1}(1)=\frac{20}{21}$, $h_{1}(3)=2$.
Similarly, for $\gamma\in[3,5]$, by Cauchy-Schwarz inequality with $p\in[0,1]$, we have

$$ $\displaystyle\Big{(}I(1)\Big{)}^{1-p}\Big{(}I(\gamma)\Big{)}^{p}$ (S70) $\displaystyle=$ $\displaystyle\left(\int_{0}^{+\infty}(s+1)^{-\frac{3}{2}}\mathrm{d}s\right)^{1 -p}\left(\int_{0}^{+\infty}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{ 2}}\mathrm{d}s\right)^{p}$ (S71) $\displaystyle\geqslant$ $\displaystyle\int_{0}^{+\infty}\Big{(}(s+1)^{-\frac{3}{2}}\Big{)}^{1-p}\Big{(} (s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}}\Big{)}^{p}\mathrm{d}s$ (S72) $\displaystyle=$ $\displaystyle\int_{0}^{+\infty}(s+1)^{-\frac{3-p+\gamma p}{2}}(s+2)^{-\frac{p- \gamma p}{2}}\mathrm{d}s.$ (S73) $$

Let $p=\frac{2}{\gamma-1}\in[0,1]$ for $\gamma\in[3,5]$, from Eq. S73 we have

$$ $\displaystyle I(\gamma)\geqslant\Big{(}I(1)\Big{)}^{\frac{3-\gamma}{2}}\Big{(} I(3)\Big{)}^{\frac{\gamma-1}{2}}=2^{\frac{3-\gamma}{2}}\left(\frac{8}{3}\right )^{\frac{\gamma-1}{2}}.$ (S74) $$

Therefore for $\gamma\in[3,5]$

$$ $\displaystyle\phi(\gamma)\geqslant 2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}2^{ \frac{3-\gamma}{2}}\left(\frac{8}{3}\right)^{\frac{\gamma-1}{2}}=\gamma\left( \frac{2}{3}\right)^{\frac{\gamma-1}{2}}=:h_{2}(\gamma)$ (S75) $$

It is easy to see that $h_{2}(\gamma)\geqslant 2$ for $\gamma\in[3,5]$.
Then by mathematical induction and Eq. S62, we have $\phi(\gamma)\geqslant 2$ for all $\gamma\geqslant 3$.
Specially, we have

$$ $\displaystyle\lim_{\gamma\rightarrow+\infty}\phi(\gamma)=2.$ (S76) $$

And specifically, for $\gamma\in\mathbb{N}$, $\gamma>1$, we analytically calculate $\phi(\gamma,T)$ for $\gamma=2n+1$ and $\gamma=2n$, respectively.
First let $\gamma=2n+1$, $n\in\mathbb{N}$.
We can see that

$$ $\displaystyle(s+2)^{-\frac{1-\gamma}{2}}=(s+2)^{n}=\sum_{k=0}^{n}C_{n}^{k}(s+1 )^{k}.$ (S77) $$

$$ $\displaystyle\int_{0}^{T}(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2} }\mathrm{d}s$ $\displaystyle=\int_{0}^{T}(s+1)^{-\frac{\gamma+2}{2}}\left(\sum_{k=0}^{n}C_{n} ^{k}(s+1)^{k}\right)\mathrm{d}s$ (S78) $\displaystyle=\sum_{k=0}^{n}\left(C_{n}^{k}\int_{0}^{T}(s+1)^{\frac{2k-\gamma- 2}{2}}\mathrm{d}s\right)$ (S79) $\displaystyle=\sum_{k=0}^{n}\left(C_{n}^{k}\frac{2}{2k-\gamma}\Big{(}(T+1)^{ \frac{2k-\gamma}{2}}-1\Big{)}\right).$ (S80) $$

Since $2k-\gamma<0$ for $k=0,1,\cdots,n$, we have $(T+1)^{\frac{2k-\gamma}{2}}\rightarrow 0$ as $T$ goes to infinity and hence

$$ $\displaystyle\;\phi(\gamma,T)$ (S81) $\displaystyle=$ $\displaystyle\;2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{\gamma}{2}}(T +2)^{\frac{1-\gamma}{2}}}+\frac{\gamma}{2}\int_{0}^{T}(s+1)^{-\frac{\gamma+2}{ 2}}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s\right)$ (S82) $\displaystyle=$ $\displaystyle\;2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{\gamma}{2}}(T +2)^{\frac{1-\gamma}{2}}}+\frac{\gamma}{2}\left(\sum_{k=0}^{n}C_{n}^{k}\frac{2 }{2k-\gamma}\Big{(}(T+1)^{\frac{2k-\gamma}{2}}-1\Big{)}\right)\right)$ (S83) $\displaystyle=$ $\displaystyle\;2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{\gamma}{2}}(T +2)^{\frac{1-\gamma}{2}}}+\left(\sum_{k=0}^{n}C_{n}^{k}\frac{2n+1}{2n-2k+1} \Big{(}1-(T+1)^{\frac{2k-\gamma}{2}}\Big{)}\right)\right).$ (S84) $$

When $T\rightarrow+\infty$, we have

$$ $\displaystyle\phi(2n+1)=2^{-n}\left(\sum_{k=0}^{n}C_{n}^{k}\frac{2n+1}{2n-2k+1 }\right).$ (S85) $$

Then let $\gamma=2n$, $n\in\mathbb{N}$, and $n\geqslant 1$.
We have

$$ $\displaystyle\;\int(s+1)^{-\frac{\gamma+2}{2}}(s+2)^{-\frac{1-\gamma}{2}} \mathrm{d}s$ (S86) $\displaystyle=$ $\displaystyle\;\int 2(s+1)^{-n-1}(\sqrt{s+2})^{2n}\mathrm{d}\sqrt{s+2}$ (S87) $\displaystyle=$ $\displaystyle\;\int 2(u^{2}-1)^{-n-1}u^{2n}\mathrm{d}u$ (S88) $\displaystyle=$ $\displaystyle\;-\frac{1}{n}u^{2n-1}(u^{2}-1)^{-n}+\frac{2n-1}{n}\int(u^{2}-1)^ {-n}u^{2n-2}\mathrm{d}u,$ (S89) $$

in which Eq. S88 is due to integration by substitution with $u=\sqrt{s+2}>1$, and Eq. S89 is due to integration by parts.
Denote by $I_{n}$ with

$$ $\displaystyle I_{n}=\int 2(u^{2}-1)^{-n-1}u^{2n}\mathrm{d}u,\quad n\geqslant 0,$ (S90) $$

then we have

$$ $\displaystyle I_{n}=-\frac{1}{n}u^{2n-1}(u^{2}-1)^{-n}+\frac{2n-1}{2n}I_{n-1}, \quad n\geqslant 1.$ (S91) $$

For $n\geqslant 1$, let $I_{n}=\frac{(2n-1)!!}{(2n)!!}A_{n}$, then we have

$$ $\displaystyle\frac{(2n-1)!!}{(2n)!!}A_{n}$ $\displaystyle=-\frac{1}{n}u^{2n-1}(u^{2}-1)^{-n}+\frac{2n-1}{2n}\frac{(2n-3)!! }{(2n-2)!!}A_{n-1},\quad n\geqslant 2,$ (S92) $\displaystyle A_{n}$ $\displaystyle=A_{n-1}-\frac{1}{n}\frac{(2n)!!}{(2n-1)!!}u^{2n-1}(u^{2}-1)^{-n} ,\quad n\geqslant 2.$ (S93) $$

Therefore for $n\geqslant 2$, we have

$$ $\displaystyle A_{n}=A_{1}-\sum_{k=2}^{n}\frac{1}{k}\frac{(2k)!!}{(2k-1)!!}u^{2 k-1}(u^{2}-1)^{-k},$ (S94) $$

and

$$ $\displaystyle I_{n}=\begin{cases}{-\frac{u}{u^{2}-1}+\frac{1}{2}\log\frac{u-1} {u+1}},&n=1,\[20.0pt] {\frac{(2n-1)!!}{(2n)!!}\left(-\frac{2u}{u^{2}-1}+\log\frac{u-1}{u+1}-\sum_{k= 2}^{n}\frac{1}{k}\frac{(2k)!!}{(2k-1)!!}\frac{u^{2k-1}}{(u^{2}-1)^{k}}\right)} ,&n\geqslant 2.\end{cases}$ (S95) $$

Therefore, for $\gamma=2$ we have

$$ $\displaystyle\phi(\gamma,T)$ $\displaystyle=2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{\gamma}{2}}(T+ 2)^{\frac{1-\gamma}{2}}}+\frac{\gamma}{2}\int_{0}^{T}(s+1)^{-\frac{\gamma+2}{2 }}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s\right)$ (S96) $\displaystyle=2^{\frac{1-\gamma}{2}}\frac{1}{(T+1)^{\frac{\gamma}{2}}(T+2)^{ \frac{1-\gamma}{2}}}$ $\displaystyle\qquad-2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\left(\frac{\sqrt{T+ 2}}{T+1}-\sqrt{2}\right)$ $\displaystyle\qquad\qquad+2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\frac{1}{2} \left(\log\frac{\sqrt{T+2}-1}{\sqrt{T+2}+1}-\log\frac{\sqrt{2}-1}{\sqrt{2}+1} \right),$ (S97) $$

and for $\gamma\geqslant 4$ we have

$$ $\displaystyle\;\phi(\gamma,T)$ (S98) $\displaystyle=$ $\displaystyle\;2^{\frac{1-\gamma}{2}}\left(\frac{1}{(T+1)^{\frac{\gamma}{2}}(T +2)^{\frac{1-\gamma}{2}}}+\frac{\gamma}{2}\int_{0}^{T}(s+1)^{-\frac{\gamma+2}{ 2}}(s+2)^{-\frac{1-\gamma}{2}}\mathrm{d}s\right)$ (S99) $\displaystyle=$ $\displaystyle\;2^{\frac{1-\gamma}{2}}\frac{1}{(T+1)^{\frac{\gamma}{2}}(T+2)^{ \frac{1-\gamma}{2}}}$ $\displaystyle\;\qquad-2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\frac{(2n-1)!!}{(2 n)!!}\left(\frac{2\sqrt{T+2}}{T+1}-2\sqrt{2}\right)$ $\displaystyle\;\qquad\qquad+2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\frac{(2n-1) !!}{(2n)!!}\left(\log\frac{\sqrt{T+2}-1}{\sqrt{T+2}+1}-\log\frac{\sqrt{2}-1}{ \sqrt{2}+1}\right)$ $\displaystyle\;\qquad\qquad\qquad-2^{\frac{1-\gamma}{2}}\frac{\gamma}{2}\frac{ (2n-1)!!}{(2n)!!}\left(\sum_{k=2}^{n}\frac{1}{k}\frac{(2k)!!}{(2k-1)!!}\Big{(} \frac{(T+2)^{\frac{2k-1}{2}}}{(T+1)^{k}}-2^{\frac{2k-1}{2}}\Big{)}\right).$ (S100) $$

When $T\rightarrow+\infty$, we have

$$ $\displaystyle\phi(2n)=\begin{cases}{2^{\frac{1}{2}-n}n\left(\sqrt{2}-\frac{1}{ 2}\log\frac{\sqrt{2}-1}{\sqrt{2}+1}\right)},&n=1,\[20.0pt] {\frac{(2n-1)!!\sqrt{2}n}{(2n)!!2^{n}}\left(\left(\sum_{k=2}^{n}\frac{1}{k} \frac{(2k)!!}{(2k-1)!!}2^{k-\frac{1}{2}}\right)+2\sqrt{2}-\log\frac{\sqrt{2}-1 }{\sqrt{2}+1}\right)},&n\geqslant 2.\end{cases}$ (S101) $$

∎

###### Proof.

### A.3 Proof of Theorem 3

###### Proof.

We first write the closed-form expressions of DDIM sampler as below:

$$ $\displaystyle\mathbf{x}_{t-1}$ $\displaystyle=\frac{\alpha_{t-1}}{\alpha_{t}}\mathbf{x}_{t}+(\sigma_{t-1}- \frac{\alpha_{t-1}}{\alpha_{t}}\sigma_{t})\bm{\epsilon}_{\theta}(\mathbf{x}_{t },c,t),$ (S102) $\displaystyle\tilde{\mathbf{x}}_{t-1}$ $\displaystyle=\frac{\alpha_{t-1}}{\alpha_{t}}\tilde{\mathbf{x}}_{t}+(\sigma_{t -1}-\frac{\alpha_{t-1}}{\alpha_{t}}\sigma_{t})(\gamma_{1}\bm{\epsilon}_{\theta }(\tilde{\mathbf{x}}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\tilde{\mathbf{ x}}_{t},t)).$ (S103) $$

Then we have

$$ $\displaystyle\Delta_{t-1}$ $\displaystyle=\mathbb{E}_{\mathbf{x}_{t}}[\mathbf{x}_{t-1}]-\mathbb{E}_{\tilde {\mathbf{x}}_{t}}[\tilde{\mathbf{x}}_{t-1}]$ (S104) $\displaystyle=\frac{\alpha_{t-1}}{\alpha_{t}}(\mathbb{E}_{\mathbf{x}_{t}}[ \mathbf{x}_{t}]-\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\tilde{\mathbf{x}}_{t}])$ $\displaystyle\qquad+(\sigma_{t-1}-\frac{\alpha_{t-1}}{\alpha_{t}}\sigma_{t})( \mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c)]-\mathbb{ E}_{\tilde{\mathbf{x}}_{t}}[\gamma_{1}\bm{\epsilon}_{\theta}(\tilde{\mathbf{x} }_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},t)]).$ (S105) $$

Note that

$$ $\displaystyle\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)=\mathbb{E}_{q(\mathbf{ x}_{0}|\mathbf{x}_{t},c)}\left[\frac{\mathbf{x}_{t}-\alpha_{t}\mathbf{x}_{0}}{ \sigma_{t}}|\mathbf{x}_{t}\right],\quad\bm{\epsilon}_{\theta}(\tilde{\mathbf{x }}_{t},c,t)=\mathbb{E}_{q(\mathbf{x}_{0}|\tilde{\mathbf{x}}_{t},c)}\left[\frac {\tilde{\mathbf{x}}_{t}-\alpha_{t}\mathbf{x}_{0}}{\sigma_{t}}|\tilde{\mathbf{x }}_{t}\right].$ (S106) $$

Therefore, by $q_{t}(\mathbf{x}_{t}|c)=\int q_{0}(\mathbf{x}_{0}|c)q_{0t}(\mathbf{x}_{t}|
\mathbf{x}_{0},c)\mathrm{d}\mathbf{x}_{0}$ and Lemma 1 we have

$$ $\displaystyle\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t} ,c,t)]=\mathbb{E}_{\mathbf{x}_{0},\mathbf{x}_{t}}\left[\frac{\mathbf{x}_{t}- \alpha_{t}\mathbf{x}_{0}}{\sigma_{t}}\right]=\frac{1}{\sigma_{t}}\mathbb{E}_{ \mathbf{x}_{t}}[\mathbf{x}_{t}]-\frac{\alpha_{t}}{\sigma_{t}}\mathbb{E}_{ \mathbf{x}_{0}}[\mathbf{x}_{0}].$ (S107) $$

Similarly, by $p_{\theta,\gamma_{1},\gamma_{0}}(\tilde{\mathbf{x}}_{t}|c)=\int q_{0}(\mathbf{
x}_{0}|c)q_{0T}(\mathbf{x}_{T}|\mathbf{x}_{0},c)p_{\theta,\gamma_{1},\gamma_{0
}}(\tilde{\mathbf{x}}_{t}|\mathbf{x}_{T},c)\mathrm{d}\mathbf{x}_{0}\mathrm{d}
\mathbf{x}_{T}$ we have

$$ $\displaystyle\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\bm{\epsilon}_{\theta}(\tilde {\mathbf{x}}_{t},c,t)]=\mathbb{E}_{\mathbf{x}_{0},\tilde{\mathbf{x}}_{t}}\left [\frac{\tilde{\mathbf{x}}_{t}-\alpha_{t}\mathbf{x}_{0}}{\sigma_{t}}\right]= \frac{1}{\sigma_{t}}\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\tilde{\mathbf{x}}_{t} ]-\frac{\alpha_{t}}{\sigma_{t}}\mathbb{E}_{\mathbf{x}_{0}}[\mathbf{x}_{0}].$ (S108) $$

Then we can simplify $\Delta_{t}$ as below:

$$ $\displaystyle\Delta_{t-1}$ $\displaystyle=\frac{\alpha_{t-1}}{\alpha_{t}}\Delta_{t}+(\sigma_{t-1}-\frac{ \alpha_{t-1}}{\alpha_{t}}\sigma_{t})(\frac{1}{\sigma_{t}}\Delta_{t}-\mathbb{E} _{\tilde{\mathbf{x}}_{t}}[(\gamma_{1}-1)\bm{\epsilon}_{\theta}(\tilde{\mathbf{ x}}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},t)])$ (S109) $\displaystyle=\frac{\sigma_{t-1}}{\sigma_{t}}\Delta_{t}-(\sigma_{t-1}-\frac{ \alpha_{t-1}}{\alpha_{t}}\sigma_{t})\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[( \gamma_{1}-1)\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},c,t)+\gamma_{0}\bm{ \epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},t)]).$ (S110) $$

$\Delta_{t}=0$ implies that $\mathbb{E}_{q_{t}(\mathbf{x}_{t}|c)}[\mathbf{x}_{t}]=\mathbb{E}_{p_{\theta,
\gamma_{1},\gamma_{0}}(\tilde{\mathbf{x}}_{t}|c)}[\tilde{\mathbf{x}}_{t}]$.
Therefore, by Eqs. S107 and S108 we have $\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)]=
\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_
{t},c,t)]$.
According to Lemma 2 and by calculating the expectation over $\mathbf{x}_{t}$ and $\tilde{\mathbf{x}}_{t}$ respectively, we have

$$ $\displaystyle\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\bm{\epsilon}_{\theta}(\tilde {\mathbf{x}}_{t},t)]$ $\displaystyle=\frac{1}{\sigma_{t}}\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\tilde{ \mathbf{x}}_{t}]-\frac{\alpha_{t}}{\sigma_{t}}\mathbb{E}_{c,\mathbf{x}_{0}, \tilde{\mathbf{x}}_{t}}[\mathbf{x}_{0}]=\frac{1}{\sigma_{t}}\mathbb{E}_{\tilde {\mathbf{x}}_{t}}[\tilde{\mathbf{x}}_{t}]-\frac{\alpha_{t}}{\sigma_{t}}\mathbb {E}_{c,\mathbf{x}_{0}}[\mathbf{x}_{0}],$ (S111) $\displaystyle\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t} ,t)]$ $\displaystyle=\frac{1}{\sigma_{t}}\mathbb{E}_{\tilde{\mathbf{x}}_{t}}[\mathbf{ x}_{t}]-\frac{\alpha_{t}}{\sigma_{t}}\mathbb{E}_{c,\mathbf{x}_{0},\mathbf{x}_{ t}}[\mathbf{x}_{0}]=\frac{1}{\sigma_{t}}\mathbb{E}_{\mathbf{x}_{t}}[\mathbf{x} _{t}]-\frac{\alpha_{t}}{\sigma_{t}}\mathbb{E}_{c,\mathbf{x}_{0}}[\mathbf{x}_{0 }].$ (S112) $$

Since $\Delta_{t}=0$, we have $\mathbb{E}_{\mathbf{x}_{t}}[\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]=\mathbb{
E}_{\tilde{\mathbf{x}}_{t}}[\bm{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t},t)]$, and thus

$$ $\displaystyle\Delta_{t-1}=-(\sigma_{t-1}-\frac{\alpha_{t-1}}{\alpha_{t}}\sigma _{t})\mathbb{E}_{\mathbf{x}_{t}}[(\gamma_{1}-1)\bm{\epsilon}_{\theta}(\mathbf{ x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)]).$ (S113) $$

∎

###### Proof.

### A.4 Proof of Theorem 4

###### Proof.

Given Eq. 27, for any $\gamma_{1}$ and $\gamma_{0}$, we have

$$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma_{1},\gamma_{0}}(\mathbf{x }_{t}|c)$ $\displaystyle=\gamma_{1}\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t}|c)+ \gamma_{0}\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t})$ (S114) $\displaystyle=-\gamma_{1}\frac{\mathbf{x}_{t}-c}{t+1}-\gamma_{0}\frac{\mathbf{ x}_{t}}{t+2},$ (S115) $\displaystyle\frac{\mathrm{d}\mathbf{x}_{t}}{\mathrm{d}t}$ $\displaystyle=-\frac{1}{2}\nabla_{\mathbf{x}_{t}}\log q_{t,\gamma_{1},\gamma_{ 0}}(\mathbf{x}_{t}|c)$ (S116) $\displaystyle=\mathbf{x}_{t}\left(\frac{\gamma_{1}}{2(t+1)}+\frac{\gamma_{0}}{ 2(t+2)}\right)-c\frac{\gamma_{1}}{2(t+1)}.$ (S117) $$

By variation of constants formula, we can analytically solve $q_{0,\gamma_{1},\gamma_{0}}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ in Eq. S117.

$$ $\displaystyle\mathbf{x}_{t}$ $\displaystyle=e^{\int_{T}^{t}\frac{\gamma_{1}}{2(s+1)}+\frac{\gamma_{0}}{2(s+2 )}\mathrm{d}s}\left(C-\int_{T}^{t}c\frac{\gamma_{1}}{2(s+1)}e^{-\int_{s}^{t} \frac{\gamma_{1}}{2(r+1)}+\frac{\gamma_{0}}{2(r+2)}\mathrm{d}r}\mathrm{d}s\right)$ (S118) $\displaystyle=(t+1)^{\frac{\gamma_{1}}{2}}(t+2)^{\frac{\gamma_{0}}{2}}\left(C- c\frac{\gamma_{1}}{2}\int_{T}^{t}(s+1)^{-\frac{\gamma_{1}+2}{2}}(s+2)^{-\frac{ \gamma_{0}}{2}}\mathrm{d}s\right),$ (S119) $$

in which $C$ is a constant to determine.
Let $t=T$, we can see that

$$ $\displaystyle C=\frac{\mathbf{x}_{T}}{(T+1)^{\frac{\gamma_{1}}{2}}(T+2)^{\frac {\gamma_{0}}{2}}}.$ (S120) $$

Therefore, we achieve the closed-form formula for $q_{0,\gamma}^{\mathrm{deter}}(\mathbf{x}_{0}|c)$ as below:

$$ $\displaystyle\mathbf{x}_{0}=2^{\frac{\gamma_{0}}{2}}\left(\frac{\mathbf{x}_{T} }{(T+1)^{\frac{\gamma_{1}}{2}}(T+2)^{\frac{\gamma_{0}}{2}}}+c\frac{\gamma_{1}} {2}\int_{0}^{T}(s+1)^{-\frac{\gamma_{1}+2}{2}}(s+2)^{-\frac{\gamma_{0}}{2}} \mathrm{d}s\right).$ (S121) $$

Since $q_{T}(\mathbf{x}_{T}|c)\sim\mathcal{N}(c,T+1)$, we can deduce that

$$ $\displaystyle\mathrm{var}_{q_{0,\gamma_{1},\gamma_{0}}^{\mathrm{deter}}( \mathbf{x}_{0}|c)}[\mathbf{x}_{0}]=2^{\gamma_{0}}(T+1)^{1-\gamma_{1}}(T+2)^{- \gamma_{0}}.$ (S122) $$

∎

###### Proof.

### A.5 Proof of Theorem 5

###### Proof.

According to Eqs. 29 and 30, we can write the variational lower bound of $p_{\theta,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0:T}|c)$ as below:

$$ $\displaystyle J_{\delta,\gamma_{1},\gamma_{0}}$ $\displaystyle=\mathbb{E}_{q_{\delta}(\mathbf{x}_{0:T}|c)}[\log q_{\delta}( \mathbf{x}_{1:T}|\mathbf{x}_{0},c)-\log p_{\theta,\gamma_{1},\gamma_{0}}( \mathbf{x}_{0:T}|c)]$ (S123) $\displaystyle=\mathbb{E}\left[-\log p_{\theta,\gamma_{1},\gamma_{0}}(\mathbf{x }_{0}|\mathbf{x}_{1},c)\right]$ $\displaystyle\qquad+\mathbb{E}\left[\sum_{t=2}^{T}D_{KL}(q_{\delta}(\mathbf{x} _{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)\|p_{\theta,\gamma_{1},\gamma_{0}}( \mathbf{x}_{t-1}|\mathbf{x}_{t},c))\right]$ $\displaystyle\qquad\qquad+C_{1},$ (S124) $$

in which $C_{1}$ is a constant not involving $\gamma_{1}$, $\gamma_{0}$, and $\theta$.

Note that $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)=\mathbb{E}_{q(\bm{\epsilon}|\mathbf
{x}_{t},c)}[\bm{\epsilon}|\mathbf{x}_{t}]$.
Hence, for $t>1$:

$$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{t},\mathbf{x}_{0}|c)}[D_{KL}(q_{ \delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)\|p_{\theta,\gamma_{1 },\gamma_{0}}(\mathbf{x}_{t-1}|\mathbf{x}_{t},c))]$ (S125) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{t},\mathbf{x}_{0}|c)}[D_{KL}(q_{ \delta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},c)\|q_{\delta}(\mathbf{ x}_{t-1}|\mathbf{x}_{t},\mathbf{f}_{\theta,\gamma_{1},\gamma_{0}}^{t}(\mathbf{ x}_{t},c),c))]$ (S126) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{t},\mathbf{x}_{0}|c)}[\|\mathbf{x}_{0 }-\mathbf{f}_{\theta,\gamma_{1},\gamma_{0}}^{t}(\mathbf{x}_{t},c)\|_{2}^{2}]$ (S127) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{\begin{subarray}{c}\mathbf{x}_{0}\sim q(\mathbf{x}_ {0}|c)\\ \bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ \mathbf{x}_{t}=\alpha_{t}\mathbf{x}_{0}+\sigma_{t}\bm{\epsilon}\end{subarray}} [\|\bm{\epsilon}-(\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_ {0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t))\|_{2}^{2}]$ (S128) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}\|_{2} ^{2}+\|\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}]$ $\displaystyle\;\qquad-2\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\langle\bm{ \epsilon},\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},t)\rangle]$ (S129) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}\|_{2} ^{2}+\|\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}]$ $\displaystyle\;\qquad-2\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\langle \mathbb{E}_{q(\bm{\epsilon}|\mathbf{x}_{t},c)}[\bm{\epsilon}|\mathbf{x}_{t}], \gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{ \theta}(\mathbf{x}_{t},t)\rangle]$ (S130) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}\|_{2} ^{2}+\|\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{2}]$ $\displaystyle\;\qquad-2\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\langle\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},c,t),\gamma_{1}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\rangle]$ (S131) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}_{ \theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}+\|\gamma_{1}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\|_{2}^{ 2}]$ $\displaystyle\;\qquad-2\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\langle\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},c,t),\gamma_{1}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t})\rangle]$ $\displaystyle\;\qquad\qquad+\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{ \epsilon}\|_{2}^{2}-\|\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}]$ (S132) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}_{ \theta}(\mathbf{x}_{t},c,t)-(\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c ,t)+\gamma_{0}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t))\|_{2}^{2}]+C_{2}$ (S133) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|(\gamma_{1}-1)\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},c,t)+\gamma_{0}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},t)\|_{2}^{2}]+C_{2},$ (S134) $$

in which Eq. S130 is from Lemma 1, and $C_{2}=\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|\bm{\epsilon}\|_{2}^{2}-\|
\bm{\epsilon}_{\theta}(\mathbf{x}_{t},c,t)\|_{2}^{2}]$ is constant not involving $\gamma_{1}$ and $\gamma_{0}$.
As for $t=1$ we have similar derivation:

$$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{1},\mathbf{x}_{0}|c)}[-\log p_{\theta ,\gamma_{1},\gamma_{0}}(\mathbf{x}_{0}|\mathbf{x}_{1},c))]$ (S135) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{q(\mathbf{x}_{1},\mathbf{x}_{0}|c)}[\|\mathbf{x}_{0 }-\mathbf{f}_{\theta,\gamma_{1},\gamma_{0}}^{t}(\mathbf{x}_{1},c)\|_{2}^{2}]+C _{3}$ (S136) $\displaystyle\propto$ $\displaystyle\;\mathbb{E}_{\begin{subarray}{c}\mathbf{x}_{0}\sim q(\mathbf{x}_ {0}|c)\\ \bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ \mathbf{x}_{1}=\alpha_{1}\mathbf{x}_{0}+\sigma_{1}\bm{\epsilon}\end{subarray}} [\|\bm{\epsilon}-(\gamma_{1}\bm{\epsilon}_{\theta}(\mathbf{x}_{1},c,1)+\gamma_ {0}\bm{\epsilon}_{\theta}(\mathbf{x}_{1},1))\|_{2}^{2}]+C_{4}$ (S137) $\displaystyle=$ $\displaystyle\;\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}[\|(\gamma_{1}-1)\bm{ \epsilon}_{\theta}(\mathbf{x}_{1},c,1)+\gamma_{0}\bm{\epsilon}_{\theta}( \mathbf{x}_{1},1))\|_{2}^{2}]+C_{5},$ (S138) $$

in which $C_{3}$, $C_{4}$, and $C_{5}$ are constants not involving $\gamma_{1}$ and $\gamma_{0}$.
∎

###### Proof.

## Appendix B Pseudo-codes of Lookup Table

We below propose the pseudo-codes to achieve the lookup table and corresponding guided sampling in Algorithms 1 and 2.

Figure: Algorithm 1 Pseudo-code to achieve lookup table of ReCFG in a PyTorch-like style.

Figure: Algorithm 2 Pseudo-code for guided sampling by lookup table of ReCFG in a PyTorch-like style.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The unbiasedness is a great property to have. However, from the toy case experiment, the proposed method seems to affect the variance recovery. Would the proposed method hurt diversity in image generation?  

2. In text-to-3D generation, e.g., DreamFusion, the CFG coefficient is a lot larger. Can the analysis explain this and can similar improvements made in these scenarios? 

3. While ReCFG’s use of a lookup table aids in theoretical correctness, it adds a layer of complexity to the sampling process. For text-to-image models, where open-vocabulary synthesis requires flexibility across diverse prompts, the need to precompute or interpolate values for novel prompts could hinder ReCFG’s feasibility. I am wondering how much is the extra computation, especially for more complicated generation tasks.
soundness: 2
strengths: The paper provides a rigorous analysis of a known issue with CFG, quantifying expectation shift and proposing ReCFG as a theoretically grounded alternative. This analysis contributes to understanding the mathematical foundation of guided sampling in diffusion models. The resulting ReCFG is designed as a post-hoc solution that can be applied to pretrained models without the need for additional training.

The experiments on the toy data is helpful for understanding the method and subsequent real data experiments also demonstrates the effectiveness.
summary: This paper introduces an enhancement to Classifier-Free Guidance (CFG), a widely used approach in diffusion model sampling, by identifying and addressing an expectation shift flaw in CFG. The authors propose "Rectified Classifier-Free Guidance" (ReCFG), a variation that relaxes the original guidance coefficient constraints to align more closely with diffusion theory. By incorporating a lookup table to adjust guidance coefficients, ReCFG aims to maintain conditional fidelity without additional retraining or loss of inference speed. The authors present theoretical derivations and empirical results demonstrating improved conditional fidelity on several state-of-the-art diffusion models, including applications on ImageNet and CC12M.
weaknesses: My main concern is the scope of the experiment and demonstrated experimental improvements. CFG has been a very important component in most AIGC model. Even GPT-type sequential image modeling uses CFG for better performance. Targeting such a component, the current work doesn't demonstrate enough practical impact in more complex generative tasks such as text to image generation. 
Performance-wise, the empirical gain measured by the FID or CLIP scores in ImageNet and CC12M are not that significant. In this case, some user study or visual comparison would be helpful in showcasing the effectiveness.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I guess one needs to solve (28) for each time step t, which may introduce more computation. Will that be an issue?
2. Whether the coefficients obtained by optimizing (37) can lead to (28) since it also take expectations with respect to t?
3. By (28) and the equation before (34), it seems that we also have $E_{x_t}(\epsilon_\theta(x_t,t,c)) = E_{x_t}(\epsilon_\theta(x_t,t)) = 0$. Now suppose you have $\Delta_t =0$. From the proof of Theorem 3, specifically (S107) and (S112), we have $E_{x_0}(x_0) = E_{c,x_0}(x_0)$. Does that mean in this case, the conditional expectation is the same as the unconditional expectation? If it is true, does that mean if $E_{x_0}(x_0) \neq  E_{c,x_0}(x_0)$, $\Delta_t \neq 0$ and you will always have expectation shift?
soundness: 3
strengths: 1. The paper provides a theoretical analysis, identifying a flaw in the popular CFG technique and proposing a mathematically grounded solution, also provides rigorous proof of the benefits of ReCFG.
2. The introduction of ReCFG offers a novel solution to the expectation shift problem associated with CFG in diffusion models.
summary: The paper examines rectified classifier-free guidance (ReCFG) for diffusion models, addressing theoretical flaws in existing models that use Classifier-Free Guidance (CFG). By introducing new coefficients for combining conditional and unconditional score functions, ReCFG adheres closely to diffusion theory, eliminating expectation shifts in conditional distributions and enhancing model fidelity without the need for retraining.
weaknesses: 1. Since it is mainly a theoretical paper, I think it would be beneficial to add more explanation after Theorem 3, while probably reduce the length of Theorem 2. 
2. If I understand correctly, the method needs to solve (28) for each time step t, which may increase the computation.
3. Although it is a theoretical paper, it would be better to add more experiment results, e.g., some illustrations of generated images.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - It is difficult to understand the connection between theorems 1 and 2 as the fact that $\mathbb{E}\_{x_t}[\nabla_{x_t} \log q_{t, \gamma}(x_t \mid c)] \neq 0$ doesn't appear anywhere in theorem 2. However, theorem 3 illustrates this difference perfectly. Why putting theorem 3 so late in the manuscript ? I believe that putting theorem 3 early would help readers better undertsand the logical flow of the paper's intuitions.
- I find the results in Table 1 and 2 very tight between the algorithms, could you provide means and standard deviations ?
- According to Table 3, it seems that the expectation ratio is often close to 1. Doesn't it advocate for $\gamma_0 = 1 - \gamma_1$ in the end ?
soundness: 4
strengths: - The paper is well written
- The proposed approach is novel
- The paper is well illustrated with intuitive theorems
summary: This paper addresses conditional generative modeling and highlights the limitations of traditional Classifier-Free Guidance (CFG). It demonstrates that denoising with CFG cannot be viewed as a reciprocal diffusion process. To tackle this issue, the authors propose a relaxed version of CFG that introduces a new type of guidance. They show that this alternative guidance can be unbiased and provide a methodology for calculating the necessary guidance parameters. The paper also illustrates the superiority of their proposed method through evaluations on standard benchmarks, showcasing its effectiveness in overcoming the limitations of classic CFG.
weaknesses: - The proposed method doesn't give a closed form expression but an estimator
- The method is computationally intensive

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * Figure 1: Is the expectation ratio essentially the softmax? The figure's message is unclear. Distinguishing the expectation ratios between the first and last sampling images is challenging based on the comparison provided. Consider providing a more detailed explanation of what the expectation ratio represents and how it's calculated and some visualization, such as using a different color scale or providing separate figures for the first and last sampling steps.

* Figure2: Seems like this method achieves unbiasness at the cost of losing diversity(much smaller variance). If caclualting KL divergence, it is hard to tell which distribution is closer to the ground-truth. 

* If relaxing the constriant of CFG: two coefficients sum to one; I doubt it increases or decreases the magnitude of the predicted noise norm, which may degenerate the sampling process. 

* In Table1\&2: are the results of the baseline CFG good enough to be persuasive? Please choose some well-accepted benchmark models results on these datasets and ensure the hyperparameters are comparable, for example, 25 or 256 sampling steps with ddpm or ddim.

* From Table3, it suggests users have to prepare the expectation ratio for each class and each sampling steps. The combination of class and sampling steps makes it too cumbersome. Furthmore, the such small digit makes people worry the numerical stability of this method. Therefore, the ablation study should be presented to prove the numerical stability of the expectation ratio. Please consider conducting some ablation studies, such as:
Comparing results when using class-specific ratios vs. a global average ratio
Analyzing how performance changes when rounding the ratios to different decimal places
Examining the impact of using fewer sampling steps on both performance and computational complexity
soundness: 3
strengths: The analysis of CFG is extensive.
summary: The paper thoroughly examines Classifier-Free Guidance (CFG) in diffusion, highlighting the bias introduced by mean shift due to violations of the zero mean assumption in data samples. It suggests adjusting the CFG coefficient to mitigate this shift. Some empirical experiments seem to support these findings, revealing the mean-shift issue and proposing an easily implementable solution to address this bias.
weaknesses: * The proposed method appears numerically unstable due to the minute coefficient deviation from the default value of 1.0, casting doubt on the experiment's reliability. Furthermore, this slight difference in coefficients may indicate that the violation of the zero-mean assumption in the original CFG has negligible practical implications.  Please consider conducting an ablation study showing how results change with small perturbations to the coefficients.

* The merits of unbiasness scarfices the diversity(dramatic smaller variance compared to the ground truth), as indicated in Figure2. Provide a quantification of the trade-off between unbiasedness and diversity would be more informative. 

* In essence, once relaxing the constriant of CFG: two coefficients sum to one; I doubt it increases or decreases the magnitude of the noise norm, which may collapse the sampling process. Providing an analysis of how the proposed method affects the noise norm throughout the sampling process, compared to standard CFG, would be informative. 

* No visual demonstration is provided. Please provide some sample images generated with standard CFG vs. the proposed method, or visualizations of the sampling process over time.