Classifier-Free Guidance is a Predictor-Corrector

## Abstract

Abstract We investigate the theoretical foundations of
classifier-free guidance (CFG).
CFG is the dominant method of conditional sampling for text-to-image diffusion models, yet
unlike other aspects of diffusion, it remains on shaky theoretical footing. In this paper, we first disprove common misconceptions,
by showing that CFG interacts differently with
DDPM ( ho2020denoising ) and DDIM ( song2021denoising ) ,
and neither sampler with CFG generates the gamma-powered distribution p ⁢ ( x | c ) γ ⁢ p ⁢ ( x ) 1 − γ 𝑝 superscript conditional 𝑥 𝑐 𝛾 𝑝 superscript 𝑥 1 𝛾 p(x|c)^{\gamma}p(x)^{1-\gamma} italic_p ( italic_x | italic_c ) start_POSTSUPERSCRIPT italic_γ end_POSTSUPERSCRIPT italic_p ( italic_x ) start_POSTSUPERSCRIPT 1 - italic_γ end_POSTSUPERSCRIPT .
Then, we clarify the behavior of CFG by showing that it is a kind of
predictor-corrector method ( song2020score ) that alternates between denoising and sharpening, which we call predictor-corrector guidance (PCG).
We prove that in the SDE limit,
CFG is actually equivalent to
combining a DDIM predictor for the conditional distribution
together with a Langevin dynamics corrector
for a gamma-powered distribution
(with a carefully chosen gamma).
Our work thus provides a lens to theoretically understand CFG
by embedding it in a broader design space
of principled sampling methods.

## 1 Introduction

Classifier-free-guidance (CFG) has become an essential part of modern
diffusion models, especially in text-to-image applications (dieleman2022guidance; rombach2022high; nichol2021glide; podell2023sdxl).
CFG is intended to improve conditional sampling, e.g. generating images conditioned on a given class label or text prompt (ho2022classifier).
The traditional (non-CFG) way to do conditional sampling
is to simply train a model for the conditional distribution $p(x\mid c)$,
including the conditioning $c$
as auxiliary input to the model.
In the context of diffusion, this means training a model
to approximate the conditional score
$s(x,t,c):=\nabla_{x}\log p_{t}(x\mid c)$
at every noise level $t$, and sampling from this model
via a standard diffusion sampler (e.g. DDPM).
Interestingly, this standard way of conditioning usually does not perform well for
diffusion models, for reasons that are unclear.
In the text-to-image case for example,
the generated samples tend to be visually incoherent and not faithful to the prompt,
even for large-scale models (ho2022classifier; rombach2022high).

Guidance methods, such as CFG and its predecessor classifier guidance (sohl2015deep; song2020score; dhariwal2021diffusion),
are methods introduced to improve the quality of conditional samples.
During training, CFG requires learning a model for both the
unconditional and conditional scores
($\nabla_{x}\log p_{t}(x)$ and $\nabla_{x}\log p_{t}(x|c)$).
Then, during sampling, CFG runs
any standard diffusion sampler (like DDPM or DDIM),
but replaces the true conditional scores with
the “CFG scores”

$$ $\displaystyle\widetilde{s}(x,t,c):=\gamma\nabla_{x}\log p_{t}(x\mid c)+(1- \gamma)\nabla\log p_{t}(x),$ (1) $$

for some $\gamma>0$.
This turns out to produce much more coherent samples in practice,
and so CFG is used in almost all
modern text-to-image diffusion models (dieleman2022guidance).
A common intuition for why CFG works
starts by observing that Equation (1) is the score of a *gamma-powered* distribution:

$$ $\displaystyle p_{t,\gamma}(x|c)$ $\displaystyle:=p_{t}(x)^{1-\gamma}p_{t}(x|c)^{\gamma},$ (2) $$

which is also proportional to $p_{t}(x)p_{t}(c|x)^{\gamma}$. Raising $p_{t}(c|x)$ to a power $\gamma>1$ sharpens the classifier around its modes, thereby emphasizing the “best” exemplars of the given class or other conditioner at each noise level. Applying CFG — that is, running a standard sampler with the usual score replaced by the CFG score at each denoising step — is supposed to increase the influence of the conditioner on the final samples.

Figure: Figure 1: CFG vs. PCG. We prove that the DDPM variant of classifier-free guidance (top) is equivalent to a kind of predictor-corrector method (bottom), in the continuous limit. We call this latter method “predictor-corrector guidance” (PCG), defined in Section 4.1. The equivalence holds for all CFG guidance strengths $\gamma$, with corresponding PCG parameter $\gamma^{\prime}=(2\gamma-1)$, as given in Theorem 3. Samples from SDXL with prompt: “photograph of a cat eating sushi using chopsticks”.
Refer to caption: extracted/5810172/fig/cfg_vs_pcg.png

However, CFG does not inherit the theoretical correctness guarantees
of standard diffusion, because the CFG scores do not necessarily correspond to
a valid diffusion forward process.
The fundamental issue (which is known, but still worth emphasizing) is that $p_{t,\gamma}(x|c)$ is not the same as the distribution obtained by applying a forward diffusion process to the gamma-powered data distribution $p_{0,\gamma}(x|c)$.
That is, letting $N_{t}[p]$ denote the distribution produced by starting from a distribution $p$ and running the diffusion forward process up to time $t$, we have

$$ $p_{t,\gamma}(x|c):=N_{t}[p_{0}(x|c)]^{\gamma}\cdot N_{t}[p_{0}(x)]^{1-\gamma} \neq N_{t}\left[p_{0}(x|c)^{\gamma}p_{0}(x)^{1-\gamma}\right].$ $$

Since the distributions $\{p_{t,\gamma}(x|c)\}_{t}$ *do not correspond to any known forward diffusion process*, we cannot properly interpret the CFG score (1) as a denoising direction; and using the CFG score in a sampling loop like DDPM or DDIM is no longer theoretically guaranteed to produce a sample from $p_{0,\gamma}(x|c)$ or any other known distribution.
Although this flaw is known in theory (e.g. du2023reduce; karras2024guiding),
it is largely ignored in practice and in much of the literature.
The theoretical foundations of CFG are thus unclear, and important questions remain open.
Is there a principled way to think about why CFG works? And what does it even mean for CFG to “work” – what problem is CFG solving?
We make progress towards understanding the foundations of CFG,
and in the process we uncover several new aspects and
connections to other methods.

- 1.
First, we disprove common misconceptions about CFG by counterexample. We show that the DDPM and DDIM variants of CFG can generate different distributions, neither of which is the gamma-powered data distribution $p_{0}(x)^{1-\gamma}p_{0}(x|c)^{\gamma}$.
- 2.
We define a
family of methods called predictor-corrector guidance (PCG),
as a natural way to approximately sample
from gamma-powered distributions.
PCG alternates
between denoising steps and Langevin dynamics steps.
Unlike typical predictor-corrector methods (song2020score), in PCG
the corrector operates on a different (sharper) distribution
than the predictor.
- 3.
We prove that in the continuous-time limit,
CFG is equivalent to PCG with a careful choice of parameters.
This gives a principled way to interpret CFG:
it is implicitly an annealed Langevin dynamics.
- 4.
For demonstration purposes, we implement the PCG sampler
for Stable Diffusion XL and observe that it produces samples
qualitatively similar to CFG, with guidance scales determined
by our theory.
Further, we explore the design axes exposed by the PCG framework, namely
guidance strength and Langevin parameters, in order to clarify their respective effects.

## 2 Preliminaries

We adopt the continuous-time
stochastic differential equation (SDE) formalism of diffusion
from song2020score. These continuous-time results can be translated to discrete-time algorithms; we give explicit algorithm descriptions for our experiments.

### 2.1 Diffusion Samplers

Forward diffusion processes start with a conditional data distribution $p_{0}(x|c)$ and gradually corrupt it with Gaussian noise, with $p_{t}(x|c)$ denoting the noisy distribution at time $t$.
The forward diffusion runs up to a time $T$ large enough that $p_{T}$ is approximately pure noise. To sample from the data distribution, we first sample from the Gaussian distribution $p_{T}$ and then run the diffusion process in reverse (which requires an estimate of the score, usually learned by a neural network). A variety of samplers have been developed to perform this reversal. DDPM (ho2020denoising) and DDIM (song2021denoising) are standard samplers that correspond to discretizations of a reverse-SDE and reverse-ODE, respectively. Due to this correspondence, we refer to the reverse-SDE as DDPM and the reverse-ODE as DDIM for short.
We will mainly consider the *variance-preserving* (VP) diffusion process
from ho2020denoising, although most of our discussion applies equally to other settings (such as variance-exploding). The forward process, reverse-SDE, and equivalent reverse-ODE for the VP conditional diffusion are (song2020score)

$$ $\displaystyle\textsf{Forward SDE}:dx$ $\displaystyle=-\frac{1}{2}\beta_{t}xdt+\sqrt{\beta_{t}}dw.$ (3) $\displaystyle\textsf{DDPM SDE}:\quad dx$ $\displaystyle=-\frac{1}{2}\beta_{t}x~{}dt-\beta_{t}\nabla_{x}\log p_{t}(x|c)dt +\sqrt{\beta_{t}}d\overline{w}$ (4) $\displaystyle\textsf{DDIM ODE}:\quad dx$ $\displaystyle=-\frac{1}{2}\beta_{t}x~{}dt-\frac{1}{2}\beta_{t}\nabla_{x}\log p _{t}(x|c)dt.$ (5) $$

The unconditional version of each sampler simply replaces $p_{t}(x|c)$ with $p_{t}(x)$.
Note that the *score* $\nabla_{x}\log p_{t}(x|c)$ appears in both (4) and (5). Intuitively, the score points in a direction toward higher probability, and so it helps to reverse the forward diffusion process. The score is unknown in general, but can be learned via standard diffusion training methods.

### 2.2 Classifier-Free Guidance

CFG replaces the usual conditional score $\nabla_{x}\log p_{t}(x|c)$ in (4) or (5) at each timestep $t$ with the alternative score $\nabla_{x}\log p_{t,\gamma}(x|c)$.
In SDE form, the CFG updates are

$$ $\displaystyle\textsf{$\textsf{CFG}_{\textsf{DDPM}}$}:\quad dx$ $\displaystyle=-\frac{1}{2}\beta_{t}x~{}dt-\beta_{t}\nabla_{x}\log p_{t,\gamma} (x|c)dt+\sqrt{\beta_{t}}d\overline{w}$ (6) $\displaystyle\textsf{$\textsf{CFG}_{\textsf{DDIM}}$}:\quad dx$ $\displaystyle=-\frac{1}{2}\beta_{t}x~{}dt-\frac{1}{2}\beta_{t}\nabla\log p_{t, \gamma}(x|c)dt,$ (7) $\displaystyle\text{where }\nabla_{x}\log p_{t,\gamma}(x|c)$ $\displaystyle=(1-\gamma)\nabla_{x}\log p_{t}(x)+\gamma\nabla_{x}\log p_{t}(x|c).$ $$

### 2.3 Langevin Dynamics

Langevin dynamics (rossky1978brownian; parisi1981correlation)
is another sampling method, which starts from an arbitrary initial distribution and iteratively transforms it into a desired one.
Langevin dynamics (LD) is given by the following SDE (robert1999monte)

$$ $\displaystyle dx$ $\displaystyle=\frac{\varepsilon}{2}\nabla\log\rho(x)dt+\sqrt{\varepsilon}dw.$ (8) $$

LD converges (under some assumptions)
to the steady-state $\rho(x)$ (roberts1996exponential). That is, letting $\rho_{s}(x)$ denote the solution of LD at time $s$, we have $\lim_{s\to\infty}\rho_{s}(x)=\rho(x)$. Similar to diffusion sampling, LD requires the score of the desired distribution $\rho$ (or a learned estimate of it).

## 3 Misconceptions about CFG

Figure: Figure 2: Counterexamples: $\textsf{CFG}_{\textsf{DDIM}}\neq\textsf{CFG}_{\textsf{DDPM}}\neq$ gamma-powered. $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$ do not generate the same output distribution, even when using the same score function. Moreover, neither generated distribution is the gamma-powered distribution $p_{0,\gamma}(x|c)$. (Left) Counterexample 1 (section 3.1): $\textsf{CFG}_{\textsf{DDIM}}$ yields a sharper distribution than $\textsf{CFG}_{\textsf{DDPM}}$, and both are sharper than $p_{0,\gamma}(x|c)$. (Right) Counterexample 2 (section 3.2): Neither $\textsf{CFG}_{\textsf{DDIM}}$ nor $\textsf{CFG}_{\textsf{DDPM}}$ yield even a scaled version of the gamma-powered distribution $p_{0,\gamma}(x|c)=\mathcal{N}(-3,1)$. The $\textsf{CFG}_{\textsf{DDPM}}$ distribution is mean-shifted relative to $p_{0,\gamma}(x|c)$. The $\textsf{CFG}_{\textsf{DDIM}}$ distribution is mean-shifted and not even Gaussian (note the asymmetrical shape).
Refer to caption: x1.png

We first observe that the exact definition of CFG matters: specifically, the sampler with which it used. Without CFG, DDPM and DDIM generate equivalent distributions. However, we will prove that with CFG, DDPM and DDIM
can generate different distributions, as follows:

###### Theorem 1 (DDIM $\neq$ DDPM; informal ) .

There exists a joint distribution $p(x,c)$ over
inputs $x\in\mathbb{R}$ and conditioning $c\in\mathbb{R}$, such that the following holds.
Consider generating a sample via CFG with conditioning $c=0$,
guidance-scale $\gamma\gg 0$, and using either DDPM or DDIM samplers.
Then, the generated distributions will be approximately

$$ $\displaystyle\widehat{p}_{\mathrm{ddpm}}\approx\mathcal{N}(0,\gamma^{-1}); \quad\widehat{p}_{\mathrm{ddim}}\approx\mathcal{N}(0,2^{-\gamma}).$ (9) $$

In particular, the DDIM variant of CFG is exponentially sharper than the DDPM variant.

Next, we disprove the misconception that CFG
generates the gamma-powered distribution data:

###### Theorem 2 (CFG $\neq$ gamma-sharpening, informal) .

There exists a joint distribution $p(x,c)$ and a $\gamma>0$
such that neither $\textsf{CFG}_{\textsf{DDIM}}$
nor $\textsf{CFG}_{\textsf{DDPM}}$ produces the gamma-powered distribution $p_{0,\gamma}(x|c)\propto p_{0}(x)^{1-\gamma}p_{0}(x|c)^{\gamma}$.

We prove both claims in the next section using simple Gaussian constructions.

###### Theorem 1 (DDIM ≠ \neq ≠ DDPM; informal ) .

###### Theorem 2 (CFG ≠ \neq ≠ gamma-sharpening, informal) .

### 3.1 Counterexample 1

We first present a setting that allows us to *exactly* solve the ODE and SDE dynamics of CFG in closed-form, and hence to find the exact distribution sampled by running CFG. This would be intractable in general, but it is possible for a specific problem, as follows.

Consider the setting where $p_{0}(x)$ and $p_{0}(x|c=0)$ are both zero-mean Gaussians, but with different variances. Specifically, $(x_{0},c)$ are jointly Gaussian, with $p(c)=\mathcal{N}(0,1)$, $p_{0}(x|c)=c+\mathcal{N}(0,1)$. Therefore

$$ $\displaystyle p_{0}(x)$ $\displaystyle=\mathcal{N}(0,2)$ $\displaystyle p_{0}(x|c=0)$ $\displaystyle=\mathcal{N}(0,1)$ $\displaystyle p_{0,\gamma}(x|c=0)$ $\displaystyle=\mathcal{N}(0,\frac{2}{\gamma+1})$ (10) $$

For this problem, we can solve $\textsf{CFG}_{\textsf{DDIM}}$ (7) and $\textsf{CFG}_{\textsf{DDPM}}$ (6) analytically; that is, we solve initial-value problems for the reversed dynamics to find the sampled distribution of $\widehat{x}_{t}$ in terms of the initial-value $x_{T}$. Applying these results to $t=0$ and averaging over the known Gaussian distribution of $x_{T}$ gives the exact distribution of $\widehat{x}_{0}$ that CFG samples. The full derivation is in Appendix A.1. The final CFG-sampled distributions are:

$$ $\displaystyle\textsf{CFG}_{\textsf{DDPM}}:\quad\widehat{x}_{0}$ $\displaystyle\sim\mathcal{N}\left(0,\frac{2-2^{2-2\gamma}}{2\gamma-1}\right)$ (11) $\displaystyle\textsf{CFG}_{\textsf{DDIM}}:\quad\widehat{x}_{0}$ $\displaystyle\sim\mathcal{N}\left(0,2^{1-\gamma}\right).$ (12) $$

This shows that for any $\gamma>1$, the $\textsf{CFG}_{\textsf{DDIM}}$ distribution is sharper than the $\textsf{CFG}_{\textsf{DDPM}}$ distribution, and both are sharper than the gamma-powered distribution $p_{0,\gamma}(x|c=0)$. (Even though the distributions all have the same mean, their different variances make them distinct.) In fact, for $\gamma\gg 1$, the variance of DDPM-CFG is approximately $\frac{2}{2\gamma-1}$, which is about twice the variance of $p_{0,\gamma}(x|c=0)$. In Figure 2, we compare the $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$ distributions – sampled using an exact denoiser (see Appendix A.6) within DDIM/DDPM sampling loops – to the unconditional, conditional, and gamma-powered distributions.

### 3.2 Counterexample 2

In the above counterexample, the $\textsf{CFG}_{\textsf{DDIM}}$, $\textsf{CFG}_{\textsf{DDPM}}$, and gamma-powered distributions had different variances but the same Gaussian form, so one might wonder whether the distributions differ only by a scale factor in general. This is not the case, as we can see in a different counterexample that reveals greater qualitative differences, in particular a symmetry-breaking behavior of CFG.

In Counterexample 2, the unconditional distribution is a Gaussian mixture with two clusters with equal weights and variances, and means at $\pm\mu$.

$$ $\displaystyle c$ $\displaystyle\in\{0,1\},\quad p(c=0)=\frac{1}{2}$ $\displaystyle p_{0}(x_{0}|c=0)$ $\displaystyle=\mathcal{N}(-\mu,1),\quad p_{0}(x_{0}|c=1)=\mathcal{N}(\mu,1)$ $\displaystyle p_{0}(x_{0})$ $\displaystyle=\frac{1}{2}p_{0}(x_{0}|c=0)+\frac{1}{2}p_{0}(x_{0}|c=1)$ (13) $$

If the means are sufficiently separated ($\mu\gg 1$), then the gamma-powered distribution for $\gamma\geq 1$ is approximately equal to the conditional distribution, i.e.
$p_{0,\gamma}(x|c)\approx p_{0}(x|c),$
due to the near-zero-probability valley between the conditional densities
(see Appendix A.2). However, for sufficiently high noise the clusters begin to merge, and $p_{t,\gamma}(x|c)\neq p_{t}(x|c)$. In particular, $p_{0,\gamma}(x|c)$ is approximately Gaussian with mean $\pm\mu$, but $p_{t,\gamma}(x|c)\neq p_{t}(x|c)$ is not.
Although we cannot solve the CFG ODE and SDE in this case, we can empirically sample the $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$ distributions using an exact denoiser and compare them to the gamma-powered distribution. In particular, we see that neither $\textsf{CFG}_{\textsf{DDIM}}$ nor $\textsf{CFG}_{\textsf{DDPM}}$ is Gaussian with mean $\pm\mu$, hence neither is a scaled version of the gamma-powered distribution. The results are shown in Figure 2.

## 4 CFG as a predictor-corrector

The previous sections illustrated the subtlety in understanding CFG.
We can now state our main structural characterization,
that CFG is equivalent to a special kind of *predictor-corrector* method (song2020score).

### 4.1 Predictor-Corrector Guidance

As a warm-up, suppose we actually wanted to sample from the
gamma-powered distribution:

$$ $\displaystyle p_{\gamma}(x|c)\propto p(x)^{1-\gamma}p(x|c)^{\gamma}.$ (14) $$

A natural strategy is to run Langevin dynamics w.r.t. $p_{\gamma}$.
This is possible in theory because
we can compute the score of $p_{\gamma}$ from the known scores of $p(x)$
and $p(x\mid c)$:

$$ $\displaystyle\nabla_{x}\log p_{\gamma}(x\mid c)=(1-\gamma)\nabla_{x}\log p(x)+ \gamma\nabla_{x}\log p(x\mid c).$ (15) $$

However this won’t work in practice, due to the well-known issue
that vanilla Langevin dynamics has impractically slow mixing times for
many distributions of interest (song2019generative).
The usual remedy for this is to use some kind of annealing,
and the success of diffusion teaches us that the diffusion process
defines a good annealing path (song2020score; du2023reduce).
Combining these ideas yields an algorithm
remarkably similar to the predictor-corrector methods introduced in song2020score.
For example, consider the following diffusion-like iteration,
starting from $x_{T}\sim\mathcal{N}(0,\sigma_{T})$ at $t=T$.
At timestep $t$,

- 1.
Predictor: Take one diffusion denoising step (e.g. DDIM or DDPM) w.r.t. $p_{t}(x\mid c)$,
using score $\nabla_{x}\log p_{t}(x\mid c)$, to move to time $t^{\prime}=t-{\Delta{t}}$.
- 2.
Corrector: Take one (or more) Langevin dynamics steps
w.r.t. distribution $p_{t^{\prime},\gamma}$,
using score
$\nabla_{x}\log p_{t^{\prime},\gamma}(x\mid c)=(1-\gamma)\nabla_{x}\log p_{t^{
\prime}}(x)+\gamma\nabla_{x}\log p_{t^{\prime}}(x\mid c).$

Figure: Figure 3: CFG is equivalent to PCG for particular parameter choices.
Refer to caption: extracted/5810172/fig/pcg_vs_cfg_diagram.png

It is reasonable to expect that running this iteration down to $t=0$
will produce a sample from approximately $p_{\gamma}(x|c)$,
since it can be thought of as annealed Langevin dynamics where the predictor
is responsible for the annealing. We name this algorithm predictor-corrector guidance (PCG).
Notably, PCG differs from the predictor-corrector algorithms
in song2020score because our predictor
and corrector operate w.r.t. *different* annealing distributions:
the predictor tries to anneal along the set of distributions
$\{p_{t}(x|c)\}_{t\in[0,1]}$,
whereas the corrector anneals
along the set $\{p_{t,\gamma}(x|c)\}_{t\in[0,1]}$.
Remarkably, it turns out that for specific choices of the denoising predictor and Langevin step size,
PCG with $K=1$ is equivalent (in the SDE limit) to CFG, but with a different $\gamma$.

### 4.2 SDE limit of PCG

Figure: Algorithm 1 $\textsf{PCG}_{\textsf{DDIM}}$, theory. (see Algorithm 2 for practical implementation.)

Consider the version of PCG defined in Algorithm 1, which uses DDIM as predictor and a particular LD on the gamma-powered distribution as corrector. We take $K=1$, i.e. a single LD step per iteration.
Crucially, we set the LD step size such that the Langevin noise scale exactly matches
the noise scale of a (hypothetical) DDPM step at the current time
(similar to du2023reduce).
In the limit as ${\Delta{t}}\to 0$, Algorithm 1 becomes the following SDE (see Appendix B):

$$ $\displaystyle dx$ $\displaystyle=\underbrace{\Delta\textsf{DDIM}(x,t)}_{\text{Predictor}}+ \underbrace{\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)}_{\text{Corrector}}=: \Delta\textsf{PCG}_{\textsf{DDIM}}(x,t,\gamma),$ (16) where $\displaystyle\Delta\textsf{DDIM}(x,t)=-\frac{1}{2}\beta_{t}(x+\nabla\log p_{t} (x|c))dt$ $\displaystyle\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)=-\frac{1}{2}\beta_{t} \Bigl{(}(1-\gamma)\nabla\log p_{t}(x)+\gamma\nabla\log p_{t}(x|c)\Bigr{)}dt+ \sqrt{\beta_{t}}d\overline{w}.$ $$

Above, $\Delta\textsf{DDIM}(x,t)$ is the *differential* of the DDIM ODE (5), i.e. the ODE can be written as $dx=\Delta\textsf{DDIM}(x,t)$.
And $\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)$, where G stands for “guidance”, is the limit as ${\Delta{t}}\to 0$ of the Langevin dynamics step in PCG, which behaves like a differential of LD (see Appendix B).

We can now show that the PCG SDE (16) matches CFG, but with a different $\gamma$. In the statement, $\Delta\textsf{CFG}_{\textsf{DDPM}}(x,t,\gamma)$ denotes the differential of the $\textsf{CFG}_{\textsf{DDPM}}$ SDE (6), similar to the notation above.
This result is trivial to prove using our definitions,
but the statement itself appears to be novel.

###### Theorem 3 (CFG is predictor-corrector) .

In the SDE limit, CFG
is equivalent to a predictor-corrector. That is, the following differentials are equal:

$$ $\displaystyle\Delta\textsf{CFG}_{\textsf{DDPM}}(x,t,\gamma)$ $\displaystyle=\Delta\textsf{DDIM}(x,t)+\Delta\textsf{LD}_{\textsf{G}}(x,t,2 \gamma-1)=:\Delta\textsf{PCG}_{\textsf{DDIM}}(x,t,2\gamma-1)$ (17) $$

Notably, the guidance scales of CFG
and the above Langevin dynamics are not identical.

###### Proof.

$$ $\displaystyle\Delta\textsf{PCG}_{\textsf{DDIM}}(x,t,\gamma)$ $\displaystyle=\Delta\textsf{DDIM}(x,t)+\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)$ $\displaystyle=-\frac{1}{2}\beta_{t}(x+(1-\gamma)\nabla\log p_{t}(x)+(1+\gamma) \nabla\log p_{t}(x|c))dt+\sqrt{\beta_{t}}d\overline{w}$ $\displaystyle=-\frac{1}{2}\beta_{t}x{\Delta{t}}-\beta_{t}\nabla_{x}\log p_{t, \gamma^{\prime}}(x|c){\Delta{t}}+\sqrt{\beta_{t}}d\overline{w},\quad\gamma^{ \prime}:=\frac{\gamma}{2}+\frac{1}{2}$ $\displaystyle=\Delta\textsf{CFG}_{\textsf{DDPM}}(x,t,\gamma^{\prime})$ $$

∎

As an aside, taking $\gamma=1$ in Theorem 3
recovers the standard fact that DDPM is equivalent, in the limit,
to DDIM interleaved with LD (e.g. karras2022elucidating).
Because for $\gamma=1$, $\textsf{CFG}_{\textsf{DDPM}}$ is just DDPM, so Theorem 3
reduces to: $\Delta\textsf{DDPM}(x,t)=\Delta\textsf{DDIM}(x,t)+\Delta\textsf{LD}_{\textsf{G
}}(x,t,1)$.
This fact, that in the non-CFG case
Langevin dynamics is equivalent to iteratively noising-then-denoising,
has been used implicitly or explicitly in a number of prior works.
For example, karras2022elucidating use a “churn” operation
in their stochastic sampler,
and lugmayr2022repaint incorporate a conceptually similar
noise-then-denoise step in their inpainting pipeline.

###### Theorem 3 (CFG is predictor-corrector) .

###### Proof.

## 5 Discussion and Related Works

There have been many recent works toward understanding CFG.
To better situate our work, it helps
to first discuss the overall research agenda.

### 5.1 Understanding CFG: The Big Picture

We want to study the question of why CFG helps in practice: specifically, why it improves both image quality and prompt adherence,
compared to conditional sampling.
We can approach this question by
applying a standard generalization decomposition.
Let $p(x|c)$ be the “ground truth” population distribution;
let $p^{*}_{\gamma}(x|c)$ be the distribution generated by the ideal CFG sampler, which exactly solves the CFG reverse SDE for the ground-truth scores (note that at $\gamma=1$, $p^{*}_{1}(x|c)=p(x|c)$); and let $\widehat{p}_{\gamma}(x|c)$ denote the distribution of the real CFG sampler, with learnt scores and finite discretization.
Now, for any image distribution
$q$, let $\textsf{PerceivedQuality}[q]\in\mathbb{R}$
denote a measure of perceived sample quality of this distribution to humans.
We cannot mathematically specify this notion of quality,
but we will assume it exists for analysis.
Notably, PerceivedQuality is *not*
a measurement of how close a distribution is
to the ground-truth $p(x|c)$ — it is possible for a generated distribution to appear even “higher quality” than the ground-truth, for example.
We can now decompose:

$$ $\displaystyle\underbrace{\textsf{PerceivedQuality}[\widehat{p}_{\gamma}]}_{ \text{Real CFG}}=\underbrace{\textsf{PerceivedQuality}[p^{*}_{\gamma}]}_{\text {Ideal CFG}}-\underbrace{\left(\textsf{PerceivedQuality}[p^{*}_{\gamma}]- \textsf{PerceivedQuality}[\widehat{p}_{\gamma}]\right)}_{\text{Generalization Gap}}.$ (18) $$

Therefore, if the LHS increases with $\gamma$,
it must be because at least one of the two occurs:

- 1.
The ideal CFG sampler improves in quality with increasing $\gamma$.
That is, CFG distorts the population distribution
in a favorable way
(e.g. by sharpening it, or otherwise).
- 2.
The generalization gap decreases with increasing $\gamma$.
That is, CFG has a type of regularization effect,
bringing population and empirical processes closer.

In fact, it is likely that both occur.
The original motivation for CG and CFG
involved the first effect: CFG was intended to produce
“lower-temperature” samples from a sharpened population distribution
(dhariwal2021diffusion; ho2022classifier).
This is particularly relevant if the model is trained on poor-quality datasets (e.g. cluttered images from the web), so we want
to use guidance to sample from a higher-quality distribution (e.g. images of an isolated subject).
On the other hand, recent studies have given evidence for the second effect.
For example, karras2024guiding argues that unguided diffusion sampling
produces “outliers,” which are avoided when using guidance — this can be
thought of as guidance reducing the generalization gap, rather than improving the
ideal sampling distribution.
Another interpretation of the second effect is that guidance
could enforce a good inductive bias:
it “simplifies” the family of possible
output distributions in some sense,
and thus simplifies the learning problem,
reducing the generalization gap. Figure 4 shows a example where this occurs.
Finally, this generalization decomposition applies
to any intervention to the SDE, not just increasing guidance strength.
For example, increasing the
Langevin steps in PCG (parameter $K$)
also shrinks the generalization gap, since
it reduces the discretization error.

Figure: Figure 4: An example where guidance benefits generalization. Suppose that the conditional distribution for $c=0$ is a a GMM with a dominant cluster, as shown in purple, and the unconditional distribution is uniform (details in Appendix A.4). We sample with DDPM using exact scores vs. scores learned by training a small MLP with early stopping. The scores are learned more accurately near the dominant cluster. (Left) For conditional sampling (no guidance), DDPM is expected to sample from the conditional distribution (purple curve). However, DDPM-with-learned-scores (orange) samples less accurately than DDPM-with-exact-scores (blue) away from the dominant cluster (where the learned scores are inaccurate) (note the prevalence of blue samples in low-probability regions). (Center) With guidance $\gamma=3$, $p_{0,\gamma}(x|c=0)$ (red) and both samplers concentrate around the dominant cluster (where the learned scores are accurate), reducing the generalization gap between the learned and exact models. (Right) Exact vs. learned condition scores $\nabla\log p(x|c=0)$.
Refer to caption: extracted/5810172/fig/prob_4.png

In this framework,
our work makes progress towards understanding
both terms on the RHS of Equation 18, in different ways.
For the first term,
we identify structural properties of ideal CFG,
by showing that $p^{*}_{\gamma}$ can be equivalently generated by
a standard technique (an annealed Langevin dynamics).
For the second term,
the PCG framework
highlights the ways in which errors in the
learned score can contribute to a
generalization gap, during both the denoising step
and the LD step (the latter would move toward an inaccurate steady-state distribution).

### 5.2 Open Questions and Limitations

In addition to the above, there are a number of other questions left open by our work.
First, we study only the stochastic variant of CFG (i.e. $\textsf{CFG}_{\textsf{DDPM}}$),
and it is not clear how to adapt our analysis to the more commonly used deterministic variant
($\textsf{CFG}_{\textsf{DDIM}}$).
This is subtle because the two CFG variants
can behave very differently in theory,
but appear to behave similarly in practice.
It is thus open to identify plausible theoretical conditions
which explain this similarity(^1^11 Curiously, $\textsf{CFG}_{\textsf{DDIM}}$ is the correct probability-flow ODE for $\textsf{CFG}_{\textsf{DDPM}}$ if and only if the true intermediate distribution at time $t$ is $p_{t,\gamma}$. However we know this is not the true distribution in general, from Section 3.); we give a suggestive experiment in Figure 6.
More broadly, it is open to find explicit characterizations of CFG’s output distribution,
in terms of the original
$p(x)$ and $p(x|c)$
— although it is
possible tractable expressions do not exist.

Finally, we presented PCG primarily as a tool to understand CFG,
not as a practical algorithm in itself.
Nevertheless, the PCG framework outlines a broad family of guided samplers,
which may be promising to explore in practice.
For example, the predictor can be any diffusion denoiser,
including CFG itself.
The corrector can operate on any
distribution with a known score, including compositional distributions as in du2023reduce, or any other distribution that might help sharpen or otherwise improve on the conditional distribution. Finally, the number of Langevin steps could be adapted to the timestep, similar to kynkaanniemi2024applying, or alternative samplers could be considered (du2023reduce; neal2012mcmc; ma2015complete).

### 5.3 Stable Diffusion Examples

Figure: Figure 5: Effect of Guidance and Correction. Each grid shows SDXL samples using $\textsf{PCG}_{\textsf{DDIM}}$, as the guidance strength $\gamma$ and Langevin iterations $K$ are varied. Left: “photograph of a dog drinking coffee with his friends”. Right: “a tree reflected in the hood of a blue car”. (Zoom in to view).
Refer to caption: extracted/5810172/fig/grid_both_narrow.png

We include several examples
running predictor-corrector guidance on
Stable Diffusion XL (podell2023sdxl).
These serve primarily to sanity-check our theory,
not as a suggestion for practice.
For all experiments, we use $\textsf{PCG}_{\textsf{DDIM}}$
as implemented explicitly
in Algorithm 2(^2^22Note that Algorithm 1 and 2 have slightly different DDIM steps, but this just corresponds to two different discretizations of the same process. Algorithm 1 uses the first-order Euler–Maruyama discretization known as “reverse SDE” (song2020score), which is convenient for our mathematical analysis. Algorithm 2 uses the original DDIM discretization (song2021denoising), equivalent to a more sophisticated integrator (lu2022dpm), which is more common in practice.).
Note that PCG offers a more flexible design space than standard CFG;
e.g. we can run multiple corrector steps for each denoising step
to improve the quality of samples (controlled by parameter $K$
in Algorithm 2).

#### CFG vs. PCG.

Figure 1 illustrates the equivalence of Theorem 3: we compare $\textsf{CFG}_{\textsf{DDPM}}$ with guidance $\gamma$
to $\textsf{PCG}_{\textsf{DDIM}}$ with exponent $\gamma^{\prime}:=(2\gamma-1)$.
We run $\textsf{CFG}_{\textsf{DDPM}}$ with 200 denoising steps,
and $\textsf{PCG}_{\textsf{DDIM}}$ with 100 denoising steps
and $K=1$ Langevin corrector step
per denoising step.
Corresponding samples appear to have
qualitatively similar guidance strengths, consistent with our theory.

#### Effects of Guidance and Corrector.

In Figure 5 we show samples from
$\textsf{PCG}_{\textsf{DDIM}}$, varying the guidance strength and Langevin iterations
(i.e. parameters $\gamma$ and $K$ respectively
in Algorithm 2).
We also include standard $\textsf{CFG}_{\textsf{DDIM}}$ samples for comparison.
All samples used 1000 denoising steps for the base predictor.
Overall, we observed that increasing Langevin steps
tends to improve the overall image quality,
while increasing guidance strength tends to
improve prompt adherence.
In particular, sufficiently many Langevin steps
can sometimes yield high-quality conditional samples,
even *without* any guidance ($\gamma=1$);
see Figure 7 in the Appendix for another
such example.
This is consistent with the observations of
song2020score on unguided predictor-corrector methods.
It is also related to the findings of du2023reduce on MCMC methods:
du2023reduce
similarly use an annealed Langevin dynamics
with reverse-diffusion annealing,
although they focus on general compositions of distributions
rather than the specific gamma-powered distribution of CFG.

Notice that in Figure 5,
increasing the number of Langevin steps
appears to also increase the “effective” guidance strength.
This is because
the dynamics does not fully mix:
one Langevin step ($K=1$) does not
suffice to fully converge the
intermediate distributions
to $p_{t,\gamma}$.

## 6 Conclusion

In this paper, we have shown that while CFG is not a diffusion sampler on the gamma-powered data distribution $p_{0}(x)^{1-\gamma}p_{0}(x|c)^{\gamma}$, it can be understood as a particular kind of predictor-corrector, where the predictor is a DDIM denoiser, and the corrector at each step $t$ is one step of Langevin dynamics on the gamma-powered noisy distribution $p_{t}(x)^{1-{\gamma^{\prime}}}p_{t}(x|c)^{\gamma^{\prime}}$, with $\gamma^{\prime}=(2\gamma-1)$. Although song2020score’s Predictor-Corrector algorithm has not been widely adopted in practice, perhaps due to its computation expense relative to samplers like DPM++ (lu2022dpmplusplus), it turns out to provide a lens to understand the unreasonable practical success of CFG.
On a practical note, PCG encompasses a rich design space of possible predictors and correctors for future exploration, that may help improve the prompt-alignment, diversity, and quality of diffusion generation.

Acknowledgements.
We thank
David Berthelot,
James Thornton,
Jason Ramapuram,
Josh Susskind,
Miguel Angel Bautista Martin,
Jiatao Gu,
Zijing Ou
and
Rob Brekelmans
for helpful discussions and feedback throughout this work.

## Appendix A 1D Gaussian Counterexamples

Figure: Figure 6: (Left) For Counterexample 1 (section 3.1), we plot the empirical and theoretical variance of the gamma-powered, $\textsf{CFG}_{\textsf{DDIM}}$, and $\textsf{CFG}_{\textsf{DDPM}}$ distributions, over a range of values of $\gamma$. The theoretical predictions are given by equations (12) and (11), and the empirical distributions are sampled using an exact denoiser. This verifies the theoretical predictions and illustrates the decreasing variance from $p_{0,\gamma}$ to $\textsf{CFG}_{\textsf{DDPM}}$ to $\textsf{CFG}_{\textsf{DDIM}}$. (Right) For counterexample 3 (section A.3 with different choices of variance ($\sigma=1$ and $\sigma=2$), we compare $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$. Increasing the variance makes the two CFG samplers more similar. Also note that the $\textsf{CFG}_{\textsf{DDIM}}$ distribution is symmetric around the center cluster, but asymmetric around the side clusters. This experiment suggests that multiple clusters and greater overlap between classes can help symmetrize and reduce the difference between $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$
Refer to caption: x3.png

### A.1 Counterexample 1 Detail

Counterexample 1 (equation 10) has

$$ $\displaystyle p_{0}(x)$ $\displaystyle\sim\mathcal{N}(0,2)$ $\displaystyle p_{0}(x|c=0)$ $\displaystyle\sim\mathcal{N}(0,1).$ $$

The $\gamma$-powered distribution is

$$ $\displaystyle p_{0,\gamma}(x|c=0)$ $\displaystyle=p_{0}(x|c)^{\gamma}p_{c=0}(x)^{1-\gamma}$ $\displaystyle\propto e^{-\frac{\gamma x^{2}}{2}}e^{-\frac{(1-\gamma)x^{2}}{4}} =e^{-\frac{(\gamma+1)x^{2}}{4}}$ $\displaystyle\sim\mathcal{N}(0,\frac{2}{\gamma+1}).$ $$

We consider the simple variance-exploding diffusion defined by the SDE

$$ $dx=\sqrt{t}dw.$ $$

The DDIM sampler is a discretization of the reverse ODE

$$ $\frac{dx}{dt}=-\frac{1}{2}\nabla_{x}\log p_{t}(x),$ $$

and the DDPM sampler is a discretization of the reverse SDE

$$ $dx=-\nabla_{x}\log p_{t,\gamma}(x)dt+d\overline{w}.$ $$

For $\textsf{CFG}_{\textsf{DDIM}}$ or $\textsf{CFG}_{\textsf{DDPM}}$, we replace the score with CFG score $\nabla_{x}\log p_{t,\gamma}(x)$.

During training we run the forward process until some time $t=T$, at which point we assume it is fully-noised, so that approximately

$$ $p_{T}(x|c=0)\sim\mathcal{N}(0,T)$ $$

(in this case the exact distribution $p_{T}(x|c=0)\sim\mathcal{N}(0,T+1)$ so we need to choose $T\gg 1$ to ensure sufficient terminal noise). At inference time we choose an initial sample $x_{T}\sim\mathcal{N}(0,T)$ and run $\textsf{CFG}_{\textsf{DDIM}}$ from $t=T\to 0$ to obtain a final sample $x_{0}$.

#### CFG DDIM subscript CFG DDIM \textsf{CFG}_{\textsf{DDIM}} CFG start_POSTSUBSCRIPT DDIM end_POSTSUBSCRIPT

For Counterexample 1, the $\textsf{CFG}_{\textsf{DDIM}}$ ODE has a closed-form solution (derivation in section A.5):

$$ $\displaystyle\textsf{CFG}_{\textsf{DDIM}}:\quad\frac{dx}{dt}$ $\displaystyle=-\frac{1}{2}\nabla_{x}\log p_{t,\gamma}(x)$ $\displaystyle=x_{t}\left(\frac{\gamma}{2(1+t)}+\frac{(1-\gamma)}{2(2+t)}\right)$ $\displaystyle\implies x_{t}$ $\displaystyle=x_{T}\sqrt{\frac{(t+1)^{\gamma}(t+2)^{1-\gamma}}{(T+1)^{\gamma}( T+2)^{1-\gamma}}}.$ $$

That is, for a particular initial sample $x_{T}$, $\textsf{CFG}_{\textsf{DDIM}}$ produces the sample $x_{t}$ at time $t$. Evaluating at $t=0$ and taking the limit as $T\to\infty$ yields the ideal denoised $x_{0}$ sampled by $\textsf{CFG}_{\textsf{DDIM}}$ given an initial sample $x_{T}$:

$$ $\displaystyle\widehat{x}_{0}^{\text{$\textsf{CFG}_{\textsf{DDIM}}$}}(x_{T})$ $\displaystyle=x_{T}\sqrt{\frac{2^{1-\gamma}}{(T+1)^{\gamma}(T+2)^{1-\gamma}}}$ $\displaystyle\to x_{T}\sqrt{\frac{2^{1-\gamma}}{T}}\quad\text{as }T\to\infty.$ $$

To get the denoised distribution obtained by reverse-sampling with $\textsf{CFG}_{\textsf{DDIM}}$, we need to average over the distribution of $x_{T}$:

$$ $\displaystyle\mathop{\mathbb{E}}_{x_{T}\sim\mathcal{N}(0,T)}[\widehat{x}_{0}^{ \text{$\textsf{CFG}_{\textsf{DDIM}}$}}(x_{T})]=\mathcal{N}(0,T\frac{2^{1- \gamma}}{T})=\mathcal{N}\left(0,2^{1-\gamma}\right).$ $$

which is equation 12 in the main text.

#### CFG DDPM subscript CFG DDPM \textsf{CFG}_{\textsf{DDPM}} CFG start_POSTSUBSCRIPT DDPM end_POSTSUBSCRIPT

$\textsf{CFG}_{\textsf{DDPM}}$ also has a closed-form solution (derived in section A.5):

$$ $\displaystyle dx$ $\displaystyle=-\nabla_{x}\log p_{t,\gamma}(x)dt+d\overline{w}$ $\displaystyle=x\left(\frac{\gamma}{(1+t)}+\frac{(1-\gamma)}{(2+t)}\right)dt+d \overline{w}$ $\displaystyle\implies x(t)$ $\displaystyle=x_{T}\frac{(1+t)^{\gamma}(2+t)^{1-\gamma}}{(1+T)^{\gamma}(2+T)^{ 1-\gamma}}+(1+t)^{\gamma}(2+t)^{1-\gamma}\sqrt{\frac{1}{2\gamma-1}}\sqrt{\left (\frac{t+1}{t+2}\right)^{1-2\gamma}-\left(\frac{T+1}{T+2}\right)^{1-2\gamma}}\xi.$ $$

Similar to the $\textsf{CFG}_{\textsf{DDIM}}$ argument, we can obtain the final denoised distribution as follows:

$$ $\displaystyle\widehat{x}_{0}^{\text{$\textsf{CFG}_{\textsf{DDPM}}$}}(x_{T})$ $\displaystyle=x_{T}\frac{2^{1-\gamma}}{(1+T)^{\gamma}(2+T)^{1-\gamma}}+2^{1- \gamma}\sqrt{\frac{1}{2\gamma-1}}\sqrt{2^{2\gamma-1}-\left(\frac{T+1}{T+2} \right)^{1-2\gamma}}\xi$ $\displaystyle\to x_{T}\frac{2^{1-\gamma}}{T}+\sqrt{\frac{2-2^{2-2\gamma}}{2 \gamma-1}}\xi\quad\text{as }T\to\infty$ $\displaystyle\implies\mathop{\mathbb{E}}_{x_{T}\sim\mathcal{N}(0,T)}[\widehat{ x}_{0}^{\text{$\textsf{CFG}_{\textsf{DDPM}}$}}(x_{T})]$ $\displaystyle=\mathcal{N}\left(0,T\left(\frac{2^{1-\gamma}}{T}\right)^{2}+ \frac{2-2^{2-2\gamma}}{2\gamma-1}\right)$ $\displaystyle\to\mathcal{N}\left(0,\frac{2-2^{2-2\gamma}}{2\gamma-1}\right),$ $$

which is equation 11 in the main text, and for $\gamma\gg 1$ becomes approximately

$$ $\mathop{\mathbb{E}}_{x_{T}\sim\mathcal{N}(0,T)}[\widehat{x}_{0}^{\text{$ \textsf{CFG}_{\textsf{DDPM}}$}}(x_{T})]\approx\mathcal{N}\left(0,\frac{2}{2 \gamma-1}\right).$ $$

In Figure 6, we confirm results (11, 12) empirically.

### A.2 Counterexample 2

Counterexample 2 (10) is a Gaussian mixture with equal weights and variances.

$$ $\displaystyle c$ $\displaystyle\in\{0,1\},\quad p(c=0)=\frac{1}{2}$ $\displaystyle p_{0}(x_{0}|c)$ $\displaystyle\sim\mathcal{N}(\mu^{(c)},1),\quad\mu^{(0)}=-\mu,\quad\mu^{(1)}=\mu$ $\displaystyle p_{0}(x_{0})$ $\displaystyle\sim\frac{1}{2}p_{0}(x_{0}|c=0)+\frac{1}{2}p_{0}(x_{0}|c=1).$ $$

We noted in the main text that if $\mu$ is sufficiently large enough that the clusters are approximately disjoint, and $\gamma\geq 1$, then $p_{0,\gamma}(x|c)\approx p_{0}(x|c)$. To see this note that

$$ $\displaystyle p_{0}(x_{0})$ $\displaystyle\approx\frac{1}{2}p_{0}(x_{0}|0)\mathbbm{1}_{x>0}+\frac{1}{2}p_{0 }(x_{0}|1)\mathbbm{1}_{x>0}$ $\displaystyle p_{0,\gamma}(x|c)$ $\displaystyle\propto p_{0}(x|c)^{\gamma}p_{0}(x)^{1-\gamma}$ $\displaystyle=p_{0}(x)\left(\frac{p_{0}(x|c)}{p_{0}(x)}\right)^{\gamma}$ $\displaystyle\propto p_{0}(x)\left(\mathbbm{1}_{\text{sign}(x)=\mu^{(c)}} \right)^{\gamma}$ $\displaystyle\approx p_{0}(x|c)\quad\text{for }\gamma\geq 1.$ $$

However, $p_{t,\gamma}(x|c)\neq p_{t}(x|c)$ since the noisy distributions do overlap/interact.

We don’t have complete closed-form solutions for this problem like we did for Counterexample 1. We have the solution for conditional DDIM for the basic VE process $dx=dw$ (using the results from the previous section):

$$ $\displaystyle\text{DDIM on $p_{t}(x|c)$: }\frac{dx}{dt}$ $\displaystyle=-\frac{1}{2}\nabla_{x}\log p_{t}(x|c)$ $\displaystyle=-\frac{1}{2(1+t)}(\mu^{(c)}-x_{t})$ $\displaystyle\implies x(t)$ $\displaystyle=\mu^{(c)}+(x_{T}-\mu^{(c)})\sqrt{\frac{1+t}{1+T}},$ $$

but otherwise have to rely on empirical results. We do however have access to the ideal conditional and unconditional denoisers via the scores (Appendix A.6):

$$ $\displaystyle\nabla_{x}\log p_{t}(x|c)$ $\displaystyle=-\frac{1}{2(1+t)}(\mu^{(c)}-x_{t})$ $\displaystyle\nabla_{x}\log p_{t}(x)$ $\displaystyle=\frac{\nabla_{x}p_{t}(x)}{p_{t}(x)}=\frac{\frac{1}{2}\sum_{c=0,1 }\nabla_{x}p_{t}(x|c)}{p_{t}(x)}.$ $$

### A.3 Counterexample 3

We consider a 3-cluster problem to investigate why $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$ often appear similar in practice despite being different in theory. Counterexample 3 (10) is a Gaussian mixture with equal weights and variances. We vary the variance to investigate its effect on CFG.

$$ $\displaystyle c$ $\displaystyle\in\{0,1,2\},\quad p(c)=\frac{1}{3}\quad\forall c$ $\displaystyle p_{0}(x_{0}|c)$ $\displaystyle\sim\mathcal{N}(\mu^{(c)},\sigma),\quad\mu^{(0)}=-3,\quad\mu^{(1) }=0,\quad\mu^{(2)}=3$ $\displaystyle p_{0}(x_{0})$ $\displaystyle\sim\frac{1}{3}p_{0}(x_{0}|c=0)+\frac{1}{3}p_{0}(x_{0}|c=1)+\frac {1}{3}p_{0}(x_{0}|c=2).$ $$

We run $\textsf{CFG}_{\textsf{DDIM}}$ and $\textsf{CFG}_{\textsf{DDPM}}$ with $\gamma=3$, for $\sigma=1$ and $\sigma=2$. Results are shown in Figure 6.

### A.4 Generalization Example 4

We consider a multi-cluster problem to explore the impact of guidance on generalization:

$$ $\displaystyle p_{0}(x)$ $\displaystyle\sim\mathcal{N}(0,10)$ $\displaystyle p_{0}(x|c=0)$ $\displaystyle\sim\sum_{i}w_{i}\mathcal{N}(\mu_{i},\sigma)$ (19) $\displaystyle\mu$ $\displaystyle=(-3,-2.5,-2,-1.5,-1,-0.5,0,0.5,1,1.5,2,2.5)$ $\displaystyle w_{i}$ $\displaystyle=0.0476\quad\forall i\neq 6;\quad w_{6}=0.476$ $\displaystyle\sigma$ $\displaystyle=0.1$ $$

Note that the unconditional distribution is wide enough to be essentially uniform within the numerical support of the conditional distribution. The conditional distribution is a GMM with evenly spaced clusters of equal variance, and all equal weights, except for a “dominant" cluster in the middle with higher weight. The results are shown in Figure 4.

### A.5 Closed-form ODE/SDE solutions

First, we want to solve equations of the general form $\frac{dx}{dt}=-a(t)x+b(t)$, which will encompass the ODEs and SDEs of interest to us. All we need for the ODEs is the special $b(t)=a(t)c$, which is easier.

The main results are

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=a(t)(c-x)$ $\displaystyle\implies x(t)$ $\displaystyle=c+(x_{T}-c)e^{A(T)-A(t)}$ (20) $\displaystyle\text{where }A(t)$ $\displaystyle=\int a(t)dt$ $$

and

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=-a(t)x+b(t)$ $\displaystyle\implies x(t)$ $\displaystyle=e^{-A(t)}(B(t)-B(T))+x_{T}e^{A(T)-A(t)}$ (21) $\displaystyle\text{where }A(t)$ $\displaystyle=\int a(t)dt,\quad B(t)=\int e^{A(t)}b(t)dt.$ $$

First let’s consider the special case $b(t)=a(t)c$, which is easier. We can solve it (formally) by separable equations:

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=a(t)(c-x)$ $\displaystyle\implies\int\frac{1}{c-x}dx$ $\displaystyle=\int a(t)dt=A(t)$ $\displaystyle\implies-\log(c-x)$ $\displaystyle=A(t)+C$ $\displaystyle\implies c-x$ $\displaystyle=e^{-A(t)-C}$ $\displaystyle\implies x(t)$ $\displaystyle=c+Ce^{-A(t)}.$ (22) $$

Next we need to apply initial conditions to get the right constants.
Remembering that we are actually sampling backward in time from initialization $x_{T}$, we can solve for the constant $C$ as follows, to obtain result (20):

$$ $\displaystyle x_{T}$ $\displaystyle=c+Ce^{-A(T)}$ $\displaystyle\implies C$ $\displaystyle=e^{A(T)}(x_{T}-c)$ $\displaystyle\implies x(t)$ $\displaystyle=c+(x_{T}-c)e^{A(T)-A(t)}.$ $$

We will apply this result to $\textsf{CFG}_{\textsf{DDIM}}$ shortly, but for now we note that for a VE diffusion $dx=\sqrt{t}dw$ on a Gaussian data distribution $p_{0}(x)\sim\mathcal{N}(\mu,\sigma)$ the above result implies the exact DDIM dynamics:

$$ $\displaystyle p_{t}(x)\sim\mathcal{N}(\mu,\sigma^{2}+t)$ $\displaystyle\text{DDIM on $p_{t}(x)$: }\frac{dx}{dt}$ $\displaystyle=-\frac{1}{2}\nabla_{x}\log p_{t}(x)$ $\displaystyle=-\frac{1}{2(\sigma^{2}+t)}(\mu-x)$ $\displaystyle A(t)$ $\displaystyle=-\frac{1}{2}\log(\sigma^{2}+t)$ $\displaystyle\implies x_{t}$ $\displaystyle=\mu+(x_{T}-\mu)e^{A(T)-A(t)}$ $\displaystyle=\mu+(x_{T}-\mu)\sqrt{\frac{\sigma^{2}+t}{\sigma^{2}+T}}.$ $$

(which makes sense since $x_{t=T}=x_{T}$ and $\frac{\sqrt{\sigma^{2}}}{\sqrt{\sigma^{2}+T}}\approx 0\implies x_{t=0}\approx\mu$).

Now let’s return to the general problem with arbitrary $b(t)$ (we need this for the SDEs). We can use an integrating factor to get a formal solution:

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=-a(t)x+b(t)$ Integrating factor: $\displaystyle e^{A(t)},\quad A(t)=\int a(t)dt$ $\displaystyle\frac{d}{dt}(x(t)e^{A(t)})$ $\displaystyle=\left(x^{\prime}(t)+a(t)x(t)\right)e^{A(t)}$ $\displaystyle=b(t)e^{A(t)}$ $\displaystyle\implies e^{A(t)}x(t)$ $\displaystyle=\int e^{A(t)}b(t)dt+C$ $\displaystyle\implies x(t)$ $\displaystyle=e^{-A(t)}\int e^{A(t)}b(t)dt+Ce^{-A(t)}.$ (23) $$

Note that if $b(t)=a(t)c$ this reduces to (22):

$$ $\displaystyle\int e^{-A(t)}e^{A(t)}b(t)dt$ $\displaystyle=ce^{-A(t)}\int a(t)e^{A(t)}dt=c$ $\displaystyle\implies x(t)$ $\displaystyle=c+Ce^{-A(t)}.$ $$

Again, we need to apply boundary conditions to get the constant, and remember that we are actually sampling backward in time from initialization $x_{T}$ to obtain result (21):

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=-a(t)x+b(t)$ $\displaystyle x_{T}$ $\displaystyle=e^{-A(T)}B(T)+Ce^{-A(T)},\quad B(t):=\int e^{A(t)}b(t)dt$ $\displaystyle\implies C$ $\displaystyle=e^{A(T)}x_{T}-B(T)$ $\displaystyle\implies x(t)$ $\displaystyle=e^{-A(t)}B(t)+(e^{A(T)}x_{T}-B(T))e^{-A(t)}$ $\displaystyle=e^{-A(t)}(B(t)-B(T))+x_{T}e^{A(T)-A(t)}.$ $$

Note that for $b(t)=a(t)c$ this reduces (20):

$$ $\displaystyle b(t)$ $\displaystyle=a(t)c\implies B(t)=ce^{A(t)}$ $\displaystyle\implies x(t)$ $\displaystyle=-ce^{-A(t)}(e^{A(t)}-e^{A(T)})+x_{T}e^{A(T)-A(t)}$ $\displaystyle=c+(x_{T}-c)e^{A(T)-A(t)}.$ $$

#### Counterexample 1 solutions

To solve the $\textsf{CFG}_{\textsf{DDIM}}$ ODE for Counterexample 1 (Equation 10) we apply result (20):

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=a(t)(c-x)\implies x(t)=c+(x_{T}-c)e^{A(T)-A(t)}$ $\displaystyle a(t)$ $\displaystyle=-\frac{\gamma}{2(1+t)}-\frac{(1-\gamma)}{2(2+t)},\quad c=0$ $\displaystyle A(t)$ $\displaystyle=-\frac{1}{2}\int\frac{\gamma}{(1+t)}+\frac{(1-\gamma)}{(2+t)}dt$ $\displaystyle=-\frac{1}{2}(\gamma\log(t+1)+(\gamma-1)\log(t+2))$ $\displaystyle\implies x_{t}$ $\displaystyle=x_{T}\sqrt{\frac{(t+1)^{\gamma}(t+2)^{1-\gamma}}{(T+1)^{\gamma}( T+2)^{1-\gamma}}}.$ $$

To solve the $\textsf{CFG}_{\textsf{DDPM}}$ SDE for Counterexample 1 (Equation 10), we first apply (21) to the SDE with $b(t)=-\xi(t)$:

$$ $\displaystyle\frac{dx}{dt}$ $\displaystyle=-a(t)x-\xi(t),\quad\langle\xi(t)\rangle=0,\quad\langle\xi(t),\xi (t^{\prime})\rangle=\delta(t-t^{\prime})$ $\displaystyle\implies x(t)$ $\displaystyle=x_{T}e^{A(T)-A(t)}+e^{-A(t)}(B(t)-B(T)),\quad A(t)=\int a(t)dt, \quad B(t)=-\int e^{A(t)}\xi(t)dt$ $\displaystyle=x_{T}e^{A(T)-A(t)}+e^{-A(t)}\sqrt{\int_{t}^{T}e^{2A(t)}dt}\xi.$ $$

Now, plugging in the DDPM drift term we find that

$$ $\displaystyle a(t)$ $\displaystyle=-\frac{\gamma}{(1+t)}-\frac{(1-\gamma)}{(2+t)}$ $\displaystyle A(t)$ $\displaystyle=-\gamma\log(1+t)-(1-\gamma)\log(2+t)$ $\displaystyle e^{A(t)}$ $\displaystyle=(1+t)^{-\gamma}(2+t)^{-1+\gamma}$ $\displaystyle\int e^{2A(t)}dt$ $\displaystyle=\int(1+t)^{-2\gamma}(2+t)^{-2+2\gamma}dt$ $\displaystyle=-\frac{1}{2\gamma-1}\left(\frac{t+1}{t+2}\right)^{1-2\gamma}$ $\displaystyle x(t)$ $\displaystyle=x_{T}e^{A(T)-A(t)}+e^{-A(t)}\sqrt{\int_{t}^{T}e^{2A(t)}dt}\xi$ $\displaystyle=x_{T}\frac{(1+t)^{\gamma}(2+t)^{1-\gamma}}{(1+T)^{\gamma}(2+T)^{ 1-\gamma}}+(1+t)^{\gamma}(2+t)^{1-\gamma}\sqrt{\frac{1}{2\gamma-1}}\sqrt{\left (\frac{t+1}{t+2}\right)^{1-2\gamma}-\left(\frac{T+1}{T+2}\right)^{1-2\gamma}}\xi.$ $$

### A.6 Exact Denoiser for GMM

For the experiments in Figure 2, we used an exact denoiser, for which we require exact conditional and unconditional scores. Exact scores are available for any GMM as follows. This is well-known (e.g. karras2024guiding) but repeated here for convenience.

$$ $\displaystyle p(x)=\sum w_{i}\phi(x;\mu_{i},\sigma_{i}),$ $\displaystyle\quad\text{where}\quad\phi(x;\mu,\sigma^{2}):=\frac{1}{\sqrt{2\pi }\sigma}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}$ $\displaystyle\implies\nabla\log p(x)$ $\displaystyle=\frac{\nabla p(x)}{p(x)}$ $\displaystyle=\frac{\sum w_{i}\nabla\phi(\mu_{i},\sigma_{i})}{\sum w_{i}\phi( \mu_{i},\sigma_{i})}$ $\displaystyle=-\frac{\sum w_{i}\left(\frac{x-\mu_{i}}{\sigma_{i}^{2}}\right) \phi(x;\mu_{i},\sigma_{i}^{2})}{\sum w_{i}\phi(\mu_{i},\sigma_{i})}.$ $$

## Appendix B PCG SDE

We want to show that the SDE limit of Algorithm 1 with $K=1$ is

$$ $\displaystyle dx$ $\displaystyle=\Delta\textsf{DDIM}(x,t)+\Delta\textsf{LD}_{\textsf{G}}(x,t, \gamma).$ $$

To see this, note that
a single iteration of Algorithm 1 with $K=1$ expands to

$$ $\displaystyle x_{t}$ $\displaystyle=x_{t+{\Delta{t}}}\underbrace{-\frac{1}{2}\beta_{t}(x_{t+{\Delta{ t}}}-\nabla\log p_{t+{\Delta{t}}}(x_{t+{\Delta{t}}}|c)){\Delta{t}}}_{\text{ DDIM step on }p_{t+{\Delta{t}}}(x+{\Delta{t}}|c)}+\underbrace{\frac{\beta_{t} \Delta{t}}{2}\nabla\log p_{t,\gamma}(x_{t}|c)+\sqrt{\beta_{t}\Delta{t}} \mathcal{N}(0,I_{d})}_{\text{Langevin dynamics on }p_{t,\gamma}(x|c)}$ $\displaystyle\implies dx$ $\displaystyle=\lim_{{\Delta{t}}\to 0}x_{t}-x_{t+{\Delta{t}}}=\underbrace{- \frac{1}{2}\beta_{t}(x_{t}-\nabla\log p_{t}(x_{t}|c))dt}_{\Delta\textsf{DDIM}( x,t)}+\underbrace{\frac{1}{2}\beta_{t}\nabla\log p_{t,\gamma}(x_{t}|c)dt+\sqrt {\beta_{t}}d\overline{w}}_{\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)}.$ $$

This concludes the proof.

A subtle point in the argument above is that $\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)$ represents the result of the Langevin step in the PCG corrector update, rather than the differential of an SDE. In Algorithm 1, $t$ remains constant during the LD iteration, and so the SDE corresponding to the LD iteration is

$$ $\displaystyle dx=\frac{1}{2}\beta_{t}\nabla\log p_{t,\gamma}(x_{t}|c)ds+\sqrt{ \beta_{t}}d\overline{w},$ (24) $$

where $s$ is an LD time-axis that is distinct from the denoising time $t$, which is fixed during the LD iteration. Thus $\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)$ is not the differential of (24) (the difference is $dt$ vs $ds)$. However, when we take an LD step of length $dt$ as required for the PCG corrector, the result is

$$ $\displaystyle\int_{0}^{dt}-\frac{\beta_{t}}{2}\nabla\log p_{t,\gamma}ds+\sqrt{ \beta_{t}}d\overline{w}=-\frac{\beta_{t}}{2}\nabla\log p_{t,\gamma}dt+\sqrt{ \beta_{t}}d\overline{w}=\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma),$ $$

so $\Delta\textsf{LD}_{\textsf{G}}(x,t,\gamma)$ represents the result of the PCG corrector update in the limit as ${\Delta{t}}\to 0$.

## Appendix C Additional Samples

Figure: Figure 7: Effect of Langevin Dynamics. PCG generations with $\gamma=1$ (no guidance) fixed and number of Langevin steps $K$ varied. The prompt is “photograph of a panda eating pizza”. Increasing the number of Langevin steps can qualitatively improve image quality, even without guidance.
Refer to caption: extracted/5810172/fig/increasing_ld_steps.png

## Appendix D Algorithms

Figure: Algorithm 2 $\textsf{PCG}_{\textsf{DDIM}}$, explicit

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The following two recent works are related to guidance in diffusion models; they focus on mixture models. "What does guidance do? A fine-grained analysis in a simple setting" and "Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models".

Algorithm 1 states that Line 4 is a DDIM step. From my understanding, DDIM (as well as DDPM) uses an exponential integrator to discretize the backward ODE (SDE). Line 4 is a Euler discretization. Some discussion might be needed.

What do we gain from writing out the SDE limit of PCG?

Are there practical reasons to sample from the gamma-powered distribution? I believe the gamma-powered distribution comes from the classifier-free guidance. In practice, people only aim to promote label alignment and keep high sample fidelity. Is it possible to go beyond the gamma-powered distribution?
soundness: 3
strengths: This paper is well written and the results appear to be correct and sound.

I am particularly appreciative of the discussions in Section 5, not only introduces relevant literature, but also touches on limitations and future directions.

Understanding classifier-free guidance from a theoretical perspective is an important direction.
summary: This paper attempts to understand classifier-free guidance from a theoretical perspective. A special characteristic of classifier-free guidance is that it introduces a strength parameter $\gamma$ so that the plug-in score function is not precisely $\nabla \log p_t(x | y)$. Although practical results demonstrate promises of this methodology, its theoretical analysis is still largely missing. This paper presents new understanding of classifier-free guidance by first pointing out that the terminal distribution is hard to find. From my reading, this result is relatively a minor contribution. More interesting results come when connecting classifier-free guidance to predictor-corrector algorithm.
weaknesses: Practical implication of the study may be limited.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: ### 1. Whether your method can boost the performance of the diffusion model compared to DDPM, DDIM, or the consistency model.

### 2. What's the benefit of your understanding of CFG? Whether your understanding of CFG can benefit the theory results of CFG in [Fu24]?

[Fu24] Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theory
soundness: 3
strengths: ### 1. Theoretical Foundation: By framing CFG within a mathematical context, the paper provides a rigorous basis for understanding its behavior and optimizing its use in diffusion models.

### 2. Experiment Validation: The paper provides the experiments to support its methodology.
summary: This paper provides a comprehensive, well-founded exploration of classifier-free guidance, establishing it as a viable, efficient alternative to classifier-based guidance methods. By grounding CFG in a predictor-corrector framework, the paper not only enhances understanding of diffusion models but also opens new paths for controlling generative models with minimal complexity.
weaknesses: ### 1. The paper explains the classifier-free guidance, but I did not see whether your method can boost the performance of the diffusion model compared to DDPM, DDIM, or the consistency model.

### 2. I do not see the benefit of your understanding of CFG. Whether your understanding of CFG can benefit the theory results of CFG?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Is it always true that  a larger $\gamma$ and more Langevin dynamic steps in the corrector can lead to sharper distribution?
soundness: 2
strengths: 1. The paper disproves the misconcepts about CFG using counterexamples.
2. The paper provides a new understanding of CFG from the perspective of predictor-corrector guidance.
summary: The paper aims to investigate the theoretical foundations of classifier-free guidance (CFG). It disproves common misconcepts by  using counterexamples to show that CFG does not generate gamma-powered distribution, and CFG interacts differently with DDPM and DDIM. The  paper shows that CFG is equivalent to a particular kind of predictor-corrector that combines one step of DDIM denoiser with one step of Langevin dynamics in the gamma-powered distribution.
weaknesses: 1. In page 3, the authors state that `` This gives a principled way to interpret CFG: it is implicitly an annealed
Langevin dynamics''.  What is the exact annealing path of the associated annealed Langevin dynamics? It seems not clear to me that CFG can be directly associated with  annealed Langevin dynamics as the predictor and corrector correspond to different limiting distributions and the corrector take only one Langevin dynamics. 

2. The interpretations of Theorem 1 and 2 are not clear stated. Is CFG-DDIM always tends to be sharper than CFG-DDPM, or it just because the special construction used in Theorem 1 and 2?

3. What is the potential usefulness of the derived results in further theoretical analysis of diffusion model?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: * Please provide the authors' responses to the points listed under "Weaknesses".
* Algorithm 2 states that the noise prediction model uses the same timestep for both the DDIM step and the Langevin dynamics step. Is this correct?
* Minor comments
  * I believe that Eq. 2 should be expressed as a proportional relationship.
  * In Line 131, the paper states that it primarily considers the VP diffusion process, but the counterexamples seem to primarily focus on the VE diffusion process.
soundness: 3
strengths: * The paper points out that the theoretical understanding of CFG is lacking, considering its widespread practical use. The analysis of how different distributions correspond to different guidance scales could be helpful for practical applications.

* The statement of CFG with DDPM through predictor-corrector sampling, where Langevin dynamics serve as the corrector, is intuitive and reasonable.
summary: This paper focuses on the theoretical understanding of classifier-free guidance (CFG), a widely used technique in conditional sampling with diffusion models. The authors argue that the theory of CFG has been somewhat misunderstood, presenting counterexamples using 1D toy models to support their claim. They show that CFG can be explained by a predictor-corrector (PC) sampling algorithm with different annealing distributions. In particular, they introduce the predictor-corrector guidance (PCG) and suggest that CFG with DDPM sampling is equivalent to PCG with DDIM sampling. In this framework, the predictor is set as DDIM and the corrector is set as Langevin dynamics with a gamma-powered distribution.
weaknesses: * The paper provides only informal theorems, so it is unclear what specific statements the authors intend to make within the scope of their work.
  * The theorems are incomplete and difficult to fully understand. In particular, the notation used in the statements is not well-defined (e.g., what is meant by c=0?), and the assumptions necessary to satisfy these theorems are not properly discussed.
  * Specifically, in my opinion, the additional claim in Theorem 1 that the DDIM variant is exponentially sharper than the DDPM variant is based solely on the counterexamples, which may lead to an overstatement in its current form.
  * For Theorem 3, the analysis only covers CFG with DDPM, so a clearer statement regarding this limitation is needed.

* While the relationship between CFG and PCG is explained, the reasons why CFG works are not adequately addressed.
  * There is a lack of sufficient analysis regarding PCG. As the authors themselves note, unlike the conventional PC algorithm, PCG operates with different annealing distributions for the predictor and corrector. Thus, the effectiveness of PCG should be explained with an analysis based on these different annealing distributions. For example, the effect of different annealing distributions on the final sampled distribution is not discussed. I believe this analysis is crucial because it ties into the argument that CFG works with a sampling distribution that deviates from the conventional intuition.
  * In explaining CFG in terms of PCG, the authors assume that the difference in timesteps between the predictor and corrector tends to zero, but the implications of this assumption are not sufficiently discussed or analyzed.
  * In Line 465, the paper mentions that CFG and PCG are qualitatively similar and claims that the results are consistent with the theory. However, looking at the quantitative metrics in Table 1, there appears to be a difference, so I question whether this statement is valid.

* CFG is known to be effective for image-condition alignment. It would be beneficial to include experimental results, such as quantitative metrics for image-text alignment in text-to-image diffusion models such as Stable Diffusion.

Meta-review: The manuscript studies classifier-free guidance for conditional generation. From a theoretical point of view, the manuscript aims to clarifies some misconception of the classifier-free guidance model in the literature, and also provides some new perspective from the connection with prediction-correction schemes. As pointed out during the discussion phase, the misconception is well understood by experts in the field and there have been previous works on this point. While the connection with prediction-correction scheme is interesting, it is unclear to the meta-reviewer whether it leads to practical better schemes. Based on these, the meta-reviewer feels that the manuscript falls short of the bar of acceptance, after carefully reading the manuscript and the discussions.

Decision: Reject