CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models

## Abstract

Abstract Classifier-free guidance (CFG) is a fundamental tool in modern diffusion models for text-guided generation. Although effective, CFG has notable drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image editing; furthermore, high guidance scales, essential for high-quality outputs, frequently result in issues like mode collapse. Contrary to the widespread belief that these are inherent limitations of diffusion models,
this paper reveals that the problems actually stem from the off-manifold phenomenon associated with CFG, rather than the diffusion models themselves.
More specifically, inspired by the recent advancements of diffusion model-based inverse problem solvers (DIS), we reformulate text-guidance as an inverse problem with a text-conditioned score matching loss and
develop CFG++, a novel approach that tackles the off-manifold challenges inherent in traditional CFG. CFG++ features a surprisingly simple fix to CFG, yet it offers significant improvements, including
better sample quality for text-to-image generation, invertibility, smaller guidance scales, reduced mode collapse, etc. Furthermore, CFG++ enables seamless interpolation between unconditional and conditional sampling at lower guidance scales, consistently outperforming traditional CFG at all scales.
Moreover, CFG++ can be easily integrated into the high-order diffusion solvers and naturally extends to distilled diffusion models.
Experimental results confirm that our method significantly enhances performance in
text-to-image generation, DDIM inversion, editing, and solving inverse problems, suggesting a wide-ranging impact and potential applications in various fields that utilize text guidance. Project Page: https://cfgpp-diffusion.github.io/ .

## 1 Introduction

Classifier-free guidance (CFG) (Ho & Salimans, ) forms the key basis of modern text-guided generation with diffusion models (Dhariwal & Nichol, ; Rombach et al., ). Nowadays, it is common practice to train a diffusion model with large-scale paired text-image data (Schuhmann et al., ), so that sampling (i.e. generating) a signal (e.g. image, video) from a diffusion model can either be done unconditionally from $p_{\theta}({\boldsymbol{x}}|\varnothing)\equiv p_{\theta}({\boldsymbol{x}})$, or conditionally from $p_{\theta}({\boldsymbol{x}}|{\boldsymbol{c}})$, where ${\boldsymbol{c}}$ is the text conditioning. Once trained, it seems natural that one would acquire samples from the conditional distribution by simply solving the probability-flow ODE or SDE sampling (Song et al., ; ; Karras et al., ) with the conditional score function. In practice, however, it is observed that the conditioning signal is insufficient when used naively. To emphasize the guidance, one uses the guidance scale $\omega>1$, where the direction can be defined by the direction from the unconditional score to the conditional score (Ho & Salimans, ).

In modern text-to-image (T2I) diffusion models, the guidance scale $\omega$ is typically set within the range of $[5.0,30]$,
referred to as the moderately high range of CFG guidance
(Chen et al., ; Podell et al., ).
The insufficiency in guidance also holds for classifier guidance (Dhariwal & Nichol, ; Song et al., ) so that a scale of 10 was used.
While using a high guidance scale yields higher-quality images with better alignment to the condition, it is also prone to mode collapse, reduces sample diversity, and yields an inevitable accumulation of errors during the sampling process. One example is DDIM inversion (Dhariwal & Nichol, ), a pivotal technique for controllable synthesis and editing (Mokady et al., ), where running the inversion process with $\omega>1.0$ leads to significant compromise in the reconstruction performance (Mokady et al., ; Wallace et al., ).
Another extreme example would be score distillation sampling (SDS) (Poole et al., ), where the guidance scale in the order of a few hundred is chosen. Using such a high guidance scale leads to better asset quality to some extent, but induces blurry and saturated results. Several research efforts have been made to mitigate this downside by exploring methods where using a smaller guidance scale suffices (Wang et al., ; Liang et al., ). Although recent progress in SDS-type methods has reduced the necessary guidance scale to a range that is similar to those of ancestral samplers, using a moderately large $\omega$ is considered an inevitable choice.

In this work, we aim to give an answer to this conundrum by revisiting the geometric view of diffusion models. In particular, inspired by the recent advances in diffusion-based inverse problem solvers (DIS) (Kadkhodaie & Simoncelli, ; Chung et al., ; Song et al., ; Kim et al., ; Chung et al., ),
we reformulate the text guidance as an inverse problem with a text-conditioned score-matching loss and derive a reverse diffusion sampling strategy by utilizing decomposed diffusion sampling (DDS) (Chung et al., ).
This results in a surprisingly simple fix of CFG to the sampling process without any computational overhead. The resulting process, which we call CFG++, works with a small guidance scale, typically $\lambda\in[0.0,1.0]$, that smoothly interpolates between unconditional and conditional sampling, with $\lambda=1.0$ having a similar effect as using CFG sampling with $\omega\sim 12.5$ at 50 neural function evaluation (NFE).
Furthermore, DDIM inversion with CFG++ is invertible up to the discretization error, simplifying image editing.
Comparing CFG++ against CFG shows that we achieve consistently better sample quality for text-to-image (T2I) generation, significantly better DDIM inversion capabilities that lead to enhanced reconstruction and editing, and enabling the incorporation of CFG guidance to diffusion inverse solvers (DIS) (Chung et al., ). While the applications of CFG++ that we show in this work are limited, we believe that our work will have a broad impact that can be applied to all applications that leverage text guidance through the traditional CFG.

Figure: Algorithm 1 Reverse Diffusion with CFG

## 2 Background

Diffusion models.
Diffusion models (Ho et al., ; Song et al., ; Karras et al., ) are generative models designed to learn the reversal of a forward noising process. This process starts with an initial distribution $p_{0}({\boldsymbol{x}})$ where ${\boldsymbol{x}}\in{\mathbb{R}}^{n}$, and progresses towards the standard Gaussian distribution $p_{T}({\boldsymbol{x}})\approx\mathcal{N}(\bm{0},{\bm{I}})$, utilizing forward Gaussian perturbation kernels. Sampling from the data distribution can be performed by solving either the reverse stochastic differential equation (SDE) or the equivalent probability-flow ordinary differential equation (PF-ODE) (Song et al., ).
For example, under the choice $p({\boldsymbol{x}}_{t}|{\boldsymbol{x}}_{0})=\mathcal{N}({\boldsymbol{x}}_{0},
\sigma_{t}^{2}{\bm{I}})$, the generative PF-ODE reads

$$ $\displaystyle d{\boldsymbol{x}}_{t}=-\sigma_{t}\nabla_{{\boldsymbol{x}}_{t}} \log p({\boldsymbol{x}}_{t})\,dt=\frac{{\boldsymbol{x}}_{t}-\mathbb{E}[{ \boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t}]}{\sigma_{t}}\,dt,\,\,\quad{ \boldsymbol{x}}_{T}\sim p_{T}({\boldsymbol{x}}_{T}),$ (1) $$

where
Tweedie’s formula (Efron, ) ${\mathbb{E}}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t}]={\boldsymbol{x}}_{t}+
\sigma_{t}^{2}\nabla_{{\boldsymbol{x}}_{t}}\log p({\boldsymbol{x}}_{t})$ is applied to achieve the second equality. When aiming for a text-conditional diffusion model that can condition on arbitrary ${\boldsymbol{c}}$, one can extend epsilon matching to a conditional one where the condition is dropped with a certain probability (Ho & Salimans, ) so that null conditioning with ${\boldsymbol{c}}=\varnothing$ is possible. The neural network architecture of ${\boldsymbol{\epsilon}}_{\theta}$ is designed so that the condition ${\boldsymbol{c}}$ can modulate the output through cross attention (Rombach et al., ).
For simplicity in notation throughout the paper, we define
${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}:={\boldsymbol{\epsilon}}_{
\theta}({\boldsymbol{x}}_{t},{\boldsymbol{c}})$ and ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}:={\boldsymbol{
\epsilon}}_{\theta}({\boldsymbol{x}}_{t},\varnothing)$ by dropping $\theta$ and ${\boldsymbol{x}}_{t}$.

Extending the result of Tweedie’s formula to the unconditional case under the variance preserving (VP) framework of DDPMs (Ho et al., ), we have

$$ $\displaystyle{\mathbb{E}}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t}, \varnothing]=\hat{\boldsymbol{x}}_{\varnothing}({\boldsymbol{x}}_{t}):=({ \boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{\alpha}_{t}}.$ (2) $$

Leveraging (2), it is common to use DDIM sampling (Song et al., ) to solve the conditional probability-flow ODE (PF-ODE) of the generative process. Specifically, a single iteration reads

$$ $\displaystyle{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor }{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}$ $\displaystyle=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}})/\sqrt{\bar{\alpha}_{t}}$ (3) $\displaystyle{\boldsymbol{x}}_{t-1}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}},$ (4) $$

where ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}:=\hat{\boldsymbol{x}}_{
\varnothing}({\boldsymbol{x}}_{t})={\mathbb{E}}[{\boldsymbol{x}}_{0}|{
\boldsymbol{x}}_{t},\varnothing]$ is the denoised signal by Tweedie’s formula, and
(4) corresponds to the renoising step.
This is repeated for $t=T,T-1,\dots,1$.

For modern diffusion models, it is common to train a diffusion model in the latent space (Rombach et al., ) with
the latent variable ${\boldsymbol{z}}$. While most of our experiments are performed with latent diffusion models (LDM), as our framework holds both for pixel- and latent-diffusion models, we will simply use the notation ${\boldsymbol{x}}$ regardless of the choice.

Classifier free guidance.
For a conditional diffusion,
ho2021classifierfree considered the sharpened posterior distribution $p^{\omega}({\boldsymbol{x}}|{\boldsymbol{c}})\propto p({\boldsymbol{x}})p({
\boldsymbol{c}}|{\boldsymbol{x}})^{\omega}$. Using Bayes rule for some timestep $t$,

$$ $\displaystyle\nabla_{\boldsymbol{x}}\log p^{\omega}({\boldsymbol{x}}_{t}|{ \boldsymbol{c}})=\nabla_{{\boldsymbol{x}}_{t}}\log p({\boldsymbol{x}}_{t})+ \omega(\nabla_{{\boldsymbol{x}}_{t}}\log p({\boldsymbol{x}}_{t}|{\boldsymbol{c }})-\nabla_{{\boldsymbol{x}}_{t}}\log p({\boldsymbol{x}}_{t}))$ (5) $$

Parametrizing the score function with ${\boldsymbol{\epsilon}}_{\theta}$, as in DDPM (ho2020denoising), we have

$$ $\displaystyle{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor }{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega} }({\boldsymbol{x}}_{t}):={\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{ \varnothing}}({\boldsymbol{x}}_{t})+\omega[{\hat{\boldsymbol{\epsilon}}_{ \boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$ (6) $$

where we introduce a compact notation ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}$ that guides the sampling from the sharpened posterior. When sampling with CFG guidance with DDIM sampling, one replaces ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$ with ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}$ for both the Tweedie estimate (3) and the subsequent update step (4), leading to Algorithm 1.

Diffusion model-based inverse problem solvers.
Diffusion model-based inverse problem solvers (DIS) aims to perform posterior sampling from an unconditional diffusion model (kadkhodaie2021stochastic; chung2023diffusion; song2023pseudoinverseguided; kim2024dreamsampler).
Specifically, for a given loss function $\ell({\boldsymbol{x}})$ which often stems from the likelihood, the goal of DIS is to address the optimization problem $\min_{{\boldsymbol{x}}\in{\mathcal{M}}}\ell({\boldsymbol{x}})$, where ${\mathcal{M}}$ represents the clean data manifold sampled from the unconditional distribution $p_{0}({\boldsymbol{x}})$. It is essential to navigate in a way that minimizes cost while also identifying the correct clean manifold.

chung2023diffusion proposed diffusion posterior sampling (DPS), where the updated estimate from the noisy sample ${\boldsymbol{x}}_{t}\in{\mathcal{M}}_{t}$ is constrained to stay on the same noisy manifold ${\mathcal{M}}_{t}$. This is achieved by computing the manifold constrained gradient (MCG) (chung2022improving) on a noisy sample ${\boldsymbol{x}}_{t}\in{\mathcal{M}}_{t}$ as
$\nabla^{mcg}_{{\boldsymbol{x}}_{t}}\ell({\boldsymbol{x}}_{t}):=\nabla_{{
\boldsymbol{x}}_{t}}\ell(\hat{\boldsymbol{x}}_{t})$,
where $\hat{\boldsymbol{x}}_{t}$ is the denoised sample in (2) through Tweedie’s formula (efron2011tweedie).
The resulting algorithm with DDIM (song2020score) can be stated as follows:

$$ $\displaystyle{\boldsymbol{x}}_{t-1}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}-\gamma_{t}\nabla_{{\boldsymbol{x}}_{t}}\ell({\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{x}}_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color [rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}},$ (7) $$

where $\gamma_{t}>0$ denotes the step size. Under the linear manifold assumption (chung2022improving; chung2023diffusion), this allows precise transition to ${\mathcal{M}}_{t-1}$. To mitigate the computational complexity and the instability of neural network backprop, chung2024decomposed shows that (7) can be equivalently represented as

$$ $\displaystyle{\boldsymbol{x}}_{t-1}$ $\displaystyle\simeq\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{x}}_{\varnothing}}-\gamma_{t}\nabla_{{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{x}}_{\varnothing}}}\ell({\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}})\right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}$ (8) $$

under further assumptions on ${\mathcal{M}}$. This method, often referred to
as the decomposed diffusion sampling (DDS), bypasses the computation of the score Jacobian, similar to poole2022dreamfusion, making it stable and suitable for large-scale medical imaging inverse problems (chung2024decomposed). In the following, we leverage the insight from DDS to propose an algorithm to improve upon the CFG algorithm.

Figure: Figure 3: Off-manifold phenomenon of CFG arise from: (a) the typical CFG scale $\omega>1.0$ which leads to extrapolation and deviation from the piecewise linear data manifold, and (b) CFG’s renoising process, which introduces a nonzero offset $\Delta^{\omega}$ from the correct manifold. CFG++ effectively mitigates all these artifacts.
Refer to caption: extracted/5841818/figs/manifold_v2.jpg

## 3 CFG++ : Manifold-Constrained CFG

### 3.1 Derivation of CFG++

Instead of uncritically adopting the sharpened posterior distribution
$p^{\omega}({\boldsymbol{x}}|{\boldsymbol{c}})\propto p({\boldsymbol{x}})p({
\boldsymbol{c}}|{\boldsymbol{x}})^{\omega}$ as introduced by ho2021classifierfree, we adopt a fundamentally different strategy by reformulating text-guidance as an optimization problem. Specifically, our focus is on identifying a loss function
$\ell({\boldsymbol{x}})$ in (8) such that, when minimized under the condition set by the text, enables the reverse diffusion process to generate samples that increasingly satisfy the text condition progressively.

One of the most significant contributions of this paper is to reveal that the text-conditioned score matching loss or alternatively, score distillation sampling (SDS) loss (poole2022dreamfusion) is ideally suited for our purpose. Specifically, we are interested in solving the following inverse problem through diffusion models:

$$ $\displaystyle\min_{{\boldsymbol{x}}\in{\mathcal{M}}}\ell_{sds}(x),\quad\ell_{ sds}({\boldsymbol{x}}):=\|{\boldsymbol{\epsilon}}_{\theta}(\sqrt{\bar{\alpha}_ {t}}{\boldsymbol{x}}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}},{ \boldsymbol{c}})-{\boldsymbol{\epsilon}}\|_{2}^{2}$ (9) $$

This implies that our goal is to identify solutions on the clean manifold
${\mathcal{M}}$ that optimally aligns with the text condition ${\boldsymbol{c}}$.

To avoid the Jacobian computation, in this paper, we attempt to solve (9) through DDS in (8).
The resulting sampling process from reverse diffusion is then given by

$$ $\displaystyle{\boldsymbol{x}}_{t-1}$ $\displaystyle=$ $\displaystyle\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}-\gamma_{t}\nabla_{{\color[rgb]{0.208,0.565,0.953}\definecolor [named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{ \varnothing}}}\ell_{sds}({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}) \right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor [named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{ \varnothing}}.$ (10) $$

By using ${\boldsymbol{x}}_{t}=\sqrt{\bar{\alpha}_{t}}{\boldsymbol{x}}+\sqrt{1-\bar{
\alpha}_{t}}{\boldsymbol{\epsilon}}$ from the clean image ${\boldsymbol{x}}\in{\mathcal{M}}$, we can equivalently write the loss as
$\ell_{sds}({\boldsymbol{x}})=\frac{\bar{\alpha}_{t}}{1-\bar{\alpha}_{t}}\|{
\boldsymbol{x}}-{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}\|^{2}$, which leads to

$$ $\displaystyle{\boldsymbol{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\left({\color[rgb ]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953 }\hat{\boldsymbol{x}}_{\varnothing}}+\lambda({\hat{\boldsymbol{x}}_{ \boldsymbol{c}}}-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}) \right)+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{0.208,0.565,0.953}\definecolor [named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{ \varnothing}}$ (11) $$

where $\lambda:=\frac{2{\bar{\alpha}_{t}}}{{1-\bar{\alpha}_{t}}}\gamma_{t}$.
Using the CFG notation ${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})
:={\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+{\lambda}[
{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\hat{
\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})]$ and ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}+\lambda({\hat{
\boldsymbol{x}}_{\boldsymbol{c}}}-{\color[rgb]{0.208,0.565,0.953}\definecolor[
named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{
\varnothing}})=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\hat{
\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t}))/
\sqrt{\bar{\alpha}_{t}}$, (11) can be equivalently represented as

$$ $\displaystyle{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x} }_{t})$ $\displaystyle=({\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol {\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t}))/\sqrt{\bar{ \alpha}_{t}}$ (12) $\displaystyle{\boldsymbol{x}}_{t-1}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^ {\lambda}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})$ (13) $$

which is summarized in Algorithm 2.
By examining Algorithm 1 and Algorithm 2, we observe that CFG and CFG++ are mostly the same, with a crucial difference in the renoising process. This surprisingly simple fix of utilizing the unconditional noise ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})$ instead of ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({
\boldsymbol{x}}_{t})$ leads to a smoother trajectory of generation, (Fig. 1 bottom) and generation with superior quality (Fig. 1 top).

Although we focus our construction of the solver on DDIM for simplicity, note that DDIM is just one way of reverse sampling. There are other widely-used solvers such as Karras Euler and its variants (karras2022elucidating), DPM-solver (lu2022dpm; lu2022dpmpp), their ancestral variants(^1^11[https://github.com/crowsonkb/k-diffusion](https://github.com/crowsonkb/k-diffusion)), etc. For most widely used solvers up to the second order, a single-step update of solving the unconditional PF-ODE can be represented as

$$ $\displaystyle{\boldsymbol{x}}_{i}={{\color[rgb]{0.208,0.565,0.953}\definecolor [named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{ \varnothing}}({\boldsymbol{x}}_{i-1})}+a_{i}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}({\boldsymbol{x}}_{i-1})+b_{i}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}({\boldsymbol{x}}_{i-2})+c_{i}{\boldsymbol{x}}_{i-1}+d_{i}{ \boldsymbol{\epsilon}},\,{\boldsymbol{\epsilon}}\sim{\mathcal{N}}(0,{\bm{I}}),$ (14) $$

with $d_{i}\neq 0$ when one uses an ancestral sampler. Note that the first right-hand side term in (14) corresponds to the denosing, whereas the rest terms describes the higher-order
corrected version of the renoising process.
As the goal of CFG++ is to optimize the denoising process under the text-guidance while keeping the renoising components equivalent to unconditional sampling, applying CFG++ to the general iteration in (14) simply leads to

$$ $\displaystyle{\boldsymbol{x}}_{i}={\color[rgb]{0.165,0.616,0.561}\definecolor[ named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{ \boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{i-1})+a_{i}{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{i-1})+b_{i}{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{i-2})+c_{i}{\boldsymbol{ x}}_{i-1}+d_{i}{\boldsymbol{\epsilon}},\,{\boldsymbol{\epsilon}}\sim{\mathcal{ N}}(0,{\bm{I}}).$ (15) $$

Moreover, CFG++ naturally extends to distilled diffusion models, such as SDXL-turbo (sauer2023adversarial) and SDXL-lightning (lin2024sdxl), where akin to (15), we use the conditional denoised estimate, but for the rest of the noise components in the Euler solver, we use the unconditional estimate. For details in how to apply CFG++ to various solvers, see Appendix A.

### 3.2 Geometry of CFG++

Mitigating off-manifold phenomenon.
In Fig. 1 (bottom),
we illustrate the evolution of the posterior mean through Tweedie’s formula during the reverse diffusion process. Notably, in the early phases of reverse diffusion sampling under CFG, there is a sudden shift in the image and intense color saturation.
Conversely, CFG++ is free from the undesirable off-manifold phenomenon. In the following, we investigate why this is the case.

Note that the denoised estimate of CFG and CFG++ at time $t$ can be equivalently represented as

$$ $\displaystyle{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor }{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({ \boldsymbol{x}}_{t})=(1-\lambda){\color[rgb]{0.208,0.565,0.953}\definecolor[ named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{ \varnothing}}({\boldsymbol{x}}_{t})+\lambda{\hat{\boldsymbol{x}}_{\boldsymbol{ c}}}({\boldsymbol{x}}_{t}),\quad{\color[rgb]{0.906,0.435,0.318}\definecolor[ named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{ \boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})=(1-\omega){\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})+\omega{\hat{ \boldsymbol{x}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t}).$ (16) $$

$\lambda,\omega\in[0,1]$ facilitates an interpolation. However, with $\omega>1.0$, CFG extrapolates beyond the unconditional and conditional estimates.
Consequently, under the assumption that the clean manifold be piecewise linear (chung2024decomposed), the conditional posterior mean estimates from CFG obtained with a guidance scale outside the range of
$[0,1]$, can easily extend beyond the piecewise linear manifold. This may lead to the estimates potentially “falling off” the data manifold, as depicted by an orange arrow pointing downwards in Fig. 3(a).
Thus, we select $\lambda\in[0,1]$ as the guidance scale for CFG++ to ensure it remains an interpolation between the unconditional and conditional estimate, thus preventing it from ‘falling off’ the clean data manifold.
An additional source of the off-manifold phenomenon in CFG occurs during the transition from the clean manifold ${\mathcal{M}}$ to the subsequent noisy manifold ${\mathcal{M}}_{t-1}$, due to a similar off-manifold phenomenon from the large guidance scale (i.e. extrapolation), illustrated in Fig. 3 (b).

Figure: Figure 4: Text-conditioned score matching loss throughout the reverse diffusion sampling for both CFG and CFG++ in SDXL. Avg. loss computed with 55 prompts from (chen2024pixartalpha).
Refer to caption: extracted/5841818/figs/align_plot3.jpg

Text-Image alignment.
In Fig. 1 (top), and Fig. 9, we observe enhanced text-to-image alignment achieved with CFG++. This enhanced alignment capability is a natural consequence of CFG++, which directly minimizes the text-conditioned score-matching loss as shown in (9). In contrast, CFG indirectly seeks text alignment through the sharpened posterior distribution
$p^{\omega}({\boldsymbol{x}}|{\boldsymbol{c}})\propto p({\boldsymbol{x}})p({
\boldsymbol{c}}|{\boldsymbol{x}})^{\omega}$. Therefore, CFG++ inherently outperforms CFG in terms of text alignment due to its fundamental design principle.
For example, Fig. 4 displays the normalized text-conditioned score matching loss, represented as $(1-\bar{\alpha}_{t})\|{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}
({\boldsymbol{x}}_{t},{\boldsymbol{c}})\|^{2}/\bar{\alpha}_{t}=\|{\boldsymbol{
x}}-{\hat{\boldsymbol{x}}_{\boldsymbol{c}}}\|^{2}$, throughout the reverse diffusion sampling process for both CFG and CFG++. The loss plot associated with CFG shows fluctuations and maintains a noticeable gap compared to CFG++ even after the completion of the reverse diffusion process. In fact, the fluctuation associated
with CFG is also related to the off-manifold issue, and from Fig. 4 we can easily see that
the off-manifold phenomenon is more dominant at early stage of reverse diffusion sampling.
Conversely, the loss trajectory for CFG++ demonstrates a much smoother variation, particularly during the early stages of reverse diffusion.

DDIM inversion.
As discussed in (song2020denoising), the denoising process for unconditional DDIM is approximately invertible, meaning that
${\boldsymbol{x}}_{t}$
can generally be recovered from
${\boldsymbol{x}}_{t-1}$
Specifically, from (3) and (4), we have the following approximate
inversion formula for unconditional DDIM:

$$ $\displaystyle{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor }{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_ {t})$ $\displaystyle=({{\boldsymbol{x}}_{t-1}-\sqrt{1-\bar{\alpha}_{t-1}}{\color[rgb] {0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})})/{\sqrt{\bar {\alpha}_{t-1}}}\simeq({{\boldsymbol{x}}_{t-1}-\sqrt{1-\bar{\alpha}_{t-1}}{ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_ {t-1})})/{\sqrt{\bar{\alpha}_{t-1}}}$ (17) $\displaystyle{\boldsymbol{x}}_{t}$ $\displaystyle=\sqrt{\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})\simeq\sqrt{ \bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({ \boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t}}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1})$ (18) $$

where the approximation arises from ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{
rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol
{x}}_{t-1})$.
A similar inversion procedure has been employed for conditional DDIM inversion under CFG By assuming ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({
\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{
pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{
\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t-1})$, and replacing ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}$ by ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}$ as detailed in
Algorithm 3.

On the other hand, by examining (12) and (13),
we can obtain the following approximate DDIM inversion formula under CFG++:

$$ $\displaystyle{\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x} }_{t})$ $\displaystyle\simeq({{\boldsymbol{x}}_{t-1}-\sqrt{1-\bar{\alpha}_{t-1}}{\color [rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_ {t-1})})/{\sqrt{\bar{\alpha}_{t-1}}}$ (19) $\displaystyle{\boldsymbol{x}}_{t}$ $\displaystyle\simeq\sqrt{\bar{\alpha}_{t}}{\hat{\boldsymbol{x}}_{\boldsymbol{c }}^{\lambda}}({\boldsymbol{x}}_{t})+\sqrt{1-\bar{\alpha}_{t}}{\hat{\boldsymbol {\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t-1})$ (20) $$

where we apply the approximation ${\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})
\simeq{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}
}_{t-1})$ alongside the
usual assumption ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{
rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol
{x}}_{t-1})$.
This formulation underpins the CFG++ guided DDIM inversion algorithm, presented in Algorithm 4.

In practice, for small time step size the error ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{
rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol
{x}}_{t-1})$ is relatively small, so the unconditional DDIM inversion by (17) and (18) lead to relatively insignificant errors.
Unfortunately, the corresponding inversion from conditional diffusion under CFG is quite distorted as noted in (mokady2023null; wallace2023edict).
In fact, this distortion is originated from the inaccuracy of the approximation
${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({
\boldsymbol{x}}_{t})\simeq{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{
pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{
\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t-1})$. More specifically,
even when ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{
rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol
{x}}_{t-1})$,
the approximation error from the CFG is given by

$$ $\displaystyle\boldsymbol{\varepsilon}_{cfg}$ $\displaystyle:={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{ \boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.906,0.435,0.318 }\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{ \epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t-1})=({\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})-{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t-1}))+\omega\big {(}\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})- \delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t-1}) \big{)}$ $\displaystyle\simeq\omega\big{(}\delta{\hat{\boldsymbol{\epsilon}}_{ \boldsymbol{c}}}({\boldsymbol{x}}_{t})-\delta{\hat{\boldsymbol{\epsilon}}_{ \boldsymbol{c}}}({\boldsymbol{x}}_{t-1})\big{)},$ (21) $$

where $\delta{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t}):={
\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[
rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})$ denotes the directional component from
unconditional to the conditional diffusion.
Since the directional component by the text guidance is not negligible,
this error becomes significant for high guidance scale $\omega$.
Accordingly, the guidance scale must
be heavily downweighted in order for inversions on real world images to be stable, thus limiting the strength of edits.
To mitigate this issue, the authors in (mokady2023null; wallace2023edict) developed
null text optimization and coupled transform techniques, respectively.

On the other hand, under the usual DDIM assumption ${\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})\simeq{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{
rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol
{x}}_{t-1})$, the approximation error of CFG++ mainly arises from (20), which is smaller than that of CFG since we have

$$ $\displaystyle\|\boldsymbol{\varepsilon}_{cfg++}\|=\lambda\|\delta{\hat{ \boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-\delta{\hat{ \boldsymbol{\epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t-1})\|<\| \boldsymbol{\varepsilon}_{cfg}\|$ (22) $$

thanks to $\lambda<\omega$.
Therefore, CFG++ significantly improves the DDIM inversion as shown Fig. 1 (middle) for representative results and Sec. 4.2 for further discussions.

**Table 1: Quantitative evaluation of 50NFE DDIM T2I with SD v1.5 on COCO 10k**
|  | $\omega=2.0,\lambda=0.2$ | $\omega=5.0,\lambda=0.4$ | $\omega=7.5,\lambda=0.6$ | $\omega=9.0,\lambda=0.8$ | $\omega=12.5,\lambda=1.0$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\downarrow$ | FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\uparrow$ |
| CFG (ho2021classifierfree) | 13.84 | 0.298 | 15.08 | 0.310 | 17.71 | 0.312 | 20.01 | 0.312 | 21.23 | 0.313 |
| CFG++ (ours) | 12.75 | 0.303 | 14.95 | 0.310 | 17.47 | 0.312 | 19.34 | 0.313 | 20.88 | 0.313 |

## 4 Experimental Results

In this section, we design experiments to show the limitations of CFG and how CFG++ can effectively mitigate these downsides. The main experiments were conducted with SD v1.5 or SDXL with 50 NFE DDIM sampling. In this regime, we searched for the matching guidance values of $\omega$ and $\lambda$ for a fair comparison. We fix $\lambda=0.2,0.4,0.6,0.8,1.0$ and find the $\omega$ values that produce the images that are of closest proximity in terms of LPIPS distance given the same seed. We found that the corresponding values were $\omega=2.0,5.0,7.5,9.0,12.5$, respectively. Some of the experiments were also conduced with distilled model such as
SDXL-turbo, lightning.

### 4.1 Text-to-Image Generation

Using the corresponding scales for $\omega$ and $\lambda$, we directly compare the performance of the T2I task using SD v1.5 and SDXL. In Tab. 1, we report quantitative metrics using 10k images generated from COCO captions (lin2014microsoft). Here,
we observe a constant improvement of the FID metric across all guidance scales (also see Fig. 13 for an apples-to-apples comparison), with approximately the same level of CLIP similarity or better.

**Table 2: Quant. eval. on accelerated T2I sampling**
| Model | Metric | CFG | CFG++ (ours) |
| --- | --- | --- | --- |
| SDXL-Lightning | FID$\downarrow$ | 59.67 | 59.21 |
| CLIP$\uparrow$ | 0.320 | 0.325 |  |
| ImageReward$\uparrow$ | 0.777 | 0.968 |  |
| SD v1.5 (DPM++ 2M) | FID$\downarrow$ | 32.72 | 32.58 |
| CLIP$\uparrow$ | 0.313 | 0.312 |  |

The improvements can also be clearly seen in Fig. 10 (SD v1.5), where the unnatural components of the generated images are corrected. Specifically, we see that unnatural depictions of human hands, and incorrect renderings of the text are corrected in CFG++, a long-standing research question in and of its own (podell2023sdxl; pelykh2024giving; chen2023textdiffuser).
We find that the improvement gain from CFG++ is even more dramatic for distilled diffusion models such as SDXL-{turbo, lightning}. We quantify the results on the same 5k prompts with 6 NFE sampling. In Fig. 5 and Fig. 11, we see significant boosts in the quality of the generated images, which is also depicted in the improvements seen in Tab. 2.

To show the compatibility of CFG++ with higher-order solvers, we experiment with DPM++2M (lu2022dpmpp) solver with 20 NFE. We note that in such low NFE regime, a CFG scale of 5.0 corresponds to the CFG++ scale of 1.0, and we have to slightly extrapolate the value of $\lambda\geq 1.0$ to achieve stronger guidance results. In Tab. 2, we again see that CFG++ yields results that are favorable to standard CFG.

Figure: Figure 5: T2I using SDXL-Lightning, 6 NFE, CFG vs CFG++.
Refer to caption: extracted/5841818/figs/t2i_sdxl_lightning.jpg

### 4.2 Diffusion image Inversion and Editing

Figure: Figure 6: Inversion and editing results. (a) Reconstructed samples after inversion by CFG and CFG++. (b) Quantitative comparison between CFG and CFG++ for reconstruction. (c) Image editing comparison via SDXL.
Refer to caption: extracted/5841818/figs/inversion_edit.jpg

We further explore the effect of CFG++ on DDIM inversion (dhariwal2021diffusion), where the source image is reverted to a latent vector that can reproduce the original image through generative process. DDIM inversion is well-known to break down in the usual CFG setting as CFG magnifies the accumulated error of each inversion step (mokady2023null), violating local linearization assumptions (wallace2023edict). We show that CFG++ mitigates this issue by improving the inversion and image editing capabilities. We evaluate our method following the experimental setups in park2024energy and kim2024dreamsampler.

Diffusion Image Inversion.
Using the matched set of scales for $\omega$ and $\lambda$, we demonstrate the effect of CFG++ on the diffusion image inversion task. Specifically, we reconstruct the images after inversion and evaluate it through PSNR and RMSE, following the methodology of wallace2023edict. In Fig. 6, we illustrate the reconstructed examples (6a) from a real image from COCO data set and computed metrics (6b) for 5k COCO-2014 (lin2014microsoft) validation set. Both qualitative and quantitative evaluations demonstrate a consistent improvement on reconstruction performance induced by CFG++.
Notably, DDIM inversion with CFG++ leads to a consistent reconstruction of the source image across all guidance scales, while DDIM inversion with CFG fails to reconstruct it, as shown in Fig. 14 for another real image.

Image Editing.
Fig. 6c compares the image editing result using CFG and CFG++ followed by image inversion with 5 NFEs. In the image editing stage, a word in the source text highlighted by green color is swapped with the target concept, and this modified text is used as the condition for sampling. CFG++ successfully edits the target concept while preserving other concepts, such as background. In particular, the water splashing in the background, which tends to disappear in the conventional CFG, is maintained through the inversion process by CFG++. This emphasizes CFG++’s superior ability to retain specific scene elements that are frequently lost in the standard CFG approach. Moreover, standard CFG’s disrupted DDIM inversion leads to saturated and less faithful editing results, whereas CFG++ delivers precise and high-fidelity edits.

### 4.3 Text-Conditioned Inverse Problems

Inverse problem involves restoring the original data ${\boldsymbol{x}}$ from a noisy measurement ${\boldsymbol{y}}={\mathcal{A}}({\boldsymbol{x}})+{\boldsymbol{n}}$, where ${\mathcal{A}}$ represents an imaging operator introduces distortions (e.g, Gaussian blur) and ${\boldsymbol{n}}$ denotes the measurement noise. Diffusion inverse solvers (DIS) address this challenge by leveraging pre-trained diffusion models as implicit priors and performing posterior sampling ${\boldsymbol{x}}\sim p({\boldsymbol{x}}|{\boldsymbol{y}})$.
Methods that leverage latent diffusion have gained recent interest (rout2024solving; song2024solving), but leveraging texts for solving these problems remains relatively underexplored (chung2023prompt; kim2023regularization). One of the main reasons for this is that naively using CFG on top of latent DIS leads to diverging samples (chung2023prompt). Several heuristic modifications such as null prompt optimization (kim2023regularization) with modified sampling schemes were needed to mitigate this drawback. This naturally leads to the question: Is it possible to leverage CFG guidance as a plug-and-play component of existing solvers? Here, we answer this question with a positive by showing that CFG++ enables the incorporation of text prompts into a standard solver. Specifically, we focus on comparing the performance of PSLD (rout2024solving) combined with CFG and CFG++ in solving linear inverse problems. This evaluation utilizes the FFHQ (karras2019style) 512x512 dataset and the text prompt "a high-quality photo of a face". Further details about the experimental settings can be found in the Appendix C.

As shown in Tab. 3, our method mostly outperforms both the vanilla PSLD and PSLD with CFG.
The superiority is also evident from the Fig. 7, where CFG++ consistently delivers high-quality reconstructions across all tasks. PSLD with CFG often suffer from artifacts and blurriness. Conversely, CFG++ achieves better fidelity, clearly distinguishing between faces and faithfully reproducing fine details like eyelids and hair texture. For more results, please refer to the Appendix D.

Figure: Figure 7: Qualitative comparison on various inverse problems using PSLD (rout2024solving) under CFG and CFG++. For more results, please refer to the Appendix D.
Refer to caption: extracted/5841818/figs/ldis_main_v3.jpg

**Table 3: Quantitative comparison (FID, LPIPS, PSNR) of PSLD, PSLD with CFG, and PSLD with CFG++ on Latent Diffusion Inverse Solver.**
|  | SR (x8) | Deblur (motion) | Deblur (gauss) | Inpaint |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ |
| PSLD | 46.24 | 0.413 | 24.41 | 97.51 | 0.500 | 21.83 | 41.65 | 0.388 | 26.88 | 10.27 | 9.36 | 30.15 |
| PSLD + CFG | 41.24 | 0.394 | 24.91 | 91.90 | 0.493 | 22.29 | 41.52 | 0.390 | 26.94 | 9.36 | 0.055 | 30.27 |
| PSLD + CFG++ (ours) | 36.58 | 0.385 | 24.87 | 65.67 | 0.482 | 21.93 | 39.85 | 0.400 | 26.90 | 9.78 | 0.052 | 30.31 |

## 5 Related Works and Discussions

kynkaanniemi2024applying observed that applying CFG guidance at the earlier stages of sampling always led to detrimental effects and drastically reduced the diversity of the samples. The guidance at the later stages of sampling had minimal effects. Drawing upon these observations, it was empirically shown that applying CFG only in the limited time interval near the middle led to the best performance. Similar observations were made in the SD community (howard2022; birch2023) in adjusting the guidance scale across $t$. These findings are orthogonal to ours in that these methods keep the sampling trajectory the same and try to empirically adjust the strength of the guidance, while we aim to design a different trajectory.
More recently, bradley2024classifier observed that the CFG score is not a valid denoising direction, showing that in the asymptotic limit, CFG sampling can be considered as a specific type of predictor-corrector (PC) sampling, dubbed PCG. PCG shares a similar spirit with CFG++ in that for the mainstream of sampling, one defers from the use of mixed CFG score (we use unconditional score, PCG uses conditional score) to avoid invalid directions. While PCG derives an interesting connection of PC sampling with CFG, the proposed algorithm does not improve the sampling performance. In contrast, CFG++ improves the sample quality without additional computation overhead.
Since CFG++ attempts to make adjustments of CFG using a linear combination of score functions, it is not surprising to see that CFG++ sampling can be achieved by setting a time varying schedule.
Specifically, by setting a time-varying schedule $\omega_{t}$ as $\omega_{t}=-\gamma_{t}/\xi_{t}$, where $\gamma_{t}=\frac{\sqrt{\bar{\alpha}_{t-1}}\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{
\bar{\alpha}_{t}}},\xi_{t}=\sqrt{1-\bar{\alpha}_{t-1}}-\frac{\sqrt{\bar{\alpha
}_{t-1}}\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}$, one can achieve the same effect with CFG as CFG++. However, this specific choice of time-varying guidance scale has not been reported in the literature, as it is a schedule that is relatively complex to be drawn heuristically. We further show in Appendix B that such choice removes undesirable oscillatory behavior in the evolution of the posterior mean.

## 6 Conclusion

This paper revises the most widespread guidance method, classifier-free guidance (CFG), highlighting that a small and reasonable guidance scale, e.g. $\lambda\in[0.0,1.0]$, might suffice successful guidance with a proper geometric correction. Observing that the original CFG suffers from off-manifold issues during sampling, we propose a simple but surprisingly effective fix. This change translates the conditional guidance from extrapolating to interpolating between unconditional and conditional sampling trajectories, leading to more interpretable guidance in contrast to the conventional CFG which relies on heuristic $\omega$ scaling. Given that CFG++ mitigates off-manifold issues, it may be beneficial for other downstream applications requiring accurate latent denoised estimate representations, e.g. DIS, as demonstrated in our experiments.

#### Acknowledgments

We thank Jinho Chang for the helpful discussions.

## Appendix A Extension of CFG++ to other solvers

In this section, we discuss ways in which we can apply CFG++ to a more diverse set of ODE/SDE solvers. We consider solving the variance exploding (VE) PF-ODE as presented in (1), as re-parametrization VP diffusion models can easily recover the VE formulation, as often implemented in widely used frameworks such as [https://github.com/crowsonkb/k-diffusion](https://github.com/crowsonkb/k-diffusion). Following the notation in lu2022dpmpp, we consider a sequence of timesteps $\{t_{i}\}_{i=0}^{M}$, where $t_{0}=T$ denotes the initial starting point of the reverse sampling (i.e. Gaussian noise).

##### Euler ( karras2022elucidating )

The construction is the same as in DDIM. We include it here for completeness. The update step reads

$$ $\displaystyle{\boldsymbol{x}}_{t_{i+1}}$ $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{i}})+\frac{{\boldsymbol{x}}_{t_{i}}-{\color[rgb] {0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318} \hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t_{i}})}{ \sigma_{t_{i}}}\cdot\sigma_{t_{i+1}},\quad$ (CFG) $\displaystyle{\boldsymbol{x}}_{t_{i+1}}$ $\displaystyle={\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t_{i}})+\frac{{\boldsymbol{x}}_{t_{i}}-{\color[rgb ]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953 }\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{i}})}{\sigma_{t_{i}} }\cdot\sigma_{t_{i+1}},\quad$ (CFG++) $$

##### Euler Ancestral

Euler Ancestral sampler follows Euler sampler, but introduces stochasticity by taking larger steps and then adding a slight amount of noise. Adjusting with CFG++ is straightforward.

$$ $\displaystyle{\boldsymbol{x}}_{t_{i+1}}$ $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{i}})+\frac{{\boldsymbol{x}}_{t_{i}}-{\color[rgb] {0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318} \hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t_{i}})}{ \sigma_{t_{i}}}\cdot(\sigma_{t_{d_{i}}}-\sigma_{t_{i}})+\sigma_{t_{i}}{ \boldsymbol{\epsilon}},\quad$ (CFG) $\displaystyle{\boldsymbol{x}}_{t_{i+1}}$ $\displaystyle={\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t_{i}})+\frac{{\boldsymbol{x}}_{t_{i}}-{\color[rgb ]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953 }\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{i}})}{\sigma_{t_{i}} }\cdot(\sigma_{t_{d_{i}}}-\sigma_{t_{i}})+\sigma_{t_{i}}{\boldsymbol{\epsilon} },\quad$ (CFG++) $$

where $t_{i}>t_{d_{i}}>t_{i+1}$ and ${\boldsymbol{\epsilon}}\sim{\mathcal{N}}(0,{\bm{I}})$.

##### DPM-solver++ 2M ( lu2022dpmpp )

Define $\sigma_{t}:=e^{-t},\,h_{i}:=t_{i}-t_{i-1},$ and $r_{i}:=h_{i-1}/h_{i}$. After initializing ${\boldsymbol{x}}_{t_{0}}$ with Gaussian noise, the first iteration is given by

$$ $\displaystyle{\boldsymbol{x}}_{t_{1}}$ $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{0}})+e^{-h_{1}}({\boldsymbol{x}}_{t_{0}}-{\color [rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol {x}}_{t_{0}}))\quad$ (CFG) $\displaystyle{\boldsymbol{x}}_{t_{1}}$ $\displaystyle={\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t_{0}})+e^{-h_{1}}({\boldsymbol{x}}_{t_{0}}-{ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{0}} ))\quad$ (CFG++) $$

The following iterations by using the standard CFG reads

$$ $\displaystyle\bm{D}_{i}$ $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{i-1}})+\frac{1}{2r_{i}}\left({\color[rgb]{ 0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318} \hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t_{i-1}})-{ \color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol {x}}_{t_{i-2}})\right),$ (23) $\displaystyle{\boldsymbol{x}}_{t_{i}}$ $\displaystyle=e^{-h_{i}}{\boldsymbol{x}}_{t_{i-1}}-(e^{-h_{i}}-1)\bm{D}_{i}.$ (24) $$

Rearranging (23), (24), we can rewrite the update steps as

$$ $\displaystyle{\boldsymbol{x}}_{t_{i}}$ $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{i-1}})-e^{-h_{i}}{\color[rgb]{0.906,0.435,0.318} \definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x} }_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t_{i-1}})+\frac{1-e^{-h_{i}}}{2 r_{i}}\left({\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor} {rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({ \boldsymbol{x}}_{t_{i-1}})-{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{i-2}})\right)+e^{-h_{i}}{\boldsymbol{x}}_{t_{i-1 }}.$ (25) $$

Notice that in order to apply CFG++ to (25), we should only keep the first term as the conditional Tweedie, and use the unconditional estimates for the rest of the components. i.e.

$$ $\displaystyle{\boldsymbol{x}}_{t_{i}}$ $\displaystyle={\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t_{i-1}})-e^{-h_{i}}{\color[rgb]{0.208,0.565,0.953 }\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x }}_{\varnothing}}({\boldsymbol{x}}_{t_{i-1}})+\frac{1-e^{-h_{i}}}{2r_{i}}\left ({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{i-1 }})-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{i-2 }})\right)+e^{-h_{i}}{\boldsymbol{x}}_{t_{i-1}}$ (26) $$

##### DPM-solver++ 2S ( lu2022dpmpp )

With the same choices of $\sigma_{t},h_{i}$, we additionally define the timesteps $\{s_{i}\}_{i=1}^{M}$ with $t_{i}>s_{i+1}>t_{i+1}$. Further, let $r_{i}=\frac{s_{i}-t_{i-1}}{t_{i}-t_{i-1}}$. Using standard CFG, the iteration reads

$$ $\displaystyle\bm{u}_{i}$ $\displaystyle=e^{-r_{i}h_{i}}{\boldsymbol{x}}_{t_{i-1}}+(1-e^{-r_{i}h_{i}}){ \color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol {x}}_{t_{i-1}})$ (27) $\displaystyle{\boldsymbol{x}}_{t_{i}}$ $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t_{i-1}})-e^{-h_{i}}{\color[rgb]{0.906,0.435,0.318} \definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x} }_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t_{i-1}})+\frac{1-e^{-h_{i}}}{2 r_{i}}\left({\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor} {rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}(\bm{u} _{i})-{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol {x}}_{t_{i-1}})\right)+e^{-h_{i}}{\boldsymbol{x}}_{t_{i-1}}$ (28) $$

Applying the general transition rule from CFG to CFG++ as introduced in (15), we can keep all of the Tweedie estimates to be unconditional, and only change the first term of (28), i.e.

$$ $\displaystyle\bm{u}_{i}$ $\displaystyle=e^{-r_{i}h_{i}}{\boldsymbol{x}}_{t_{i-1}}+(1-e^{-r_{i}h_{i}}){ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{i-1 }})$ (29) $\displaystyle{\boldsymbol{x}}_{t_{i}}$ $\displaystyle={\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({ \boldsymbol{x}}_{t_{i-1}})-e^{-h_{i}}{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x} }_{\varnothing}}({\boldsymbol{x}}_{t_{i-1}})+\frac{1-e^{-h_{i}}}{2r_{i}}\left( {\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}(\bm{u}_{i}) -{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t_{i-1 }})\right)+e^{-h_{i}}{\boldsymbol{x}}_{t_{i-1}}.$ (30) $$

For the ancestral version of DPM-solver++ 2S, we can follow the general transition rule, as was shown for Euler $\rightarrow$ Euler Ancestral.

## Appendix B Evolution of the posterior mean: CFG vs. CFG++

Recall that one can equivalently view the evolution of the posterior mean ${\mathbb{E}}[{\boldsymbol{x}}_{0}|{\boldsymbol{x}}_{t}]$ rather than the noisy variables ${\boldsymbol{x}}_{t}$.
In this context, we derive the sequential evolution of conditional posterior mean ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}$ and ${\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{
0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}$ through time $t$ to further understand the underlying behavior of the proposed sampling.

###### Proposition 1 .

Let $d{\boldsymbol{z}}({\boldsymbol{x}}_{t}):={\boldsymbol{z}}({\boldsymbol{x}}_{t}
)-{\boldsymbol{z}}({\boldsymbol{x}}_{t+1})$ denote the discrete time evolution of some random variable ${\boldsymbol{z}}$ at time $t$. Then, the evolution of ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}$ of CFG and ${\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{
0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}$ of CFG++ is given by

$$ $\displaystyle d{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t})$ $\displaystyle=\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}d{ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_ {t})+\omega(\Delta({\boldsymbol{x}}_{t},{\boldsymbol{c}})-\Delta({\boldsymbol{ x}}_{t+1},{\boldsymbol{c}}))$ (31) $\displaystyle d{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t})$ $\displaystyle=\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}} \underbrace{d{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor }{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({ \boldsymbol{x}}_{t})}_{\text{uncond. shift}}+\lambda\underbrace{\Delta({ \boldsymbol{x}}_{t},{\boldsymbol{c}})}_{\text{cond. shift}},$ (32) $$

where $\Delta({\boldsymbol{x}}_{t},{\boldsymbol{c}}):={\hat{\boldsymbol{x}}_{
\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{0.208,0.565,0.953}
\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{x}
}_{\varnothing}}({\boldsymbol{x}}_{t})$.

###### Proof.

We start by writing the iteration from $t+1\rightarrow t$

$$ $\displaystyle{\boldsymbol{x}}_{t}=\sqrt{\bar{\alpha}_{t}}{\color[rgb]{ 0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318} \hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t+1})+\sqrt{ 1+\bar{\alpha}_{t}}{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{ \boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t+1}).$ (33) $$

The Tweedie estimate for the next step is then written as

$$ $\displaystyle{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor }{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}({ \boldsymbol{x}}_{t})$ $\displaystyle=\frac{{\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb] {0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318} \hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})}{ \sqrt{\bar{\alpha}_{t}}}$ (34) $\displaystyle=\frac{\sqrt{\bar{\alpha}_{t}}{\color[rgb]{0.906,0.435,0.318} \definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x} }_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t+1})+\sqrt{1-\bar{\alpha}_{t}} ({\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({ \boldsymbol{x}}_{t+1})-{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{ \boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t}))}{\sqrt{\bar{\alpha}_{t}}}$ (35) $\displaystyle={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t+1})+\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{ \alpha}_{t}}}\Bigl{[}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{ \varnothing}}({\boldsymbol{x}}_{t+1})-{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})+\omega({\hat{\boldsymbol{ \epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t+1})-{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t+1}))$ $\displaystyle\phantom{={\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t+1})+\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{ \alpha}_{t}}}\Bigl{[}{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{ \varnothing}}({\boldsymbol{x}}_{t+1})-{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})}-\omega({\hat{\boldsymbol{ \epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t}))\Bigr{]}.$ (36) $$

Using the relation ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\omega}}({
\boldsymbol{x}}_{t})=-(\sqrt{\bar{\alpha}_{t}}{\color[rgb]{0.906,0.435,0.318}
\definecolor[named]{pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}
}_{\boldsymbol{c}}^{\omega}}({\boldsymbol{x}}_{t})-{\boldsymbol{x}}_{t})/\sqrt
{1-\bar{\alpha}_{t}}$, we have

$$ $\displaystyle d{\color[rgb]{0.906,0.435,0.318}\definecolor[named]{ pgfstrokecolor}{rgb}{0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \omega}}({\boldsymbol{x}}_{t})$ $\displaystyle=\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}d{ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_ {t})+\omega\left({\hat{\boldsymbol{x}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t}) -{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t}) \right)-\omega\left({\hat{\boldsymbol{x}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{ t+1})-{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t+1}) \right).$ (37) $$

Similarly, for CFG++,

$$ $\displaystyle{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor }{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}({ \boldsymbol{x}}_{t})$ $\displaystyle=\frac{{\boldsymbol{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\color[rgb] {0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561} \hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t})} {\sqrt{\bar{\alpha}_{t}}}$ (38) $\displaystyle=\frac{\sqrt{\bar{\alpha}_{t}}{\color[rgb]{0.165,0.616,0.561} \definecolor[named]{pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x} }_{\boldsymbol{c}}^{\lambda}}({\boldsymbol{x}}_{t+1})+\sqrt{1-\bar{\alpha}_{t} }({\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_ {t+1})-{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb} {0.165,0.616,0.561}\hat{\boldsymbol{\epsilon}}_{\boldsymbol{c}}^{\lambda}}({ \boldsymbol{x}}_{t}))}{\sqrt{\bar{\alpha}_{t}}}$ (39) $\displaystyle={\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t+1})+\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{ \alpha}_{t}}}\left[{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{ pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{ \varnothing}}({\boldsymbol{x}}_{t+1})-{\color[rgb]{0.208,0.565,0.953} \definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953}\hat{\boldsymbol{ \epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t})-\lambda({\hat{\boldsymbol{ \epsilon}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{\color[rgb]{ 0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{0.208,0.565,0.953} \hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_{t}))\right].$ (40) $$

Hence,

$$ $\displaystyle d{\color[rgb]{0.165,0.616,0.561}\definecolor[named]{ pgfstrokecolor}{rgb}{0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{ \lambda}}({\boldsymbol{x}}_{t})$ $\displaystyle=\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}d{ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_ {t})+\lambda({\hat{\boldsymbol{x}}_{\boldsymbol{c}}}({\boldsymbol{x}}_{t})-{ \color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.208,0.565,0.953}\hat{\boldsymbol{x}}_{\varnothing}}({\boldsymbol{x}}_{t})),$ (41) $$

∎

Proposition 1 implies that the CFG++ update of ${\color[rgb]{0.165,0.616,0.561}\definecolor[named]{pgfstrokecolor}{rgb}{
0.165,0.616,0.561}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\lambda}}$ is decomposed into two shift terms: 1) $d{\color[rgb]{0.208,0.565,0.953}\definecolor[named]{pgfstrokecolor}{rgb}{
0.208,0.565,0.953}\hat{\boldsymbol{\epsilon}}_{\varnothing}}({\boldsymbol{x}}_
{t})$ represents the difference between consecutive unconditional scores (i.e. unconditional shift), and 2) $\Delta({\boldsymbol{x}}_{t},{\boldsymbol{c}})$ denotes the direction of conditional guidance at time $t$ (i.e. conditional shift).
The conditional shift term is multiplied by a small interpolation factor $\lambda$ in the case of CFG++, inducing a small nudge toward the condition.

CFG ${\color[rgb]{0.906,0.435,0.318}\definecolor[named]{pgfstrokecolor}{rgb}{
0.906,0.435,0.318}\hat{\boldsymbol{x}}_{\boldsymbol{c}}^{\omega}}$, on the other hand, has the same unconditional shift, but has an oscillatory behavior for the conditional shift.
The difference between CFG/CFG++ sampling arises from the unexpected additional shift from the previous timestep $t+1$ that exists for the CFG decomposition: $-\Delta({\boldsymbol{x}}_{t+1},{\boldsymbol{c}})$, and the scaling constant $\omega$.The initial conditional shift $\omega\Delta({\boldsymbol{x}}_{t},{\boldsymbol{c}})$ with a large $\omega$ pushes the trajectory off the manifold, but cancels some of its effects by subtracting the conditional shift from the previous step $\omega\Delta({\boldsymbol{x}}_{t+1},{\boldsymbol{c}})$. The compounded vector $\Delta({\boldsymbol{x}}_{t},{\boldsymbol{c}})-\Delta({\boldsymbol{x}}_{t+1},{
\boldsymbol{c}})$ does induce a nudge closer to the condition but requires a large value of $\omega$ to have a meaningful effect, and thus is hard to interpret.

###### Proposition 1 .

###### Proof.

## Appendix C Experimental details

### C.1 Text-Conditioned Inverse Problems

Problem settings. We evaluate our approach across following degradation types: 1) Super-resolution with a scale factor of x8, 2) Motion deblurring from an image convolved with a 61 x 61 motion kernel, randomly sampled with an intensity value $0.3^{2}$, 3) Gaussian deblurring from an image convolved with a 61 x 61 kernel with an intensity value 0.5, and 4) Inpainting from 10-20% free-form masking, as implemented in (saharia2022palette). In inpainting evaluations, regions outside the mask are overlaid with the ground truth image.

Datasets, Models. For evaluation, we use the FFHQ (karras2019style) 512x512 dataset and follow (chung2023diffusion) by selecting the first 1,000 images for testing. For the pre-trained latent diffusion model including baseline methods, we choose SD v1.5 trained on the LAION dataset. As a baseline for latent DIS, we consider PSLD (rout2024solving). It enforces fidelity by projecting onto the subspace of ${\mathcal{A}}$ during the intermediate step between decoding and encoding, in conjunction with the DPS (chung2023diffusion) loss term. For gradient updates in both vanilla PSLD and PSLD with CFG, we use static step sizes of $\eta=1.0$ and $\gamma=0.1$ as recommended in (rout2024solving). For CFG scale $\omega$, we applied the corresponing scales that we found to be corresponded for CFG++ scale $\lambda$. Please refer to the Tab. 4 for the hyperparameters used for PSLD with CFG++.

**Table 4: Hyperparameters for PSLD (rout2024solving) with CFG++ and corresponding CFG scale $\omega$.**
|  | SR(x8) | Deblur(motion) | Deblur(gauss) | Inpaint |
| --- | --- | --- | --- | --- |
| $\eta$ | 1.3 | 0.4 | 1.0 | 1.0 |
| $\gamma$ | 0.1 | 0.025 | 0.12 | 0.1 |
| $\lambda$ | 0.1 | 0.2 | 0.2 | 0.6 |
| $\omega$ | 1.5 | 2.0 | 2.0 | 7.5 |

## Appendix D Further experimental results

### D.1 T2I

Since the range of guidance scales for CFG and CFG++ is different, we matched the guidance values of $\omega$ and $\lambda$ for comparison by computing the LPIPS distance using the same seed.

Fig. 12 shows the clean estimates at each diffusion sampling time $t$, computed using Tweedie’s formula, both with and without CFG. It is evident that the original CFG induces significant error, particularly at earlier times, leading to off-manifold samples. However, the CFG++ addresses this issue by adjusting the DDIM re-noising step and the guidance scale.

Figure: Figure 9: Enhanced T2I results by SDXL ($\omega=9.0,\lambda=0.8$) with CFG++. Under CFG, the lion cub is not visible (top-left), the dog appears with two tails (top-right), the goggles have an unusual shape (bottom-left), and the tree trunk is folded (bottom-right). These artifacts are absent in those produced by CFG++.
Refer to caption: extracted/5841818/figs/t2i_sdxl.jpg

Figure: Figure 10: T2I using SD v1.5, CFG vs CFG++ ($\omega=9.0,\lambda=0.8$). Unnatural depictions of human hands, and incorrect renderings of the text by CFG are corrected in CFG++.
Refer to caption: extracted/5841818/figs/t2i_main2.jpg

Figure: Figure 11: T2I using SDXL-Lightning, 6 NFE, CFG vs CFG++. The overall image quality and sophistication have improved with CFG++.
Refer to caption: extracted/5841818/figs/t2i_sdxl_lightning_suppl2.jpg

Figure: Figure 12: The discrete evolution of the posterior mean in CFG and CFG++. Denoised estimates in latent space are decoded into pixel space at each timestep using SDXL.
Refer to caption: extracted/5841818/figs/tweedie_progression.jpg

Figure: Figure 13: Interpolating behavior of T2I along various guidance scales using SD v1.5 with CFG and CFG++
Refer to caption: extracted/5841818/figs/t2i_scale_match.jpg

Figure: Figure 14: Real image inversion results at various CFG and CFG++ scales using SDv1.5. The image is from COCO dataset. We use the text prompt “a white dog is sleeping on a street and a bicycle” during the inversion.
Refer to caption: extracted/5841818/figs/real_inv.png

### D.2 Real Image Editing

In Fig. 15-17, we provide qualitative comparison on real image editing via DDIM inversion with CFG and CFG++. For the DDIM inversion stage, we use "a photography of [source concept]" as conditioning prompt. For the sampling stage, we swap [source concept] to [target concept] and use it as conditioning prompt for generation. For example, in Fig. 15, we set [source concept] to "dog" and [target concept] to "cat".
For all experiments, we set the guidance scale as $\omega=9.0$ and $\lambda=0.8$ as described in the main paper.
The comparison demonstrates that CFG++ successfully edits the given image which was not possible by CFG. This results also support our claim on reduced error during DDIM inversion by CFG++.

Figure: Figure 15: Comparison on real image editing via DDIM inversion under CFG and CFG++ using SDXL. Cat $\rightarrow$ Dog.
Refer to caption: extracted/5841818/figs/real_edit1.jpg

Figure: Figure 16: Comparison on real image editing via DDIM inversion under CFG and CFG++ using SDXL. Horse $\rightarrow$ Zebra.
Refer to caption: extracted/5841818/figs/real_edit2.jpg

Figure: Figure 17: Comparison on real image editing via DDIM inversion under CFG and CFG++ using SDXL. Cat $\rightarrow$ Cat with glasses.
Refer to caption: extracted/5841818/figs/real_edit3.jpg

### D.3 Text-Conditioned Inverse Problems

In Fig. 18-21, we display additional qualitative comparison for text-conditioned inverse problem solver with CFG and CFG++. Experiments are conducted with FFHQ (512x512) validation set and CFG++ consistently leads to better reconstruction of true solution for various tasks.

Figure: Figure 18: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Super-Resolution (x8).
Refer to caption: extracted/5841818/figs/ldis_sup_sr.jpg

Figure: Figure 19: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Motion Deblurring.
Refer to caption: extracted/5841818/figs/ldis_sup_mb.jpg

Figure: Figure 20: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Gaussian Deblurring.
Refer to caption: extracted/5841818/figs/ldis_sup_gb.jpg

Figure: Figure 21: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Inpainting.
Refer to caption: extracted/5841818/figs/ldis_sup_fip.jpg

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * Having the average performance across the different corruption in Tab. 3 would help. Here, the proposed method seems more efficient than CFG to retrieve the clean image according to the FID metric but not so much according to the LPIPS one. Do the authors have some justifications?
* It is unclear why the performances of CFG drop while augmenting the NFE in Fig. 6 (b).
soundness: 2
strengths: The paper is generally well-written with a useful high-level preliminary section. One of the strengths of the method is its simplicity while being well justified. The paper proposes many experiments showing relative gains compared to CFG.
summary: This paper proposes to tackle Classifier Free Guidance's (CFG) limitations, particularly its lack of invertibility and mode collapse. The authors formulate the hypothesis that these limitations come from the off-manifold phenomenon. The proposed method leverages recent approaches developed for inverse problem-solving through diffusion models. The proposed method named CFG++ is compared with CFG for text-to-image generation, image edition or image restoration.
weaknesses: * The CFG method formulation seems different than in the original paper see eq (6) from (Ho and Salimans, 2021). Moreover, the $w=0$ setting in the experiments should be equivalent to no guidance at all when using CFG according to the eq (6) (from the CFG++ method). In Fig. 6, however, the images appear edited with $w=0$ indicating some contradiction.
* The ImageReward metric is not defined. The FID and CLIP metrics are very similar between CFG and CFG++ especially when using SD v1.5. In Tab. 1 the FID and CLIP are also very close for both methods. This tends to indicate that the gains are relatively marginal even in a low NFE setting.   
* The statement that the proposed CFG++ is invertible and brings more diversity in the generation is not evaluated.

**Typos**

* The variable $x_c$ is not introduced in line 228
* Table 1., column 2, Clip score arrow does not show the correct direction
* line 238 ``a crucial difference in the **renoising** process'' shoulnd't it be denoising?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. SDXL-Turbo does not make use of guidance_scale or negative_prompt. Instead, they disable it with guidance_scale=0.0. I would like to know how to verify the validity of CFG/CFG++ in this case.

2.  Does this method also work for negative prompts?
soundness: 3
strengths: 1. CFG++ introduces a novel approach by redefining classifier-free guidance as a constrained manifold problem, which effectively improves the generation quality by staying within the data manifold.
2. This paper validates the effectiveness of the proposed method on standard benchmarks and different tasks (e.g., text-to-image generation, inversion, editing).
3. The paper is well-written and easy to follow.
summary: The paper introduces CFG++, a novel approach to fixing the off-manifold issues that can occur in CFG (classifier-free guidance) during sampling. They reformulate CFG as a manifold-constrained problem, transforming the conditional guidance from extrapolation to interpolation between unconditionally sampled trajectories and conditionally sampled trajectories, thus making the guidance more interpretable. Experimental results show that the method can reduce the artifact, solve the DDIM inversion problem, and be integrated into the high-order diffusion solvers.
weaknesses: 1. There are a few minor issues in the tables, such as the direction of the arrow in the fourth column of CLIP in Table 1 (which should be as high as possible) and the missing ImageReward metric for SD v1.5 in Table 2. 

2. This paper focuses on addressing the inability to perform successful DDIM inversion with classifier-free guidance, so it should include a comparison of the results with this class of methods like [1].

3. I'm a little suspicious of ''Contrary to the widespread belief that these are inherent limitations of diffusion models, this paper reveals that these problems actually stem from outrageous phenomena associated with CFGs...'' in the Abstract. As far as I know, current work has generally recognized and worked for the CFG problem.  Perhaps rewording would better reflect the originality of the paper.

[1] Mokady, Ron, et al. "Null-text inversion for editing real images using guided diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Only diffusion-/score matching-based generative models are discussed in this work. Could you please **provide some similar derivations for the recent flow matching-based generative models, such as SD3 and FLUX**? I believe similar conclusions stand for flow-based methods, and it will make this work more comprehensive.

2. Is it possible to **disentangle two noises used in the denoising and renoising process**? For example, we can set different hyperparameters $\lambda_1$ and $\lambda_2$ for $(\epsilon_c-\epsilon)$, and use this term in different weights for denoising and renoising respectively. Setting $\lambda_1=\lambda_2$ is the same as CFG and setting $\lambda_2=0$ is the same as CFG++.

3. Is it possible to **provide derivations of CFG++ for other diffusion solvers**, except DDIM? Extensions of CFG++ to other solvers in Appendix A are more like intuitive understanding, rather than derivations from inverse problems and SDS loss, as done for DDIM.

4. Essentially, **CFG++ can be written as reweighted CFG whose $\omega$ varies along the sampling process** (let $s\coloneqq t-1$ and $\epsilon\coloneqq \epsilon_\emptyset$ for easier LaTeX rendering in OpenReview):

- For DDIM CFG: $$\mathbf{x}_s=\frac{\sqrt{\bar{\alpha}_s}}{\sqrt{\bar{\alpha}_t}}\mathbf{x}_t-\frac{\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}-\sqrt{(1-\bar{\alpha}_s)\bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}}(\omega\epsilon_c-(\omega-1)\epsilon)$$

- For DDIM CFG++: $$\mathbf{x}_s=\frac{\sqrt{\bar{\alpha}_s}}{\sqrt{\bar{\alpha}_t}}\mathbf{x}_t-\frac{\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}}{\sqrt{\bar{\alpha}_t}}(\lambda\epsilon_c-(\lambda-1)\epsilon)+\sqrt{1-\bar{\alpha}_s}\epsilon$$
$$=\frac{\sqrt{\bar{\alpha}_s}}{\sqrt{\bar{\alpha}_t}}\mathbf{x}_t-\frac{\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}-\sqrt{(1-\bar{\alpha}_s)\bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}}(\frac{\lambda\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}}{\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}-\sqrt{(1-\bar{\alpha}_s)\bar{\alpha}_t}}\epsilon_c-\frac{(\lambda-1)\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}+\sqrt{(1-\bar{\alpha}_s)\bar{\alpha}_t}}{\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}-\sqrt{(1-\bar{\alpha}_s)\bar{\alpha}_t}}\epsilon)$$

So, $$\omega=\frac{\lambda\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}}{\sqrt{(1-\bar{\alpha}_t)\bar{\alpha}_s}-\sqrt{(1-\bar{\alpha}_s)\bar{\alpha}_t}}.$$

As discussed in Sec. 5, previous studies also propose adjusting the guidance scale across timesteps. However, it may be incorrect to claim that "these findings are orthogonal to ours and keep the sampling trajectory the same ... CFG++ is designing a different trajectory".
soundness: 4
strengths: Overall, this manuscript is excellent, and I hope it gets accepted. Below, I outline some strengths that support this view:

1. **Well-written**: The paper is well-written with coherent logic and clear notation, making it easy to understand and follow.

2. **Thorough Experiments**: The work studies many downstream tasks, including text-to-image generation for SDv1.5, SDXL, and their distilled versions, as well as DDIM inversion and editing, PF-ODE trajectories, and text-conditioned inverse problems. The authors provide extensive quantitative and qualitative results to show the superiority of the proposed method.

3. **Intriguing Derivation and Analysis**: The derivation of CFG++ is intriguing by treating text-conditioned generation as an inverse problem and utilizing SDS as the loss function. Moreover, the authors present various perspectives to analyze CFG++, including manifold geometry, score matching loss throughout the denoising process, and the evolution of the posterior mean. These analyses are insightful and helpful in understanding the underlying mechanisms of CFG++.

4. **Simple Yet Widely Applicable Method**: The proposed method essentially modifies the re-noising process of the original CFG, but it achieves improvements across various downstream tasks and could be potentially effective for other tasks relevant to the ICLR community. Additionally, the DDIM solver, as well as other popular diffusion solvers like EDM and DPM-Solver, are derived, establishing CFG++ as a general method.
summary: This work introduces an enhanced version of the widely used classifier-free guidance (CFG) technique, CFG++. By formulating conditional generation as an inverse problem and utilizing score distillation sampling (SDS) loss, CFG++ improves various tasks, including text-to-image generation, DDIM inversion-based editing and text-conditioned inverse problems.
weaknesses: Generally, I think there are no obvious weaknesses in this work. However, there are some questions and please refer to the Questions part.

---

confidence: 5
contribution: 1
initial_rating: 8
presentation: 2
questions: **Question**: Could you elaborate on the specific mechanisms through which off-manifold behavior impacts the performance of traditional Classifier-Free Guidance (CFG)?  A detailed explanation of how off-manifold behavior is identified and measured within the context of your experiments would enhance understanding. Including visualizations or theoretical examples could further clarify this concept.

**Question**: What are the reasons behind selecting the specific datasets used in your experiments? Are there plans to test CFG++ on a wider array of datasets or tasks?  Expanding the experimental framework to include diverse datasets and applications would strengthen the validity of your findings. Consider including datasets from different domains (e.g., text-to-image generation, video synthesis) to demonstrate the versatility of CFG++.

**Question**: Why were certain state-of-the-art methods, such as Stable Diffusion and other recent advances in guided diffusion, excluded from your comparative analysis?  Including a more comprehensive set of comparisons with these methods would provide a clearer context for evaluating the performance of CFG++. Detailed performance metrics and qualitative results could enrich the discussion.

**Question**: Can you provide more in-depth mathematical justifications or derivations that support the effectiveness of CFG++ in addressing the limitations of traditional CFG?  A rigorous theoretical framework would enhance the paper's credibility. Detailed mathematical formulations explaining how CFG++ operates under manifold constraints could help bridge the gap between theory and practice.

 **Question**: How does CFG++ perform in terms of computational efficiency and scalability with larger models or datasets?  Including a discussion on scalability, including any computational overhead observed during experiments, would be valuable. Experiments assessing performance on larger models could illustrate the practical applicability of CFG++ in real-world scenarios.

 **Question**: Have you considered conducting user studies to evaluate the perceived quality and usefulness of the outputs generated by CFG++?  Incorporating qualitative assessments from human evaluators could provide insights into the model's effectiveness from a user perspective. Such studies could help highlight the practical implications of using CFG++ in various applications.


**Question**: What future research directions do you foresee emerging from your findings regarding CFG++ and manifold constraints in diffusion models?  A discussion on potential extensions or related areas of research would help contextualize your contributions within the broader landscape of machine learning, inviting collaboration and further inquiry.
soundness: 2
strengths: The paper exhibits a high degree of originality by introducing the CFG++ framework, which fundamentally reinterprets the limitations of traditional Classifier-Free Guidance (CFG). Instead of merely enhancing existing methodologies, the authors propose that many issues attributed to diffusion models arise from off-manifold behavior. This shift in perspective not only challenges established assumptions but also provides a fresh lens through which to analyze and improve guidance mechanisms in generative models. By integrating recent advancements in inverse problem-solving within diffusion contexts, the authors demonstrate a creative synthesis of ideas that broadens the scope of CFG applications.


The quality of the research is evident in both its theoretical foundations and empirical validation. The authors articulate their arguments clearly, supported by robust mathematical formulations and well-structured experiments. The comparative analysis illustrates the advantages of CFG++ over prior methods, effectively showcasing the model’s performance improvements in terms of reduced artifacts and enhanced image quality. Additionally, the use of rigorous benchmarks for evaluation strengthens the credibility of the findings. The paper’s methodology is sound, making it a valuable contribution to the field.


The clarity of the writing is commendable, as the authors navigate complex concepts with precision. The introduction provides a succinct overview of the issues addressed and the motivations behind the proposed solution. Throughout the paper, technical jargon is well-defined, ensuring accessibility for readers with varying levels of expertise. Figures and diagrams are effectively utilized to illustrate key points, facilitating a better understanding of the model's mechanics and results. Overall, the paper is well-organized, which enhances its readability and facilitates comprehension.
summary: The paper presents CFG++, an innovative approach designed to enhance the performance of diffusion models by addressing limitations associated with traditional Classifier-Free Guidance (CFG). The authors argue that several drawbacks of CFG, such as the lack of invertibility in DDIM (Denoising Diffusion Implicit Models) and the issues of mode collapse when high guidance scales are used, arise from the off-manifold phenomena related to CFG rather than the inherent characteristics of diffusion models themselves.
weaknesses: One notable weakness is the limited scope of experiments presented. While the authors provide comparative results demonstrating the efficacy of CFG++, the evaluation primarily focuses on specific datasets or tasks. To enhance the robustness of their claims, the authors could include a broader range of benchmarks across diverse domains. For instance, incorporating tasks from different image generation contexts (e.g., artistic style transfer, super-resolution) would provide a more comprehensive assessment of CFG++’s performance and generalizability. The authors should extend their experimental framework to include varied datasets and tasks, thereby demonstrating the versatility and applicability of CFG++ in different settings.


Although the paper claims improvements over prior methods, it lacks a detailed comparison with a broader array of state-of-the-art techniques in diffusion models. For example, a comparison with models like **Stable Diffusion** or recent advancements in **Guided Diffusion** could contextualize the advantages of CFG++ more clearly.  The authors should include direct comparisons with a wider selection of contemporary approaches, utilizing standardized metrics (such as FID scores) to provide a clearer understanding of CFG++’s standing in the current landscape.

While the paper introduces the concept of manifold constraints and off-manifold behavior, the theoretical underpinnings could be further strengthened. The authors briefly mention these concepts but do not provide a rigorous mathematical framework that details how CFG++ operates under these constraints or why these adjustments yield better results. A more thorough theoretical analysis, including mathematical proofs or derivations that illustrate the benefits of operating within manifold constraints, would bolster the paper's credibility and understanding.


The paper does not address how CFG++ scales with increasing model size or complexity. As generative models continue to grow, understanding how CFG++ performs under these conditions is crucial. There is a lack of discussion regarding potential computational overhead or challenges in real-world applications. Including analyses or experiments that assess the scalability of CFG++ with larger models or datasets, along with a discussion on computational efficiency, would provide valuable insights for practitioners considering its application.

The paper primarily focuses on quantitative metrics, such as image quality, but does not incorporate user-centric evaluations. Assessments based on human judgment, such as user studies to evaluate the perceived quality or usefulness of generated images, could provide a more holistic view of CFG++’s effectiveness.  The authors should consider conducting qualitative studies where human evaluators assess the output of CFG++ compared to other methods. This could provide insights into the model's real-world applicability and user satisfaction.

### Conclusion
While the paper "CFG++: Manifold-Constrained Classifier-Free Guidance for Diffusion Models" presents significant contributions, addressing these weaknesses would enhance its overall impact and clarity. By broadening experimental evaluations, strengthening theoretical foundations, and considering scalability and user-centric perspectives, the authors can more effectively support their claims and goals.

Meta-review: This paper works on the Classifier-free guidance (CFG) in diffusion models for text-guided generation. It tackles the the off-manifold problem in CFG by proposing a simple revision to the original CFG, resulting in improvements with better sample quality for text-to-image generation. Experimental results demonstrate the effectiveness in text-to-image generation, DDIM inversion, editing, and image inverse problems. Four reviewers (hzx1, 7gyf, GmUK,NrB8) gave scores of 6, 6, 8, 1 after rebuttal. The first three reviewers are positive on the simplicity and experimental justification of improvement over CFG. There are some debates on the fourth reviewer' comments in the post-rebuttal phase. The reviewer NrB8 initially raised several questions and the authors answered these questions in the rebuttal. There is controversy on the reviewer NrB8 's question in the discussion phase. After discussion, the reviewer NrB8 rated score of 1 in the final decision. The AC has asked reviewer NrB8 to provide reasons of decision and remaining concerns on  the authors' responses, but he/she did not respond to the authors' rebuttal. Therefore,  the rating of reviewer NrB8 is not fully supportive. Considering the first three reviewers' positive final decisions, the paper can be accepted, but the authors are suggested to further revise the paper considering these discussions.

Decision: Accept (Poster)