CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models
Authors: Hyungjin Chung
Jeongsol Kim
Geon Yeong Park
Hyelin Nam
Jong Chul Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
Manifold
Classifier-free guidance
Primary area: generative models

Abstract: Classifier-free guidance (CFG) is a fundamental tool in modern diffusion models for text-guided generation. Although effective, CFG has notable drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image editing; furthermore, high guidance scales, essential for high-quality outputs, frequently result in issues like mode collapse. Contrary to the widespread belief that these are inherent limitations of diffusion models, this paper reveals that the problems actually stem from the off-manifold phenomenon associated with CFG, rather than the diffusion models themselves. More specifically, inspired by the recent advancements of diffusion model-based inverse problem solvers (DIS),  we reformulate text-guidance as an inverse problem with a text-conditioned score matching loss and develop CFG++, a novel approach that tackles the off-manifold challenges inherent in traditional CFG. CFG++ features a surprisingly simple fix to CFG, yet it offers significant improvements, including better sample quality for text-to-image generation, invertibility, smaller guidance scales,  reduced etc. Furthermore, CFG++ enables seamless interpolation between unconditional and conditional sampling at lower guidance scales, consistently outperforming traditional CFG at all scales. Moreover, CFG++ can be easily integrated into the high-order diffusion solvers and naturally extends to distilled diffusion models. Experimental results confirm that our method significantly enhances performance in text-to-image generation, DDIM inversion, editing, and solving inverse problems, suggesting a wide-ranging impact and potential applications in various fields that utilize text guidance. Project Page: https://cfgpp-diffusion.github.io/anon

Published as a conference paper at ICLR 2025
CFG++: MANIFOLD-CONSTRAINED CLASSIFIER FREE GUIDANCE FOR DIFFUSION MODELS
Hyungjin Chung", Jeongsol Kim", Geon Yeong Park", Hyelin Nam", Jong Chul Ye KAIST
* Equal Contribution
Mnj.chung, jeongsol, pky3436, hyelin. nam, jong. ye weeeeeestt.ac.... kc.
CFG
CFG++
Source Image
CFG
CFG (2.0)
CFG++ (0.2)
CFG
CFG+
"beautiful lady, freckles, big smile, blue eyes, short ginger hair, wearing floral vest top, soft light, dark gray background"
Figure 1: (Top) Comparison of T2I results by SDXL-Turbo for the prompt "kayak in the water, optical color, aerial view, rainbow". The CFG-guided image has significant artifacts, which are reduced in the CFG++ version. (Middle) DDIM Inversion results under CFG show noticeable artifacts at various CFG scales, which are significantly reduced by CFG++. (Bottom) The evolution of denoised estimates differs between CFG and CFG++ CFG exhibits sudden shifts and intense color saturation early in reverse diffusion, while CFG++ transitions smoothly from low to high-resolution.
ABSTRACT
Classifier-fre free guidance (CFG) is a fundamental tool in modern diffusion mod- els for text-guided generation. Although effective, CFG has notable drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image editing; furthermore, high guidance scales, essential for high-quality outputs, frequently

---

Published as a conference paper at ICLR 2025
result in issues like mode collapse. This paper reveals that the problems may stem from the off off-manifold phenomenon associated with CFG, rather than the diffusion models themselves. More specifically, inspired by the recent advance- ments of diffusion model-based inverse problem solvers (DIS), we reformulate text-guidance as inverse problem with text-conditioned score matching loss and develop CFG++, novel approach that tackles the off-manifold challenges inherent in traditional CFG. CFG++ features surprisingly simple fix CFG, yet it offers significant improvements, including better sample quality for text-to- image generation, invertibility, smaller guidance scales, etc. Furthermore, CFG++ enables seamless interpolation between unconditional and conditional sampling at lower guidance scales, consistently outperforming traditional CFG at all scales. Moreover, CFG++ can be easily integrated into high-order diffusion solvers and naturally extends to distilled diffusion models. Experimental results confirm that our method significantly enhances performance in -ot-tt--maae generation, DDIM inversion, editing. and solving inverse problems, suggesting a wide-ranging impact and potential applications in various fields that utilize text guidance. Project Page: ht https://cfffp di ffusion. gi thub.
INTRODUCTION
Classifier-free guidance (CFG) (Ho & Salimans, 2021) forms the key basis of modern text- eext-guided generation with diffusion models (Dhariwal & Nichol, 2021; Rombach et al., 2022). Nowadays, is common practice to train diffusion model with large-scale paired text-image data (Schuhmann et al., 2022), so that sampling (i.e. generating) signal (e.g. image, video) from a diffusion model can either be done unconditionally from pe (e (2) po(x), or conditionally from pe pe (lcc,, where c is the text conditioning. Once trained, it seems natural that one would acquire samples from the conditional distribution by simply solving the probability-
ODE or SDE sampling (Song et al., 2021b;a; Karras et al., 2022) with the conditional score function. In practice, however, it is observed that the conditioning signal is insufficient when used naively. To emphasize the guidance, one uses the guidance scale w > 1, where the direction can be defined by the direction from the unconditional score the conditional score (Ho & Salimans, 2021).
In modern text-to-image (T2I) diffusion models, the guidance scale w is typically set within the range of [5.0, 30), referred as the moderately high range of CFG guidance (Chen et al., 2024; Podell et al., 2023). The insufficiency in guidance also holds for classifier guidance (Dhariwal & Nichol, 2021: Song et al., 2021b) so that a scale of 10 was used. While using high guidance scale yields higher-quality images with better alignment to the condition, it is also prone to mode collapse, reduces sample diversity, and yields an inevitable accumulation of errors during the sampling process. One example is DDIM inversion (Dhariwal & Nichol, 2021), a pivotal technique for controllable synthesis and editing (Mokady et al., 2023), where running the inversion process with w > 1.0 leads to significant compromise in the reconstruction performance (Mokady et al., 2023; Wallace et al., 2023). Another extreme example would be score distillation sampling (SDS) (Poole et al., 2022), where the guidance scale in the order of few hundred is chosen. Using such a high guidance scale leads to better asset quality to some extent, but induces blurry and saturated results. Several research efforts have been made to mitigate this downside by exploring methods where using a smaller guidance scale suffices (Wang et al., 2024; Liang et al., 2023). Although recent progress in SDS-type methods has reduced the necessary guidance scale to range that similar to those of ancestral samplers, using moderately large w is considered an inevitable choice.
In this work, we aim to give an answer to this conundrum by revisiting the geometric view of diffusion models. In particular, inspired by the recent advances in diffusion- based inverse problem solvers (DIS) (Kadkhodaie& & Simoncelli, 2021: Chung et al., 2023a; Song et al., 2023; Kim et al., 2024b; Chung et al., 2024), we reformulate the text guidance as an inverse problem with a text-conditioned score-matching loss and derive a reverse diffusion sampling strategy by utilizing decomposed diffusion sampling (DDS) (Chung et al., 2024). This results in surprisingly simple fix of CFG to the sampling process without any computational overhead. The resulting process, which we call CFG++, works with a small guidance scale, typically x [0.0, 1.0), that smoothly interpolates between unconditional and conditional sampling, with = 1.0 having a similar effect as using CFG sampling with w ~ 12.5 at 50 neural function evaluation (NFF). Furthermore, CFG++ reduces the inversion
2

---

Published as a conference paper at ICLR 2025
Algorithm Reverse Diffusion with CFG
Algorithm 2 Reverse Diffusion with CFG++
Require: ær (O Ia), S WER
Require: øy ~ (O La), E [0.1]
1: for == T' to do
for = 7 to do
2: e (m+)
(m:) + [(((((()))) ( (g(())))
2: (xx)) ] ég (xx) + \(Ec((x)) (x(1))
4z (2i) +- (0) VI- a.c)(
3: #2(x)) t (æ) vI and (21)))
4: 2t-1 = v 1 (ara) + vT -0-1 (3t)
4: 21-1 van 1a (xx)) v1 12212 (x:)
5: end for
5: end for
6: return 20
return mn
Figure 2: Comparison between reverse diffusion process by CFG and CFG++ CFG++ proposes simple but surprisingly effective fix: using éΦ (zx) instead of Eg (zx) in updating øt-1-
error, enhancing and simplifying image reconstruction, as well editing. Comparing CFG++ against CFG shows that we achieve consistently better sample quality for text-to-image (T2I) generation, significantly better DDIM inversion capabilities that lead enhanced reconstruction and editing. and enabling the incorporation of CFG guidance to diffusion inverse solvers (DIS) (Chung et al., 2023a). While the applications of CFG++ that we show in this work are limited, believe that our work will have a broad impact that can be applied to all applications that leverage text guidance through the traditional CFG.
BACKGROUND
Diffusion models. Diffusion models (Ho et al., 2020; Song al., 2021b; Karras et al., 2022) generative models designed to learn the reversal of a forward noising process. This process starts with an initial distribution po( where æ IR", and progresses towards the standard Gaussian distribution (x) = (0, I). utilizing forward Gaussian perturbation kernels. Sampling from the data distribution can be performed by solving either the reverse stochastic differential equation (SDE) or the equivalent probability-flow ordinary differential equation (PF-ODE) (Song et al., 2021b). For example, under the choice p( (20) = (aro, af I), the generative PF-ODE reads
dx. = - Va, log p(x.) di 2t. EF [r)]
dt,
Pr ~ Pr (PT),
(1)
where Tweedie's 's formula (Efron, 2011) E| x.] second equality. When aiming for text-conditional diffusion model that can condition on arbitrary c, one can extend epsilon matching
x; + V x. log p(x,) applied to achieve the
conditional one where the condition is dropped with certain probability (Ho & Salimans, 2021) so that null conditioning with neural network architecture of 6g designed so that the condition e can modulate the output through cross attention (Rombach et al., 2022). For simplicity in notation throughout the paper, we define éc := Eg c) and Ég == E Ø) by dropping and Zt.
Ø is possible. The
Extending the result of Tweedie's 's formula to the unconditional case under the variance preserving (VP) framework of DDPMs (Ho et al., 2020), we have
Eizoo| Øt, Ø 2g == (a: vic
Leveraging (2), it is common to DDIM sampling (Song al., 2021b) to solve the conditional probability-llow ODE (PF-ODE) of the generative process. Specifically, a single iteration reads
a = (a. /1 /1-a (.t22//Va \/va.
- 1 to + /I a1-162
(4)
where to := (x,)) = E po xr, Ø] is the denoised signal by Tweedie's formula, and (4) corre- sponds to the renoising step This is repeated for t T, I,...,
For modern diffusion models, is common to train a diffusion model in the latent space (Rombach al., 2022) with the latent variable z. While most of our experiments are performed with latent diffusion models (L.M)), as our framework holds both for pixel- and latent-diffusion models, we will simply use the notation æ regardless of the choice.
Classifier free guidance. For a conditional diffusion, Ho & Salimans (2021) considered the sharpened posterior distribution ydd (x)c) x p(z)p(c|x)* Using Bayes rule for some timestep
Va po (x,(c) Va, log p(x,) + w Va, logp(x Vz, log log:(()))
(5)
Siiiiarly, define &e (x( F [T.
3

---

Published as a conference paper at ICLR 2025
(a)
(b)
4
34-1
M2
M,
&g
M
CFG+
CFG
M,-1
CFG++
CFG
x2
M
Figure 3: Off-manifold phenomenon of CFG arise from: (a) the typical CFG scale w > 1.0 which leads to extrapolation and deviation from the piecewise linear data manifold, and (b) CFG's renoising process, which introduces a nonzero offset A from the correct manifold. CFG++ effectively mitigates all these artifacts.
Parametrizing the score function with 60, as in DDPM (Ho et al., 2020), we have
Ec (xx) == (xt) + (()e((;)) - éØ (xx)]
(6)
where we introduce a compact notation dg that guides the sampling from the sharpened posterior. When sampling with CFG guidance with DDIM sampling, one replaces é9 with é for both the Tweedie estimate (3) and the subsequent update step (4), leading to Algorithm 1.
Diffusion model-based inverse problem solvers. Diffusion model-based inverse problem solvers (DIS) aims to perform posterior sampling from an unconditional diffusion model (Kadkhodaie & Simoncelli, 2021; Chung et al., 2023a; Song et al., 2023; Kim et al., 2024b). Specifically, for a given loss function f(x) which often stems from the likelihood, the goal of DIS is to address the optimization problem minge, EM f(x), where M represents the clean data manifold sampled from the unconditional distribution Po(2). It is essential navigate in way that minimizes cost while also identifying the correct clean manifold.
Chung et al. (2023a) proposed diffusion posterior sampling (DPS), where the updated estimate from the noisy sample æ E M, is constrained to stay on the same noisy manifold Mr This is achieved by computing the manifold constrained gradient (MCG) (Chung al., 2022) noisy sample 2e E Mr as Ving f(xx) == Va, {(t),, where ã; is the denoised sample in (2) through Tweedic's s
formula (Efron, 2011). The resulting algorithm with DDIM (Song al., 2021b) can be stated as follows:
xz-1 = V ق (ág - 7: Vz, ((x))) 11-
(7)
where 7/ > 0 denotes the step size. Under the linear manifold assumption (Chung et al., 2022: 2023a), this allows precise transition to M To mitigate the computational complexity and the instability of neural network backprop, Chung et al. (2024) shows that (7) can be equivalently represented as
2t-1 V @ - 7+ ((#x)) + / 0t-16
under further assumptions on M. This method, often referred to as the decomposed diffusion sampling (DDS), bypasses the computation of the score Jacobian, similar to Poole et al. (2022), making it stable and suitable for large-scale medical imaging inverse problems (Chung et al., 2024). In the following, we leverage the insight from DDS to propose algorithm to improve upon the CFG algorithm.
CFG++ MANIFOLD-CONSTRAIII CFG
3.1 DERIVATION OF CFG++
Instead of uncritically adopting the sharpened posterior distribution (*(()) p(æ) as introduced by Ho & Salimans (2021), we adopt a fundamentally different strategy by reformulating text- text-guidance an optimization problem. Specifically, in (8) such that, when minimized under the condition set by the text, enables the reverse diffusion process to generate samples that increasingly satisfy the text condition progressively.
focus is identifying loss function {(x)
One of the most significant contributions of this paper is to reveal that the text-conditioned score matching loss or alternatively, score distillation sampling (SDS) loss (Poole al., 2022) is ideally
4

---

Published as a conference paper at ICLR 2025
suited for purpose. Specifically, we are interested in solving the following inverse problem through diffusion models:
min (ads(()), leds (ar) == [ a + v dr,,e c) -
<l1 <
(9)
=EM
This implies that our goal is to identify solutions on the clean manifold M that optimally aligns with the text condition c.
To avoid the Jacobian computation, in this paper, we attempt to solve (9) through DDS in (8). The resulting sampling process from reverse diffusion is then given by
1 Vau-1 @ 7
V ae sads (+x) + v 16
(10)
By using 2t yā x + vI aGE from the clean image E .M, we can equivalently write the loss x lel1? which leads to
as Cads (2) = I-
de
It-1 vo,-i (0 + M t2)) v 01-16
(11)
+ where λ = 20: Yt. Using the CFG notation (2x)) == éي (2:)) + X((c(x)) (ي and ã
1-a;
A(xe 2t)= (ate vi aré (xx)) (ax)))/
(11) be equivalently represented as
(x1) (x) -1- (x:))))
(12)
xt-1 = vā (xx)) + /I - -m
(13)
which is summarized in Algorithm 2. By examining Algorithm and Algorithm 2, we observe that CFG and CFG++ are mostly the same, with a crucial difference in the renoising process. This surprisingly simple fix of utilizing the unconditional noise (x,) instead of (x²) leads to a smoother trajectory of generation, (Fig. bottom) and generation with superior quality (Fig. top).
Although we focus our construction of the solver on DDIM for simplicity, note that DDIM is just one way of reverse sampling. There are other widely-used solvers such as Karras Fuler and its variants (Karras al., 2022), DPM-solver (Lu et al., 2022a:b), their ancestral variants?, etc. For most widely used solvers up to the second order, a single-step update solving the unconditional PF-ODE can be represented
x; a (I;-1) + a (x:-1) + b (x;-2) + Q: + die, E ~~ NO,,
(14)
with d, f O when one uses an ancestral sampler. Note that the first right-hand side term in (14) corresponds the denosing. whereas the rest terms describes the higher-order corrected version of the renoising process. As the goal of CFG+ is to optimize the denoising process under the text- rext-guidance while keeping the renoising components equivalent to unconditional sampling, applying CFG++ to the general iteration in (14) simply leads to
x; (( AV (a._)) (x:-1))) + a (x. b, (x,-2) + C,8;-1 + die
= -2 + a ( + b, (2;-2) + - + d,e, e ~~ (),
(15)
Moreover, CFG++ naturally extends to distilled diffusion models, such SDXL-turbo (Sauer et al., 2023) and SDXI--lghhtning (I.in et al., 2024), where akin to (15), we use the conditional denoised estimate, but for the rest of the noise components in the Fuler solver, we use the unconditional estimate. For details in how to apply CFG++ to various solvers, see Appendix A.
3.2 GEOMETRY OF CFG++
Mitigating off-manifold phenomenon. In Fig. I (bottom), we illustrate the evolution of the posterior mean through Tweedie's 's formula during the reverse diffusion process. Notably, in the early phases of reverse diffusion sampling under CFG, there is sudden shift in the image and intense color saturation. Conversely, CFG++ is free from the undesirable off-manifold phenomenon. In the following, investigate why this is the case.
Note that the denoised estimate of CFG and CFG++ at time / can be equivalently represented as (xx)) = (1 - w)oo (x:)) + (w.) (16)
(xx) = (1 - x) + (xx))
5

---

Published as a conference paper at ICLR 2025
OFG+
((alte)) -d
0.0
1000
800
600
400
200
"kayak in the waler, optical color, perial view, rainbow"
Diffusion time
Figure 4: Text-conditioned score matching loss throughout the reverse diffusion sampling for both CFG and CFG++ in SDXL. Avg. loss computed with 55 prompts from (Chen et al., 2024).
m, 1) facilitates interpolation. However, with 1.0, CFG extrapolates beyond the unconditional and conditional estimates. Consequently, under the assumption that the clean manifold be piecewise linear (Chung et al., 2024), the conditional posterior mean estimates from CFG obtained with a guidance scale outside the range of [0, 1), can easily extend beyond the piecewise linear manifold. This may lead to the estimates potentially "falling off" the data manifold, as depicted by an orange arrow pointing downwards in Fig. 3(a). Thus, we select À E [0, 1] as the guidance scale for CFG++ ensure it remains interpolation between the unconditional and conditional estimate, thus preventing it from 'falling off' the clean data manifold. An additional source of the off-manifold phenomenon CFG occurs during the transition from the clean manifold M the subsequent noisy manifold M,-1, due to a similar off off-manifold phenomenon from the large guidance scale (i.c. extrapolation). illustrated in Fig. 3 (b).
Text-Image alignment. In Fig. I (top), and Fig. 10, we observe enhanced text-to-ing text-to-image alignment achieved with CFG++. This enhanced alignment capability natural consequence of CFG++ which directly minimizes the text-conditioned score-matching loss as shown in (9). In contrast, CFG indirectly seeks text alignment through the sharpened posterior distribution yf (*(ee) xx p(=))p(c(<r) Therefore, CFG++ inherently outperforms CFG in terms of text alignment due to its fundamental design principle. For example, Fig. displays the normalized text-conditioned score matching loss, represented (1 ā ae]lle ca e)ll / ||æ lell2 throughout the reverse diffusion sampling process for both CFG and CFG++ The loss plot associated with CFG shows fluctuations and maintains noticeable gap compared to CFG++
after the completion of the reverse diffusion process. In fact, the fluctuation associated with CFG is also related to the off-manifold issue, and from Fig. 4 we can easily see that the off-manifold phenomenon is more dominant at early stage of reverse diffusion sampling. Conversely, the loss trajectory for CFG++ demonstrates much smoother variation, particularly during the early stages of reverse diffusion.
DDIM inversion. As discussed in (Song et al., 2021a), the denoising process for unconditional DDIM approximately invertible, meaning that æ; can generally be recovered from æ;- Specifically, from (3) and (4), we have the following approximate inversion formula for unconditional DDIM:
a (xt))= (at-1 V 04 1 (x:))) xt = Va (x.) ) + vI (xz) (x;) + VI āi (x)-1
(#t-1 V1 0 0: ( (R:-1))/ Va-1 01 Vou
(17)
(18)
where the approximation arises from (x)) ~ é ( A similar inversion procedure has been employed for conditional DDIM inversion under CFG by assuming (x:) 2 (Ix-1), and replacing é by e.- as detailed in Algorithm 3.
On the other hand, by examining (12) and (13), inversion formula under CFG++:
obtain the following approximate DDIM
2/2 (xx) = (x:-1 V1 11 (m-
(19)
St va + v1 o. (xt-1)
(20)

---

Published as a conference paper at ICLR 2025
2.0,
5,0,A
04
7.5.A
125,A
Method
IR
CUPP
IR
CLIP1
CLIP
IR
CLPP
IR
(No.& Salmons, 2021)
13.84
0,298
0.235
15.08
0.310
0.068
17.71
0,312
0,152
20.01
0.312
0.170
21.23
0.313
0.192
12.75
0.303
.0.218
14.95
0.310
0,071
17.47
0.312
0.156
19.34
0.313
0.208
2008
0.133
0.194
Table 1: Quantitative evaluation of 50 NFE DDIM T21 with SD v1.5 on COCO 10k
where we apply the approximation (xx) ~ é (xx-1) alongside the usual assumption éω (2t) ~ éω (zt-1). This formulation underpins the CFG++ guided DDIM inversion algorithm, presented in Algorithm 4.
In practice, for small time step size the error čω (2:e) ( is relatively small, so the un- conditional DDIM inversion by (17) and (18) lead to relatively insignificant errors. Unfortunately, the corresponding inversion from conditional diffusion under CFG quite distorted as noted in (Mokady et al., 2023; Wallace et al., 2023). In fact, this distortion is originated from the inaccuracy of the approximation 6 (ax)) 2% (1)-1)- More specifically, even when ég (xt) eg (2 the approximation error from the CFG is given by
Ecfg := ~" (xt)) ~" (2x-1) = (x,) - én (3) + w (dée((x)) )
= ((88c(())) - (
(21)
where 52.( (x( (x1) denotes the directional component from unconditional to the conditional diffusion. Since the directional component by the text guidance is not negligible, this error becomes significant for high guidance scale w. Accordingly, the guidance scale must be heavily downweighted in order for inversions on real world images to be stable, thus limiting the strength of edits. To mitigate this issue, the authors in (Mokady al., 2023; Wallace et al., 2023) developed null text optimization and coupled transform techniques, respectively.
On the other hand, under the usual DDIM assumption è (2x)) 2g (1;-1). the approximation error of CFG+ mainly arises from (20), which is smaller than that of CFG since we have
|Eafg++|| || = X (()) 5é.(#;;1)) < ||Eefal|
(22)
thanks to À < W. Therefore, CFG++ significantly improves the DDIM inversion as shown Fig. 1 (middle) for representative results and Sec. 4.2 for further discussions.
4 EXPERIMENTAL RESULTS
In this section, we design experiments to show the limitations of CFG and how CFG++ can effectively mitigate these downsides. The main experiments were conducted with SD v1.5 or SDXL with 50 NFE DDIM sampling. In this regime,
searched for the matching guidance values of w and À for a fair comparison. We fix A = 0.2, 0.4, 0.6, 0.8, 1.0 and find the w values that produce the images that are of closest proximity in terms of LPIPS distance given the same seed. We found that the corresponding values were w = 2.0, 5.0, 7.5, 9.0, 12.5, respectively. Some of the experiments were also conducted with distilled model such as SDXL-turbo, lightning.
4.1 TEXT-TO-IMAGE GENERATION
Model
Metric
CPG CFG++ (ours)
Using the corresponding scales for and X, directly compare the performance of the T2I task using SD v1.5 and SDXL. In Tab. 1, re- port quantitative metrics using 10k images gen- erated from COCO captions (Lin al., 2014). Here, we observe a constant improvement of the FIID metric across all guidance scales (also see Fig. 15 for an apples-to-apples comparison), with approximately the same level of CLIP sim- ilarity or better. Additionally, we evaluate Im- ageReward (IR) (Xu et al., 2024), trained on human judgment data for better correlation, and observe improvements in this metric as well. The improvements can also be clearly seen in Fig. 11 (SD v1.5), where the unnatural components of the generated images are corrected. Specifically, we see that unnatural depictions of human hands, and incorrect renderings of the text are corrected in CFG++,
FID
59.67
59,21
SDXL-Turbo
CLIP
0.320
0.325
IewgeRewardf
0.777
0.968
FIDJ
56.11
55,19
SDXL-Lightning
CLIP
0.322
0.324
ImwaeReward
0.691
0.829
FID
32.72
32.58
SD v1. (DPM++ 2M)
CLIP
0.313
0.312
IewgeReward 1 0.086
0.023
Table 2: Quant. eval. on accelerated T21 sampling
7

---

Published as a conference paper at ICLR 2025
small cactus with happy face in the sahara desert.
game on refants a bethroom mirror"
"A paintaar study hard.... many concepts the sir. ...
'A transparent scuppture duck made out of glass,
CFG
CFG++
Turbo
Lighlning
Figure 5: T2I using SDXL- (turbo, lightning), 6 NFE, CFG CFG++.
long-standing research question in and of its own (Podell al., 2023; Pelykh et al., 2024; Chen et al., 2023). We find that the improvement gain from CFG++ even more dramatic for distilled diffusion models such as SDXL- (turbo, lightning). We quantify the results on the same 5k prompts with NFE sampling. In Fig. 5, Fig. 12 and Fig. 13 we see significant boosts in the quality of the generated images, which is also depicted in the improvements seen Tab. 2.
To show the compatibility of CFG++ with higher- solvers, we experiment with the DPM++ 2M (Lu al., 2022b) solver using 20 NFE. We note that in such low NFE regime, a CFG scale of 5.0 corresponds to CFG++ scale of 1.0, and we have to slightly extrapolate the value of À 2 1.0 to achieve stronger guidance results. In Tab. 2, we observe that CFG++ broadly comparable to CFG. This outcome differs from the consistent improvement of CFG++ over CFG observed with 50 NFE DDIM sampling. We attribute this to two factors: (1) the overall image quality for 20 NFE DPM++ 2M is generally lower compared to 50 NFE DDIM, leading noisier quantitative metrics, particularly for metrics like ImwgeReward that are sensitive to subtle quality changes; and (2) the difference between CFG and CFG++ + tends to be less pronounced in low NEE regimes without distillation, as stronger guidance effects emerge with higher NFEs.
4.2 DIFFUSION IMAGE INVERSION AND EDITING
We further explore the effect of CFG++ DDIM inversion (Dhariwal & Nichol, 2021), where the source image is reverted to a latent vector that can reproduce the original image through generative process. DDIM inversion is well-known to break down in the usual CFG setting CFG magnifies the accumulated eтror of each inversion step (Mokady et al., 2023), violating local lincarization assumptions (Wallace al., 2023). We show that CFG++ mitigates this issue by improving the inversion and image editing capabilities. We evaluate our method following the experimental setups in Park al. (2024) and Kim et (2024b).
Diffusion Image Inversion. Using the matched set of scales for w and X, we demonstrate the effect of CFG++ on the diffusion image inversion task. Specifically, we reconstruct the images after inversion and evaluate it through PSNR and RMSE, following the methodology of Wallace et al. (2023). In Fig. 6, we illustrate the reconstructed examples (6a) from a real image from COCO data set and computed metrics (6b) for 5k COCO-20 2014 (Lin et al., 2014) validation set. Both qualitative and quantitative evaluations demonstrate a consistent improvement on reconstruction performance induced by CFG++. Notably, DDIM inversion with CFG++ leads to consistent reconstruction of the source image across all guidance scales, while DDIM inversion with CFG fails to reconstruct it, shown in Fig. 16 for another real image.
Image Editing, Fig. 6c compares the image editing result using CFG and CFG++ followed by image inversion with 5 NFEs. In the image editing stage, word in the source text highlighted by green
8

---

Published as a conference paper at ICLR 2025
(9)
Source Image
CFG (w)0.0)
CFG (2m2.))
CFQ+ (2=0.2)
(c) Synthelic Image
Cuto orange kitton, sliding down an aqua siide, hapey exciled, in front aecilement and scared in the eye. water splashing
7.5
Whtnn "pupps" CFG (ar 9,01
CFi+ to = o.
NFE-6
321 20
34130 50
86125
85130
101125
02128 64
08115
68153
201323
"chhen' - "baby lon"
Figure 6: Inversion and editing results. (a) Reconstructed samples after inversion by CFG and CFG++. (b) Quantitative comparison between CFG and CFG++ for reconstruction. (c) Image editing comparison via SDXL.
color is swapped with the target concept, and this modified text is used the condition for sampling. CFG++ successfully edits the target concept while preserving other concepts, such as background. In particular, the water splashing in the background, which tends disappear in the conventional CFG, maintained through the inversion process by CFG++. This emphasizes CFG++'s superior ability retain specific scene elements that frequently lost in the standard CFG approach. Moreover, standard CFG's disrupted DDIM inversion leads to saturated and less faithful editing results, whereas CFG++ delivers precise and high-fidelity edits.
4.3 TEXT-CONDITIONEL INVERSE PROBLEMS
Inverse problem involves restoring the original data æ from a noisy measurement y -A(x) n. where A represents an imaging operator introduces distortions (e.g, Gaussian blur) and denotes the measurement noise. Diffusion inverse solvers (DIS) address this challenge by leveraging pre- trained diffusion models implicit priors and performing posterior sampling æ p(zly). Methods that leverage latent diffusion have gained recent interest (Rout et al., 2024; Song et al., 2024), but leveraging texts for solving these problems remains relatively underexplored (Chung et al., 2023b; Kim et al., 2023). One of the main reasons for this is that naively using CFG on top of latent DIS leads diverging samples (Chung et al., 2023b). Several heuristic modifications such as null prompt optimization (Kim et al., 2023) with modified sampling schemes were needed to mitigate this drawback. This naturally leads the question: Is it possible to leverage CFG guidance hou-aand-pl component of existing solvers? Here, we answer this question with a positive by showing that CFG++ enables the incorporation of text prompts into a standard solver. Specifically, we focus on comparing the performance of PSLD (Rout et al., 2024) combined with CFG and CFG++ solving linear inverse problems. This evaluation utilizes the FFHQ (Karras er al., 2019) 512x512 datasct and the text prompt "a high-quality photo of a face". Further details about the experimental settings can be found in the Appendix
As shown in Tab. 3, our method mostly outperforms both the vanilla PSLD and PSLD with CFG. The superiority is also evident from the Fig. 7, where CFG++ consistently delivers high-quality reconstructions across all tasks. PSLD with CFG often suffer from artifacts and blurriness. Conversely, CFG++ achieves better fidelity, clearly distinguishing between faces and faithfully reproducing fine details like eyelids and hair texture. For more results, please refer to the Appendix F.

---

Published as a conference paper at ICLR 2025
GT
Degraded
CFG
CFG++
SR(x8)
Motion Blur
Figure 7: CFG and CFG++. For more results, please refer the Appendix F.
Qualitative comparison on various inverse problems using PSLD (Rout et al., 2024) under
SR (x8)
Deblor (motion))
Deblur (gauss)
Inpaint
Method
FID
LPIPS I.
PSNR
FID 1
LPIPS L
PSNR
FID J
LPIPS
PSNR
FID 4
LPIPS
PSNR
PSLD
46.24
0413
24.41
97.51
0.500
21.83
41.65
0.388
26.88
10.27
0.083
30.15
4 CPC
41.24
0.394
24.91
91.90
0.193
22.29
41.52
0.390
26.94
9.36
0.055
30.27
PSI CRTi+ (nuns)
39.85
0.400
26.90
9.78
Table 3:
Quantitative comparison (FID, LPIPS, PSNR) of PSLD, PSLD with CFG, and PSLD with CFG+ on Latent Diffusion Inverse Solver.
RELATED WORKS AND DISCUSSIONS
Kynkäänniemi al. (2024) observed that applying CFG guidance at the earlier stages of sampling always led to detrimental effects and drastically reduced the diversity of the samples. The guidance the later stages of sampling had minimal effects. Drawing upon these observations, it was empirically shown that applying CFG only in the limited time interval the middle led to the best performance. Similar observations were made in the SD community (Howard & Prashanth, 2022: Birch, 2023) in adjusting the guidance scale across t. These works try empirically adjust the strength of the guidance, while we derive our trajectory in a principled manner. More recently. Bradley & Nakkiran (2024) observed that the CFG score
a valid denoising direction, showing that in the asymptotic corrector (PC) sampling, dubbed PCG. PCG shares a similar spirit with CFG++ in that for the mainstream of sampling, one defers from the use of mixed CFG score (we use unconditional score, PCG uses conditional score) to avoid invalid directions. While PCG derives an interesting connection of PC sampling with CFG, the proposed algorithm does not improve the sampling performance. In contrast, CFG++ improves the sample quality without additional computation overhead. Since CFG++ attempts to make adjustments of CFG using a linear combination of score functions, it is not surprising to see that CFG++ sampling can be achieved by setting a time varying schedule. Specifically, by setting a time-varying schedule
limit, CFG sampling can be considered as a specific type of predictor predictor-corrector
Wi as We - (VI (VI- - δt-1
one can achieve the same effect with CFG as CFG++
Details shown in App. D. However, this specific choice of time-varying guidance scale has not been reported in the literature, as it is a schedule that is relatively complex to be drawn heuristically. We further show in Appendix C that such choice removes undesirable oscillatory behavior in the evolution of the posterior mean.
6 CONCLUSION
This paper revises the most widespread guidance method, classifier-free guidance (CFG), highlighting that a small and reasonable guidance scale, e.g. [0.0, 1.0), might suffice successful guidance with proper geometric correction. Observing that the original CFG suffers from off- manifolld issues during sampling, we propose a simple but surprisingly effective fix. This change translates the condi- tional guidance from extrapolating to interpolating between unconditional and conditional sampling trajectories, leading to more interpretable guidance in contrast to the conventional CFG which relies on heuristic w scaling. Given that CFG++ mitigates off-manifold issues. it may be beneficial for other downstream applications requiring accurate latent denoised estimate representations, e.g. DIS, as demonstrated in our experiments.
10

---

Jeremy Howard and Rekil Prashanth. Adjusting guidance weight function of time. https: 2022. Idea mentioned on Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea under Grant RS-2024- 00336454 and by the Institute for Information & Communications Technology Planning & Evaluation (ITT)) grant funded by the Korea government (MSIT) (RS-2019-11190055, Artificial Intelligence Graduate School Program, KAIST).
REFERENCES
Alex Birch. Turning off classifier-free guidance at low noise levels. https://twitter.ccmm 2023. Idea mentioned on Twitter.
Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is predictor-corrector. arXiv preprint arXiv:2408.0900 2024.
Jingye Chen, Yupan Huang. Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters. Thirty-sevent Conference on Neural Information Processing Systems, 2023. URL https::/openreet ew. net::oorum2.dkeeeRRoom
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreww http://ppeerriieeeeee//rrrrrrrre
net wtt.:nte/oorrmm/iieeeetteeeeee
Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreviiee wttpiiewwweefffrruu wwt..e/ooromiidnnnjoo@eel
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. International Conference on Learning Representations, 2023a. URL hwtps:t//tpeeeeee http::://ppeeevveeee wttpreeeeieeeeeefrreeeeeeee www.nneeeetmniiiimmmmmm (inoorggmidd
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. arXiv preprint arXiv:2310.01110, 2023b.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Decomposed diffusion sampler for accelerating large-scale inverse problems. In The Twelfth International Conference on Learning Representations, 2024. URL https::tp///t.... /tpp::://////eeeeieeeeee net/ stttfformmdddddshhttttt
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Bradley Efron. Tweedie's ' formula and selection bias. Journal the American Statistical Association, 106(496);1602-1614, 2011.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021. URL https://openrevii/ net/ Forum?id=qw8AAxxYYb.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
//twitter wtateerrooo/eeeee Twitter.
Zahra Kadkhodaie and Eero P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: / //tps////e////.w..///m net net:/forumiddexShNNBuuu
11

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurlPS, 2022.
Beomsu Kim, Yu-Guan Hsieh, Michal Klein, Marco Cuturi, Jong Chul Ye, Bahjat Kawar, and James Thortoo.. Simple reflow: Improved techniques for fast flow models. arXiv preprint arXiv:2410.07815. 2024a.
Jeongsol Kim, Geon Yeong Park. Hyungjin Chung, and Jong Chul Ye. Regularization by texts for latent diffusion inverse solvers. arXiv preprint 1. 2023.
Jeongsol Kim, Geon Yeong Park, and Jong Chul Ye. Dreamsampler: Unifying diffusion sampling and score distillation for image manipulation. arXiv preprint arXiv:2403. /10000./1455, 2024b.
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724 2024.
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint 1xXiv:331111124. 2023.
Shanchuan Lin, Anran Wang, and Xiao Yang. Saxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402. 13929, 2024.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Cheng Lu. Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022a. URL V.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX10:221..01095 2022b.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEFJCVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye. Energy-based cross attention for bayesian context update in text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Anton Pelykh, Ozge Mercanoglu Sincan, and Richard Bowden. Giving a hand to diffusion mod- els: two-stage approach to improving conditional human image generation. arXiv preprint 1031v:2403.10731 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXivv2307.01955, 2023.
Ben Poole. Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. Xrvivv 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695. 2022.
12

---

Published as a conference paper at ICLR 2025
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprini arXiv:2311. 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations, 2024. URL j8hdRqoUhN.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference Learning Representations, ICLR, 2021a.
Jaaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guidde diffusion models for inverse problems. In International Conference on Learning Representations, 2023. URL https::/opeerevieee
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR, 2021b.
Bram Wallace, Akash Gokul, and Nikhil Naik. EDICT: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 22532-22541, 2023.
Zhengyi Wang, Cheng Lu. Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: figh-fiieeity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Algorithm 3 DDIM Inversion with CFG
Algorithm 4 DDIM Inversion with CFG++
Require: 2o. 0 I
Require: æo, À [0,
I: for to T
for i = to
do
2: cc (xt) (21))) + [((((())))) éΦ (2x
2:
(xx)) cx + ((6n((s)) En(x)))
3: * (x+) + (ote vI de ( (2r))// Vā: 0r
3: (2(x)) t (ær vI a.ce (xx))/ vas
4: 2:41 = vau+ de (z)) + vI a1+1 (2))
4: m++1 / (mre) v1 6
5: end for
5: end for
return 27
return at
Figure 8: Comparison between DDIM inversion. CFG++ proposes simple yet effective fix: using Eo (2t) instead of E (zx) in Tweedie's denoising step.
A EXTENSION OF CFG++ TO OTHER SOLVERS
In this section, diseuss ways in which solvers. We consider solving the variance exploding (VE) PF-ODE as presented in (1), as re- parametrization VP diffusion models can easily recover the VE formulation, as often implemented in widely used frameworks such as https: /github....ttteegggggg thub. com/ http///oowoo////// berrsokkkkkkiiffffe diffusion Following the notation Lu al. (2022b), we consider sequence timesteps {tz {t+) where to T denotes the initial starting point of the reverse sampling (i.e. Gaussian noise).
apply CFG++ to diverse set of ODE/SDE
Euler (Karras et al., 2022) The construction is the same as in DDIM. We include it here for completeness. The update step reads
Zt+t1 = 2e (xt,,) + Tr. # (3:t:)
Ot+t1
(CFG)
tt
1611 = (x1,) + Ft. te (x
011117
(CFG++)
Te:
Euler Ancestral Euler Ancestral sampler follows Euler sampler, but introduces stochasticity by taking larger steps and then adding slight amount of noise. Adjusting with CFG+ is straightforward.
1 # (3)t;) Ct, a (ar.), (atu, -r;)) + ot.6
(CFG)
Tt Thas x (xt,) + Ft; (arts) (ora, ot,) + t,
(CFG++)
0f:
where t, > td. > t++1 and e N)0,
DPM-solver++ 2M (Lu al., 2022b) Define or == e- h4 = t. 4 4-11 and r, = h4-1/hi-
After initializing It, with Gaussian noise, the first iteration is given by
St; 2 (xtt) + h1 (att x (at,))
(CFG)
3t, a (2tt)) + -he -xt Dg (z),)) The following iterations by using the standard CFG reads
(CFG++)
D; a% (ft:-1) + 2r; St. hi Ei. (e- 1)D:
(# (#%( 2: (ft;-2))
(23)
(24)
Rearranging (23), (24), we can rewrite the update steps as
(mt.)) ) + 1
Ft, = th (
( (Zt..)) - (2 + a
(25)
2r;
Notice that in order to apply CFG++ to (25), we should only keep the first term as the conditional Tweedie, and use the unconditional estimates for the rest of the components. i.e.
1-c- hh
2t, (x hhe (x).,,)
(#0()))))) .2(xx,,))) + Di (26)
2r;
14

---

Published as a conference paper at ICLR 2025
DPM-solver++ 2S (Lu et al., 2022b) With the same choices of oe hi additionally define the timesteps {st [s with > 8141 441 Further, let 5:-f; iteration reads
Using standard CFG, the 4--4-1
U, - 2 0t-11 + (1 rihi) (zt._1)
(27)
h
Ft. = 2e (I))_1) - - te (xt;
+
( (u:)
2 (2t_))) + -h zta
(28)
2r:
Applying the general transition rule from CFG to CFG++ introduced in (15), we can keep all of the Tweedie estimates to be unconditional, and only change the first term of (28), i.e.
u, = e - 1t
e er.hi (x)..1)
(29)
1 1 eh
Zt. = Ee (2t - e hi th (x)
(u.)) Do (IL..)) + t x
(30)
2r:
For the ancestral version of DPM-solver++ 2S, we can follow the general transition rule, as was shown for Euler - Euler Ancestral.
B EXTENSION OF CFG++ TO FLOW MATCHING
Here, we generalize CFG++ with flow matching which subsumes diffusion models as specific instantiations. Let Po and Pi be two data distributions on R" and Qoi denotes a coupling of po and pr. Then, the flow matching loss defined as follows:
FFM (0; Qoi) E ) )~QQ
io.oo.t.umii (-mmit()) ( æo, vo (o),,))))
(31)
where 2o R (0, + R refers velocity parameterized by 6, ( (x. y)
||
y and
2, = (1 - t) tæ This objective can be reformulated in the form of denoising as follows:
4m : Qun) = E an Qo (00ooo((0m0))
ÉMSE (zo, Do (o(
(32)
where Do( x2 t t) serves denoiser function. Then, derive an ODE which translates samples from po and p1 given a vector field Vo:
Dz Do(xI, t) di, LE (0,1),
dx, = (2 l)di
(33)
t
which resembles (1) with choice of t. For text-conditional flow, CFG can be applied to (33) by solving the following ODE (Kim et al., 2024a):
dæ; = Uo (00( t, Ø) + wVs (Zi, t, c) - vo(a,, 2))] dt = 5% (x,),t, t E (0,
(34)
We similarly define a (ax))
16 (x:), ãe (ax)) = n. vv L, L) tū (ax)), and
a (ax)) æt tū (xx)) Based this ODE,
reproduce results from diverse set of solvers in Sec. A. For example, single update step of Euler solver reads:
0t,41 = -- (xt,) + It. xdd (xx.)
t.41.
(CFG)
Fo4 = (2 Et.
(0)4) -441-
(CFG++)
t
This implics that CFG++ is potentially compatible with various flow-based generative models.
C EVOLUTION OF THE POSTERIOR MEAN: CFG vs. CFG++
Recall that one can equivalently view the evolution the posterior mean E|x, [x,] rather than the noisy variables E₁. In this context, derive the sequential evolution of conditional posterior mean a and a through time to further understand the underlying behavior of the proposed sampling.
15

---

Published as a conference paper at ICLR 2025
Proposition 1. Let (z(x)) == z(x1) =xx+11 denote the discrete time evolution of some random variable z at time f. Then, the evolution of a of CFG and a: of CFG++ is given by
-1-
dæ (x))
dê (a:) w(((xx c) A(x(+1,)))
(35)
(
dē (xt)
+ c)
(36)
Vit
uncond. shift
cond. shift
where A(z,, == (x())) - 2z (xt).
Proof. We start by writing the iteration from + 1-7 V Va (I141) + /1 + (244))
(37)
The Tweedie estimate for the next step is then written as
2e (xx vI 0 (a::)
(38)
vā v Varua (2(+1) + v1 an(e (2(41) (xx,))
(39)
Vār
=x0(1)11) + VT VT- (2(4))))) - (x)) + w(Cc(1141) 20 (2(+1))
Vā
w((((())) - (x(())
(40)
Using the relation 2: (xt) (Vār (xx)) xt)\ /01vvV/ cis we have
vI vI- αi
d
dê (xx) + w (#c( (x()) ( (2(+1))
(41)
Vāa
Similarly, for CFG++
2t VI a
(xx)
(42)
Vat
(2 + VI Ci (E (I(+1) (x())
(43)
Vār
(2041) + -- αe
Tē (1141) - (ar))) - -((x((x)) - -
(44)
Vot
Hence,
VI- ac Vou.
di (x.
dē (xx) x((t((x)) 2e (2:)),
(45)
Proposition I implies that the CFG++ update of a is decomposed into shift terms: dē (x;) represents the difference between consecutive unconditional scores (i.e. unconditional shift), and 2) A(x; c) denotes the direction of conditional guidance time (i.e. conditional shift). The conditional shift term is multiplied by small interpolation factor À in the case of CFG++, inducing small nudge toward the condition.
CFG eh on the other hand, has the same unconditional shift, but has oscillatory behavior for the conditional shift. The difference between CFG/CFG++ sampling arises from the unexpected additional shift from the previous timestep é 1+1 that exists for the CFG decomposition: -A(x, c). and the scaling constant w. The initial conditional shift wA (z,, e) with a large w pushes the trajectory off the manifold, but cancels some of its effects by subtracting the conditional shift from the previous step w (x c). The compounded vector A (I,, c) A ( 1, c) does induce a nudge closer to the condition but requires large value of w have meaningful effect, and thus is hard interpret.
16

---

Published as a conference paper at ICLR 2025
Continuous Transition of CFG Coefficients
(CFG
0.12
1.0.0 1.0.=110000 (CP))))
0.10
Coefficient Value
CFG++ (w1=1.0, w2=0.0)
(w1110, w2-1.0)
0.00
200
600
Time Steps
Figure 9: Continuous transition between CFG and CFG++ schedule
D CFG++ AS REWEIGHTED CFG
One can rewrite the update step of CFG and CFG++ succinctly as
ai ae + V āG-
(46)
Ct /1 - ā. (2te)
vI 1
(47)
Vor
where for CFG, w/ = and for CFG++ w' = 0. Note that we are using the same notation w here, but the guidance scale (represented as À in the previous sections), much smaller. One can derive
wV1 āelCe éΦ)
Ft-1 Vdi-1 B2
+ /1 ā,1168 / Cr- (Ee (ê))
(48)
Vai
WVT arlbe -ē0)
-Voor-1me + V - -
+ v ju- - (0) (49)
vai
1 Út-1
= 21-1,0 + (Be - EΦ)
w'-1-an a w/
(50)
Here, we see that the guidance scale used in the original CFG is a composition of two time-dependent functions. When composited with the same strength for w' and w, depicted in Fig. (b). the guidance scale of CFG peaks at the earlier stages of sampling, drops down, then rises back up. The sudden peak in the earlier stages may explain the unnatural saturation the earlier stages of sampling using CFG. In contrast, the guidance scale of CFG++ has convex-like function, where the scale gradually increases to some value.
By enforcing that the guidance scales integrate to 1 (i.e. enforce it as PDF), can generalize CFG/CFG++, and scrutinize the behavior when we interpolate between these two by setting 0 < w/ w. The guidance scale functions for these interpolations shown in Fig. 9. In Tab. 4, that the image quality measured by FID and ImageReward on COCO- data, gradually gets better transition from CFG++ CFG.
17

---

Published as a conference paper at ICLR 2025
W 0.0 (CFG++)
w = 0.4
w/ 0.6
w/ 0.8
w 1.0 (CFG)
ImageReward |
-0.112
-0.125 66.60
-0.126
-0.153
-0.420
FID
66.52
66.78
67.50
68.10
Table 4: Quantitative results of COCO-1k by interpolating between CFG and CFG++
E EXPERIMENTAL DETAILS
E. TEXT-CONDITIONED INVERSE PROBLEMS
Problem settings. We evaluate our approach across following degradation types: 1) Super-resolution with scale factor x8, 2) Motion deblurring from image convolved with a 61 x 61 motion kernel, randomly sampled with an intensity value 0.3². Gaussian deblurring from image convolved with X 61 kernel with an intensity value 0.5, and 4) Inpainting from 10-20% free-form masking, implemented in (Saharia et al., 2022). In inpainting evaluations, regions outside the mask are overlaid with the ground truth image.
Datasets, Models. For evaluation, use the FFHQ (Karras et al., 2019) 512x512 dataset and follow (Chung et al., 2023a) by selecting the first 1,000 images for testing. For the pre-trained latent diffusion model including baseline methods, choose SD v1.5 trained on the LAION dataset. As baseline for latent DIS, we consider PSLD (Rout et al., 2024). enforces fidelity by projecting onto the subspace of A during the intermediate step between decoding and encoding, in conjunction with the DPS (Chung et al., 2023a) loss term. For gradient updates in both vanilla PSLD and PSLD with CFG, use static step sizes of 17 1.0 and y 0.1 recommended in (Rout al., 2024). For CFG scale w. we applied the corresponing scales that we found to be corresponded for CFG++ scale X. Please refer the Tab. 5 for the hyperparameters used for PSLD with CFG++
SR(x8)
Deblur(motion)
Deblur(gauss)
Inpaint
1.3
0.4
1.0
1.0
7
0.025
0.12
2.0
2.0
Table 5: Hyperparameters for PSLD (Rout et al., 2024) with CFG++ and corresponding CFG scale w.
F FURTHER EXPERIMENTAL RESULTS
F.I T2I
Since the range of guidance scales for CFG and CFG++ is different, we matched the guidance values of w and for comparison by computing the LPIPS distance using the same seed.
Fig. 14 shows the clean estimates at each diffusion sampling time 4, computed using Tweedie 's formula, both with and without CFG. evident that the original CFG induces significant error, particularly earlier times, leading to off-manifold samples. However, the CFG++ addresses this issue by adjusting the DDIM re-noising step and the guidance seale.
F.2 REAL IMAGE EDITING
In Fig. 17-19, we provide qualitative comparison on real image editing via DDIM inversion with CFG and CFG++. For the DDIM inversion stage, we use "a photogr aphy of [source concept]" as conditioning prompt. For the sampling stage, we swap [source concept] to [target concept] and use it as conditioning prompt for generation. For example, in Fig. 17, we set [source concept] to "dog" and [target concept] to "cat". For experiments, we set the guidance scale as w = 9.0 and 0.8 as described in the main paper. The comparison demonstrates that CFG++ successfully edits the given image which was not possible by CFG. This results also support our claim on reduced error during DDIM inversion by CFG++.
18

---

Published as a conference paper at ICLR 2025
"selfie woman and her lion cub the plains"
"dog jumping front of moon gate, purple flowers, snowy mountain"
CFG
CFG++
"woman sniper, wearing soviet army uniform, snow ground"
"fisherman sitting on a tropical beach at sunset with bending palm trees"
CFG
CFG++
Figure 10: Enhanced T2I results by SDXL (w 9.0, = 0.8) with CFG++ Under CFG, the lion cub is not visible (top-left), the dog appears with two tails (top-right), the goggles have an unusual shape (bottom-left), and the tree trunk is folded (bottom-right), These artifacts are absent in those produced by CFG++
F.3 CENDICONDD INVERSE PROBLEMS
In Fig. 20-23, display additional qualitative comparison for text-conditioned inverse problem solver with CFG and CFG++ Experiments are conducted with FFHQ (512x512) validation set and CFG++ consistently leads to better reconstruction of true solution for various tasks.
19

---

Published as a conference paper at ICLR 2025
"A wearing suit taking self portrait with camera"
coffee shop with neon sign ... little girl with red hair...
"A painter study hard to leam how to draw
COFFEE
CFG
Coffe
CFG++
Figure 11: T2I using SD v1.5, CFG vs CFG++ 9.0, X 0.8). Unnatural depictions human hands, and incorrect renderings of the text by CFG are corrected in CFG++
"Large brick building with a clock tower at its height."
"SSpectacuaar Tiny World in the Transparent Jar...
"A silhouette of grand piano overlooking dusky cityscape....
CFG
CFG++
Figure 12: T21 using SDXL-Turbo, 6 NFE, CFG vs CFG++ The overall image quality and sophisti-
cation have improved with CFG++. DreamShaper XL was used for both images and metries in main part.
20

---

Published as a conference paper at ICLR 2025
"An alpaca made of colorful building blocks, cyberpunk."
"Floating, colossal, futuristic statue in the sky, ...."
detailed painting sea captain, steering his ship..."
CFG
CFG++
Figure 13: T21 using SDXL-Lightning, 6 NFE, CFG CFG++ The overall image quality and sophistication have improved with CFG++. Leosam's HolldWorld XL was used for both images and metrics in main part.
21

---

Published as a conference paper at ICLR 2025
1=900
800
700
600
400
CFG++
"A small cactus with happy face in the Sahara desert."
CFG
CFG++
*A baby painter trying very simple picture, white background"
CFQ
CFG+
"A blue jay standing on large basket of rainbow macarons."
CFG+
"A transparent sculpture of duck made out of glass. The sculpture is in front of painting of a landscape."
CFG++
"A 4k dair image of a lemur wearing red magician hat and blue coat performing magic tricks with cards in garden."
Figure 14: The discrete evolution of the posterior mean in CFG and CFG++ Denoised estimates in latent space decoded into pixel space at each timestep using SDXL.
22

---

Published as a conference paper at ICLR 2025
2.
9.6
NEW RILE
NOEW SEPENTOR
FYJEW SUTSENTEL
The AlmMnn ROO tonstralior
The attirat 1io0 tenst allsr
OOL
CFG
NOEW REP
NOEW SETSERTOL
NOEW SUPENTEL
FORE SUTSENTEL
the alt West ROC tenstralion
WA
CFG++
OZ
0.4
U.6
0.8
Figure 15: Interpolating behavior of T2I along various guidance scales using SD v1.5 with CFG and CFG++
Image
OU 3.0
7.5
9.0
(CFG)
0.2
2 0.4
2 0.6
= 0.8
(CFC
Figure 16: Real image inversion results various CFG and CFG++ scales using SDv1.5. 1.5. The image is from COCO dataset. We use the text prompt "a white dog is sleeping street and bicycle" during the inversion.
23

---

Published as a conference paper at ICLR 2025
Source
CFG
CFG++
Source
CFG
CFG++
Cat - Dog
Figure 17: Comparison on real image editing via DDIM inversion under CFG and CFG++ using SDXL. Cat - Dog.
Source
CFG
CFG++
Source
CFG
CFG++
Horse -> Zebra
Figure 18: Comparison on real image editing via DDIM inversion under CFG and CFG++ using SDXL. Horse Zebra.
24

---

Published as a conference paper at ICLR 2025
Source
CFG
CFG++
Source
CFG
CFG++
Cat - Cat wt glasses
Figure 19: Comparison on real image editing via DDIM inversion under CFG and CFG++ using
SDXL. Cat - Cat with glasses.
GT
Degraded
CFG
CFG++
Figure 20: Results of PSLD using CFG and CFG++ the FFHQ dataset at Super- Resoluttso (x8).
25

---

Published as a conference paper at ICLR 2025
GT
Degraded
CFG
CFG++
ia
Figure 21: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Motion Deblurring.
GT
Degraded
CFG
CFG++
Figure 22: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Gaussian Deblurring
26

---

Published as a conference paper at ICLR 2025
GT
Degraded
CFG
CFG++
Figure 23: Results of PSLD using CFG and CFG++ on the FFHQ dataset at Inpainting.
27


Reviews:
[{"review_id": "gIBxxSD2Q0", "reviewer": "Reviewer_hzx1", "summary": "This paper proposes to tackle Classifier Free Guidance's (CFG) limitations, particularly its lack of invertibility and mode collapse. The authors formulate the hypothesis that these limitations come from the off-manifold phenomenon. The proposed method leverages recent approaches developed for inverse problem-solving through diffusion models. The proposed method named CFG++ is compared with CFG for text-to-image generation, image edition or image restoration.", "questions": "* Having the average performance across the different corruption in Tab. 3 would help. Here, the proposed method seems more efficient than CFG to retrieve the clean image according to the FID metric but not so much according to the LPIPS one. Do the authors have some justifications?\n* It is unclear why the performances of CFG drop while augmenting the NFE in Fig. 6 (b).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is generally well-written with a useful high-level preliminary section. One of the strengths of the method is its simplicity while being well justified. The paper proposes many experiments showing relative gains compared to CFG.", "weaknesses": "* The CFG method formulation seems different than in the original paper see eq (6) from (Ho and Salimans, 2021). Moreover, the $w=0$ setting in the experiments should be equivalent to no guidance at all when using CFG according to the eq (6) (from the CFG++ method). In Fig. 6, however, the images appear edited with $w=0$ indicating some contradiction.\n* The ImageReward metric is not defined. The FID and CLIP metrics are very similar between CFG and CFG++ especially when using SD v1.5. In Tab. 1 the FID and CLIP are also very close for both methods. This tends to indicate that the gains are relatively marginal even in a low NFE setting.   \n* The statement that the proposed CFG++ is invertible and brings more diversity in the generation is not evaluated.\n\n**Typos**\n\n* The variable $x_c$ is not introduced in line 228\n* Table 1., column 2, Clip score arrow does not show the correct direction\n* line 238 ``a crucial difference in the **renoising** process'' shoulnd't it be denoising?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xhDDmyaJgy", "reviewer": "Reviewer_7gyf", "summary": "The paper introduces CFG++, a novel approach to fixing the off-manifold issues that can occur in CFG (classifier-free guidance) during sampling. They reformulate CFG as a manifold-constrained problem, transforming the conditional guidance from extrapolation to interpolation between unconditionally sampled trajectories and conditionally sampled trajectories, thus making the guidance more interpretable. Experimental results show that the method can reduce the artifact, solve the DDIM inversion problem, and be integrated into the high-order diffusion solvers.", "questions": "1. SDXL-Turbo does not make use of guidance_scale or negative_prompt. Instead, they disable it with guidance_scale=0.0. I would like to know how to verify the validity of CFG/CFG++ in this case.\n\n2.  Does this method also work for negative prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. CFG++ introduces a novel approach by redefining classifier-free guidance as a constrained manifold problem, which effectively improves the generation quality by staying within the data manifold.\n2. This paper validates the effectiveness of the proposed method on standard benchmarks and different tasks (e.g., text-to-image generation, inversion, editing).\n3. The paper is well-written and easy to follow.", "weaknesses": "1. There are a few minor issues in the tables, such as the direction of the arrow in the fourth column of CLIP in Table 1 (which should be as high as possible) and the missing ImageReward metric for SD v1.5 in Table 2. \n\n2. This paper focuses on addressing the inability to perform successful DDIM inversion with classifier-free guidance, so it should include a comparison of the results with this class of methods like [1].\n\n3. I'm a little suspicious of ''Contrary to the widespread belief that these are inherent limitations of diffusion models, this paper reveals that these problems actually stem from outrageous phenomena associated with CFGs...'' in the Abstract. As far as I know, current work has generally recognized and worked for the CFG problem.  Perhaps rewording would better reflect the originality of the paper.\n\n[1] Mokady, Ron, et al. \"Null-text inversion for editing real images using guided diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KpLD5XdYnn", "reviewer": "Reviewer_GmUK", "summary": "This work introduces an enhanced version of the widely used classifier-free guidance (CFG) technique, CFG++. By formulating conditional generation as an inverse problem and utilizing score distillation sampling (SDS) loss, CFG++ improves various tasks, including text-to-image generation, DDIM inversion-based editing and text-conditioned inverse problems.", "questions": "1. Only diffusion-/score matching-based generative models are discussed in this work. Could you please **provide some similar derivations for the recent flow matching-based generative models, such as SD3 and FLUX**? I believe similar conclusions stand for flow-based methods, and it will make this work more comprehensive.\n\n2. Is it possible to **disentangle two noises used in the denoising and renoising process**? For example, we can set different hyperparameters $\\lambda_1$ and $\\lambda_2$ for $(\\epsilon_c-\\epsilon)$, and use this term in different weights for denoising and renoising respectively. Setting $\\lambda_1=\\lambda_2$ is the same as CFG and setting $\\lambda_2=0$ is the same as CFG++.\n\n3. Is it possible to **provide derivations of CFG++ for other diffusion solvers**, except DDIM? Extensions of CFG++ to other solvers in Appendix A are more like intuitive understanding, rather than derivations from inverse problems and SDS loss, as done for DDIM.\n\n4. Essentially, **CFG++ can be written as reweighted CFG whose $\\omega$ varies along the sampling process** (let $s\\coloneqq t-1$ and $\\epsilon\\coloneqq \\epsilon_\\emptyset$ for easier LaTeX rendering in OpenReview):\n\n- For DDIM CFG: $$\\mathbf{x}_s=\\frac{\\sqrt{\\bar{\\alpha}_s}}{\\sqrt{\\bar{\\alpha}_t}}\\mathbf{x}_t-\\frac{\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}-\\sqrt{(1-\\bar{\\alpha}_s)\\bar{\\alpha}_t}}{\\sqrt{\\bar{\\alpha}_t}}(\\omega\\epsilon_c-(\\omega-1)\\epsilon)$$\n\n- For DDIM CFG++: $$\\mathbf{x}_s=\\frac{\\sqrt{\\bar{\\alpha}_s}}{\\sqrt{\\bar{\\alpha}_t}}\\mathbf{x}_t-\\frac{\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}}{\\sqrt{\\bar{\\alpha}_t}}(\\lambda\\epsilon_c-(\\lambda-1)\\epsilon)+\\sqrt{1-\\bar{\\alpha}_s}\\epsilon$$\n$$=\\frac{\\sqrt{\\bar{\\alpha}_s}}{\\sqrt{\\bar{\\alpha}_t}}\\mathbf{x}_t-\\frac{\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}-\\sqrt{(1-\\bar{\\alpha}_s)\\bar{\\alpha}_t}}{\\sqrt{\\bar{\\alpha}_t}}(\\frac{\\lambda\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}}{\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}-\\sqrt{(1-\\bar{\\alpha}_s)\\bar{\\alpha}_t}}\\epsilon_c-\\frac{(\\lambda-1)\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}+\\sqrt{(1-\\bar{\\alpha}_s)\\bar{\\alpha}_t}}{\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}-\\sqrt{(1-\\bar{\\alpha}_s)\\bar{\\alpha}_t}}\\epsilon)$$\n\nSo, $$\\omega=\\frac{\\lambda\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}}{\\sqrt{(1-\\bar{\\alpha}_t)\\bar{\\alpha}_s}-\\sqrt{(1-\\bar{\\alpha}_s)\\bar{\\alpha}_t}}.$$\n\nAs discussed in Sec. 5, previous studies also propose adjusting the guidance scale across timesteps. However, it may be incorrect to claim that \"these findings are orthogonal to ours and keep the sampling trajectory the same ... CFG++ is designing a different trajectory\".", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, this manuscript is excellent, and I hope it gets accepted. Below, I outline some strengths that support this view:\n\n1. **Well-written**: The paper is well-written with coherent logic and clear notation, making it easy to understand and follow.\n\n2. **Thorough Experiments**: The work studies many downstream tasks, including text-to-image generation for SDv1.5, SDXL, and their distilled versions, as well as DDIM inversion and editing, PF-ODE trajectories, and text-conditioned inverse problems. The authors provide extensive quantitative and qualitative results to show the superiority of the proposed method.\n\n3. **Intriguing Derivation and Analysis**: The derivation of CFG++ is intriguing by treating text-conditioned generation as an inverse problem and utilizing SDS as the loss function. Moreover, the authors present various perspectives to analyze CFG++, including manifold geometry, score matching loss throughout the denoising process, and the evolution of the posterior mean. These analyses are insightful and helpful in understanding the underlying mechanisms of CFG++.\n\n4. **Simple Yet Widely Applicable Method**: The proposed method essentially modifies the re-noising process of the original CFG, but it achieves improvements across various downstream tasks and could be potentially effective for other tasks relevant to the ICLR community. Additionally, the DDIM solver, as well as other popular diffusion solvers like EDM and DPM-Solver, are derived, establishing CFG++ as a general method.", "weaknesses": "Generally, I think there are no obvious weaknesses in this work. However, there are some questions and please refer to the Questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2Kcfqb0BL", "reviewer": "Reviewer_NrB8", "summary": "The paper presents CFG++, an innovative approach designed to enhance the performance of diffusion models by addressing limitations associated with traditional Classifier-Free Guidance (CFG). The authors argue that several drawbacks of CFG, such as the lack of invertibility in DDIM (Denoising Diffusion Implicit Models) and the issues of mode collapse when high guidance scales are used, arise from the off-manifold phenomena related to CFG rather than the inherent characteristics of diffusion models themselves.", "questions": "**Question**: Could you elaborate on the specific mechanisms through which off-manifold behavior impacts the performance of traditional Classifier-Free Guidance (CFG)?  A detailed explanation of how off-manifold behavior is identified and measured within the context of your experiments would enhance understanding. Including visualizations or theoretical examples could further clarify this concept.\n\n**Question**: What are the reasons behind selecting the specific datasets used in your experiments? Are there plans to test CFG++ on a wider array of datasets or tasks?  Expanding the experimental framework to include diverse datasets and applications would strengthen the validity of your findings. Consider including datasets from different domains (e.g., text-to-image generation, video synthesis) to demonstrate the versatility of CFG++.\n\n**Question**: Why were certain state-of-the-art methods, such as Stable Diffusion and other recent advances in guided diffusion, excluded from your comparative analysis?  Including a more comprehensive set of comparisons with these methods would provide a clearer context for evaluating the performance of CFG++. Detailed performance metrics and qualitative results could enrich the discussion.\n\n**Question**: Can you provide more in-depth mathematical justifications or derivations that support the effectiveness of CFG++ in addressing the limitations of traditional CFG?  A rigorous theoretical framework would enhance the paper's credibility. Detailed mathematical formulations explaining how CFG++ operates under manifold constraints could help bridge the gap between theory and practice.\n\n **Question**: How does CFG++ perform in terms of computational efficiency and scalability with larger models or datasets?  Including a discussion on scalability, including any computational overhead observed during experiments, would be valuable. Experiments assessing performance on larger models could illustrate the practical applicability of CFG++ in real-world scenarios.\n\n **Question**: Have you considered conducting user studies to evaluate the perceived quality and usefulness of the outputs generated by CFG++?  Incorporating qualitative assessments from human evaluators could provide insights into the model's effectiveness from a user perspective. Such studies could help highlight the practical implications of using CFG++ in various applications.\n\n\n**Question**: What future research directions do you foresee emerging from your findings regarding CFG++ and manifold constraints in diffusion models?  A discussion on potential extensions or related areas of research would help contextualize your contributions within the broader landscape of machine learning, inviting collaboration and further inquiry.", "limitations": "", "rating": 1, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper exhibits a high degree of originality by introducing the CFG++ framework, which fundamentally reinterprets the limitations of traditional Classifier-Free Guidance (CFG). Instead of merely enhancing existing methodologies, the authors propose that many issues attributed to diffusion models arise from off-manifold behavior. This shift in perspective not only challenges established assumptions but also provides a fresh lens through which to analyze and improve guidance mechanisms in generative models. By integrating recent advancements in inverse problem-solving within diffusion contexts, the authors demonstrate a creative synthesis of ideas that broadens the scope of CFG applications.\n\n\nThe quality of the research is evident in both its theoretical foundations and empirical validation. The authors articulate their arguments clearly, supported by robust mathematical formulations and well-structured experiments. The comparative analysis illustrates the advantages of CFG++ over prior methods, effectively showcasing the model’s performance improvements in terms of reduced artifacts and enhanced image quality. Additionally, the use of rigorous benchmarks for evaluation strengthens the credibility of the findings. The paper’s methodology is sound, making it a valuable contribution to the field.\n\n\nThe clarity of the writing is commendable, as the authors navigate complex concepts with precision. The introduction provides a succinct overview of the issues addressed and the motivations behind the proposed solution. Throughout the paper, technical jargon is well-defined, ensuring accessibility for readers with varying levels of expertise. Figures and diagrams are effectively utilized to illustrate key points, facilitating a better understanding of the model's mechanics and results. Overall, the paper is well-organized, which enhances its readability and facilitates comprehension.", "weaknesses": "One notable weakness is the limited scope of experiments presented. While the authors provide comparative results demonstrating the efficacy of CFG++, the evaluation primarily focuses on specific datasets or tasks. To enhance the robustness of their claims, the authors could include a broader range of benchmarks across diverse domains. For instance, incorporating tasks from different image generation contexts (e.g., artistic style transfer, super-resolution) would provide a more comprehensive assessment of CFG++’s performance and generalizability. The authors should extend their experimental framework to include varied datasets and tasks, thereby demonstrating the versatility and applicability of CFG++ in different settings.\n\n\nAlthough the paper claims improvements over prior methods, it lacks a detailed comparison with a broader array of state-of-the-art techniques in diffusion models. For example, a comparison with models like **Stable Diffusion** or recent advancements in **Guided Diffusion** could contextualize the advantages of CFG++ more clearly.  The authors should include direct comparisons with a wider selection of contemporary approaches, utilizing standardized metrics (such as FID scores) to provide a clearer understanding of CFG++’s standing in the current landscape.\n\nWhile the paper introduces the concept of manifold constraints and off-manifold behavior, the theoretical underpinnings could be further strengthened. The authors briefly mention these concepts but do not provide a rigorous mathematical framework that details how CFG++ operates under these constraints or why these adjustments yield better results. A more thorough theoretical analysis, including mathematical proofs or derivations that illustrate the benefits of operating within manifold constraints, would bolster the paper's credibility and understanding.\n\n\nThe paper does not address how CFG++ scales with increasing model size or complexity. As generative models continue to grow, understanding how CFG++ performs under these conditions is crucial. There is a lack of discussion regarding potential computational overhead or challenges in real-world applications. Including analyses or experiments that assess the scalability of CFG++ with larger models or datasets, along with a discussion on computational efficiency, would provide valuable insights for practitioners considering its application.\n\nThe paper primarily focuses on quantitative metrics, such as image quality, but does not incorporate user-centric evaluations. Assessments based on human judgment, such as user studies to evaluate the perceived quality or usefulness of generated images, could provide a more holistic view of CFG++’s effectiveness.  The authors should consider conducting qualitative studies where human evaluators assess the output of CFG++ compared to other methods. This could provide insights into the model's real-world applicability and user satisfaction.\n\n### Conclusion\nWhile the paper \"CFG++: Manifold-Constrained Classifier-Free Guidance for Diffusion Models\" presents significant contributions, addressing these weaknesses would enhance its overall impact and clarity. By broadening experimental evaluations, strengthening theoretical foundations, and considering scalability and user-centric perspectives, the authors can more effectively support their claims and goals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)