Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models
Authors: Seyedmorteza Sadat
Otmar Hilliges
Romann M. Weber
Venue: ICLR 2025 Poster (2025)
Keywords: classifier-free guidance
diffusion models
Primary area: generative models
TL;DR: We propose a method to mitigate oversaturation and artifacts of  high classifier-free guidance scales in diffusion models.

Abstract: Classifier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models. While a high guidance scale is generally required to enhance these aspects, it also causes oversaturation and unrealistic artifacts. In this paper, we revisit the CFG update rule and introduce modifications to address this issue. We first decompose the update term in CFG into parallel and orthogonal components with respect to the conditional model prediction and observe that the parallel component primarily causes oversaturation, while the orthogonal component enhances image quality. Accordingly, we propose down-weighting the parallel component to achieve high-quality generations without oversaturation. Additionally, we draw a connection between CFG and gradient ascent and introduce a new rescaling and momentum method for the CFG update rule based on this insight. Our approach, termed adaptive projected guidance (APG), retains the quality-boosting advantages of CFG while enabling the use of higher guidance scales without oversaturation. APG is easy to implement and introduces practically no additional computational overhead to the sampling process. Through extensive experiments, we demonstrate that APG is compatible with various conditional diffusion models and samplers, leading to improved FID, recall, and saturation scores while maintaining precision comparable to CFG, making our method a superior plug-and-play alternative to standard classifier-free guidance.

Published as a conference paper at ICLR 2025
ELIMINATING OVERSATURATION AND ARTIFACTS OF HIGH GUIDANCE SCALES IN DIFFUSION MODELS
Seyedmorteza Sadat', Otmar Hilliges', Romann M. Weber
'ETH Zürich, "DisneeyeesearcciSttuios [seyedmorteza. sadat, otmar hilliges) ctt.e (romann. wober @di wnw..aeyeeearrcchccco com com
ABSTRACT
flassffier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models. While high guidance scale is generally required to enhance these aspects, it also causes oversaturation and unrealistic artifacts. In this paper, we revisit the CFG update rule and introduce modifications address this issue. We first decompose the update term CFG into parallel and orthogonal components with respect to the conditional model prediction and observe that the parallel component primarily causes oversaturation, while the orthogonal component enhances image quality. Accordingly, we propose Nown-weighting the parallel component to achieve high- quality generations without oversaturation. Additionally, we draw a connection between CFG and gradient ascent and introduce a new rescaling and momentum method for the CFG update rule based on this insight. Our approach, termed adaptive projected guidance (APG), retains the quality-boosting advantages of CFG while enabling the use of higher guidance scales without oversaturation. APG is easy to implement and introduces practically no additional computational overhead to the sampling process. Through extensive experiments, that APG is compatible with various conditional diffuusin models and samplers, leading improved FID, recall, and saturation scores while maintaining precision comparable to CFG, making our method a superior plug-and-ply -lay alternative to standard classifier-fre guidance.'
demonstrate
CFG
APG (Ours)
Figure 1: Classifier-free guidance is essential for generating high-quality images but causes over- saturation and unrealistic artifacts in the outputs. We introduce APG, novel method that keeps the quality of CFG but significantly reduces its harmful oversaturation. Both images are generated with Stable Diffusion XL (Podell et al., 2023) and a guidance scale of 15.
All visual results in the paper best viewed in color and when zoomed in.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) are a class of generative models that learn the data distribution by reversing forward process that adds noise to the data until the samples are indistinguishable from pure noise. Although the theory suggests that simulating the backward process in diffusion models should result in correct sampling from the data distribution, unguided sampling from diffusion models often results in low-quality images that do not align well with the input condition. Accordingly, classifier-free guidance (Ho & Salimans, 2022) has been established as essential tool in modern diffusion models for increasing the quality of generations and the alignment between the condition and the generated image, albeit at the cost of reduced diversity (Ho & Salimans, 2022; Sadat et al., 2024a).
Modern text-to- -oxt-to-image models, such as Stable Diffusion (Rombach et al., 2022), generally require high guidance scales in order for the generations to have better quality and align well with the input prompt. However, high guidance scales often result in oversaturated colors and simplified image compositions (Saharia al., 2022b; Kynkäänniemi et al., 2024). Despite these disadvantages, high CFG scales are still used in practice due to their superior image quality compared to alternatives.
In this paper, we analyze the update rule of CFG and show that with few modifications to how the CFG update is applied at inference, we can vastly mitigate the oversaturation and artifacts of high guidance scales, First, we show that the CFG update rule can be decomposed into two components, one that is parallel to the conditional model prediction, and one that is orthogonal to this prediction. We show that the orthogonal element mainly responsible for improving image quality, while the parallel part primarily adds contrast and saturation to the output. To the best of our knowledge, this is the first study that disentangles these two effects in CFG.
Additionally, we establish connection between the CFG update rule and stochastic gradient ascent. This insight leads us to explore a rescaled version of the CFG update direction and incorporate a momentum term, similar to adaptive optimization methods. The rescaling is motivated by the need to control large update norms, which can cause significant drifts in the sampling process. To prevent this, constrain the updates to lie within sphere. For the momentum term, unlike with traditional optimization, we apply a negative value to introduce a repulsive effect between consecutive updates, effectively down-weighting components already present previous steps. We refer to this as reverse momentum. By combining rescaling, reverse momentum, and projection, we introduce a new method, called adaptive projected guidance (APG), which allows the use of higher guidance scales without oversaturation or degradation in image quality.
Through extensive experiments with several diffusion models, such EDM2 (Karras et al., 2023) and Stable Diffusion (Rombach et al., 2022), we demonstrate that APG can utilize high guidance scales without encountering oversaturation. As a result, we conclude that APG significantly expands the usable guidance range in practice and mitigates the harmful effects of CFG at high guidance scales. Our quantitative analysis shows that replacing CFG with APG improves FID, recall, and saturation scores while maintaining precision similar to CFG. Furthermore, when combined with Stable Diffuusio 3 (Esser et al., 2024), APG enhances the consistency of text rendering in generated images. We also demonstrate that APG is compatible with distilled models that use fewer sampling steps, such as SDXL-Lightning (Lin et al., 2024b). A representative visual comparison between CFG and APG is shown in Figure 1.
RELATED WORK
Score-based diffusion models (Song& & Ermon, 2019; Song al., 2021b; Sohl-Diekstein et al., 2015; Ho et al., 2020) learn data distributions by reversing a forward diffusion process that gradually corrupts data into Gaussian noise. These models have rapidly outperformed previous generative modeling methods in terms of fidelity and diversity (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), setting new benchmarks across various domains. They have achieved state-of-the-ard results in unconditional image generation (Dhariwal & Nichol, 2021; Karras et al., 2022), text-to-image generation (Ramesh et al., 2022: Saharia et al., 2022b: Balaji et al., 2022; Rombach et al., 2022; Podell et al., 2023; Yu et al., 2022), video generation (Blattmann et al., 2023b;a; Gupta et al., 2023), maageeto-imagee translation (Saharia et al., 2022a: Liu et al., 2023a), and audio generation (Chen et al., 2021; Kong al., 2021; Huang et al., 2023).
2

---

Published as a conference paper at ICLR 2025
Since the introduction of the DDPM model (Ho et al., 2020), numerous advancements have been made, such as improved network architectures (Hoogeboom et al., 2023; Karras et al., 2023; Peebles & Xie, 2022; Dhariwal & Nichol, 2021), enhanced sampling algorithms (Song et al., 2021a; Karras et al., 2022: Liu et al., 2022b; Lu et al., 2022a; Salimans & Ho, 2022), and new training techniques (Nichol & Dhariwal, 2021; Karras et al., 2022; Song et al., 2021b; Salimans & Ho, 2022; Rombach et al., 2022). Despite these advancements, diffusion guidance, including both classifier and classifier-free guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), remains crucial in enhancing generation quality and improving alignment between the condition and the output image (Nichol et al., 2022), albeit at the cost of reduced diversity and oversaturated outputs.
A recent line of work, such as CADS (Sadat et al., 2024a) and interval guidance (IG) (Kynkäänniemi et al., 2024), has focused on enhancing the diversity of generations at higher guidance scales. In contrast, our proposed method, APG, specifically addresses the oversaturation issue in CFG, as these diversity-boosttin
methods still struggle with oversaturation at higher guidance scales. In Appendix C.I, we demonstrate that APG can be combined with CADS and IG to achieve diverse generations without encountering oversaturation problems.
Dynamic thresholding (Saharia et al., 2022b) was introduced to mitigate the saturation effect in CFG, but it is not directly applicable to latent diffusion models (since it assumes pixel values are between [-1,1] and tends to produce images lacking in detail. Another approach, CFG Rescale (Lin et al., 2024a), aims to reduce overexposure in generated images by rescaling the standard deviation of the predictions after applying CFG. However, we demonstrate that method is noticeably more effective at reducing oversaturation compared to CFG Rescale.
Orthogonal projection has been explored in the context of text-to-3D generation (Armandpour et al., 2023) and non-linear guidance (Zheng & Lan, 2024), but none of these methods tackle the saturation issue at higher guidance scales. We also demonstrate that naive projection has minimal impac on CFG behavior, as it must be applied to the denoised predictions to be effective. Additionally, we incorporate rescaling and reverse momentum to further mitigate the adverse effects of CFG higher guidance scales. We show that APG can be applied to various conditional diffusion models while adding practically no overhead to the sampling process.
BACKGROUND
We provide a brief overview of diffusion models in this section. Let æ ~ (a)m( represent a data point, and let 2; = x + a ( describe a forward process of the diffusion model that introduces noise to the data, where t E [o, 1] is the time step. Here, (t) is the noise schedule, which determines the amount of information destroyed at each time step t, with (0) = 0 and (1) mmax Karras et al. (2022) demonstrated that this forward process is equivalent to the following ordinary differential equation (ODE):
dz, = (t)oo xx log (z,)df,
where (z)) denotes the time- denendennddnt distribution of noisy samples, with po Paata and P1 (O,, of I). With access the deeedeeendent score function V., log Pt (2x), one can sample from the data distribution Pdata by solving the ODE backward in time (from ( = 1 to / = 0). The unknown score function Vz, log (z() is estimated using a neural denoiser Do f), which trained to predict the clean samples æ from the corresponding noisy samples 2t. This framework also allows for conditional generation by training a denoiser Dg Ze, t, y) that incorporates additional input signals y, such as class labels or text prompts.
Classifier- guidance (CFG) CFG is an inference method designed to enhance the quality of generated outputs by combining the predictions of a conditional model and an unconditional model (Ho & Salimans, 2022). Given null condition Y = Ø for the unconditional case, CFG modifies the denoiser's output at each sampling step as follows:
Dera (2,, 4 y) = Do (z,,,, I, Youll) + w w(Do(z,, y) - Do 4, Mmmn)),
where = represents the non-guided case. The unconditional model Dø(z,, E, Ymull) is trained
by randomly applying the null condition Ynull = Ø to the denoiser's 's input for a portion of training. Alternatively, separate denoiser can be trained to estimate the unconditional score in Equation (2) (Karras et al., 2023). Similar to the truncation method used in GANs (Brock et al., 2019), CFG improves the quality of images but reduces diversity (Murphy, 2023).
3

---

Published as a conference paper at ICLR 2025
4 ADAPTIVE PROJECTED GUIDANCE
We now present our method for addressing oversaturation and artifacts in CFG at high guidance scales. Let AD,- Da i, y) - Dø(zt, ; ynall) be the CFG update direction time step t. Note that Equation (2) can now be written as
DCFG t, = De(zi, t, + (w 1)AD,,
(See Appendix A for the derivation.) We use Equation (3) for the rest of this paper to motivate our changes. APG has three elements: (1) projection, (2) rescaling. and (3) reverse momentum. We
discuss each component below.
Orthogonal projection First, note that we can decompose AD, into two different components: AD which is parallel to Do(z,, 1. y), and AD,, which is orthogonal Do(z,, y), i.e., AD; AD + ADD. We can compute AD, via orthogonal projection, with
(AD,, Dø 1,1 AD (Do Z1, v)) y), (z( f, y)) Do(z,,t,9)
We then have ADF AD, - AD We observe that the orthogonal component is chiefly respon- sible for improvements in image quality, while the parallel component increases saturation in the generations shown in Figure 2.
Accordingly, we modify the update direction to form (D, = AD, + n where 7 S is a hyperparameter. Note that (1) is identical to the unmodified CFG update direction described above. We show that reducing the strength of the parallel component (i.e. setting r close to zero) significantly reduces saturation and results in more realistic generations higher guidance scales.
The intuition behind the saturating effect of the parallel component is helped by thinking of the output Do &,, f, y) an image with a typical range of values.² When update parallel to this image is added, it serves to create a "gain," " pushing the values toward the extremes of their range. This gain effect can be seen by direct calculation:
IAD!! De t, v) (w - 1)AD! 1 1w (w--w) DD 3)|l Do(zi,t,y).
(5)
where we note that the term in brackets on the right-hand side is greater than one for w >> 1. Thus, this term only adds saturation to the predictions (z y) during each inference step, much like multiplying pixel values by number greater than one. We show in Section 5.2 that reducing r) and leaning more heavily on the orthogonal component significantly attenuates this saturation side effect in generations while maintaining the quality-boosting benefits of CFG.
Adding rescaling Next, we argue that the CFG update rule in Equation (3) can be interpreted as one step of gradient ascent on the &2 distance between the conditional and unconditional prediction, i.e., one step of gradient ascent on De (D)(z,, v) De (t. t, Youu) |1 with a learning rate of (See Appendix A for proof.) Inspired by this interpretation and normalized gradient ascent, we rescale the CPG update rule at each step to regulate the impact of each update. Specifically, we constrain D, to be inside a sphere with radius via
AD, + AD; min L, |
(6)
where r is a hyperparameter. This rescaling ensures that the CFG update AD, stays closer to Do 1, y). limiting drift each sampling step if || is large. As demonstrated in Section 5.2, this adjustment improves both FID and recall.
Adding reverse momentum Finally, leveraging the connection to gradient ascent, we introduce reverse momentum term to the CFG update rule. We define the momentum for the CFG update direction as AD,- +- AD,- - 3AD, where AD, = 0 initially. The momentum term accounts for the average values of past updates; however, unlike standard optimization methods, we use negative
This intuition also holds for the image-like representations in latent diffusion models.
4

---

Published as a conference paper at ICLR 2025
(a) w/o CFG
(b) CFG
(c) CFG with AD!
CFG with AD;
Figure 2: Influence of the parallel and orthogonal components (AD) and ADF) in CFG. (a) The generation without CFG lacks quality and detail. (b) Applying CFG increases quality but introduces oversaturation. (c) Applying CFG only with the parallel component AD! barely changes the output quality compared to (a) and only increases saturation. (d) Applying CFG with only the orthogonal part ADz enhances image quality without causing oversaturation.
(a) w/o CFG
(b) CFG
APG (Ours)
Figure 3: Illustrating the effect of APG on generated images. (a) Sampling without guidance leads to low-quality generations. (b) CFG improves image quality but causes oversaturation. (c) Using APG instead of CFG results in high-quality generations without oversaturation.
momentum strength B < 0. Intuitively, this pushes the model away from previous CFG update directions and encourages the model to focus more on the current update direction. As shown in Section 5.2, incorporating reverse momentum further enhances image quality (i.c., lower FID scores).
APG is easy implement, and we provide the source code in Algorithm (appendix). As shown in Section 5.2, it is crucial to convert the diffusion model's ' outputs (e.g., predicted noise) into the denoised prediction (z f, y) in order to perform the projection. Further details on obtaining Dø(z,, f, y) for common prediction types are discussed in Appendix B. Figure 3 demonstrates that using APG instead of CFG produces high-quality generations without oversaturation or undesirable artifacts associated with high guidance scales.
5 EXPERIMENTS AND RESULTS
Setup We mainly experiment with text-to-image generation with Stable Diffusion (Rombach et al., 2022) and dass-conditional ImageNet (Russakovsky al., 2015) generation using EDM2 (Karras et al., 2023) and DiT-X1/2 (Peebles & Xie, 2022). For all experiments, we use the default diffusion sampler from each model (e.g., Euler scheduler for Stable Diffusion XL) along with pretrained checkpoints and corresponding codebases to ensure consistency in weights and the sampling process with the original frameworks.
primary metric for evaluating both the quality and diversity of generated images due to its alignment with human judgment. Since FID is sensitive small implementation details, ensure that all models are evaluated under the same setup. For completeness, we also report precision (Kynkäänniemi et al., 2019) additional quality metric and recall (Kynkäänniemi et al., 2019) diversity metric.
Distribution metrics We use Fréchet Inception Distance (FID) (Heusel et al., 2017)
5

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
CFG
APG (Ours)
CFG
APG OOurs)
Figure 4: Class-conditional generation results using EDM2 with = 4. APG significantly reduces saturation in the generations while keeping the high-quality global structure of each image.
CFG
APG (Ours)
CFG
APG (Ours)
CFG
APG (Ours)
CFG
APG (Ours)
Figure 5:
Text-to- to-image generation results using Stable Diffusion XL with w = 15. APG produces more realistic images compared oversaturated outputs of CFG.
Color metrics While FID measures the overall quality of generated images, we introduce specific metrics directly assess saturation and contrast. To measure saturation, convert each image from RGB to HSV and compute the mean of the saturation channel. We define contrast (also known as RMS contrast) the standard deviation of pixel values after converting the image grayscale. The final metrics are derived by averaging the saturation and contrast values across all generated images.
5.1 MAIN RESULTS
Qualitative results Figures and present our qualitative results comparing APG with CFG for EDM2 and Stable Diffusion XL. We observe that, compared CFG, APG generates more realistic images with noticeably lower saturation. Furthermore, APG appears to produce fewer artifacts the final outputs, illustrated in Figure 6. Additional visual results can be found in Appendix E.

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
A horse
small cactus with happy face in the Sahara desert
CFG
APG (Ours)
CFG
APG (Ours)
A ballet dancer next to a waterfall
A cat with queen hat
Figure 6: Examples of artifacts in the CFG outputs that can be solved by using APG. We that for all images, APG outputs follow the prompt with a more globally consistent generation.
Table 1:
Quantitative comparison between CFG and APG. APG consistently improves FIID, recall and color metrics while maintaining similar or better precision compared to CFG.
Model
Guidance
FID |
Precision t
Recall f
Saturation J
Contrast
CPG
10.42
0.85
0.48
0.46
0.27
EDM2-S (w
APG (Ours)
6.49
0.85
0.62
0.33
0.21
CFG
8.65
0.84
0.57
0.37
0.23
EDM2-XXI.
APG (Ours)
4.94
0.83
0.67
0.31
0.21
CRG
19.14
0.35
0.37
0.25
DIT-XL/2 (w
APG (Ours)
9.34
0.89
0.56
0.30
0.20
CFG
27.53
0.65
0.41
0.36
0.27
Stable Diffusion 2.1 (w 10)
APG (Ours)
22.21
0.67
0.49
0.27
0.22
CFG
26.29
0.62
0.49
0.28
0.24
Stable Diffusion XL (w 15)
APG (Ours)
25.35
0.64
0.50
0.18
0.17
Quantitative results We next present a quantitative comparison between APG and CFG in Table 1. The table shows that APG outperforms CFG across multiple models, consistently achieving better FID and recall scores, as well as lower saturation and contrast. Moreover, APG demonstrates similar precision to CFG, indicating that the reduction in saturation does not compromise the quality of individual samples.
Distribution of pixel values Figure 7 presents the kernel density estimate (KDE) plot of RGB and saturation values for 100 images generated using CFG and APG, along with KDE plots for 100 real samples drawn from the evaluation subset of ImageNet. Compared CFG, APG plots are more broadly distributed across the spectrum with less concentration at the extremes. This indicates that images generated with APG closer to real data in terms of saturation and color composition.
APG vs guidance scale In Figure 8, we demonstrate that the guidance scale increases, APG consistently achieves lower FID and higher recall while maintaining similar or better precision compared to CFG. Additionally, CFG exhibits increasing saturation at higher guidance scales, whereas APG maintains relatively constant saturation level. Therefore, APG allows the usage of higher guidance scales, achieving better FID and diversity without oversaturation.
7

---

Published as a conference paper at ICLR 2025
KDF of pixel values
KDE saturation values
CRG
Density
Deasity
0.3
50
100
150
200
200
-01
0.2
0.1
0.5
0.5
0.7
0.8
0.9
1.1
Figure 7: Kernel density estimates of pixel and saturation values for two sets of samples generated with CFG and APG. Compared CFG, images generated with APG show less concentration around saturated pixels, indicated by the spikes at the extreme values in both plots.
FID
Procodin
Rocal
Saturation
CTC And ithaa
06-
0,33
Figure 8: Comparison between CFG and APG as the guidance scale increases. APG offers better FID and recall while maintaining similar or better precision CFG higher guidance scales.
CFG
APG (Ours)
CFG
APG (Ours)
A pink dog
A plate of fries
Figure 9: Showcasing the compatibility of APG with distilled diffusion models using SDXXL- Lightning. Compared to CFG, using APG does not result in degradation in the output quality.
Improving diversity While APG is designed to address oversaturation at high guidance scales, we also observed that can enhance the diversity of generations. As shown in Table and Figure 8, APG improves distribution coverage (i.e., higher recall) while maintaining precision comparable to CFG. Additional qualitative results illustrating the enhanced diversity are provided in Figure 17 (appendix).
Using APG with distilled models A common issue with CFG that it degrades the quality of final outputs when applied to distilled models with fewer sampling steps (e.g., 8-step SDXIL-Lightning (Lin et al., 2024c)). In this section, we show that APG does encounter this problem and can be effectively applied to distilled models. Figure 9 demonstrates that replacing CFG with APG significantly improves generation quality. Extended results with additional models are provided in Appendix C.3, along with more visual examples in Appendix E.
Next, we demonstrate that integrating APG with Stable Diffusion (Esser et al., 2024) enhances the consistency of text rendering in generated images. As shown in Figure 10, APG produces more accurate spelling in the generated images compared to standard CFG. More visual results given in Figure 20 (appendix).
Text spelling with Stable Diffusion 3
8

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
a ccow is sinnging
Cow singing
LL WAY
STOP
ALL WAY
A sign that says "A cow is singing"
A stop sign with "ALL WAY" written below it
CFG
APG (Ours)
CFG
APG (Ours)
KEEP OFF THE GRASS
KECP OFASS
Hello worlld
HELLO WORLD
The words "KEEP OFF THE GRASS" written brick
A holding sign that says "hello world"
Figure 10: Comparison of CFG and APG for text quality in generated images using Stable Diffusion 3 (Esser et al., 2024). In contrast to CFG, APG consistently produces correct spellings.
CFG Rescale
APG (Ours)
CFG Rescale
APG (Ours)
CFG Rescale
APG (Ours)
CFG Rescale
APG (Ours)
Figure 11: Comparison between APG and CFG Rescale using Stable Diffusion XL. CFG Rescale is unable to solve the saturation issue high guidance scales compared with APG.
Comparison with CFG Rescale CFG Rescale was introduced (Lin et al., 2024a) as a method to reduce saturation at high guidance scales. In this section, demonstrate that APG is more effective than CFG Rescale. The comparison in Figure shows that APG outputs have significantly less saturation and realistic than those produced with CFG Rescale.

---

Published as a conference paper at ICLR 2025
(a) CFG
(b) +Projection (eg f, y))
(c) +Projection (Do(#1, t, v))
Figure 12: The importance of projecting onto the denoised samples. When performing projection w.r.t. the predicted noise (b), the outputs are barely different than standard CFG (a). However, projecting onto denoised samples (c) more effectively reduces saturation.
Computational cost of APG The computational cost of APG practically identical to that of CFG, the rescaling and projection steps incur negligible overhead compared to querying the denoiser. Specifically, in the case of Stable Diffusion XL, the forward pass through the diffusion network takes approximately 130 milliseconds requires only about 0.45 milliseconds.
RTX 3090 GPU for single image, while the guidance step
5.2 ABLATION STUDIES
We now present our ablation studies in this section. The ex- periments are based on class-conditions generation using the EDM2 model (Karras al., 2023), with FID as the primary metric justify our design choices. First, Table 2 highlights the importance of each component APG. We observe that removing projection, rescaling, reverse momentum results in higher FID scores. Additionally, note that the projection component is primarily responsible for reducing saturation while rescaling and reverse momentum mainly improve FID and recall. Appendix C.8 gives extended ablation results on the effect of each component in APG.
Table Importance different com- ponents in APG.
Config
Recall
Saturation |
APG (w 4) 6.49
0.62
0.33
wro projection
6.63
0.60
0.37
rescaling
7.93
0.56
0.34
wro momentum
6.85
0.61
0.33
Importance of the model prediction type While CFG works the same across all model predic- tion types, we observed that our method performs best when applied to the denoised predictions Dø Et, f, y), rather than, for example, the noise prediction e0(z,, t,,t,3 y). This is illustrated in Fig- ure 12, where projecting onto 60(2,, t, y) produces results nearly identical to CFG, while projecting onto (2 t, y) significantly reduces saturation. Note that discussed in Appendix B, this is not bottleneck for APG as various prediction types can be readily converted to Do(z,, t, y) at each step.
CONCLUSION AND DISCUSSION
In this work, we investigated the oversaturation effect of high CFG scales and introduced new method, adaptive projected guidance (APG), that achieves the same quality-boostinn benefits as CFG without causing oversaturation. The key idea behind APG is to project the CFG update onto denoised prediction of the diffusion model Do(z,, l. y) and remove or down-weight the component parallel to that prediction. Additionally, by linking CFG to gradient ascent, we demonstrated that its performance can be further enhanced by incorporating rescaling and reverse momentum. Through extensive experiments, we showed that APG improves FID, recall, and saturation metrics compared to CFG, while maintaining similar better precision. Thus, APG offers plug-and-plas alternative to standard CFG capable of delivering superior results with practically no additional computational overhead. Like CFG, challenges remain accelerating APG so that the sampling cost approaches that of the unguided sampling (i.e., removing the need to query the diffusion network twice at each sampling step). We consider this a promising direction for future research.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
As generative modeling continues evolve, the potential create fake erroneous data increases. While advancements in Al-generated content can enhance efficiency and foster creativity, it is crucial address the associated ethical concerns. For more detailed discussion on ethics and creativity in computer vision, we recommend Rostamzadeh et al. (2021).
REPRODUCIBILITY STATEMENT
This work builds on the official implementations of the pretrained models referenced in the main text. The source code for implementing APG is provided in Algorithm 1, and Appendix D outlines additional implementation details, including the hyperparameters used in the main experiments.
REFERENCES
Mohammadreza Armandpour, Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. CORR, abs/2304. 004968, 2023. doi: 10.48550/ARXIV.2304.00 04966. URL
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text-to- image diffusion models with an ensemble of expert denoisers. CoRR, abs/2211.01324, 2022. doi:
10.48550/arXiv.2211 301324. URL http:://c...rg/0044555a/xkx.1.1
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, abs/2311.15127, 2023a. doi: 10.4855/ARXIV.231..51111 URL https://doi.org/ http://dooi.r//1O... 1o5.dd/...rr/1144455/i//iiii 2311.15127.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 22563-22575, 2023b.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreeiew.eeel forum?id-B 1xsp//d9 xsqj09Fm.
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-( Fixart-(\delta): Fast and controllable image generation with latent consistency models. arXiv preprint ar arXiv:2401.055 1...020..522,... 2024.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthe- sis. Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- mual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 1 afstsaffasstrt..hhatt 1
8794, 2021. URL https:/proeeddnss.nerri..cccc
1/has/
Patrick Esser. Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403. 03206, 2024.
11

---

Published as a conference paper at ICLR 2025
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626-6637, 2017. URL http://pwweeediiggs.uttiiccc/cpp 0ul694407cc00ffff687133600000000
Abstracctttttttt
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abs/2207.12598. 2022.
doi: 10. 00.48550/rxxvv.227.. 125988 URL. http::/doii.org: wwtp:////g.....55/aaa.i...... 12598.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL hitps:
100 10179cc4b-Abstracc
Emiel Hoogeboom, Jonathan Jonathann Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. CoRR, abs/2301. 100//330..119,,,,, 1093, 2023. doi: 10.485550a2x1vv3300.1000 URL
https://doi.org/10.. http://dbooor//100055555vxxx...m 48550//aXiv.2301......
Qingqing Huang, Daniel Park, Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Havnø Frank, Jesse H. Engel, Quoe V. Le. William Chan, and Wei Han. Noise2music: Text-conditioned music generation with diffusion models. CoRR, abs/2302.03917, 2023. doi: 10448500arrXi.22022.3391 URL https://doi.orr//10485550 arXiv.2302.03917.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. 2022. URL http:///peeeeveeeeeeee/mm////tot
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models, 2023.
Diederik P. Kingma and Ruiqi Gao. Vdm++ Variational diffusion models for high-quality synthesis, 2023.
Zhifeng Kong. Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 1021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview. hel/forum?id-a-xtte
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Im- proved precision and recall metric for assessing generative models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 3929-3938, 2019. URL http://ppooceddinnnneiripppppp Abstratt.htttt.h
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. CoRR, 4bs/24444.0724,, 2024. doi: 1044850/ARXXV.24400000
URL https://doi.or//
(0-488550axXX122444007772
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps flawed. In IEEE/CVF Winter Conference Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA, Jamuary 3-8, 2024, pp. 2392-5399, IEEE, 2024a. doi: 10.1 2.1100/WACV57000 S00.ACCVV77000.22200333333 2022.00532. URL https://doi.or//////i http://..11///////00000 071.577771..2200333333
12

---

Published as a conference paper at ICLR 2025
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXi::2402.13929 2024b.
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion
distillation. CoRR, abs/2402.13929, 2024c. //doi.org//0..44... 000.08850000/1112222..22202
doi: 1044550/ARXXV.440..5922 URL https:
Tsung-YV Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014. doi: 10.1007/ 3-319-10602- 48.
978-3-319- 10602- _\_48. URL
Guan-Horn Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. 12s: Image-to-imaae schrödinger bridge. CoRR, abs/2302.05872, 2023a. doi: 10.48550/u2Xi.2222..5555 URL https:///oii. wttp://i...rrr1....555..050 http://00055550001/2222..05..000
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. 2022a. URL https///peenteiiee..e//onnnmnnn
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL https://openevvieewee///oooo ?id= PIKWVd2yBkY.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. 2023b. URL
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022a. URL http://papers.nnss.co/oo httppaper_filesppper////2/ 260a 1000144ccc22@@aaBBaaeeeeeeeeees Abstract-Conffrenncc
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. CoRR, 0bs/2211.01105, 2022b. doi: (0048550/a2Xi..2211.0000
URL https://doi.org/ 100.0.../55////...222111 1,000000 1,01095.
Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8 8162-8111 8171, PMLR, 2021. URL http://proceedings.mii mlr. mlr.press/ v139/nccho22 ha.mll.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 Proceedings of Machine Learning Research, pp. 16784-16804, PMLR, 2022. URL httpss//prooeeeingg..iii plr.prssss v162/michol22a.htm
William Peebles and Saining Xie. Scalable diffusion models with transformers. CoRR, abs/2212.097488, 2022. doi: 10.4855Muxii.2212..0099 URL https///ti.org/11485555axx 2212.09748.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952. 2023. doi: 10.48550/ARX1.22000011959 URL https:
13

---

Published as a conference paper at ICLR 2025
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.00112) 2022. doi: 10.48550/ arXiv.2204.06122. URL http://d....rg0000444055050..220
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. doi: 10. (0.1199/CVPP52228222220000 URL http::://oi.rg//////// CVPR552688.0022.00044
Negar Rostamzadeh, Emily Denton, and Linda Petrini. Ethics and creativity in computer vision. CoRR, abs/2112.03111, 2021. URL http:://rxiv.oor//bb/1111 1 12.03111.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211-25,, 2015. doi:
10.1007//1123--05--08
URL
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through condition-annealee sampling. The Twelfth International Conference on Learning Representations, 2024a. URL https://peenevvi..eeeffnnn id=zMoNrajk2X.
Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, and Romann M Weber. No training, no problem: Rethinking classifier-free free guidance for diffusion models. arXiv preprint arXiv:2407.026877 2024b.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In Munkhtsetseg Nandigjav, Niloy J. Mitra, and Aaron Hertzmann (eds.), SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 11, 2022, pp. 15:1-15:14 ACM, 2022a. doi: 00.145/35282333553755 URL https://doi.org// 1144/3528833.3530777
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022b. URL http://papers.nng hatpssccppaprffll//sspeee www.traat-Cooeeeeneeeee 7755aaadaachh2220000555hhh000a
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL http:///oooaneeeewweeerfmm///llp
sd community. Sdx1 flash in collaboration with project fluently. https///huggingfecccccooooommmmn sdxl-flash. Accessed: 2024-09-08.
Jascha
Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. 37:2256-2265. 2015. URL http://proceediing. mttpppess//37/soolldiiotttttnt 1...ff html.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL http:://openeeiiew.eet/ooommmnno hiaes//owneeeoommommmsssgiiiaatt giarCHLP.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019. December 8-14 2019, Vancouver, BC, Canada, pp. 11895-11907, 2019. URL haap//00le222744075557111c1c1212
14

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL http:///peneeevevneeffooonmmmnio PAC
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Trans. Mach. Learn. Res., 2022, 2022. URL http://openneview.nee Yonunkid-AFDDYYKKNND.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. CoRR, abs/2302.04867, 2023. doi: 10.445500arXx..2322..444
URL http://dw...rr11000.550550//.aaa
Candi Zheng and Yuan Lan. Characteristic guidance: Non-linear correction for diffusion model large guidance scale. 2024. URL https//pppoaavvooweeeeeoouuttttt
15

---

Published as a conference paper at ICLR 2025
A DETAILS ON CFG AS GRADIENT ASCENT
In this section, we discuss how CFG can be interpreted as one step of gradient ascent. To begin, note that the CFG update rule can be expressed as:
Dera(ze, (z, t, y) = Do(z,, t, Ymull) + w Do(zz,
t, (,
- Do(zi, Mmmll))
(7)
w:Do(zt; 4,)) + (1 - w) Do Zt. L, ymull)
(8)
-D(z,,,, + (w (z t, (,u) + (1 w) Do t, Umoll)
(9)
Do(ai,t,y) + (ou-1)(Do(zz)
Do(ze, t, Ymall))
(10)
Do(2,, AAD:
(11)
where y = w - 10 and AD, = Do(z,, t,y) - Do(z,,t, t, Ymal). Next, observe that we can write:
Do Dø(zi, Yrull) V (o(2
|Do | Do(zs, t. youn)ll
(12)
Thus, if we define the CFG objective function
fCFG (Do (.), Pz(z 1, Mmull) = Do (,((z,t,)) - Do(zi, L, fman)))
(13)
the CFG update rule becomes equivalent to:
DCFG(Ee,te t, y) (z, y, + V (D( fCFG (DFG(DD(#)) (De , t, B), Do Dz(z,, t, youu)).
(14)
Hence, we have shown that the CFG update rule corresponds to a single step of gradient ascent with respect to the objective function fCFG (Do(z,, y), Do (zi, t, yaull)).
This interpretation motivated us to incorporate rescaling into standard CFG. Since the objective function fCrG (Do(z,, y), Do(2,, Ymun)) does not have maximum, the CFG update step may result in arbitrary drift from (z(z,, t, y) By applying rescaling, we constrain the CFG update to remain within ball of limited radius around Dz(2,, t, y). The reverse momentum method is similarly inspired by this interpretation, where each update is pushed away from previous predictions.
B DENOISED PREDICTION FOR DIFFERENT DIFFUSION MODELS
We next briefly outline the process of computing the denoised prediction Dø(zt, f, y) for various diffusion models. For further details, refer readers to Kingma & Gao (2023). In the following, let æ represent the clean data, y a condition or class, and ~ NO I) the noise. Given noisy sample 2, time step I, the objective is to recover the clean data æ that produced z,. The denoised version of 2t, which approximates æ, is estimated by neural network, denoted as De I, y). Before applying APG, we always convert all model predictions to Do( t, y). This conversion is compatible with most samplers based on the denoising framework, such as EDM (Karras et al., 2022) and DPM++ (Lu al., 20226). The conversions for various models are derived below, and summary is provided in Table 3.
DDPM For models using the DDPM framework (Ho et al., 2020), the forward diffusion process is defined z, a,æ + o,C, where of + of = 1. These models typically predict the total added noise e via a neural network 60 62 L, y). Given the prediction of the model, the denoised prediction be estimated via
Do 1, Z, (2,,
(15)
Ot
the model predicts the velocity QgE T2C, we have
21
04
az af
0,xx
04
a2
(16)
at
This leads the following formulation for the denoised prediction:
D0 (,
(17)
16

---

Published as a conference paper at ICLR 2025
Table Summary of calculating denoised predictions Da(z y) for different diffusion models.
Config
Forward process
Model prediciton
Denoised prediction Do(z,,4,3)
DDPM
0,4 dye
c0(2,,
(z; f.
DDPM
0, + OGE
a 02,
EDM
(t)e
Fo (o()in(t) Cuoisa
Catip (t)zz Cout (f) Fo (t) Cuoiso (t),
Rectified flow
(1 - f)x tc
00(2,, t,
2 100 (z+, (,
EDM framework For the EDM framework (Karras al., 2022), the forward process is described by & x - ( and the denoised prediction Dø(z,, i, y) is formulated via
Do(2,, 1, = Cakip (t) + Cout! (t) Fo (f) Cuoise (f), y),
(18)
where (e (t) Cuoise (t), 3) is the output of the neural network. The EDM framework uses (t) x t; thus, a and & can be used interchangeably in this framework.
Rectified flow models For rectified flow models (Liu al., 2023b), such as Stable Diffusion (Esser et al., 2024), the forward process is given by 2t = (1 t)a: + te. The model predicts the velocity field given by v = x. Accordingly, we have
x 2t (1 1)a x.
Zt
1)a la
21 x
(19)
t
t
t
Thus, the denoised prediction can be determined by:
Do(z),t,))
(2,(,e,,,)) 2t 0,v0(ze,t,,))
(20)
where we define oe t
This section demonstrates the effect of applying APG on toy example to illustrate the differences between APG and CFG. We use mixture of two nigh-dimensional Gaussians as the data distribution, which allows us to analytically compute the score functions during the diffusion process, eliminating potential errors introduced by denoiser network. Specifically, the data distribution Pdata is defined as:
1 N(1 Pdata (x) 2
( (T))()))))))))))))))())) 2 (12,000)
(21)
where f41 = -2, 2), n1 2., for each component. Accordingly. the conditional distributions are equal to
., 2). and = 0.25. We use dimensionality of 500
P (æly y = 1) = of) (x)
Pdata
and Pdata y = 2) = (N (x)
(22)
The sampling results are shown in Figure 13 (visualizing the first two dimensions of each Gaussian). When CFG is applied with high guidance scale, results drift toward regions less likely according to the data distribution. In contrast, applying APG corrects this drift and improves mode coverage. While this simplified example, we argue that a similar phenomenon occurs when applying CFG to images, leading to artifacts and oversaturation in the final outputs.
C ADDITIONAL EXPERIMENTS
Additional experiments and ablation studies are included in this section. Unless stated otherwise, the experiments are conducted using class-conditional ImageNet (Russakovsky al., 2015) generation.
C.I COMPATIBILITY WITH CADS AND IG
We first demonstrate that APG compatible with CADS (Sadat et al., 2024a) and interval guidance (IG) (Kynkäänniemi al., 2024), both of which designed enhance the diversity of generations at high guidance scales. The results, shown in Table 4, indicate that replacing CFG with APG leads improved FID, recall, and saturation scores for both methods.
17

---

Published as a conference paper at ICLR 2025
(a) Real data
(b) CFG
(c) APG (Ours)
Figure 13: Visualizing the effect of APG on the sampling process using toy problem. The real samples from the data distribution are shown in (a). When sampling with high guidance, CFG leads drift away from the mean of the data distribution and results in reduced mode coverage in the generated samples (b). In contrast, sampling with APG eliminates the drift and increases the coverage of the distribution (c). We used the EDM sampler (Karras al., 2022) for this experiment.
Table 4: Compatibility of APG with CADS (Sadat al., 2024a) and IG (Kynkäänniemí et al., 2024). Combining APG with other methods that improve diversity results in better FID than each method in isolation.
(a) CADS
(b) Interval guidance (IG)
Precision
Recall
Saturation
Contrast
Guidance
Precision
Recall
Saturation
Contrast
CFG
10.42
0.85
0.48
0.46
0.27
CPG
10.42
0.85
0.48
0.46
0.27
*CADS
8.65
0.85
0.56
0.43
0,26
+G
7.49
0.84
0.60
0.39
0:25
+both
3.56
0.84
0.64
0.32
0.21
+both
5.29
0.84
0.65
0.33
0.22
FID
Precision
Recall
Saturation
13
#8 Steps
Steps
Steps
# Steps
Figure 14: Comparison of CFG and APG across different numbers of sampling steps. APG consis- tently achieves better FID and recall while maintaining comparable superior precision CFG.
C.2 APG VS NUMBER OF SAMPLING STEPS
We present the performance comparison between APG and CFG across different numbers of sampling steps using the EDM2 model. Figure 14 indicates that APG consistently provides better FID, recall, and saturation while maintaining the same level of precision.
C.3 USING APG WITH DISTILLED MODELS
In this section, show the compatibility of APG with distilled models using PIXART-Ō (Chen et al., 2024), SDXL-Lightning (Lin et al., 2024c), and SDXL-Flash (sd community). Consistent with the main text, Figure 15 demonstrates that replacing CFG with APG significantly improves generation quality and saturation level across all models. This also consistent with Figure 14, where APG outperforms CFG at fewer sampling steps (e.g., 8-16).
18

---

Published as a conference paper at ICLR 2025
PIXART-O
CFG
APG (Ours)
CFG
APG (Ours)
A blue bird
basket macarons
SDXL-Lightning
CFG
APG (Ours)
CFG
APG (Ours)
pink dog
plate of fries
SDXL-Flash
CFG
APG (Ours)
CFG
APG (Ours)
bouquet flowers
red bear
Extended results showcasing the compatibility of APG with distilled diffusion models. Compared to CFG, using APG does not lead to degradation in the output quality.
Figure 15:
C.4 COMPATIBILITY WITH DIFFERENT SAMPLERS
While the main experiment in Table used the default sampler of each model, we next separately show that APG is compatible with different sampling algorithms widely used with diffusion models. As shown in Table 5, using APG with different samplers results in improved FID, recall, and saturation scores, consistent with the main findings in Table 1. We used class-conditional generation using DIT-XL/2 for this experiment.
C.5 COMPATIBILITY WITH ICG
Independent condition guidance (ICG) (Sadat et al., 2024b) is a method to apply CFG without the need to query unconditional model. In Table 6, show that APG is compatible with ICG, and
19

---

Published as a conference paper at ICLR 2025
Table Impact of using APG with popular diffusion samplers using the class-conditional ImageNet model (DiT-XL/2), Compared to CFG, APG showes improved metrics across all samplers.
APG (Ours)
CFG
Sampler
FID
Recall +
Saturation
FID 4
Recall 4
Saturation 4
DDIM (Song al., 1021a)
6.69
0.62
0.30
17.45
0.38
DPM+ (Lu et al., 2022b)
6.87
0.62
0.32
17.65
0.38
0.43
SDE-DPM++ (Lu et al., 2022b)
8.53
0.57
0.32
19.01
0.36
0.43
PNDM (Liu al., 2022a)
5.37
0.68
0.32
16.50
0.40
0.43
UniPC (Zhao ct al., 2023)
6.91
0.62
0.32
17.65
0.38
0.43
Table 6: Compatibility of APG and ICG. Combining APG with ICG significantly improves FID,
recall, and saturation scores while maintaining similar precision.
Guidance
FID
Precision
Recall f
Saturation 4
Contrast
ICG
17.63
0.32
0.49
0.28
+APG (Ours)
5.73
0.85
0.63
0.33
0.22
Table 7: Compatibility APG and TSG with. Combining APG with TSG improves FID, recall, and saturation scores while maintaining similar precision.
Guidance
FID
Precision f
Recall f
Saturation +
Contrast
TSG
14.00
0.81
0.52
0.37
0.28
+APG (Ours)
5.84
0.81
0.66
0.30
0.20
similar to CFG, using APG with ICG results in improved FID, recall, and saturation scores while maintaining similar precision. We use class-conditional ImageNet generation with EDM2-S for this experiment.
C.6 COMPATIBILITY WITH TSG
Time-step guidance (TSG) (Sadat et al., 2024b) is an extension of CFG that leverages the time- step information learned by the diffusion model to enhance the quality generations. We next demonstrate that applying the update rule in APG further improves the performance of TSG. Table 7 shows that APG improves FID, recall, and saturation metrics, while maintaining similar precision to TSG. This experiment is based class-conditional ImageNet generation using DiT-XI//2.
C.7 ALIGNMENT WITH THE CONDITION
We next demonstrate that replacing CFG with APG does not compromise the alignment between the input condition and the output, To validate this, measure the classification ac- curacy of the generated results for the ImageNet task and the CLIP score for Stable Diffusion. The results in Table 8 show that both CFG and APG achieve comparable alignment met- rics. Thus, APG reduces saturation and improves FID without compromising condition alignment.
Table 8: Condition alignment com- parison between CFG and APG.
Alignment metric
CFG
APG
Class Accuracy 7
0.97
0.96
C.8 EXTENDED ABLATION STUDIES
Effect of the parallel component We next demonstrate the effect of r on the generated images in Table 9a. As hypothesized in Section 4, increasing the strength of the parallel component leads to higher saturation levels and increased FID. We recommend setting r = 0 by default and only increasing if more saturation is desired in the generated images.
20

---

Published as a conference paper at ICLR 2025
Table Ablation study examining various design elements in APG.
(a) Influence of
(b) Impact of rescaling
(c) Effect of momentum 8
FID
Saturation 1
FID 1
Rocall 7
FID 1
Rocall
0.0
6.49
0.33
0.25
7.45
0.72
-1.5
13.38
0.73
0.25
6.49
0.34
2.5
6.49
0.62
-0.75
6.49
0.62
0.5
6.49
0.36
7.97
0.57
0.0
6.84
0.60
0.37
7,93
Table 10: Hyperparameters used in the main experiment (Table 1).
Model
7
B
EDM2-S
4
0
2.5
-0.75
EDM2-XI.
2
0
2.5
-0.75
DiT-XL/2
4
0
-0.50
Stable Diffusion 2.1
10
0
-0.75
Stable Diffusion XL
15
0
15
-0.50
Effect of the rescaling threshold The effect of the rescaling radius the generated images is shown in Table 9b. Excessive rescaling degrades image quality, while high values of r result in no noticeable change, the rescaling function approaches the identity function. Therefore, midrange values for r yield better FID scores. We suggest observing the norm of A D, during the inference process and choosing in a way that comparable (on average) the norm of AD;
Effect of the momentum strength Table 9c shows the effect of momentum strength 3 on generation quality. Note Negative values for 3 result in better FID compared to positive momentum, and excessive momentum degrades image quality. This aligns with our hypothesis that moving away from the previous directions helps limit the drift that can occur during sampling with higher guidance scales. Empirically, found that 3 E [-0.75, -0.25] works well in most setups.
D IMPLEMENTATION DETAILS
We provide the code for APG in Algorithm 1. Compared to CFG, APG only includes few additional lines of code without noticeable computational overhead. As discussed in Section 4, we always convert the predictions of the diffusion model Do I, y). compute the guided prediction, and convert back to the initial output type at each sampling step.
We mainly the ADM evaluation suite (Dhariwal & Nichol, 2021) for computing FID, precision, and recall. The FID is computed using 10,000 generated images and the whole training set for class- conditional ImageNet models. For text-to-image models, the FID is evaluated using the evaluation subset of MS COCO 2017 (Lin et al., 2014). The hyperparameters used for the main experiment are given in Table 10.
E MORE VISUAL RESULTS
This section presents extended visual comparisons between APG and CFG. Additional results using EDM2 are provided in Figure 16, with example of how APG enhances diversity shown in Figure 17. Further images for Stable Diffusion 2.1 and Stable Diffusion XI. are included in Figures 18 and 19. Moreover, Figure 20 illustrates how APG improves text spelling in Stable Diffusion 3. Finally, more examples of APG applied to distilled models are shown in Figures 21 to 23.
21

---

Published as a conference paper at ICLR 2025
Algorithm PyTorch implementation of APG.
import torch
class MomentumBuffer:
def init _nitt_seet (self, momentum: :loat):
melf.momentum momentum
self. seli.running_avee average
def update update_value: corch.Tenson new_average self molmmmmme * self. running_average
self. relf.running update_value new_average
def project(
v0: forch.Tenson [B, C, H, W]
vi: torch.Tensor, [B, C, H, W]
dtype v0. dtype
v0, vl v0. v0.double(), vl.double()
vi torch.nn. wtw.:.aw....tiioo.....ooo nowemiiiee (v1, dim=1-1, -2, -31)
vi_parallel = (vC v1) sum (din=[-1, -2, -3], keepdin-True) vl
v0_orthogonal vo vo_parallel
return v0 parallel. (dtype), v0 v0_orthogonal.tee to (dtype)
def senetiiv__rojeeted_uiiieee
pred_cond: torch. Tensor,
[B, C,
pred pred_uncond: torch. Tensor, # [B, C, E, W] guidance_scale: float,
nomentum_buffer: MomentumHuffer None, eta: float 1.0, thrm_hhreshold: float 0.0,
diff pree__cond pred_uncont if momentun_buffer is not None: momentum buffer. update (difE) diff conettum_bbufer...ummin.....eee
if norm_threshold 0:
dif:_norm likes_likeele iike (diff)
ones torch.ones__1 oetttoons_____
(p=2, dim=1-1, -2, -31, keepdim=True)
scale_factor torch.minimus (ones, norm_threshold diff dirf_norm)
diff diff scale_factor
diff_paralled, diff_orthogonal = project (diff, pred_cond)
normalized_update diff_orthogonal + eta diff siff_parallel
pred guided = pred_comt + qqudance__scale 1) * limmaliizd_updtttt
return pred_guided
22

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
CFG
APG (Ours)
CFG
APG (Ours)
Figure 16: More visual results comparing APG and CFG using EDM2.
23

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
Figure 17: Showcasing the diversity of generations after using APG. APG removes oversaturation issues while improving diversity w.r.t. the overall image composition.
CFG
APG (Ours)
CFG
APG (Ours)
Astronaut on the Moon
A hurger
CFG
APG (Ours)
CFG
APG (Ours)
A tiger
A lion
Figure 18: More visual examples comparing CFG and APG using Stable Diffusion 2.1.
24

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
4k dslr photo raccoon wearing an astronaut A highly detailed paper origami Dachshund on a table next to porcelain teapot, 4k dslr.
helmet, photorealistic.
CFG
APG (Ours)
CFG
APG (Ours)
A fox
A golden retriever
Figure 19: More visual examples comparing CFG and APG using Stable Diffusion XL.
CFG
APG (Ours)
CFG
APG (Ours)
DIFFUSION
Difuusion
A t-shirt with "Archacology Rocks!" written it
storefront with "Diffusion" written it
CFG
APG (Ours)
CFG
APG (Ours)
BUY MILLK
BUY MILK
VERY DEEP LLEARNING
Very Deep Learning
A green sign that says "Very Deep Learning"
A yellow sticky note with "BUY MILK" written on
Figure 20: Additional visual examples the quality of rendering text using Stable Diffusion 3. Compared to CFG, APG results show more consistent spellings.
25

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
Image of corgi
Astronaut Mars during sunset
CFG
APG (Ours)
CFG
APG (Ours)
SHA
lion playing guitar
beautiful outdoor scene
Figure 21: More visual examples comparing CFG and APG using PIXART-6.
CFG
APG (Ours)
CFG
APG (Ours)
Astronaut riding horse
An elephant under the
CFG
APG (Ours)
CFG
APG (Ours)
A statue a greek man
A plate steak
Figure 22: More visual examples comparing CFG and APG using SDXL-Flash.
26

---

Published as a conference paper at ICLR 2025
CFG
APG (Ours)
CFG
APG (Ours)
A cave under the ocean
A glass of red wine
CFG
APG (Ours)
CFG
APG (Ours)
basket red fruits
yellow panda
Figure 23: More visual examples comparing CFG and APG using SDXL-Lightning
27


Reviews:
[{"review_id": "2hIXVN0Boa", "reviewer": "Reviewer_xLYc", "summary": "This work aims to reduce the oversaturation in the current CFG approach. The authors decompose the CFG update term into two components—parallel and orthogonal—and both theoretically and empirically demonstrate the role of each component. Based on these observations, they redesign the weighting of each component and additionally propose an adaptive rescaling strategy.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The intuition and theory are well-balanced.\n- The results are quite promising. Their guidance does not drastically alter the image content, which aligns well with their claims.\n- They demonstrate robustness across various types of diffusion models.", "weaknesses": "- I do not view this as a straightforward rescaling of CFG. However, it might be open to interpretation, and I would be interested to hear the authors’ perspective on this.\n\n- Which model was used for Figure 8? \n- Further analysis on the rescale weight would be helpful—does it show any trends with respect to t? \n\n- *(As a suggestion)* Including illustrations of the geometry, along with the related intuitions, would further enhance the clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEUCGu1Ql5", "reviewer": "Reviewer_mQei", "summary": "The paper introduces a novel method called Adaptive Projected Guidance (APG) to address the issue of oversaturation and unrealistic artifacts that occur when using high guidance scales in diffusion models. Specifically, the authors propose a modification to the classifier-free guidance (CFG) update rule by decomposing the update term into parallel and orthogonal components relative to the conditional model prediction. The authors conduct extensive experiments to support their claims.", "questions": "Please refer to the Weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a new method, APG, that addresses a significant problem in diffusion models—oversaturation and artifacts associated with high guidance scales. The approach is innovative and directly tackles a well-known issue in the field.\n2. The authors provide extensive experimental results to validate their claims. They demonstrate the effectiveness of APG across various models and samplers, showing improvements in FID, recall, and saturation scores.\n3. The paper provides a clear comparison between APG and CFG, both qualitatively and quantitatively. The visual results are particularly compelling in showing the reduction in oversaturation and artifacts.", "weaknesses": "1. The final example generated by APG in Figure 10 has a logical error, where the cat has only one paw.\n2. For Table 1, the authors should consider adding HPSv2 and ImageReward metrics to evaluate the generation performance.\n3. For Table 1, why is the guidance scale of Stable Diffusion XL so large (w=15)? When reducing the guidance scale, are the performance improvements limited? The authors should provide more quantitative results.\n4. The authors claim that the orthogonal component is chiefly responsible for improvements in image quality, while the parallel component increases saturation in the generations. How many examples do the authors observe? Are there any theories to support this observation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tgHPhKcZfM", "reviewer": "Reviewer_LhNz", "summary": "This paper presents a method called Adaptive Projected Guidance (APG) to address the issues of oversaturation and artifacts that occur when using high guidance scales in Classifier-Free Guidance (CFG) for diffusion models. The authors decompose the CFG update rule into two components: a parallel component responsible for oversaturation and an orthogonal component that enhances image quality. They propose down-weighting the parallel component to reduce oversaturation. Additionally, they introduce rescaling and reverse momentum to improve sampling. Extensive experiments demonstrate that APG can improve FID, recall, and saturation scores while maintaining precision, making it a superior alternative to standard CFG.", "questions": "See the questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. APG is presented as a plug-and-play method with virtually no additional computational overhead, making it attractive for practical use in various diffusion models.\n2. The method of down-weighting the parallel component in CFG is novel and interesting, effectively reducing oversaturation and preserving image quality at high guidance scales.", "weaknesses": "1. In Figure 8, could the authors explain why the proposed APG shows lower precision and higher FID compared to CFG when the guidance scale  w  is below certain values (e.g.,  w = 2.5 , where APG precision is lower than CFG)?\n2. While reverse momentum is introduced, the intuition behind why it improves the image generation quality could be expanded. More detailed theoretical or experimental exploration of this component would clarify its importance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)