Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

Authors: Seyedmorteza Sadat, Otmar Hilliges, Romann M. Weber
Primary area: generative models
Keywords: classifier-free guidance, diffusion models

Abstract:
Classifier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models. While a high guidance scale is generally required to enhance these aspects, it also causes oversaturation and unrealistic artifacts. In this paper, we revisit the CFG update rule and introduce modifications to address this issue. We first decompose the update term in CFG into parallel and orthogonal components with respect to the conditional model prediction and observe that the parallel component primarily causes oversaturation, while the orthogonal component enhances image quality. Accordingly, we propose down-weighting the parallel component to achieve high-quality generations without oversaturation. Additionally, we draw a connection between CFG and gradient ascent and introduce a new rescaling and momentum method for the CFG update rule based on this insight. Our approach, termed adaptive projected guidance (APG), retains the quality-boosting advantages of CFG while enabling the use of higher guidance scales without oversaturation. APG is easy to implement and introduces practically no additional computational overhead to the sampling process. Through extensive experiments, we demonstrate that APG is compatible with various conditional diffusion models and samplers, leading to improved FID, recall, and saturation scores while maintaining precision comparable to CFG, making our method a superior plug-and-play alternative to standard classifier-free guidance.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729986024943

thread_type:
official_review

<thread_object>
creator: reviewer_lhnz

<content>
summary:
This paper presents a method called Adaptive Projected Guidance (APG) to address the issues of oversaturation and artifacts that occur when using high guidance scales in Classifier-Free Guidance (CFG) for diffusion models. The authors decompose the CFG update rule into two components: a parallel component responsible for oversaturation and an orthogonal component that enhances image quality. They propose down-weighting the parallel component to reduce oversaturation. Additionally, they introduce rescaling and reverse momentum to improve sampling. Extensive experiments demonstrate that APG can improve FID, recall, and saturation scores while maintaining precision, making it a superior alternative to standard CFG.

strengths:
1. APG is presented as a plug-and-play method with virtually no additional computational overhead, making it attractive for practical use in various diffusion models.
2. The method of down-weighting the parallel component in CFG is novel and interesting, effectively reducing oversaturation and preserving image quality at high guidance scales.

weaknesses:
1. In Figure 8, could the authors explain why the proposed APG shows lower precision and higher FID compared to CFG when the guidance scale  w  is below certain values (e.g.,  w = 2.5 , where APG precision is lower than CFG)?
2. While reverse momentum is introduced, the intuition behind why it improves the image generation quality could be expanded. More detailed theoretical or experimental exploration of this component would clarify its importance.

questions:
See the questions in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730028379234

thread_type:
official_review

<thread_object>
creator: reviewer_mqei

<content>
summary:
The paper introduces a novel method called Adaptive Projected Guidance (APG) to address the issue of oversaturation and unrealistic artifacts that occur when using high guidance scales in diffusion models. Specifically, the authors propose a modification to the classifier-free guidance (CFG) update rule by decomposing the update term into parallel and orthogonal components relative to the conditional model prediction. The authors conduct extensive experiments to support their claims.

strengths:
1. The paper introduces a new method, APG, that addresses a significant problem in diffusion models—oversaturation and artifacts associated with high guidance scales. The approach is innovative and directly tackles a well-known issue in the field.
2. The authors provide extensive experimental results to validate their claims. They demonstrate the effectiveness of APG across various models and samplers, showing improvements in FID, recall, and saturation scores.
3. The paper provides a clear comparison between APG and CFG, both qualitatively and quantitatively. The visual results are particularly compelling in showing the reduction in oversaturation and artifacts.

weaknesses:
1. The final example generated by APG in Figure 10 has a logical error, where the cat has only one paw.
2. For Table 1, the authors should consider adding HPSv2 and ImageReward metrics to evaluate the generation performance.
3. For Table 1, why is the guidance scale of Stable Diffusion XL so large (w=15)? When reducing the guidance scale, are the performance improvements limited? The authors should provide more quantitative results.
4. The authors claim that the orthogonal component is chiefly responsible for improvements in image quality, while the parallel component increases saturation in the generations. How many examples do the authors observe? Are there any theories to support this observation?

questions:
Please refer to the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730290517679

thread_type:
official_review

<thread_object>
creator: reviewer_xlyc

<content>
summary:
This work aims to reduce the oversaturation in the current CFG approach. The authors decompose the CFG update term into two components—parallel and orthogonal—and both theoretically and empirically demonstrate the role of each component. Based on these observations, they redesign the weighting of each component and additionally propose an adaptive rescaling strategy.

strengths:
- The intuition and theory are well-balanced.
- The results are quite promising. Their guidance does not drastically alter the image content, which aligns well with their claims.
- They demonstrate robustness across various types of diffusion models.

weaknesses:
- I do not view this as a straightforward rescaling of CFG. However, it might be open to interpretation, and I would be interested to hear the authors’ perspective on this.

- Which model was used for Figure 8? 
- Further analysis on the rescale weight would be helpful—does it show any trends with respect to t? 

- *(As a suggestion)* Including illustrations of the geometry, along with the related intuitions, would further enhance the clarity.

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732445182923

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message to All Reviewers

comment:
We thank the reviewers once again for their positive assessment of our paper’s contribution and presentation and for their helpful comments. We hope that our replies have satisfactorily addressed the reviewers’ questions and further strengthened their assessment of our work. We remain open to discussion should any issues remain.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents a novel method called Adaptive Projected Guidance (APG) to address the issue of oversaturation and unrealistic artifacts that occur when using high guidance scales in diffusion models. To do so, the authors propose a modification to the classifier-free guidance (CFG) update rule by decomposing the update term into parallel and orthogonal components relative to the conditional model prediction. Experimental results have verified the superior performance of the developed method.