Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

## Abstract

Abstract Classifier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models. While a high guidance scale is generally required to enhance these aspects, it also causes oversaturation and unrealistic artifacts. In this paper, we revisit the CFG update rule and introduce modifications to address this issue. We first decompose the update term in CFG into parallel and orthogonal components with respect to the conditional model prediction and observe that the parallel component primarily causes oversaturation, while the orthogonal component enhances image quality. Accordingly, we propose down-weighting the parallel component to achieve high-quality generations without oversaturation. Additionally, we draw a connection between CFG and gradient ascent and introduce a new rescaling and momentum method for the CFG update rule based on this insight. Our approach, termed adaptive projected guidance (APG) , retains the quality-boosting advantages of CFG while enabling the use of higher guidance scales without oversaturation. APG is easy to implement and introduces practically no additional computational overhead to the sampling process. Through extensive experiments, we demonstrate that APG is compatible with various conditional diffusion models and samplers, leading to improved FID, recall, and saturation scores while maintaining precision comparable to CFG, making our method a superior plug-and-play alternative to standard classifier-free guidance. 1 1 1 All visual results in the paper are best viewed in color and when zoomed in.

## 1 Introduction

Diffusion models (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ) are a class of generative models that learn the data distribution by reversing a forward process that adds noise to the data until the samples are indistinguishable from pure noise. Although the theory suggests that simulating the backward process in diffusion models should result in correct sampling from the data distribution, unguided sampling from diffusion models often results in low-quality images that do not align well with the input condition. Accordingly, classifier-free guidance (Ho & Salimans, ) has been established as an essential tool in modern diffusion models for increasing the quality of generations and the alignment between the condition and the generated image, albeit at the cost of reduced diversity (Ho & Salimans, ; Sadat et al., ).

Modern text-to-image models, such as Stable Diffusion (Rombach et al., ), generally require high guidance scales in order for the generations to have better quality and align well with the input prompt. However, high guidance scales often result in oversaturated colors and simplified image compositions (Saharia et al., ; Kynkäänniemi et al., ). Despite these disadvantages, high CFG scales are still used in practice due to their superior image quality compared to alternatives.

In this paper, we analyze the update rule of CFG and show that with a few modifications to how the CFG update is applied at inference, we can vastly mitigate the oversaturation and artifacts of high guidance scales. First, we show that the CFG update rule can be decomposed into two components, one that is parallel to the conditional model prediction, and one that is orthogonal to this prediction. We show that the orthogonal element is mainly responsible for improving image quality, while the parallel part primarily adds contrast and saturation to the output. To the best of our knowledge, this is the first study that disentangles these two effects in CFG.

Additionally, we establish a connection between the CFG update rule and stochastic gradient ascent. This insight leads us to explore a rescaled version of the CFG update direction and incorporate a momentum term, similar to adaptive optimization methods. The rescaling is motivated by the need to control large update norms, which can cause significant drifts in the sampling process. To prevent this, we constrain the updates to lie within a sphere. For the momentum term, unlike with traditional optimization, we apply a *negative* value to introduce a repulsive effect between consecutive updates, effectively down-weighting components already present in previous steps. We refer to this as *reverse momentum*. By combining rescaling, reverse momentum, and projection, we introduce a new method, called adaptive projected guidance (APG), which allows the use of higher guidance scales without oversaturation or degradation in image quality.

Through extensive experiments with several diffusion models, such as EDM2 (Karras et al., ) and Stable Diffusion (Rombach et al., ), we demonstrate that APG can utilize high guidance scales without encountering oversaturation. As a result, we conclude that APG significantly expands the usable guidance range in practice and mitigates the harmful effects of CFG at high guidance scales. Our quantitative analysis shows that replacing CFG with APG improves FID, recall, and saturation scores while maintaining precision similar to CFG. Furthermore, when combined with Stable Diffusion 3 (Esser et al., ), APG enhances the consistency of text rendering in generated images. We also demonstrate that APG is compatible with distilled models that use fewer sampling steps, such as SDXL-Lightning (Lin et al., ). A representative visual comparison between CFG and APG is shown in [Figure 1](#S0.F1).

## 2 Related work

Score-based diffusion models (Song & Ermon, ; Song et al., ; Sohl-Dickstein et al., ; Ho et al., ) learn data distributions by reversing a forward diffusion process that gradually corrupts data into Gaussian noise. These models have rapidly outperformed previous generative modeling methods in terms of fidelity and diversity (Nichol & Dhariwal, ; Dhariwal & Nichol, ), setting new benchmarks across various domains. They have achieved state-of-the-art results in unconditional image generation (Dhariwal & Nichol, ; Karras et al., ), text-to-image generation (Ramesh et al., ; Saharia et al., ; Balaji et al., ; Rombach et al., ; Podell et al., ; Yu et al., ), video generation (Blattmann et al., ; ; Gupta et al., ), image-to-image translation (Saharia et al., ; Liu et al., ), and audio generation (Chen et al., ; Kong et al., ; Huang et al., ).

Since the introduction of the DDPM model (Ho et al., ), numerous advancements have been made, such as improved network architectures (Hoogeboom et al., ; Karras et al., ; Peebles & Xie, ; Dhariwal & Nichol, ), enhanced sampling algorithms (Song et al., ; Karras et al., ; Liu et al., ; Lu et al., ; Salimans & Ho, ), and new training techniques (Nichol & Dhariwal, ; Karras et al., ; Song et al., ; Salimans & Ho, ; Rombach et al., ). Despite these advancements, diffusion guidance, including both classifier and classifier-free guidance (Dhariwal & Nichol, ; Ho & Salimans, ), remains crucial in enhancing generation quality and improving alignment between the condition and the output image (Nichol et al., ), albeit at the cost of reduced diversity and oversaturated outputs.

A recent line of work, such as CADS (Sadat et al., ) and interval guidance (IG) (Kynkäänniemi et al., ), has focused on enhancing the diversity of generations at higher guidance scales. In contrast, our proposed method, APG, specifically addresses the oversaturation issue in CFG, as these diversity-boosting methods still struggle with oversaturation at higher guidance scales. In [Section C.1](#A3.SS1), we demonstrate that APG can be combined with CADS and IG to achieve diverse generations without encountering oversaturation problems.

Dynamic thresholding (Saharia et al., ) was introduced to mitigate the saturation effect in CFG, but it is not directly applicable to latent diffusion models (since it assumes pixel values are between $[-1,1]$) and tends to produce images lacking in detail. Another approach, CFG Rescale (Lin et al., ), aims to reduce overexposure in generated images by rescaling the standard deviation of the predictions after applying CFG. However, we demonstrate that our method is noticeably more effective at reducing oversaturation compared to CFG Rescale.

Orthogonal projection has been explored in the context of text-to-3D generation (Armandpour et al., ) and non-linear guidance (Zheng & Lan, ), but none of these methods tackle the saturation issue at higher guidance scales. We also demonstrate that naive projection has minimal impact on CFG behavior, as it must be applied to the denoised predictions to be effective. Additionally, we incorporate rescaling and reverse momentum to further mitigate the adverse effects of CFG at higher guidance scales. We show that APG can be applied to various conditional diffusion models while adding practically no overhead to the sampling process.

## 3 Background

We provide a brief overview of diffusion models in this section. Let ${\bm{x}}\sim p_{\textnormal{data}}({\bm{x}})$ represent a data point, and let ${\bm{z}}_{t}={\bm{x}}+\sigma(t){\bm{\epsilon}}$ describe a forward process of the diffusion model that introduces noise to the data, where $t\in[0,1]$ is the time step. Here, $\sigma(t)$ is the noise schedule, which determines the amount of information destroyed at each time step $t$, with $\sigma(0)=0$ and $\sigma(1)=\sigma_{\textnormal{max}}$. Karras et al. () demonstrated that this forward process is equivalent to the following ordinary differential equation (ODE):

$$ $\mathrm{d}{\bm{z}}_{t}=-\dot{\sigma}(t)\sigma(t)\operatorname{\nabla}_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})\mathrm{d}t,$ (1) $$

where $p_{t}({\bm{z}}_{t})$ denotes the time-dependent distribution of noisy samples, with $p_{0}=p_{\textnormal{data}}$ and $p_{1}=\mathcal{N}(\mathbf{0},\sigma_{\textnormal{max}}^{2}\mathbf{I})$. With access to the time-dependent score function $\nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})$, one can sample from the data distribution $p_{\textnormal{data}}$ by solving the ODE backward in time (from $t=1$ to $t=0$). The unknown score function $\nabla_{{\bm{z}}_{t}}\log p_{t}({\bm{z}}_{t})$ is estimated using a neural denoiser $D_{{\bm{\theta}}}({\bm{z}}_{t},t)$, which is trained to predict the clean samples ${\bm{x}}$ from the corresponding noisy samples ${\bm{z}}_{t}$. This framework also allows for conditional generation by training a denoiser $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ that incorporates additional input signals ${\bm{y}}$, such as class labels or text prompts.

##### Classifier-free guidance (CFG)

CFG is an inference method designed to enhance the quality of generated outputs by combining the predictions of a conditional model and an unconditional model (Ho & Salimans, ). Given a null condition ${\bm{y}}_{\textnormal{null}}=\varnothing$ for the unconditional case, CFG modifies the denoiser’s output at each sampling step as follows:

$$ $\hat{D}_{\textrm{CFG}}({\bm{z}}_{t},t,{\bm{y}})=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})+w\left(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})\right),$ (2) $$

where $w=1$ represents the non-guided case. The unconditional model $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})$ is trained by randomly applying the null condition ${\bm{y}}_{\textnormal{null}}=\varnothing$ to the denoiser’s input for a portion of training. Alternatively, a separate denoiser can be trained to estimate the unconditional score in [Equation 2](#S3.E2) (Karras et al., ). Similar to the truncation method used in GANs (Brock et al., ), CFG improves the quality of images but reduces diversity (Murphy, ).

## 4 Adaptive projected guidance

We now present our method for addressing oversaturation and artifacts in CFG at high guidance scales.
Let $\Delta D_{t}=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})$ be the CFG update direction at time step $t$. Note that [Equation 2](#S3.E2) can now be written as

$$ $\displaystyle\hat{D}_{\textrm{CFG}}({\bm{z}}_{t},t,{\bm{y}})$ $\displaystyle=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+(w-1)\Delta D_{t}.$ (3) $$

(See [Appendix A](#A1) for the derivation.) We use [Equation 3](#S4.E3) for the rest of this paper to motivate our changes. APG has three elements: (1) projection, (2) rescaling, and (3) reverse momentum. We discuss each component below.

##### Orthogonal projection

First, note that we can decompose $\Delta D_{t}$ into two different components: $\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}$, which is parallel to $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$, and $\Delta D_{t}^{\perp}$, which is orthogonal to $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$, i.e., $\Delta D_{t}=\Delta D_{t}^{\perp}+\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}$. We can compute $\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}$ via orthogonal projection, with

$$ $\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}=\frac{\left\langle\Delta D_{t},D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})\right\rangle}{\left\langle D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})\right\rangle}D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}).$ (4) $$

We then have $\Delta D_{t}^{\perp}=\Delta D_{t}-\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}$. We observe that the orthogonal component is chiefly responsible for improvements in image quality, while the parallel component increases saturation in the generations as shown in [Figure 3](#S4.F3).

Accordingly, we modify the update direction to form $\Delta D_{t}(\eta)=\Delta D_{t}^{\perp}+\eta\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}$, where $\eta\leq 1$ is a hyperparameter. Note that $\Delta D_{t}(1)$ is identical to the unmodified CFG update direction described above. We show that reducing the strength of the parallel component (i.e. setting $\eta$ close to zero) significantly reduces saturation and results in more realistic generations at higher guidance scales.

The intuition behind the saturating effect of the parallel component is helped by thinking of the output $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ as an image with a typical range of values.(^2^22This intuition also holds for the image-like representations in latent diffusion models.) When an update parallel to this image is added, it serves to create a “gain,” pushing the values toward the extremes of their range. This gain effect can be seen by direct calculation:

$$ $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+(w-1)\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}=\left[1+(w-1)\frac{\left\lVert\Delta D_{t}^{{\mkern 3.0mu\vphantom{\perp}\vrule depth=0.0pt\mkern 2.0mu\vrule depth=0.0pt\mkern 3.0mu}}\right\rVert}{\left\lVert D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})\right\rVert}\right]D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),$ (5) $$

where we note that the term in brackets on the right-hand side is greater than one for $w>1$.
Thus, this term only adds saturation to the predictions $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ during each inference step, much like multiplying pixel values by a number greater than one. We show in [Section 5.2](#S5.SS2) that reducing $\eta$ and leaning more heavily on the orthogonal component significantly attenuates this saturation side effect in generations while maintaining the quality-boosting benefits of CFG.

##### Adding rescaling

Next, we argue that the CFG update rule in [Equation 3](#S4.E3) can be interpreted as one step of gradient ascent on the $\ell_{2}$ distance between the conditional and unconditional prediction, i.e., one step of gradient ascent on $\frac{1}{2}\left\lVert D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})\right\rVert^{2}$ with a learning rate of $w-1$. (See [Appendix A](#A1) for proof.) Inspired by this interpretation and normalized gradient ascent, we rescale the CFG update rule at each step to regulate the impact of each update. Specifically, we constrain $\Delta D_{t}$ to be inside a sphere with radius $r$ via

$$ $\Delta D_{t}\leftarrow\Delta D_{t}\cdot\min\left(1,\frac{r}{\left\lVert\Delta D_{t}\right\rVert}\right),$ (6) $$

where $r$ is a hyperparameter. This rescaling ensures that the CFG update $\Delta D_{t}$ stays closer to $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$, limiting drift at each sampling step if $\left\lVert\Delta D_{t}\right\rVert$ is large. As demonstrated in [Section 5.2](#S5.SS2), this adjustment improves both FID and recall.

##### Adding reverse momentum

Finally, leveraging the connection to gradient ascent, we introduce a reverse momentum term to the CFG update rule. We define the momentum for the CFG update direction as $\overline{\Delta D_{t}}\leftarrow\Delta D_{t}+\beta\overline{\Delta D_{t}}$, where $\overline{\Delta D_{t}}=0$ initially. The momentum term accounts for the average values of past updates; however, unlike standard optimization methods, we use a *negative* momentum strength $\beta<0$. Intuitively, this pushes the model away from previous CFG update directions and encourages the model to focus more on the current update direction. As shown in [Section 5.2](#S5.SS2), incorporating reverse momentum further enhances image quality (i.e., lower FID scores).

APG is easy to implement, and we provide the source code in [Algorithm 1](#alg1) (appendix). As shown in [Section 5.2](#S5.SS2), it is crucial to convert the diffusion model’s outputs (e.g., predicted noise) into the denoised prediction $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ in order to perform the projection. Further details on obtaining $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ for common prediction types are discussed in [Appendix B](#A2). [Figure 3](#S4.F3) demonstrates that using APG instead of CFG produces high-quality generations without oversaturation or the undesirable artifacts associated with high guidance scales.

Figure: (a) w/o CFG
Refer to caption: /html/2410.02416/assets/figures/edm/method/no_cfg_new.jpg

## 5 Experiments and results

##### Setup

We mainly experiment with text-to-image generation with Stable Diffusion (Rombach et al., ) and class-conditional ImageNet (Russakovsky et al., ) generation using EDM2 (Karras et al., ) and DiT-XL/2 (Peebles & Xie, ). For all experiments, we use the default diffusion sampler from each model (e.g., Euler scheduler for Stable Diffusion XL) along with pretrained checkpoints and corresponding codebases to ensure consistency in weights and the sampling process with the original frameworks.

##### Distribution metrics

We use Fréchet Inception Distance (FID) (Heusel et al., ) as our primary metric for evaluating both the quality and diversity of generated images due to its alignment with human judgment. Since FID is sensitive to small implementation details, we ensure that all models are evaluated under the same setup. For completeness, we also report precision (Kynkäänniemi et al., ) as an additional quality metric and recall (Kynkäänniemi et al., ) as a diversity metric.

##### Color metrics

While FID measures the overall quality of generated images, we introduce specific metrics to directly assess saturation and contrast. To measure saturation, we convert each image from RGB to HSV and compute the mean of the saturation channel. We define contrast (also known as RMS contrast) as the standard deviation of pixel values after converting the image to grayscale. The final metrics are derived by averaging the saturation and contrast values across all generated images.

### 5.1 Main results

Figure: Figure 4: Class-conditional generation results using EDM2 with $w=4$. APG significantly reduces saturation in the generations while keeping the high-quality global structure of each image.
Refer to caption: /html/2410.02416/assets/figures/edm/class_88_base.jpg

##### Qualitative results

[Figures 5](#S5.F5) and [5](#S5.F5) present our qualitative results comparing APG with CFG for EDM2 and Stable Diffusion XL. We observe that, compared to CFG, APG generates more realistic images with noticeably lower saturation. Furthermore, APG appears to produce fewer artifacts in the final outputs, as illustrated in [Figure 7](#S5.F7). Additional visual results can be found in [Appendix E](#A5).

Figure: A horse
Refer to caption: /html/2410.02416/assets/figures/sdxl/horse2-base.jpg

##### Quantitative results

We next present a quantitative comparison between APG and CFG in [Table 1](#S5.T1). The table shows that APG outperforms CFG across multiple models, consistently achieving better FID and recall scores, as well as lower saturation and contrast. Moreover, APG demonstrates similar precision to CFG, indicating that the reduction in saturation does not compromise the quality of individual samples.

**Table 1: Quantitative comparison between CFG and APG. APG consistently improves FID, recall and color metrics while maintaining similar or better precision compared to CFG.**
| Model | Guidance | FID $\downarrow$ | Precision $\uparrow$ | Recall $\uparrow$ | Saturation $\downarrow$ | Contrast $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| EDM2-S ($w=4$) | CFG | 10.42 | 0.85 | 0.48 | 0.46 | 0.27 |
| APG (Ours) | 6.49 | 0.85 | 0.62 | 0.33 | 0.21 |  |
| EDM2-XXL ($w=2$) | CFG | 8.65 | 0.84 | 0.57 | 0.37 | 0.23 |
| APG (Ours) | 4.94 | 0.83 | 0.67 | 0.31 | 0.21 |  |
| DiT-XL/2 ($w=4$) | CFG | 19.14 | 0.92 | 0.35 | 0.37 | 0.25 |
| APG (Ours) | 9.34 | 0.89 | 0.56 | 0.30 | 0.20 |  |
| Stable Diffusion 2.1 ($w=10$) | CFG | 27.53 | 0.65 | 0.41 | 0.36 | 0.27 |
| APG (Ours) | 22.21 | 0.67 | 0.49 | 0.27 | 0.22 |  |
| Stable Diffusion XL ($w=15$) | CFG | 26.29 | 0.62 | 0.49 | 0.28 | 0.24 |
| APG (Ours) | 25.35 | 0.64 | 0.50 | 0.18 | 0.17 |  |

Figure: Figure 8: Kernel density estimates of pixel and saturation values for two sets of samples generated with CFG and APG. Compared to CFG, images generated with APG show less concentration around saturated pixels, indicated by the spikes at the extreme values in both plots.

##### Distribution of pixel values

[Figure 8](#S5.F8) presents the kernel density estimate (KDE) plot of RGB and saturation values for 100 images generated using CFG and APG, along with KDE plots for 100 real samples drawn from the evaluation subset of ImageNet. Compared to CFG, APG plots are more broadly distributed across the spectrum with less concentration at the extremes. This indicates that images generated with APG are closer to real data in terms of saturation and color composition.

##### APG vs guidance scale

In [Figure 9](#S5.F9), we demonstrate that as the guidance scale increases, APG consistently achieves lower FID and higher recall while maintaining similar or better precision compared to CFG. Additionally, CFG exhibits increasing saturation at higher guidance scales, whereas APG maintains a relatively constant saturation level. Therefore, APG allows the usage of higher guidance scales, achieving better FID and diversity without oversaturation.

Figure: Figure 9: Comparison between CFG and APG as the guidance scale increases. APG offers better FID and recall while maintaining similar or better precision to CFG at higher guidance scales.

Figure: A pink dog
Refer to caption: /html/2410.02416/assets/figures/sdxl_lightning/pink_dog_base.jpg

##### Improving diversity

While APG is designed to address oversaturation at high guidance scales, we also observed that it can enhance the diversity of generations. As shown in [Table 1](#S5.T1) and [Figure 9](#S5.F9), APG improves distribution coverage (i.e., higher recall) while maintaining precision comparable to CFG. Additional qualitative results illustrating the enhanced diversity are provided in [Figure 21](#A5.F21) (appendix).

##### Using APG with distilled models

A common issue with CFG is that it degrades the quality of final outputs when applied to distilled models with fewer sampling steps (e.g., 8-step SDXL-Lightning (Lin et al., )). In this section, we show that APG does not encounter this problem and can be effectively applied to distilled models. [Figure 11](#S5.F11) demonstrates that replacing CFG with APG significantly improves generation quality. Extended results with additional models are provided in [Section C.3](#A3.SS3), along with more visual examples in [Appendix E](#A5).

##### Text spelling with Stable Diffusion 3

Next, we demonstrate that integrating APG with Stable Diffusion 3 (Esser et al., ) enhances the consistency of text rendering in generated images. As shown in [Figure 13](#S5.F13), APG produces more accurate spelling in the generated images compared to standard CFG. More visual results are given in [Figure 27](#A5.F27) (appendix).

Figure: A sign that says “A cow is singing”
Refer to caption: /html/2410.02416/assets/figures/sd3/cow_singing_base.jpg

Figure: Figure 14: Comparison between APG and CFG Rescale using Stable Diffusion XL. CFG Rescale is unable to solve the saturation issue at high guidance scales compared with APG.
Refer to caption: /html/2410.02416/assets/figures/cfg_rescale/lion_cfg_rescale.jpg

##### Comparison with CFG Rescale

CFG Rescale was introduced in (Lin et al., ) as a method to reduce saturation at high guidance scales. In this section, we demonstrate that APG is more effective than CFG Rescale. The comparison in [Figure 14](#S5.F14) shows that APG outputs have significantly less saturation and are more realistic than those produced with CFG Rescale.

##### Computational cost of APG

The computational cost of APG is practically identical to that of CFG, as the rescaling and projection steps incur negligible overhead compared to querying the denoiser. Specifically, in the case of Stable Diffusion XL, the forward pass through the diffusion network takes approximately 130 milliseconds on an RTX 3090 GPU for a single image, while the guidance step requires only about 0.45 milliseconds.

### 5.2 Ablation studies

We now present our ablation studies in this section. The experiments are based on class-conditional generation using the EDM2 model (Karras et al., ), with FID as the primary metric to justify our design choices. First, [Table 2](#S5.T2) highlights the importance of each component in APG. We observe that removing projection, rescaling, or reverse momentum results in higher FID scores. Additionally, note that the projection component is primarily responsible for reducing saturation while rescaling and reverse momentum mainly improve FID and recall. [Section C.8](#A3.SS8) gives extended ablation results on the effect of each component in APG.

##### Importance of the model prediction type

While CFG works the same across all model prediction types, we observed that our method performs best when applied to the denoised predictions $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$, rather than, for example, the noise prediction ${\bm{\epsilon}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$. This is illustrated in [Figure 15](#S5.F15), where projecting onto ${\bm{\epsilon}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ produces results nearly identical to CFG, while projecting onto $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ significantly reduces saturation. Note that as discussed in [Appendix B](#A2), this is not a bottleneck for APG as various prediction types can be readily converted to $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ at each step.

Figure: (a) CFG
Refer to caption: /html/2410.02416/assets/figures/ablations/projection/cfg.jpg

## 6 Conclusion and discussion

In this work, we investigated the oversaturation effect of high CFG scales and introduced a new method, adaptive projected guidance (APG), that achieves the same quality-boosting benefits as CFG without causing oversaturation. The key idea behind APG is to project the CFG update onto the denoised prediction of the diffusion model $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ and remove or down-weight the component parallel to that prediction. Additionally, by linking CFG to gradient ascent, we demonstrated that its performance can be further enhanced by incorporating rescaling and reverse momentum. Through extensive experiments, we showed that APG improves FID, recall, and saturation metrics compared to CFG, while maintaining similar or better precision. Thus, APG offers a plug-and-play alternative to standard CFG capable of delivering superior results with practically no additional computational overhead. Like CFG, challenges remain in accelerating APG so that the sampling cost approaches that of the unguided sampling (i.e., removing the need to query the diffusion network twice at each sampling step). We consider this a promising direction for future research.

#### Ethics Statement

As generative modeling continues to evolve, the potential to create fake or erroneous data increases. While advancements in AI-generated content can enhance efficiency and foster creativity, it is crucial to address the associated ethical concerns. For a more detailed discussion on ethics and creativity in computer vision, we recommend Rostamzadeh et al. ().

#### Reproducibility Statement

This work builds on the official implementations of the pretrained models referenced in the main text. The source code for implementing APG is provided in [Algorithm 1](#alg1), and [Appendix D](#A4) outlines additional implementation details, including the hyperparameters used in the main experiments.

## Appendix A Details on CFG as gradient ascent

In this section, we discuss how CFG can be interpreted as one step of gradient ascent. To begin, note that the CFG update rule can be expressed as:

$$ $\displaystyle\hat{D}_{\textrm{CFG}}({\bm{z}}_{t},t,{\bm{y}})$ $\displaystyle=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})+w(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}}))$ (7) $\displaystyle=wD_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+(1-w)D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})$ (8) $\displaystyle=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+(w-1)D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+(1-w)D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})$ (9) $\displaystyle=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+(w-1)\left(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})\right)$ (10) $\displaystyle=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+\gamma{\Delta D_{t}},$ (11) $$

where $\gamma=w-1$, and $\Delta D_{t}=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})$. Next, observe that we can write:

$$ $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})=\operatorname{\nabla}_{D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})}\left[\frac{1}{2}\left\lVert D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})\right\rVert^{2}\right].$ (12) $$

Thus, if we define the CFG objective function as

$$ $f_{\mathrm{CFG}}(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}}))=\frac{1}{2}\left\lVert D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})-D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})\right\rVert^{2},$ (13) $$

the CFG update rule becomes equivalent to:

$$ $\hat{D}_{\textrm{CFG}}({\bm{z}}_{t},t,{\bm{y}})=D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})+\gamma\operatorname{\nabla}_{D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})}f_{\mathrm{CFG}}(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}})).$ (14) $$

Hence, we have shown that the CFG update rule corresponds to a single step of gradient *ascent* with respect to the objective function $f_{\mathrm{CFG}}(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}}))$.

This interpretation motivated us to incorporate rescaling into standard CFG. Since the objective function $f_{\mathrm{CFG}}(D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}_{\textnormal{null}}))$ does not have a maximum, the CFG update step may result in arbitrary drift from $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$. By applying rescaling, we constrain the CFG update to remain within a ball of limited radius around $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$. The reverse momentum method is similarly inspired by this interpretation, where each update is pushed away from previous predictions.

## Appendix B Denoised prediction for different diffusion models

We next briefly outline the process of computing the denoised prediction $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ for various diffusion models. For further details, we refer readers to Kingma & Gao (). In the following, let ${\bm{x}}$ represent the clean data, ${\bm{y}}$ a condition or class, and ${\bm{\epsilon}}\sim\mathcal{N}\left(\boldsymbol{0},{\boldsymbol{I}}\right)$ the noise. Given a noisy sample ${\bm{z}}_{t}$ at time step $t$, the objective is to recover the clean data ${\bm{x}}$ that produced ${\bm{z}}_{t}$. The denoised version of ${\bm{z}}_{t}$, which approximates ${\bm{x}}$, is estimated by a neural network, denoted as $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$. Before applying APG, we always convert all model predictions to $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$. This conversion is compatible with most samplers based on the denoising framework, such as EDM (Karras et al., ) and DPM++ (Lu et al., ). The conversions for various models are derived below, and a summary is provided in [Table 3](#A2.T3).

##### DDPM

For models using the DDPM framework (Ho et al., ), the forward diffusion process is defined as ${\bm{z}}_{t}=\alpha_{t}{\bm{x}}+\sigma_{t}{\bm{\epsilon}}$, where $\sigma_{t}^{2}+\alpha_{t}^{2}=1$. These models typically predict the total added noise ${\bm{\epsilon}}$ via a neural network ${\bm{\epsilon}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$. Given the prediction of the model, the denoised prediction can be estimated via

$$ $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})=\frac{{\bm{z}}_{t}-\sigma_{t}{\bm{\epsilon}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})}{\alpha_{t}}.$ (15) $$

If the model predicts the velocity ${\bm{v}}=\alpha_{t}{\bm{\epsilon}}-\sigma_{t}{\bm{x}}$, we have

$$ ${\bm{v}}=\alpha_{t}\frac{{\bm{z}}_{t}-\alpha_{t}{\bm{x}}}{\sigma_{t}}-\sigma_{t}{\bm{x}}=\frac{\alpha_{t}{\bm{z}}_{t}-\alpha_{t}^{2}{\bm{x}}-\sigma_{t}^{2}{\bm{x}}}{\sigma_{t}}=\frac{\alpha_{t}{\bm{z}}_{t}-{\bm{x}}}{\sigma_{t}}.$ (16) $$

This leads to the following formulation for the denoised prediction:

$$ $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})=\alpha_{t}{\bm{z}}_{t}-\sigma_{t}{\bm{v}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}).$ (17) $$

##### EDM framework

For the EDM framework (Karras et al., ), the forward process is described by ${\bm{z}}_{t}={\bm{x}}+\sigma(t){\bm{\epsilon}}$, and the denoised prediction $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ is formulated via

$$ $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})=c_{\mathrm{skip}}(t){\bm{z}}_{t}+c_{\mathrm{out}}(t)F_{{\bm{\theta}}}(c_{\mathrm{in}}(t){\bm{z}}_{t},c_{\mathrm{noise}}(t),{\bm{y}}),$ (18) $$

where $F_{{\bm{\theta}}}(c_{\mathrm{in}}(t){\bm{z}}_{t},c_{\mathrm{noise}}(t),{\bm{y}})$ is the output of the neural network. The EDM framework uses $\sigma(t)\propto t$; thus, $\sigma$ and $t$ can be used interchangeably in this framework.

##### Rectified flow models

For rectified flow models (Liu et al., ), such as Stable Diffusion 3 (Esser et al., ), the forward process is given by ${\bm{z}}_{t}=(1-t){\bm{x}}+t{\bm{\epsilon}}$. The model predicts the velocity field given by ${\bm{v}}={\bm{\epsilon}}-{\bm{x}}$. Accordingly, we have

$$ $\displaystyle{\bm{v}}$ $\displaystyle={\bm{\epsilon}}-{\bm{x}}=\frac{{\bm{z}}_{t}-(1-t){\bm{x}}}{t}-{\bm{x}}=\frac{{\bm{z}}_{t}-(1-t){\bm{x}}-t{\bm{x}}}{t}=\frac{{\bm{z}}_{t}-{\bm{x}}}{t}.$ (19) $$

Thus, the denoised prediction can be determined by:

$$ $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})={\bm{z}}_{t}-t{\bm{v}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})={\bm{z}}_{t}-\sigma_{t}{\bm{v}}_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}}),$ (20) $$

where we define $\sigma_{t}=t$.

Table: Table 3: Summary of calculating denoised predictions $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$ for different diffusion models.

This section demonstrates the effect of applying APG on a toy example to illustrate the differences between APG and CFG. We use a mixture of two high-dimensional Gaussians as the data distribution, which allows us to analytically compute the score functions during the diffusion process, eliminating potential errors introduced by a denoiser network. Specifically, the data distribution $p_{\textnormal{data}}$ is defined as:

$$ $p_{\textnormal{data}}({\bm{x}})=\frac{1}{2}\mathcal{N}\left({\bm{\mu}}_{1},\sigma^{2}{\boldsymbol{I}}\right)({\bm{x}})+\frac{1}{2}\mathcal{N}\left({\bm{\mu}}_{2},\sigma^{2}{\boldsymbol{I}}\right)({\bm{x}}),$ (21) $$

where ${\bm{\mu}}_{1}=[-2,-2,\dots,-2]$, ${\bm{\mu}}_{2}=[2,2,\dots,2]$, and $\sigma=0.25$. We use a dimensionality of 500 for each component. Accordingly, the conditional distributions are equal to

$$ $p_{\textnormal{data}}({\bm{x}}\left|\right.y=1)=\mathcal{N}\left({\bm{\mu}}_{1},\sigma^{2}{\boldsymbol{I}}\right)({\bm{x}})\qquad\mathrm{and}\qquad p_{\textnormal{data}}({\bm{x}}\left|\right.y=2)=\mathcal{N}\left({\bm{\mu}}_{2},\sigma^{2}{\boldsymbol{I}}\right)({\bm{x}}).$ (22) $$

The sampling results are shown in [Figure 16](#A2.F16) (visualizing the first two dimensions of each Gaussian). When CFG is applied with a high guidance scale, it results in a drift toward regions less likely according to the data distribution. In contrast, applying APG corrects this drift and improves mode coverage. While this is a simplified example, we argue that a similar phenomenon occurs when applying CFG to images, leading to artifacts and oversaturation in the final outputs.

Figure: (a) Real data
Refer to caption: /html/2410.02416/assets/x1.png

## Appendix C Additional experiments

Additional experiments and ablation studies are included in this section. Unless stated otherwise, the experiments are conducted using class-conditional ImageNet (Russakovsky et al., ) generation.

### C.1 Compatibility with CADS and IG

We first demonstrate that APG is compatible with CADS (Sadat et al., ) and interval guidance (IG) (Kynkäänniemi et al., ), both of which are designed to enhance the diversity of generations at high guidance scales. The results, shown in [Table 4](#A3.T4), indicate that replacing CFG with APG leads to improved FID, recall, and saturation scores for both methods.

**Table 4: Compatibility of APG with CADS (Sadat et al., ) and IG (Kynkäänniemi et al., ). Combining APG with other methods that improve diversity results in better FID than each method in isolation.**
| Guidance | FID $\downarrow$ | Precision $\uparrow$ | Recall $\uparrow$ | Saturation $\downarrow$ | Contrast $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| CFG | 10.42 | 0.85 | 0.48 | 0.46 | 0.27 |
| +CADS | 8.65 | 0.85 | 0.56 | 0.43 | 0.26 |
| +APG | 6.49 | 0.85 | 0.62 | 0.33 | 0.21 |
| +both | 5.56 | 0.84 | 0.64 | 0.32 | 0.21 |

### C.2 APG vs number of sampling steps

We now present the performance comparison between APG and CFG across different numbers of sampling steps using the EDM2 model. [Figure 17](#A3.F17) indicates that APG consistently provides better FID, recall, and saturation while maintaining the same level of precision.

Figure: Figure 17: Comparison of CFG and APG across different numbers of sampling steps. APG consistently achieves better FID and recall while maintaining comparable or superior precision to CFG.

### C.3 Using APG with distilled models

In this section, we show the compatibility of APG with distilled models using PixArt-$\delta$ (Chen et al., ), SDXL-Lightning (Lin et al., ), and SDXL-Flash (). Consistent with the main text, [Figure 19](#A3.F19) demonstrates that replacing CFG with APG significantly improves generation quality and saturation level across all models. This is also consistent with [Figure 17](#A3.F17), where APG outperforms CFG at fewer sampling steps (e.g., 8-16).

Figure: A blue bird
Refer to caption: /html/2410.02416/assets/figures/pixart_lcm/bird_base_v2.jpg

### C.4 Compatibility with different samplers

While the main experiment in [Table 1](#S5.T1) used the default sampler of each model, we next separately show that APG is compatible with different sampling algorithms widely used with diffusion models. As shown in [Table 5](#A3.T5), using APG with different samplers results in improved FID, recall, and saturation scores, consistent with the main findings in [Table 1](#S5.T1). We used class-conditional generation using DiT-XL/2 for this experiment.

**Table 5: Impact of using APG with popular diffusion samplers using the class-conditional ImageNet model (DiT-XL/2). Compared to CFG, APG showes improved metrics across all samplers.**
|  | APG (Ours) | CFG |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Sampler | FID $\downarrow$ | Recall $\downarrow$ | Saturation $\downarrow$ | FID $\downarrow$ | Recall $\downarrow$ | Saturation $\downarrow$ |
| DDIM (Song et al., ) | 6.69 | 0.62 | 0.30 | 17.45 | 0.38 | 0.42 |
| DPM++ (Lu et al., ) | 6.87 | 0.62 | 0.32 | 17.65 | 0.38 | 0.43 |
| SDE-DPM++ (Lu et al., ) | 8.53 | 0.57 | 0.32 | 19.01 | 0.36 | 0.43 |
| PNDM (Liu et al., ) | 5.37 | 0.68 | 0.32 | 16.50 | 0.40 | 0.43 |
| UniPC (Zhao et al., ) | 6.91 | 0.62 | 0.32 | 17.65 | 0.38 | 0.43 |

### C.5 Compatibility with ICG

Independent condition guidance (ICG) (Sadat et al., ) is a method to apply CFG without the need to query an unconditional model. In [Table 6](#A3.T6), we show that APG is compatible with ICG, and similar to CFG, using APG with ICG results in improved FID, recall, and saturation scores while maintaining similar precision. We use class-conditional ImageNet generation with EDM2-S for this experiment.

**Table 6: Compatibility of APG and ICG. Combining APG with ICG significantly improves FID, recall, and saturation scores while maintaining similar precision.**
| Guidance | FID $\downarrow$ | Precision $\uparrow$ | Recall $\uparrow$ | Saturation $\downarrow$ | Contrast $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| ICG | 17.63 | 0.85 | 0.32 | 0.49 | 0.28 |
| +APG (Ours) | 5.73 | 0.85 | 0.63 | 0.33 | 0.22 |

### C.6 Compatibility with TSG

Time-step guidance (TSG) (Sadat et al., ) is an extension of CFG that leverages the time-step information learned by the diffusion model to enhance the quality of generations. We next demonstrate that applying the update rule in APG further improves the performance of TSG. [Table 7](#A3.T7) shows that APG improves FID, recall, and saturation metrics, while maintaining similar precision to TSG. This experiment is based on class-conditional ImageNet generation using DiT-XL/2.

**Table 7: Compatibility of APG and TSG with. Combining APG with TSG improves FID, recall, and saturation scores while maintaining similar precision.**
| Guidance | FID $\downarrow$ | Precision $\uparrow$ | Recall $\uparrow$ | Saturation $\downarrow$ | Contrast $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| TSG | 14.00 | 0.81 | 0.52 | 0.37 | 0.28 |
| +APG (Ours) | 5.84 | 0.81 | 0.66 | 0.30 | 0.20 |

### C.7 Alignment with the condition

We next demonstrate that replacing CFG with APG does not compromise the alignment between the input condition and the output. To validate this, we measure the classification accuracy of the generated results for the ImageNet task and the CLIP score for Stable Diffusion. The results in [Table 8](#A3.T8) show that both CFG and APG achieve comparable alignment metrics. Thus, APG reduces saturation and improves FID without compromising condition alignment.

### C.8 Extended ablation studies

##### Effect of the parallel component

We next demonstrate the effect of $\eta$ on the generated images in [Table 9(a)](#A3.T9.st1). As hypothesized in [Section 4](#S4), increasing the strength of the parallel component leads to higher saturation levels and increased FID. We recommend setting $\eta=0$ by default and only increasing it if more saturation is desired in the generated images.

##### Effect of the rescaling threshold

The effect of the rescaling radius $r$ on the generated images is shown in [Table 9(b)](#A3.T9.st2). Excessive rescaling degrades image quality, while high values of $r$ result in no noticeable change, as the rescaling function approaches the identity function. Therefore, midrange values for $r$ yield better FID scores. We suggest observing the norm of $\Delta D_{t}$ during the inference process and choosing $r$ in a way that is comparable (on average) to the norm of $\Delta D_{t}$.

##### Effect of the momentum strength

[Table 9(c)](#A3.T9.st3) shows the effect of momentum strength $\beta$ on generation quality. Note
Negative values for $\beta$ result in better FID compared to positive momentum, and excessive momentum degrades image quality. This aligns with our hypothesis that moving away from the previous directions helps limit the drift that can occur during sampling with higher guidance scales. Empirically, we found that $\beta\in[-0.75,-0.25]$ works well in most setups.

**Table 9: Ablation study examining various design elements in APG.**
| $\eta$ | FID $\downarrow$ | $Saturation$↓ |
| --- | --- | --- |
| 0.0 | 6.49 | 0.33 |
| 0.25 | 6.49 | $0.34$ |
| 0.5 | 6.49 | $0.36$ |
| 1.0 | 6.63 | $0.37$ |

## Appendix D Implementation details

We provide the code for APG in [Algorithm 1](#alg1). Compared to CFG, APG only includes a few additional lines of code without noticeable computational overhead. As discussed in [Section 4](#S4), we always convert the predictions of the diffusion model to $D_{{\bm{\theta}}}({\bm{z}}_{t},t,{\bm{y}})$, compute the guided prediction, and convert it back to the initial output type at each sampling step.

We mainly use the ADM evaluation suite (Dhariwal & Nichol, ) for computing FID, precision, and recall. The FID is computed using $10000$ generated images and the whole training set for class-conditional ImageNet models. For text-to-image models, the FID is evaluated using the evaluation subset of MS COCO 2017 (Lin et al., ). The hyperparameters used for the main experiment are given in [Table 10](#A4.T10).

**Table 10: Hyperparameters used in the main experiment ([Table 1](#S5.T1)).**
| Model | $w$ | $\eta$ | $r$ | $\beta$ |
| --- | --- | --- | --- | --- |
| EDM2-S | 4 | 0 | 2.5 | $-0.75$ |
| EDM2-XL | 2 | 0 | 2.5 | $-0.75$ |
| DiT-XL/2 | 4 | 0 | 5 | $-0.50$ |
| Stable Diffusion 2.1 | 10 | 0 | 7.5 | $-0.75$ |
| Stable Diffusion XL | 15 | 0 | 15 | $-0.50$ |

Figure: Algorithm 1 PyTorch implementation of APG.

## Appendix E More visual results

This section presents extended visual comparisons between APG and CFG. Additional results using EDM2 are provided in [Figure 20](#A5.F20), with an example of how APG enhances diversity shown in [Figure 21](#A5.F21). Further images for Stable Diffusion 2.1 and Stable Diffusion XL are included in [Figures 23](#A5.F23) and [25](#A5.F25). Moreover, [Figure 27](#A5.F27) illustrates how APG improves text spelling in Stable Diffusion 3. Finally, more examples of APG applied to distilled models are shown in [Figures 29](#A5.F29), [33](#A5.F33) and [31](#A5.F31).

Figure: Figure 20: More visual results comparing APG and CFG using EDM2.
Refer to caption: /html/2410.02416/assets/figures/edm/appendix/class_40_cfg.jpg

Figure: Figure 21: Showcasing the diversity of generations after using APG. APG removes oversaturation issues while improving diversity w.r.t. the overall image composition.
Refer to caption: /html/2410.02416/assets/figures/dit/cls_33_cfg_old.jpg

Figure: Astronaut on the Moon
Refer to caption: /html/2410.02416/assets/figures/sd2/astronaut_base.jpg

Figure: A 4k dslr photo of a raccoon wearing an astronaut helmet, photorealistic.
Refer to caption: /html/2410.02416/assets/figures/sdxl/appendix/sdxl_rackon_base.jpg

Figure: A t-shirt with “Archaeology Rocks!” written on it
Refer to caption: /html/2410.02416/assets/figures/sd3/arch_base.jpg

Figure: Image of a corgi
Refer to caption: /html/2410.02416/assets/figures/pixart_lcm/appendix/corgi_base.jpg

Figure: Astronaut riding a horse
Refer to caption: /html/2410.02416/assets/figures/sdxl_flash/appendix/grid-0026.jpg

Figure: A cave under the ocean
Refer to caption: /html/2410.02416/assets/figures/sdxl_lightning/appendix/grid-0010.jpg

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses section.
soundness: 3
strengths: - The intuition and theory are well-balanced.
- The results are quite promising. Their guidance does not drastically alter the image content, which aligns well with their claims.
- They demonstrate robustness across various types of diffusion models.
summary: This work aims to reduce the oversaturation in the current CFG approach. The authors decompose the CFG update term into two components—parallel and orthogonal—and both theoretically and empirically demonstrate the role of each component. Based on these observations, they redesign the weighting of each component and additionally propose an adaptive rescaling strategy.
weaknesses: - I do not view this as a straightforward rescaling of CFG. However, it might be open to interpretation, and I would be interested to hear the authors’ perspective on this.

- Which model was used for Figure 8? 
- Further analysis on the rescale weight would be helpful—does it show any trends with respect to t? 

- *(As a suggestion)* Including illustrations of the geometry, along with the related intuitions, would further enhance the clarity.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Please refer to the Weakness section.
soundness: 4
strengths: 1. The paper introduces a new method, APG, that addresses a significant problem in diffusion models—oversaturation and artifacts associated with high guidance scales. The approach is innovative and directly tackles a well-known issue in the field.
2. The authors provide extensive experimental results to validate their claims. They demonstrate the effectiveness of APG across various models and samplers, showing improvements in FID, recall, and saturation scores.
3. The paper provides a clear comparison between APG and CFG, both qualitatively and quantitatively. The visual results are particularly compelling in showing the reduction in oversaturation and artifacts.
summary: The paper introduces a novel method called Adaptive Projected Guidance (APG) to address the issue of oversaturation and unrealistic artifacts that occur when using high guidance scales in diffusion models. Specifically, the authors propose a modification to the classifier-free guidance (CFG) update rule by decomposing the update term into parallel and orthogonal components relative to the conditional model prediction. The authors conduct extensive experiments to support their claims.
weaknesses: 1. The final example generated by APG in Figure 10 has a logical error, where the cat has only one paw.
2. For Table 1, the authors should consider adding HPSv2 and ImageReward metrics to evaluate the generation performance.
3. For Table 1, why is the guidance scale of Stable Diffusion XL so large (w=15)? When reducing the guidance scale, are the performance improvements limited? The authors should provide more quantitative results.
4. The authors claim that the orthogonal component is chiefly responsible for improvements in image quality, while the parallel component increases saturation in the generations. How many examples do the authors observe? Are there any theories to support this observation?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: See the questions in the weakness section.
soundness: 3
strengths: 1. APG is presented as a plug-and-play method with virtually no additional computational overhead, making it attractive for practical use in various diffusion models.
2. The method of down-weighting the parallel component in CFG is novel and interesting, effectively reducing oversaturation and preserving image quality at high guidance scales.
summary: This paper presents a method called Adaptive Projected Guidance (APG) to address the issues of oversaturation and artifacts that occur when using high guidance scales in Classifier-Free Guidance (CFG) for diffusion models. The authors decompose the CFG update rule into two components: a parallel component responsible for oversaturation and an orthogonal component that enhances image quality. They propose down-weighting the parallel component to reduce oversaturation. Additionally, they introduce rescaling and reverse momentum to improve sampling. Extensive experiments demonstrate that APG can improve FID, recall, and saturation scores while maintaining precision, making it a superior alternative to standard CFG.
weaknesses: 1. In Figure 8, could the authors explain why the proposed APG shows lower precision and higher FID compared to CFG when the guidance scale  w  is below certain values (e.g.,  w = 2.5 , where APG precision is lower than CFG)?
2. While reverse momentum is introduced, the intuition behind why it improves the image generation quality could be expanded. More detailed theoretical or experimental exploration of this component would clarify its importance.

Meta-review: This work presents a novel method called Adaptive Projected Guidance (APG) to address the issue of oversaturation and unrealistic artifacts that occur when using high guidance scales in diffusion models. To do so, the authors propose a modification to the classifier-free guidance (CFG) update rule by decomposing the update term into parallel and orthogonal components relative to the conditional model prediction. Experimental results have verified the superior performance of the developed method.

Decision: Accept (Poster)