No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models

Authors: Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, Romann M. Weber
Primary area: generative models
Keywords: diffusion models, classifier-free guidance

Abstract:
Classifier-free guidance (CFG) has become the standard method for enhancing the quality of conditional diffusion models. However, employing CFG requires either training an unconditional model alongside the main diffusion model or modifying the training procedure by periodically inserting a null condition.  There is also no clear extension of CFG to unconditional models. In this paper, we revisit the core principles of CFG and introduce a new method, independent condition guidance (ICG), which provides the benefits of CFG without the need for any special training procedures. Our approach streamlines the training process of conditional diffusion models and can also be applied during inference on any pre-trained conditional model. Additionally, by leveraging the time-step information encoded in all diffusion networks, we propose an extension of CFG, called time-step guidance (TSG), which can be applied to *any* diffusion model, including unconditional ones. Our guidance techniques are easy to implement and have the same sampling cost as CFG. Through extensive experiments, we demonstrate that ICG matches the performance of standard CFG across various conditional diffusion models. Moreover, we show that TSG improves generation quality in a manner similar to CFG, without relying on any conditional information.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730127269924

thread_type:
official_review

<thread_object>
creator: reviewer_c3pf

<content>
summary:
Reviewing the classifier-free guidance (CFG), this paper found that the unconditional score in CFG can be derived by a random label vector y_hat (termed ICG) without training the unconditional model by setting the y_null label. Furthermore, the authors stimulate the guidance by introducing a perturbated time-step embedding based on the original t embedding (termed TSG). Experiments show that ICG achieves similar results to CFG, and TSG improves the FID for conditional and unconditional tasks.

strengths:
1. The idea of ICG is novel and theoretically simple to implement. ICG provides a new perspective on CFG.

2. The presentation is good and the paper is easy to follow.

3. Extensive experiments show consistent effectiveness of the proposed two methods

weaknesses:
1. My major concern lies in the proposed TSG:
- the authors explain TSG from its connection to Langevin dynamics, but the connection is built up on Taylor approximation where $\tilde{t} = t + \Delta{t}$, and $\Delta{t}$ should be sufficiently small. But in practice, the authors use $\Delta{t}=st^\alpha n$ where $n \sim N(0, I), s=2, \alpha=1$ and $t$ is the time-step, therefore I do think $\Delta{T}$ is sufficiently small anymore.
- I suspect the effectiveness of TSG comes from the error correction, rather than the guidance perspective, according to the results shown in Table 4.
- Will TSG work in the diffusion models that do not embed class labels into timestep (for example, UViT)?

2. please provide the NFE and samplers used to report FID throughout the paper. 

3. some writing issues in section 3:
- According to eq 6 of the CFG paper [1], eq 4 in this paper looks wrong, eq 4 should be $\hat{D} = D_\theta (z_t, t, y) + w(D_\theta(z_t, t, y) - D_\theta(z_t, t, y_null))$.
- undefined notation $\beta_t$ in eq 2


[1] Ho, Jonathan, and Tim Salimans. "Classifier-free diffusion guidance." arXiv preprint arXiv:2207.12598 (2022).

questions:
The authors mention that there are two methods to decide the random vector $\hat{y}$: draw from Gaussian distribution or a random class label, so that $\nabla_{z_t} log p_t (z_t | \hat{y}) = \nabla_{z_t} log p_t (z_t)$. I wonder how much would the conditional score $\nabla_{z_t} log p_t (z_t | \hat{y})$ change in practice given different samples $\hat{y}_1, \hat{y}_2, \hat{y}_3 ...$ and the same $z_t$

I would increase my rating if some of my doubts and concerns were resolved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730492076915

thread_type:
official_review

<thread_object>
creator: reviewer_ryva

<content>
summary:
Classifier-free guidance (CFG) is a common method for improving the quality of samples generated from diffusion models. One limitation of CFG is the added training cost due to the requirement for both a conditional and an unconditional model. The authors propose Independent Condition Guidance (ICG), claiming that it provides the same advantages of CFG on image generation quality without additional training costs. They also propose Time-Step Guidance (TSG), a guidance algorithm based on a mixture of score estimates given both perturbed and unperturbed time-step values, which overcomes CFG’s reliance on the availability of class labels. TSG is based on a mixture of scores pertaining to a perturbed and unperturbed time-step value. The authors empirically validate their methods against CFG and un-guided baselines, demonstrating improvements in image quality over unguided sampling that are comparable in magnitude to CFG, while avoiding its limitations.

strengths:
1. The paper is generally well written and easy to follow.
2. The introduction and related works sections provided a good justification for the limitations of CFG and the benefits of solutions like ICG and TSG. There is certainly a potential for this paper to provide a meaningful contribution to the field of diffusion-guidance, if my concerns about the theoretical claims and empirical validation are adequately addressed by the authors.
3. The experiments are for the most part appropriate and the ablation study is useful to understand the effect of some of the hyperparameter choices.

weaknesses:
1. A central claim is that ICG provides the same benefits of CFG without the need for additional training requirements. This hinges on the validity of the theoretical claim in Section 4, which avoids the need for a potentially expensive marginalization over classes. However, the theoretical justification appears insufficient. It seems incorrect to state that sampling a class label $\hat{y}$ independently from $z_t$ at time-step $t$ implies the equivalence $\nabla_{z_t} \log p_t(\hat{y} \mid z_t) = \nabla_{z_t} \log p_t(z_t)$, since conditional independencies are a property of the diffusion model parametrized by $\theta$, irrespective of what distribution $y$ was drawn from to compute the sum in $(6)$. Given the conditional score estimate parametrized by $\theta$, $z_t$ should be dependent on $y$. The authors later allude to a procedure to "bootstrap the score", but this is not reflected in their Algorithm 1. Moreover the authors describe the correct equivalence in Appendix A, which involves marginalizing over $y$, but they don't explain how this relates to the seemingly incorrect equivalence relation (7).
2. Empirical evidence supporting TSG is limited. There are no baseline comparisons other than an unguided model. The baseline comparisons for ICG are also limited given that other methods have been published with similar motivations as ICG.
3. The choice of distribution from which $\hat{y}$ is sampled seems non-trivial, and although the authors attempted to shed some light on this in their ablation study, it is unclear how such a distribution should be chosen in general, and how sensitive are their results to the choice of distribution.
4. Several hyperparameters are introduced in the paper, but the description of the hyperparameter tuning procedure seems incomplete. Importantly, on which dataset were hyperparemeters tuned, and using what evaluation criterion?
5. Code is not made publicly available, which diminishes the paper's impact and impairs reproducibility. The provided pseudocode does not seem to entirely capture the design choices that the authors made as part of their experiments.

questions:
1. Please correct or clarify the theoretical justification behind ICG.
2. Are there not other perturbation-based guidance methods, such as the ones discussed in the Related Works section (in the same spirit as SAG), that TSG could be compared to?
3. Why not include a comparison between ICG and CADS alone (Appendix B only includes ICG+CADS)? It seems like it shares many similarities with ICG. If this is the case, why is this comparison not included in the main paper?
4. The connection between TSG and Langevin dynamics is interesting, but it is not clear how the hypothesized benefits of TSG relate to, or differ from, the theoretical benefits underlying the Langevin diffusion SDE that is implicit in the SDE corresponding to the forward diffusion process.
5. Can the authors better justify the need for two hyperparameters $s$ and $\alpha$ in TSG, and how they tuned these hyperparameters (and on which dataset)?
6. The conditional and unconditional samples from the ICG method seem to have slightly more contrast and are more saturated than the CFG baseline. If the authors agree with my impression, can they hypothesize as to why this should the case?
7. While the authors may be able to justify why ICG could theoretically perform as well as CFG, I don't understand why ICG would be expected to perform better than CFG, as concluded from the plots in Figure 3. Even if the authors can justify this, it would be helpful to understand how sensitive is this result is senstive to hyperparameter choices and random seeds.
8. Figure 5: I did not understand the rationale for testing ICG against the unguided baseline as opposed to ICG against CFG on the text condition. Can the authors clarify?

Minor:
9. Can the authors revisit Section 3,4, and 5, and ensure all variables and functions are properly defined? E.g. $\epsilon$ is undefined. This would help improve the readability.
10. It seems misleading to say that Equation (1) is "equivalent to" Equation (2), and that the denoiser $D_\theta$ "approximates" the score function.
11. It would be helpful to refer to the appendix sections in relevant sections of the main text.


---Post-rebuttal: The authors have addressed the majority of my questions and concerns in their responses to my review and in their global comment. I have therefore increased my rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675197756

thread_type:
official_review

<thread_object>
creator: reviewer_8vmw

<content>
summary:
This paper proposes two methods, independent condition guidance (ICG) and time-step guidance (TSG), for conditional and unconditional sampling from a diffusion model, which are summarized below. 

**ICG**

They propose ICG in place of classifier-free guidance (CFG) such that instead of learning two separate models or having two different tasks, one conditional and one conditional, they learn a single conditional model with only a conditional task without making use of null tokens. They do based on the insight that:

1. A conditional model can be turned into an unconditional model by noting that:
    1. the marginal distribution $p(x_t)$ is equal to the expectation $E_{y \sim p(y)}[p(x|y)]$ which can be estimated using sampling a single independent sample $y \sim p_{data}(y)$, which implies that $s_\theta(x_t) \approx s_\theta(x_t, y)$. 

They propose ICG to “streamline” the training of conditional model and improve efficiency (line 534) and as an alternative to CFG, which requires either separate models or a single model with a null token, representing unconditional generation. With ICG, the authors show that training a conditional model is sufficient and the CFG update can be replaced by:

$D_{ICG} = D(z_t, t, \widehat{y}) + w_{ICG}(D(z_t, t, y) - D(z_t, t, \widehat{y}))$ 

where $\widehat{y}$ is a Gaussian vector or a class sampled independent of $y$ at each time-step $t$.

**TSG**

TSG is based on the insight that score model architectures add the class-conditioning embedding into the time-step embedding. Therefore, similar to ICG, they perturb the time-step $t$ by $\Delta t \sim N(0, \sigma)$ TSG then uses a linear combination of $D_\theta(x_t, t)$ and $D_\theta(x_t, t + \Delta t)$ as the “score” network update. The justification of TSG is based on a connection to Langevin dynamics. 

In practice, they do not perturb the time, but similar to ICG they perturb the time-step embedding vector by $s t^{\alpha} n$, where $s, t, \alpha$ selected to match the mean and scale of the “real” time-step embedding vectors and $n \sim N(0, 1)$, and they propose the following update instead of CFG:

$D_{TSG} = D(z_t, \tilde{t}, y) + w_{TSG}(D(z_t, t, y) - D(z_t, \tilde{t}, y))$ where $\tilde{t}$ is the perturbed embedding.

strengths:
The paper is well-written and the authors perform extensive experiments and provide algorithmic and experimental details to reproduce their experiments. 

**Experiments**

The experiments show that ICG and CFG perform identically and ICG “simulates the behavior of CFG across several conditional models” (line 249). They also run a number of ablations, for instance, they compare using an independent Gaussian vector versus sampling $y \sim p(y)$ in ICG, providing stronger experimental evidence.

weaknesses:
The main question left un-answered by this work is whether a marginal model, either trained separately or with a null token, is necessary for a sampling method like CFG:
1. An analysis of the variance of the estimate $s_\theta(x_t \mid y)$ is missing from the paper. 
2. An analysis of the impact of the variance on sampling is also missing from the paper
3. Is there a trade-off between the slower convergence of training with a null token with estimation of the marginal model?

See the questions section.

The authors could improve the paper by providing a proof (or an empirical analysis) for the sampling procedure for a toy distribution where the $p_{data} = 0.5 N(-a, 1) + 0.5 N(a, 1)$. For instance, with a mixture of Gaussians as the data distribution, the marginal and conditional score functions are computable in closed form and a toy experiment, where various trade-offs can be examined, could make the paper stronger. 

TSG
For connecting TSG to Langevin dynamics, the authors show that eq 9 resembles a Langevin dynamics step. However, by using the perturbed TSG denoising model $\widehat{D}$  to define the score $\nabla_{z_t} \log \widehat{p}(z_t)$, the connection seems circular. Could the authors clarify what the connection to Langevin dynamics is and the relevance of that connection?

questions:
The authors present an estimate of the marginal model $p(x_t)$, where instead of marginalizing over all possible $y$ by learning a score model with a null token, they use a Monte Carlo estimate. However, the variance of the estimate of the marginal score and it’s impact on sampling has not been addressed in the paper. 

1. Supports of $p(x_t | y=1)$ and $p(x_t | y=0)$ are different: In case of two classes and when $p(x_t | \widehat{y})$ is low, in such a case the ICG term is the score of $p(x_t | \widehat{y})^{1 - w_{ICG}} p(x_t | y)^{w_{ICG}}$, if $p(x_t | \widehat{y})$ is low then its score’s magnitude is high and would dominate the ICG update, potentially pushing in the direction of $p(x_0 | \widehat{y})$
    1. will more steps be required for ICG sampling compared to CFG? 
    2. What will happen when the support for $x_t$ all $t \in [0, T]$ is disjoint for the two classes, such that $p(x_t | \widehat{y}) = 0$ when $\widehat{y}$ is the wrong class? For instance, if $p_{data} = 0.5  N(-5, 1) + 0.5 N(5, 1)$, and the model prior is a centered Gaussian.
2. If the classes are imbalanced then sampling from the majority class can bias the sampling process when the user wishes to sample from the minority class. For instance, if $p(y = 0 | x_0) = 0.99$ and $p(y = 1 | x_0)= 0.01$,  and ICG samples $\widehat{y}$ with an equal probability. The authors should consider modifying the algorithm to sample from $p(y)$, when available, and not a uniform distribution. 

Some simple theoretical guidance with/or toy examples should suffice for an explanation. Potentially with $p(x_0)$  as a mixture of Gaussians in 2d and the authors could vary the guidance scale while sampling.

1. For text to image sampling with ICG, do the authors recommend sampling random text prompts to estimate the marginal model? 
2. For the text conditional experiments in Table 3, can the authors also include CFG performance as a baseline?
3. For the ControlNet experiment, I believe the authors describe training with an empty string for 50% of their text prompts. See section 3.3 in the ControlNet paper. Do the authors use a random text and/or image prompt for the ControlNet model to define a marginal model? 
4. In line 053, the authors claim that there is no clear extension of CFG to unconditional generation:
    1. However, CFG requires training a marginal model, which can be sampled easily. Moreover, one could randomly sample a label y to do generation as well.  
    2. Can the authors provide details about unconditional sampling with ICG? The update defined in algorithm 1, requires a condition $y$ as input. is it just randomly sampling a label and then running ICG?
    
    $D_{ICG} = D(z_t, t, \widehat{y}) + w_{ICG}(D(z_t, t, y) - D(z_t, t, \widehat{y}))$
</content>
</thread_object>

</thread>
<thread>
creation date:
1732368859983

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Response to Reviewers

comment:
We thank all reviewers for their time and for providing valuable feedback to enhance the argumentation surrounding ICG. In response to the reviewers’ comments and suggestions, we offer the following detailed explanation regarding the bias of ICG. Based on this constructive exchange, we will modify Section 4 (and the appendix) in the final version of the paper to incorporate the points discussed during the rebuttal.

### **Bounding the Error on the Unconditional Score Approximation**
It can be shown (see Equation 15 in our paper) that
$$
\\mathbb{E}\_{p(y|z)} \\left[ \\nabla_z \\log p(z|y) \\right] = \\nabla_z \\log p(z).
$$
When the expectation is instead taken over the distribution $q(y)$, we have
$$
\\mathbb{E}\_{q(y)} \\left[ \\nabla_z \\log p(z|y) \\right] = \\nabla_z \\log p(z) - \\nabla_z \\mathcal{D}\_{\\text{KL}}(q(y)\\|p(y|z)).
$$

*Proof.* We can write $\\nabla_z \\log p(z|y) = \\nabla_z \\log p(z,y) - \\nabla_z \\log p(y) = \\nabla_z \\log p(z,y)$, which in turn can be written as $\\nabla_z \\log p(z,y) = \\nabla_z \\log p(z) + \\nabla_z \\log p(y|z)$. Taking the expectation over $q(y)$, we have
$$
\\mathbb{E}\_{q(y)} \\left[ \\nabla_z \\log p(z,y) \\right] = \\nabla_z \\log p(z) + \\mathbb{E}\_{q(y)} \\left[ \\nabla_z \\log p(y|z) \\right] = \\nabla_z \\log p(z) + \\text{Error}.
$$
Now consider the KL divergence between $q(y)$ and $p(y|z)$:
$$
\\mathcal{D}\_{\\text{KL}}(q(y)\\|p(y|z)) = \\int q(y) \\left[ \\log q(y) - \\log p(y|z) \\right] \\text{d}y.
$$
Taking the gradient with respect to $z$, we have 
$$
\\nabla_z \\mathcal{D}\_{\\text{KL}}(q(y)\\|p(y|z)) = -\\int q(y) \\nabla_z \\log p(y|z) \\mathrm{d}y = -\\mathbb{E}\_{q(y)} \\left[ \\nabla_z \\log p(y|z) \\right],
$$
which is the negative of the error term above, proving the result. $\\Box$

Now note that early in sampling, the input $z$ is so noisy that the mutual information between $y$ and $z$ is negligible. In this case, $p(y|z)$ is very close to $p(y)$. (Think of it as a classifier with noisy input.) Accordingly, if we choose $q(y) = p(y)$, the expected error almost vanishes early in the sampling. Toward the end, the bias of the ICG estimate increases as $p(y)$ moves further from $p(y|z)$. However, since guidance is more important in early and mid stages of sampling [1, 2], we hypothesize that any increasing error in the unconditional score estimate is largely ignored by the model. We believe this is one of the core reasons why ICG gives similar results to CFG in practice.

[1] Balaji, Yogesh, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis et al. "ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers." *arXiv preprint arXiv:2211.01324* (2022).

[2] Castillo, Angela, Jonas Kohler, Juan C. Pérez, Juan Pablo Pérez, Albert Pumarola, Bernard Ghanem, Pablo Arbeláez, and Ali Thabet. "Adaptive guidance: Training-free acceleration of conditional diffusion models." *arXiv preprint arXiv:2312.12487* (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1740847561037

thread_type:
public_comment

<thread_object>
creator: ~seyedmorteza_sadat1

<content>
title:
Camera-ready update

comment:
We would like to thank again the reviewers and the AC for supporting our submission. For the camera-ready version, we have reverted to our paper’s original title before submission, which also matches its arXiv version, as it more directly highlights our core contributions, particularly in rethinking classifier-free guidance without additional training requirements, while staying within the original scope of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The manuscript proposes a new approach for diffusion models that achieves the benefits of Classifier-Free Guidance (CFG) without the need to train both a conditional and unconditional model. Two novel methods are introduced: Independent Condition Guidance (ICG) and Time-Step Guidance (TSG). ICG eliminates the special training procedures required by CFG, thereby simplifying the training process for conditional diffusion models. TSG extends guidance techniques to unconditional diffusion models by leveraging time-step perturbation. Both methods are easy to implement, computationally efficient, and empirically validated to perform on par with or better than CFG across multiple tasks and benchmarks.

By addressing the training inefficiencies of CFG, ICG streamlines the implementation of conditional diffusion models without compromising performance, making it particularly relevant for large-scale and resource-constrained applications. TSG, on the other hand, extends guidance techniques to unconditional models. The experimental validation is thorough and well-executed, showing consistent performance across various models and tasks. The proposed techniques are straightforward to integrate into existing frameworks, ensuring their immediate applicability and impact for the broader community.

The reviewing panel unanimously recommends acceptance of this paper. The proposed methods address meaningful limitations of current techniques while maintaining computational efficiency and delivering strong empirical performance. The contributions are novel, broadly applicable, and of significant interest to the community.