No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models
Authors: Seyedmorteza Sadat
Manuel Kansy
Otmar Hilliges
Romann M. Weber
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
classifier-free guidance
Primary area: generative models
TL;DR: We propose two methods that enable the use of classifier-free guidance without training an auxiliary unconditional model or relying on any conditional information.

Abstract: Classifier-free guidance (CFG) has become the standard method for enhancing the quality of conditional diffusion models. However, employing CFG requires either training an unconditional model alongside the main diffusion model or modifying the training procedure by periodically inserting a null condition.  There is also no clear extension of CFG to unconditional models. In this paper, we revisit the core principles of CFG and introduce a new method, independent condition guidance (ICG), which provides the benefits of CFG without the need for any special training procedures. Our approach streamlines the training process of conditional diffusion models and can also be applied during inference on any pre-trained conditional model. Additionally, by leveraging the time-step information encoded in all diffusion networks, we propose an extension of CFG, called time-step guidance (TSG), which can be applied to *any* diffusion model, including unconditional ones. Our guidance techniques are easy to implement and have the same sampling cost as CFG. Through extensive experiments, we demonstrate that ICG matches the performance of standard CFG across various conditional diffusion models. Moreover, we show that TSG improves generation quality in a manner similar to CFG, without relying on any conditional information.

Published as a conference paper at ICLR 2025
No TRAINING, No PROBLEM: RETHINKING -RESS GUIDANCE FOR DIFFUSION MODELS
Seyedmorteza Sadat', Manuel Kansy', Otmar Hilliges', Romann M. Weber?
'ETH Zürich, "DisneyResearcchStuuiios [seyedmorteza. sadat, manuel kansy, otmar, otwar.hiiliiin....o.........iiee www.:hiillig..... (iiff 0inf::iiin......eeeee (romann. weber) @di edw.sneyreeercch.com com
ABSTRACT
Classifier-free guidance (CFG) has become the standard method for enhancing the quality of conditional diffusion models. However, employing CFG requires either training an unconditional model alongside the main diffusion model or modifying the training procedure by periodically inserting null condition. There is also no clear extension of CFG to unconditional models. In this paper, we revisit the core principles of CFG and introduce a new method, independent condition guidance (ICG), which provides the benefits of CFG without the need for any special training procedures. Our approach streamlines the training process of conditional diffusion models and can also be applied during inference on any pre- trained conditional model. Additionally, by leveraging the time-step information encoded in all diffusion networks, we propose an extension of CFG, called time- step guidance (TSG), which can be applied to any diffusion model, including unconditional ones. Our guidance techniques are easy to implement and have the same sampling cost as CFG. Through extensive experiments, we demonstrate that ICG matches the performance of standard CFG across various conditional diffusion models. Moreover, we show that TSG improves generation quality in a manner similar to CFG, without relying on any conditional information.
INTRODUCTION
Diffusion models have recently emerged as the main methodology behind many successful generative models (Sohl-Dickstein al., 2015; Ho al., 2020; Dhariwal & Nichol, 2021; Rombach et al., 2022; Song & Ermon, 2019; Song et al., 2021b). At the core of such models lies diffusion process that gradually adds noise to the data until the corrupted points are indistinguishable from pure noise. During inference, denoiser trained to reverse this process is used to gradually refine pure-noise samples until they resemble the clean data. While the theory suggests that standard sampling from diffusion models should yield high-quality images, this does not generally hold in practice, and guidance methods are often required to increase the quality of generations, albeit at the expense of diversity (Dhariwal & Nichol, 2021; Ho & Salimans, 2022). Classifier guidance (Dhariwal & Nichol, 2021) introduced this quality-boosting concept by utilizing the gradient of a classifier trained on noisy images increase the flss--lieelhood of generated samples. Later, classifier-free guidance (CFG) (Ho & Salimans, 2022) was proposed, allowing diffusion models to simulate the same behavior as classifier guidance without using an explicit classifier. Since then, CFG has been applied other conditional generation tasks, such as text-to- -o--teg-imgeee synthesis (Nichol et al., 2022) and text-to-3D generation (Poole al., 2023).
In addition to CFG's trading diversity for quality, it has the following two practical limitations. First, it requires a dedicated, pre-defined training process on an auxiliary task in order to learn the unconditional score function. This typically involves training separate unconditional model or, more commonly, randomly dropping the conditioning vector and replacing with a null vector during training. This approach reduces training efficiency, as the model now needs to be trained on two different tasks. Moreover, replacing the condition may not be straightforward when multiple conditioning signals such text, images, and audio- used simultancously when the null vector (often the zero vector) carries specific meaning. We demonstrate that this dedicated auxiliary training process is unnecessary. A second limitation is that there has been clear way extend

---

Published as a conference paper at ICLR 2025
the benefits of classifier-fres ffreee guidance beyond conditional models to unconditional generation. We introduce a method that closes this gap.
We revisit the principles behind classifier-free free guidance and show both thcoretically and empirically that similar quality -boasiig behavior can be achieved without the need for additional auxiliary training of an unconditional model. The main idea is that by using a conditioning vector independent of the input data, the conditional score function becomes equivalent to the unconditional score. This equivalen
insight leads us propose independent condition guidance (ICG), a method that replicates the behavior of CFG at inference time without requiring auxiliary training of an unconditional model, i.., without needing explicit access to the unconditional score function. In Section 6.1, we show that the auxiliary training of the unconditional model in CFG can be detrimental to training efficiency, and similar or better performance can be achieved by training only a purely conditional model and using ICG instead.
Inspired by the above, we also introduce novel technique to extend classifier-free guidance to more general setting that includes unconditional generation. We argue that by using a perturbed version of the time-step embedding in diffusion models, one can create a guidance signal similar to CFG to improve the quality of generations. This method, which we call time-step guidance (TSG), aims to improve the accuracy of denoising at each sampling step by leveraging the time-step information learned by the diffusion model to steer sampling trajectories toward better noise-removal paths.
ICG and TSG are easy to implement, do not require additional fine-tuning of the underlying diffusion models, and have the same sampling cost CFG. Through extensive experiments, we empirically verify that: 1) ICG offers performance similar to CFG and can be readily applied to models that are not trained with the CFG objective in mind, such as EDM (Karras al., 2022); and 2) TSG improves output quality in a manner similar to CFG for both conditional and unconditional generation.
The core contributions of work as follows: (i) We revisit the principles of classifier-free guidance and offer an efficient, theoretically motivated method to employ CFG without requiring any auxiliary training of an unconditional model, greatly simplifying the training process of conditional diffusion models and improving training efficiency relative to the standard approach. (ii) We offer an extension of CFG that is generally applicable to all diffusion models, whether conditional or uncondi- tional. (iii We demonstrate empirically that our guidance techniques achieve the quality-boosting benefits of CFG across various setups and network architectures.
2 RELATED WORK
Score-based diffusion models (Song & Ermon, 2019; Song et al., 2021b; Sohl-Dickstein et al., 2015; Ho et al., 2020) learn the data distribution by reversing a forward diffusion process that progressively transforms the data into Gaussian noise. These models have quickly surpassed the fidelity and diversity of previous generative modeling methods (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), achieving state-of-the-ee results in various domains, including unconditional image generation (Dhariwal & Nichol, 2021; Karras et al., 2022), text-to-image generation (Ramesh et al., 2022; Saharia et al., 2022b; Balaji al., 2022; Rombach et al., 2022; Podell et al., 2023; Yu et al., 2022), video generation (Blattmann et al.. 2023b;a; Gupta et al., 2023). image-to-ingg to-iiage translation (Saharia al., 2022a; Liu et al., 2023), motion synthesis (Tevet et al., 2023; Tseng al., 2023), and audio generation (Chen et al., 2021; Kong et al., 2021; Huang et al., 2023).
Since the development of the DDPM model (Ho at. 2020), many advancements have been proposed including improved network architectures (Hoogeboom et al., 2023; Karras et al., 2023; Peebles & Xie. 2022; Dhariwal & Nichol, 2021), sampling algorithms (Song et al., 2021a; Karras al., 2022; Liu et al., 2022; Lu et al., 2022a; Salimans & Ho, 2022), and training methods (Nichol & Dhariwal, 2021; Karras et al., 2022; Song et al., 2021b; Salimans Ho, 2022; Rombach al., 2022). Despite these recent advances, diffusion guidance, including classifier and classifier- free guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), still plays an essential role in improving the quality of generations well as increasing the alignment between the condition and the output image (Nichol et al., 2022).
SAG (Hong al., 2022) and PAG (Ahn al., 2024) have recently been proposed increase the quality of bNsed diffusion models by modifying the predictions of the self-attention layers. Our method is complementary to these approaches, one can combine ICG updates with the update
2

---

Published as a conference paper at ICLR 2025
signal from the perturbed attention modules (Hong et al., 2022). In addition, make assumptions about the network architecture.
Another line of work includes guiding the generation of the diffusion model with diffcrentiable loss function or an off-the-shelf classifier (Song et al., 2023; Chung et al., 2022; Yu ct al., 2023; Bansal primarily focused solving inverse problems,
et al., 2023; He et al., 2023). These methods typically with unconditional models, while we are instead concerned with achieving the benefits of CFG in conditional models without any additional training requirements. With TSG, we also generalize our approach to extend CFG-like benefits to unconditional models.
Perturbing the condition vector is employed in CADS (Sadat et al., 2024) increase the diversity of generations. CADS differs from ICG in focusing on the conditional branch to improve diversity, while ICG concerned with the unconditional branch to simulate CFG. Since CADS is designed to enhance the diversity of CFG, it can be used alongside ICG to improve the diversity of output at high guidance scales (see Appendix C).
BACKGROUND
This section provides overview of diffusion models. Let æ ~> (aam be data point, t E [0, 1] be the time step, and 2, = + (t)E be the forward process of the diffusion model that adds noise to the data. Here (t) is the noise schedule and determines how much information is destroyed at each time step t, with (0) = 0 and (1) = max. Karras et al. (2022) showed that this forward process corresponds to the ordinary differential equation (ODE)
dz = (1)) Vz, logp;(2zdd
or, equivalently, stochastic differential equation (SDH) given by
dz = (t)oo((t) v. log pr(z)) dt 3()) V log + (23(t)o((t) duy.
(2)
Here dui; is the standard Wiener process, and pe(zt) ) is the time-dependent distribution of noisy samples, with Pu Paata and P1 (0, of I) Assuming that we have access to the time- dependent score function Vz, log Pt (=+),
sample from the data distribution Pdata by solving the ODE or SDE backward time (from = 1 to 1 = 0). The unknown score function V s: log Pr (2,) (z,, corresponding noisy samples 2, The framework allows for conditional generation by training denoiser Do Zt, f, y) that accepts additional input signals y. such as class labels or text prompts.
estimated via neural denoiser (= t) that is trained to predict the clean samples æ from the
Training objective Given noisy sample at time step the denoiser Dø(zi, t, y) with parameters can be trained with the standard MSE loss (also called denoising score matching loss)
arg min E, [Do 4,y x: x=]²
The denoiser approximates the time dependent conditional score function Vz, log (r via
(z:p((z:)l)) Do(zi, t,
21
(1)2
Classifier-free guidance (CFG) CFG is an inference method for improving the quality of generated outputs by mixing the predictions of a conditional and an unconditional model (Ho Salimans, 2022). Specifically, given a null condition ynull Ø corresponding the unconditional case, CFG modifies the output of the denoiser at each sampling step according to
(5) where WCRG = corresponds to the non-guided case. The unconditional model Do(=,, f, Ynal) is trained by randomly assigning the null condition Youlll Ø to the input of the denoiser with probability p. where we normally have p E [0.1, 0.2]. One can also train a separate denoiser to estimate the unconditional score in Equation (5) (Karras et al., 2023). Similar to the truncation method in GANs (Brock et al., 2019), CFG increases the quality of individual images at the expense of less diversity (Murphy, 2023).
Do t, y) = De( f, Bmon) + WCFG De t, y) De t, tomm)),
3

---

Published as a conference paper at ICLR 2025
4 REVISITING CLASSIFIER-FREE GUIDANCE
We now show how a conditional model can be used to simulate the behavior of classifier-free guidance, without needing any auxiliary training learn the unconditional score function. The analysis in this section is inspired by Sadat et al. (2024).
First, note that at each time step t, classifier free guidance implicitly uses the conditional score Zz, log pa ( y) and the unconditional score 2z, log pr(z)) to guide the sampling process. From Bayes' theorem, we can write pt (Zyl y) 2e se (*t)) which gives us
p((3)
2x log p. (1 y) Zz log
V 2, log (1 zz)..
(6)
Next, assume that we replace the condition y with random vector (()) that is independent of the input z₁. In this case, the "classifier" pt (ū) 24) is equal to ( which gives us
Zx, log (z 9) R V (x Vx, log ( V logp:(zz).
This suggests that we can estimate the unconditional score purely based on the conditional model by replacing the condition y with an independent vector ÿ. We argue as result that there is no need to train separate model Dø(z,, t. youn) apply classifier-fres guidance, as we can use the conditional model itself to predict the score of the unconditional distribution as long as we pick an input condition that is independent of z,. We call this method independent condition guidance (ICG) for the rest of the paper.
In reality, the scores in Equation (7) are approximated by finite-capacity parametric models on dependent data drawn from Pt (z. y) In this case, Dø(z,, t, ÿ) will not necessarily exactly equal Do(z, L, Yoan) since independent ý and 2, are technically out of distribution relative to what the model was trained on. We have not found this to be an issue in practice, and further, the expected error in the unconditional score estimate can be made arbitrarily small through both the capacity of Do and the choice of (g)). We provide analysis in the appendix.
Implementation details In practice, we experiment with two options for the independent condition. First, ý can be drawn from a Gaussian distribution with a suitable standard deviation so that ý matches the scale of the actual conditioning vector y. Second, random condition from the conditioning space, such as a random class label or random clip tokens, can be chosen as the independent ý. We show in Section 7 that both methods perform similarly. However, there may be slight preference for the random condition over Gaussian noise, as it stays closer to the conditioning distribution that the diffusion model was trained on.
5 TIME-STEP GUIDANCE
Inspired by ICG, we next offer an extension of classifier-free guidance that can be used with any model, including unconditional networks. We begin our analysis with class-conditional models and subsequently extend it to a more general setting.
In the class-conditional case, the embedding vector of the class typically added to the embedding vector of the time step to compute the input condition of the diffusion network. Hence, in practice, CFG essentially uses the outputs of the diffusion network for two different input embeddings and takes their difference as the update direction. Thus, might directly utilize the time-step embedding of each diffusion model as a means to define a similar guidance signal. This leads to novel method that we refer to time-step guidance (TSG), which, like CFG, increases the quality of generations but, unlike CFG, is applicable even to unconditional models.
In this method, we compute the model outputs for the clean time-step embedding and perturbed embedding and use their difference to guide the sampling. More specifically, at each time step t, we update the output via
Do () = D=( + wTsG (Do () Do 1)),
(8)
where is the perturbed version of t. The intuition behind TSG is that at each time step t, altering the time-step embedding of the network leads denoised outputs with either insufficient or excessive
Howeeer, at the highest noise scales, pe(ze, y) 23 ((()))
4

---

Published as a conference paper at ICLR 2025
noise removal (see Figure 10 in the appendix). Consequently, these outputs can be exploited to prevent the network from going toward undesirable predictions, thus increasing the accuracy of the score predictions at each time step. As show below, TSG is related to stochastic Langevin dynamies in terms of the first-order approximation, and hence, is expected to improve generation quality.
Connection to Langevin dynamics Let i = t + At. where At is small perturbation. Using @Dalz..0) Al. Hence, Da 1) Do(z.,
Taylor expansion, we get Da(z;, Dz + @Do(z..)) At. Based on Equation (4), the score function is equal to
8t
- wrss)) 8t
VA, (o. = Vx, (z.) + 1 - WTsG @Do () At.
o(1)
ŌL
Now, if we follow the Fuler sampling step for solving Equation (1), i.e. we define the update rule as #t-1 zt + The vx log (z()),, then the modified sampling step after time-step guidance will be equal to
1 WTSG 0Do(z., t) At.
-t-1 2, 7. Z., (z,) + 17: ot)
(10)
Assuming that AL is Gaussian random variable with zero mean, the update rule resembles Langevin dynamics step, where the noise strength is determined based on the network behavior as represented by @Du(z.t) As Langevin dynamics is known to increase the quality of sampling from 2t a given distribution by compensating for the errors happening at each sampling step, we argue that TSG also behaves similarly in terms of first-order approximation.
Implementation details In practice, we implement TSG by perturbing the time-step embedding with zero-mean Gaussian noise according to femb temb st" where n NN0, I) and st" determines the noise scale at each time step t. We choose s and a such that the scale of the noise portion becomes comparable the scale of the time-step embedding lemb- Empirically, we also find that it is sometimes beneficial to apply the perturbed embeddings only to a portion of layers in the diffusion network, e.g., using femb for the first 10 layers and fmmb for the rest of layers. We provide ablations these hyperparameters in Section 7.
6 EXPERIMENTS
In this section, rigorously evaluate ICG and demonstrate its ability simulate the behavior of CFG across several conditional models. Additionally, we show that TSG improves the quality of both conditional and unconditional generations compared to the non-guided sampling baseline.
Setup All experiments are conducted via pre-trained checkpoints provided by official implementa- tions. We use the recommended sampler that comes with each model, such as the EDM sampler for EDM networks (Karras et al., 2022), DPM++ (Lu et al., 2022b) for Stable Diffusion (Rombach et al., 2022), and DDPM (Ho et al., 2020) for DīT-XL/2 (Peebles & Xie, 2022).
Evaluation We use Fréchet Inception Distance (FID) (Heusel et al., 2017) the main metric to measure both quality and diversity due to its alignment with human judgment. As FID known to be sensitive to small implementation details, we ensure that models under comparison follow the same evaluation setup. For completeness, we also report precision (Kynkäääniemi et al., 2019) as standalone quality metric and recall (Kynkäänniemi et al., 2019) diversity metric whenever possible. FDDINOV2 (Stein et al., 2024) is also reported for the EDM2 model (Karras et al., 2023).
6.1 COMPARISON BETWEEN ICG AND CFG
Qualitative results The qualitative comparisons between ICG and CFG are given in Figure 1 for Stable Diffusion (Rombach et al., 2022) and DiT-XL/2 (Peebles & Xie, 2022) models. Figure also shows a comparison between the EDM2 model (Karras et al., 2023) guided with a separate unconditional module vs ICG. We observe that both ICG and CFG improve image quality, and the outputs of ICG and CFG are almost identical. This empirical evidence agrees with our theoretical justification provided in Section 4.
5

---

Published as a conference paper at ICLR 2025
w/o guidance
CFG
ICG (Ours)
Stable Diffusion
w/o guidance
CFG
ICG (Ours)
(b) DiT-XL/2
Figure 1: Comparison between CFG and ICG for (a) Stable Diffusion (Rombach al., 2022) and (b) DiT-XL/2 (Peebles & Xie, 2022). Both CFG and ICG significantly improve the image quality of the baseline. Also note the similarity between the outputs of CFG and ICG, confirming our theoretical analysis in Section 4.
w/o guidance
CFG
ICG (Ours)
Figure 2: Comparison between the EDM2 model (Karras et al., 2023) guided with another uncondi- tional module and ICG. observe that using ICG leads to similar generations as CFG, and both methods significantly improve output quality compared to sampling without guidance.
Quantitative results We now show that ICG and CFG both result in similar performance metrics across several conditional models. As shown in Table compared to CFG, ICG achieves better or similar performance across all metrics.? In Table 2, we also compare the effect of ICG EDM (Karras al., 2022) and EDM2 (Karras al., 2023) models that were not trained with the CFG objective. The table shows that ICG performs similarly to guiding the generations with separate unconditional module.
Effect of removing the CFG objective from training In this experiment, we demonstrate that the training component allocated to the CFG objective (i.e., label dropping) is unnecessary. and better results are obtained by training a purely conditional model and guiding the generations inference with ICG. Using a DíT model for glass-conditional ImageNet generation, Figure 3 shows that the purely conditional model consistently outperforms standard training with label dropping (p 0.1) across all checkpoints. Consequently, training resources can be reallocated to the conditional part, leading to either faster convergence (by approximately 30%) or a better model (with around a 20% reduction in FID) with the same number of training iterations.
FFor MDM (Tevet et al., 2023), recall is not available, and R-precision is reported similar the paper.

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison between CFG and ICG. ICG able to achieve similar metrics to standard CFG by extracting the unconditional score from the conditional model itself.
Model
Architecture
Guidance
FID J
Precision
Recall f
CFG
20.13
0.69
0.54
Stable Diffusion (Rombach et al., 2022)
UNet
ICG (Ours)
20.05
0.69
0.53
CFG
5,56
0.81
0.66
DiT-XI./2 (Pechles & Xie, 2022)
Transformer
ICG (Ours)
5.50
0.83
0.65
CFG
14.61
0.93
0.02
Pose-to-Image (Sadat et al., 2024)
UNet
ICG (Ours)
13.46
0.94
0.03
CFG
0.65
0.73
MDM (Tevet et al., 2023)
Transformer
ICG (Ours)
0.47
0.71
Table 2: Quantitative comparison between CFG and ICG for EDM networks. Although these models are not trained with the CFG objective, guiding their generations using a separate unconditional module results in similar outcomes to using ICG.
Model
Dataset
Guidance
FID
FDDINOv2
CFG
3.36
79.94
EDM2-XS (Karras al., 2023)
Imagenet
ICG (Ours)
3.35
79.54
CFG
1.87
EDM (Karras al., 2022)
CIFAR-10
ICG (Ours)
1.87
FD
IS
Precision
Recall
18
(Dun)
DE
10
13
10
120
501
Training Iter. (x10")
Training Iter. (x10')
Training Iter. (x100*)
Training Iter. (x100))
Figure 3: Comparison of CFG and ICG during training of DiT model on ImageNet. Compared to standard CFG with label dropping, using ICG with purely conditional model achieves better FID across all checkpoints. This indicates that the iterations spent on the CFG objective could be better allocated to training the conditional score, ultimately leading to better model.
Varying the Guidance Scale Next, we demonstrate that by varying the guidance scale of ICG, we can increase the quality of outputs in a manner similar to standard CFG. As shown in Figure 4, increasing the guidance scale improves precision but reduces recall. The FID plots also form U-shaped curve, consistent with what we expect from standard CFG.
Results for ControlNet We also show that ICG can be used for improving the quality of image- conditioned models as well. We use ControlNet (Zhang & Agrawala, 2023) as an example in this section since it is not trained with the CFG objective on the image condition input. That is, it only applies CFG to the text component of the condition. Our results are given in Figure 5. We see that without any text prompt, ICG significantly improves the quality of generations over the base sampling.
6.2 EFFECTIVENESS OF TIME-STED GUIDANCE
Lastly, we show the effectiveness of time-step guidance in improving generation quality without relying on any information about the conditioning signal. The qualitative results are given Figure 6. We can see that TSG increases the image quality of both conditional and unconditional sampling. Table also presents the quantitative evaluation of TSG for both conditional and unconditional generation. Similar to CFG, using TSG significantly improves FID by trading diversity with quality. Finally, Figure shows how TSG behaves as we increase the guidance scale. We observe that similar to CFG, TSG also has U-shaped plot for the FID the guidance scale increases.
7

---

Published as a conference paper at ICLR 2025
FID
Precision
Recall
127
Stade Dilinios
00
0.50
Figure 4: Behavior of ICG as the guidance scale increases. Similar to CFG, ICG trades diversity
(lower recall) for quality (higher precision) at higher guidance scales.
w/ guidance
ICG
Posc-to-image generation
w/o guidance
ICG
(b) Depth Depth-to-iman to-image generation
Figure Image-conditionel generation with ControlNet (without prompt). ICG significantly
increases the quality of generations by applying guidance to the image condition.
8

---

Published as a conference paper at ICLR 2025
w/o guidance
TSG (Ours)
w/o guidance
TSG (Ours)
(a) DiT-XL/2 unconditional
(b) Stable Diffusion unconditional
w/o guidance
TSG (Ours)
w/o guidance
TSG (Ours)
(c) DIT-XL/2 conditional
(d) Stable Diffusion conditional
Figure 6: Effectiveness of TSG to improve the quality of both unconditional and conditional genera- tion across two different models: DiT-XL/2 (Peebles & Xie, 2022) for cass-coonitional generation, and Stable Diffusion (Rombach et al., 2022) for text-conditional generation.
Table 3: Quantitative comparison between the baseline sampling of the diffusion models and sampling with TSG. TSG significantly boosts quality (lower FID) across various setups.
Model
Architecture
Type
Guidance
FID
Precision
Recall
69.38
0.42
0.49
Unconditional
TSG (Ours)
56.65
0.54
0.54
Stable Diffussoo (Rombach et al., 2022)
UNet
X
36.63
0.48
0.64
Text-conditional
TSG (Ours)
22.17
0.62
0.59
X
48.67
0.48
0.59
Unconditional
TSG (Ours)
29.03
0.69
0.55
DIT-XL/2 (Peebles Xic, 2022)
Transformer
X
15.49
0.64
0.74
Class-conditional
(Ours)
6.39
0.82
FID
Precision
Recall
870
0.7
Figure 7: Behavior of TSG as the guidance scale increases for DiT-XL/2, Similar CFG, TSG also significantly improves FID by trading diversity (recall) with quality (precision).
Combining TSG and ICG We also demonstrate that ICG and TSG can be complementary to each other when combined at the proper scale. The quantitative results of this experiment are presented in Table with visual example given in Figure 8. The table indicates that the combination of ICG and TSG outperforms each method in isolation in terms of FID, and all guided sampling algo- rithms significantly outperform the non-guided baseline.
Table 4: Compatibility of ICG and TSG
ICG
TSG
FID
Precision
Recall
15.49
0.64
0.74
x
6.47
0.77
0.69
x
9.55
0.70
0.71
5.76
0.82
0.65

---

Published as a conference paper at ICLR 2025
w/o guidance
ICG
ICG + TSG
Figure 8: Visual example of combining ICG and TSG. The combination yields better visual genera- tions compared to the baseline and using ICG alone.
Table 6: Ablation study examining various design elements in TSG.
(a) Influence of the noise scale
(b) Influence of a
(c) Maximum layer index
FID 1
Precision
Recall
FID J.
Precision 1
Recall 1
Index
FID 1.
Precision 1
Recall 1
1.0
10,23
0.69
0.35
0.75
7.22
0.82
0.62
5
7.84
0.75
0.69
2.0
6.85
0.80
0.69
1.0
6,39
0.84
0.65
10
6.85
0.79
0.65
2.5
7.94
0.80
1.25
6.47
0.78
0.66
7.65
0.82
0.65
ABLATION STUDIES
next present the ablation studies on the effect of random conditioning ICG and the hyperparam- eters in TSG. All ablations are conducted using the DiT-XL/2 model (Peebles & Xie, 2022).
The choice of random condition in ICG We first show that both Gaussian noise and random conditions can be used for estimating the unconditional part in ICG, The quantitative results are given in Table 5. The table shows that both methods are viable options for simulating classifier-free guidance without training.
Table 5: Ablation on the choice of inde- pendent condition ICG.
ICG method
FID 1
Precision 1
Recall 1
Gaussian noise
5.50
0.83
0.65
Random condition
5.55
0.83
0.65
Impact of hyperparameters in time-step guidance This ablation study explores the effect of hyperparameters TSG. The results are presented in Table 6. We observe that as we introduce more perturbation into the time-step embedding of the model, in the form of higher noise scale (Table 6a), lower power a (Table 6b), or higher layer index (Table 6c), precision improves while recall decreases. This suggests that the amount of perturbation should be balanced for a good trade-off between diversity and quality. We also empirically observed that adding too much noise to the time-step embedding hurts image quality.
DISCUSSION AND CONCLUSION
In this paper, we revisited the core aspects classifier-free guidance and showed that by replacing the conditional vector in a trained conditional diffusion model with an independent condition, we can efficiently estimate the score of the unconditional distribution. We then introduced independent condition guidance (ICG), novel method that simulates the same behavior need to learn an unconditional model during training. Inspired by this, we also proposed time-step guidance (TSG) and demonstrated that the time-step information learned by the diffusion model can be leveraged to enhance the quality of generations, even for unconditional models. Our experiments indicate that ICG performs similarly standard CFG and alleviates the need consider the CFG objective during training. Thus, ICG streamlines the training of conditional models and improves training efficiency. Additionally, we verified that TSG also improves generation quality manner similar to CFG, without relying any conditional information. As with CFG, challenges remain accelerating the proposed methods narrow the gap between the cost of guided and unguided sampling (i.e., eliminating the need to query the diffusion model twice per sampling step); we view this topic as promising avenue for further research.
CFG without the
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
As generative modeling advances, the creation and spread of fabricated inaccurate data become easier. Thus, while improvements in Al-generated content can boost productivity and creativity, it is crucial consider the associated risks and ethical implications. For more detailed discussion on the ethics and creativity in computer vision, refer readers to Rostamzadeh et al. (2021).
REPRODUCIBILITY STATEMENT
This work is based on the official implementations of the pretrained models referenced in the main text. The exact algorithms for ICG and TSG are provided Algorithms and 2, with corresponding pseudocode shown in Figures 11 and 12. Additional implementation details, including the specific hyperparameters used to generate the results this paper, are discussed in Appendix G.
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, Seonhwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-atteetion guidance. CoRR, abs/2403. 17377, 2024. doi: 1048550ARXXV:2403.17777 URL
Yogesh Balaji, Seungjun Nah, Xun Huang. Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text-to- image diffusion models with ensemble of expert denoisers. CoRR, abs/2211.01322 2022. doi:
10.488550/rXivv2211 01324. URL
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, 0bss2311..11221 2023a. doi: 1.5550ARRXIV231111151111 URL https:/ddoiorr/111455555aa 2311.15127.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563 -25755, 2023b.
Andrew Brock. Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://ppenreiiew.nee/ (ooum?id-BBssjj09Fm xsqj09Fm.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. 2021. URL http:///pennevieeeeetf/mmmm/m hidd
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffussonn models beat gans on image synthe- sis. In Marc" Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- nual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 8780-8794, 2021. URL htpp://pooeddng..nurps.cccoooppp 99nd22alccc9taaaas55777000000 1f50c0 df5cfa- A Abstract.html.
1/hash/

---

Published as a conference paper at ICLR 2025
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 2023.
Yutong He, Naoki Murata, Chich- Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao. Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. arXiv preprint arXiv:2311.16422 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 2017, Long Beach, CA, USA. pp. 6626-6637, 2017. URL (16644077e00fff66871166000000009
Abstract.html.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abwp:/00./222225//0//00 2022. doi: 10.48550//rxiv.2207.. 12598. URL
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: Attp://0607001100000094444/-bba0
Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. CoRR, abs/2210.00939, 2022. doi: 10,48550/arXivv 2210.00939. URL https::/doi..rr// 1ttp://0.44884 085555/02xxx2222000009333
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. CoRR, abs/2301.1 1099, 2023. doi: 10.48500a2iiv2201.11000 URL
https://doi.org/ wtt//1..485500/// 40v55//axivv.231..111
Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Havnø Frank, Jesse H. Engel, Quoe V. Le, William Chan, and Wei Han. Noise2music: Text-conditioned music generation with diffusion models. CoRR, abs/2302.03917, 2023. doi: 10448500arXx..222200397 URL https://doi.orr wrgg//..4/55000 arXiv.2302.03917.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. 2022. URL
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models, 2023.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Im- proved precision and recall metric for assessing generative models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ach--Buc, Emily B. Fox, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 3929-3938, 2019. URL 004445510eed09088227000333333331 0at
/9hash/////
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference. Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part V. volume 8693 of Lecture Notes in Computer Science, pp. 755-755, Springer, 2014. doi: 10.1007/ 978-3-319. 10602- 48. URL
12

---

Published as a conference paper at ICLR 2025
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A. Theodorou, Weili Nie, and Anima Anandkumar. sbsb: Image-to-imagee schrödinger bridge. CORR, /t/ss///..000722,, 2023. doi: 10.48550%arX1w.2202.8888 URL
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httpss//opennevii..ee///onmmnn PIKWVd2yBKY.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver: A fast ODE solver for diffusion probabilistie model sampling in around 10 steps. In NeurlPS, 2022a. URL http://papers.nisscccpppp httpppppr..ssssppppe/22/aaaa 260a 44cce22a9aaaafalccceffeeeeeee Abstrract-Cooffeeeeee...
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. CoRR, abs/2211.00000 2022b. doi: 10.485500/rXiw.2211 2.101095. URL htttp:://doi..or/110. 48550//rxiv.221 http::/doo..o0110044555550022222
1.00000
Kevin Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL
http://probml..uig http://probml.iiiubbbbboo....... github.oo/book...
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8162-81 8171. PMLR, 2021. URL http://procedingg.mlrrrrrer v1 1339/icchol2la.htm
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 167846666600 16804. PMLR, 2022. URL https://proceedings.. wwt.ggssmm..psssss..ee.. v162/nichol22a.htm!!
William Peebles and Saining Xie. Scalable diffusion models with transformers. CoRR, abs/2212.099748, 2022. doi: 10448550/axX..2212.09745 URL http:://doo.org/114455555500 2212.09748.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01955 001952, 2023. //doi.org/ www.r//////00000 20.488500ar2ii.23077.000 01992..
doi: 10448550ARXXI22300000999 URL https:
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:///peemeeveewee//dfiittt FjNys5c7Vyy.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.06111 2022. doi: 10.48550/ arXiv.2204.06125 URL
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. doi: 10. 10.11999CPR2228822222001444 URL https://doi.org/11... CVVR55268.20222.000
Negar Rostamzadeh, Emily Denton, and Linda Petrini. Ethics and creativity in computer vision. CoRR, abs/21 2021. URL https://anxiv.or//ab//111
httpg///aaaa//221111111111 12.0311..
13

---

Published as a conference paper at ICLR 2025
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through oudition-annealle sampling. The Tweifth International Conference Learning Representations, 2024. URL https///peeneviee.n////omm id=zMoNrajk2X.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-imat diffusion models. In Munkhtsetseg Nandigjav, Niloy Mitra, and Aaron Hertzmann (eds.), SIGGRAPH '22: Special Interest Group Computer Graphics and Inteructive Techniques Conference, Vancouver, BC, Canada, August 7- 11, 2022, pp. 15:1-15:10. ACM, 2022a. doi: 10. 1111144522223335305555 URL http:://doi.org//// 1145/3528233.353073
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-ine to-image diffusion models with deep language understanding. 2022b. URL http:/ppapers.nipp httpppppprrniiissc.pppeere files/apper/2222/aassl
W9SSscadacc00122000f55eefeeodded --bttttctbttrattt-o -ostaaaat--ooomeeeeeeeee Conference.htm
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Teuth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https//oppaneeieewwettrrommmnmii diiiiooouumddd-Tdilllpppo..o..
Jascha Sohl-Dickstein. Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. 37:2256-2265, 2015. URL http://proceedings. altepress/337/ssoh-dickssss html.
Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https///peemeeiiw.eeeffoomm Sid=
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-plan controllable generation. In International Conference on Machine Learning, pp. 32483-324222 -2499.9 PMLR, 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11895-11907, 2019. URL. https:/pproeedings.eerrij www.nneeuuuupppppeeee pearips.cc/paper/2019
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria. May 3-7, 2021. OpenReview.net, 2021b. URL http::///eecrceiiooooooooommmn/p
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Haim Bermano. Human motion diffuusin model. 2023. URL http:///ooeeeeevvv..eee/////d/ss /SS
Jonathan Tseng, Rodrigo Castellon, and C. Karen Liu. EDGE: editable dance generation from music. In IEEF/CVE Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 448-458, IEEE, 2023. doi: 10. 10.1109CCVR32799.20230000 URL https://doi.org/ 0.. 1091000 STT.R52729.202300000 2023,00051.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Trans. Mach. Learn. Res., 2022, 2022. URL https//opeeneviee.neel Fonun:id-AFDeYYKiNND
14

---

Published as a conference paper at ICLR 2025
Jiwen Yu, Yinhuai Wang. Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 23174-23184 23484,220 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. CoRR, abs/2302.05543. 2023. doi: (0.48550/RRXI:223000 AR311.232222225555 URL https::/doi.or//10.44553 arXiw.2302.05543.
15

---

Published as a conference paper at ICLR 2025
A ANALYSIS OF THE ERROR IN ESTIMATING THE UNCONDITIONAL SCORE
In this section, we provide another perspective on why training the conditional score is sufficient for computing the unconditional score. For case of exposition. assume that the model's objective is to directly learn the conditional score, s0 y) V log p(z y), from paired data jointly drawn from p(z, y). This can be achieved by directly using denoising score matching (Song al., 2021b) or by training a denoiser (z(z, y) and converting its outputs to the corresponding score function via Equation (4). The question then boils down whether the unconditional score, V= logp(z), can be recovered from the conditional score.
Lemma A.I. The unconditional score, V= l(z), is equal the conditional expectation of the conditional score, V pozy).
Proof. By direct calculation, we have that
V p(z)
V logg(p)))==
(11)
p(z)
V ʃ p(y)p(=lu)d
(12)
p(z)
/
ply)
V ydd
(13)
p(z)
/
p(y)p(z(y) v)
p(=)p(z1 y) V= p(z y)dy
(14)
f p(ul=) V. log p y)dy.
(15)
The last term is the conditional expectation of the conditional score, proving the result.
This shows that under sufficient data and optimization, the unconditional score at each time step is theoretically available from the conditional score through (conditional) marginalization. It is also therefore the case that a condition drawn from p(y|z) will produce an unbiased estimate of the unconditional score.
We propose drawing conditions randomly from a distribution q(y). which not necessarily equal to p(y]z). Below we characterize the approximation error incurred by this choice. Theorem A.2. When the expectation is taken over the distribution q(y), we have
Eg(y) V po (=|ly)] V logp(z) PxL (g(z) (((y)))).
That is, the approximation error is bounded by the gradient of the KL divergence between q(y) and p)y|z).
Proof. We can write
V log p(2y)) V logp(z,y) V logp(u) V log p(z, v),
which in turn can be written as
V log, y) Vz logp(z) + V= (ogl())z).
Taking the expectation over q(y), have
[V log y)] = v p(=) + Ey(w) V logp(y|2)
V= p(z) + Error.
Now consider the KL divergence between (()) and p(y)z)):
(xl((()))p(()z) / ((v) log q(v) - log p(v)=)).
Taking the gradient with respect to z, we have
V V*Praa((v)lp(U = / ( log logg(y)z)dy = -EgG) V log
which is the negative of the term above, proving the result.
16

---

Published as a conference paper at ICLR 2025
B ANALYSIS OF THE ICG ESTIMATOR FOR MIXTURES OF GAUSSIANS
We now analyze the ICG estimator using a mixture of Gaussian distributions so that the ground truth scores are analytically available. Assume we have the data distribution given by Pamm(a)) (z5 fo. I) + 025.V f1. I). This gives us Pdau(22 y = 0) = po(2:) = N(x; Ma. I) and Pdata (arly y 1) = (x)) Nx 11, Given the forward process 2₁ x 1 we have po(2))
N Mo, (1 + o)) and P1 (z,) (z; Ml. (1 + of) Let s(2,, y) be the conditional score
function =z, log Pa (e v) In this case, the conditional score functions are given by
44 x
and s(z,,1=1> 141 x
8(2,, = 0)
(16)
1 + of
o
The unconditional score function is equal to
s(z,) = 0.5 v. po 21) + 0.5 V. P1
(17)
0.5po 21) + 0.5p1 (z1)
V NN: Mo, (1 + 1) + V
(18)
po (2(z)) + pi (2t)
(z)
No x + Po(z)) + Pi (1() o
Pi (1()))
11 x
(19)
(o(z)) + (z() 1 1+ o?
The CFG update direction is therefore given by
s y = 0) s(z))
140 x
140
p1 (2,)
(20)
1 1+ o?
p=(2+)
+ P1 (z+)
1 0f
po(2)) + p1(
1 oî
P1 (1()
140
(1(z))
P1 Ø8
(21)
po(z,) + p1 (z:)
1 of
(z.),) + pi (z+) 1 of
pI (z,)
N4 14
(22)
po(z.) + pi (z() 1+ of
For ICG, assume that the random condition ý is sampled from (()). Therefore, the update rule given by ICG is equal to
Ú 0,
5 y = 0) s(z., 9)
(23)
Ma-n T-
4 = 1.
This means that on expectation, the update direction of ICG is equal to
[ = 0) s( 0)] = { = 1) 14D 11
(24)
1 + +
This example shows that in theory, we can set q(ū = 1) (s pp ( and the ICG estimator becomes an unbiased estimate of the CFG update direction. In practice and for real-world models, we noted that it is sufficient to set g()) to either a uniform distribution over the space of the conditions, or equal to a Gaussian distribution with suitable standard deviation.
Pr (z.)
C COMPATIBILITY OF ICG WITH CADS
We show that ICG is compatible with CADS (Sadat et al., 2024), and CADS can be used on top of ICG increase the diversity of generations. An example of applying CADS to ICG is shown in Figure 9, and the quantitative results are given in Table 7 for the DiT-XL/2 model. As ICG behaves similarly to the standard CFG, applying CADS increases diversity with minimal drop in quality.
Table 7: Effectiveness of CADS on ICG.
Guidance
FID
Precision
Recall
ICG +CADS
20.56
0.89 0.78
0.32
8.83
0.61
D INTUITION BEHIND TSG
This section provides more intuition on time-step guidance. We demonstrate that if we perturb the time step with positive or negative constant (using i 8 or i 8) to guide the sampling, it results
17

---

Published as a conference paper at ICLR 2025
ICG
+CADS
ICG
+CADS
Figure 9: Similar to CFG, ICG is compatible with CADS, and CADS can be used to increase the diversity of ICG higher guidance scales. Samples are generated from the DiT-XL/2 model.
8
1
TSG
Figure 10: Intuition behind TSG: Using lower time steps for guidance causes excessive noise removal (soft outputs), while higher time steps cause insufficient noise removal (noisy images). TSG employs both directions to improve output quality.
Table 8: Comparison between TSG and other guidance methods. TSG achieves better quality compared to SAG and PAG while requiring no specific assumption about the underlying architecture of the diffusion model.
Guidance
FID
Precision
Recall
Unguided
69.38
0.42
0.49
SAG (Hong al., 2022)
59.34
0.48
0.53
PAG (Ahn et al., 2024)
62.85
0.42
0.51
TSG (ours)
56.65
0.54
0.54
in insufficient or excessive noise removal in final generations. As shown in Figure 10, using lower time steps causes the model to perform excessive noise removal (soft outputs), while using higher time steps forces the model to perform insufficient noise removal (noisy images). TSG uses both directions to prevent the outputs from moving toward these undesirable paths, thereby increasing the quality of the generations.
E COMPARISON BETWEEN TSG AND OTHER METHODS
In this section, we compare the performance of TSG with other guidance mechanisms that improve the generation quality of diffusion models without relying any condition. We use self-attention guidance (SAG) (Hong et al., 2022), and perturbed- -attrndtd-ttttiionn guidance (PAG) (Ahn et al., 2024) as two representative methods in this category. The results given in Table 8. We use Stable Diffusion 2.1 for this parameters and the default hyperparameters for all methods. We note that TSG is more effective improving the quality of generations, providing better FID compared SAG and PAG.
18

---

Published as a conference paper at ICLR 2025
Table 9: Comparison between unguided sampling and TSG using different number of sampling steps. TSG is able to achieve significantly better FID compared to both unguided baselines.
Guidance
# Steps
FID
Precision
Recall
Unguided
100
15.49
0.6382
0.7437
Unguided
200
12.94
0.6683
0.7387
TSG
100
6.39
0.8198
0.6489
F INCREASING THE NUMBER OF SAMPLING STEPS
Since TSG increases the number of sampling steps (similar to CFG), natural question is whether the same behavior can be achieved by simply increasing the number of sampling steps in the unguided sampling baseline. Our results in Table 9 indicate that this approach performs significantly worse than TSG, suggesting that, like CFG, TSG alters the sampling trajectories toward higher-quality generations. Note that TSG achieves a significantly better FID compared both unguided sampling baselines. This aligns with findings from the CFG literature, where guided sampling outperforms unguided baselines even with many sampling steps (e.g., 1000 steps) (Dhariwal & Nichol, 2021; Ho & Salimans, 2022).
G IMPLEMENTATION DETAILS
The sampling details for ICG and TSG are provided in Algorithms 1 and 2. Both algorithms are straightforward to implement and require minimal code changes compared to the base sampling. The pseudocode for implementing ICG and TSG is also included in Figures 11 and 12. Additionally, the hyperparameters used our experiments are listed in Tables and 11. The CADS experiment was conducted with linear schedule using T1 0.5, 72 0.9, and SCADS 0.15. Lastly, please note that we did not perform an exhaustive grid search on the parameters of TSG, and better configurations likely exist for each model.
For the TSG noise schedule, we experimented with a constant and a power schedule, as shown in Figure 12, and found that both work similarly. We recommend using the power schedule as offers more flexibility over the scale of the noise at each L. The constant schedule is technically a special case of the power schedule, where the exponent is zero. We also found it useful apply TSG only at intervals during the sampling, i.c., for t E [ min T m. where Tmin and Taax are hyperparameters. Also, when limiting TSG to only portion of layers the diffusion model, we used the first N layers of transformer-basee based architectures and the first N layers of the encoder and decoder in -Net-based architectures. We chose to scale the amount of noise based on the standard deviation of the time-step embedding (see Figures 11 and 12) for more fine-grained control over the scale.
We primarily use the ADM evaluation script (Dhariwal & Nichol, 2021) for computing FID, precision, and recall to ensure fair comparison across experiments. For class-conditional models, the FID is computed between 10,000 (for DiT-XL/2) or 50,000 (For EDM and EDM2) generated images and the full training dataset. For text-to-image models, we use the evaluation subset of MS COCO 2017 (Lin et al., 2014 4) as the ground truth for captions and images.
H MORE VISUAL RESULTS
This section presents additional visual results for our guidance methods. More results on ICG are provided in Figure 13, while additional results for TSG shown in Figures 14 and 15. Consistent with the main results of the paper, ICG produces similar outcomes to CFG, and TSG consistently enhances the quality compared the baseline. Figure 16 provides examples of the effectiveness of TSG based on Stable Diffusion XL (SDXL) (Podell et al., 2023). Finally, Figure 17 shows qualitative comparison between unguided sampling and sampling with TSG for several latent diffusion models from Rombach et al. (2022).
19

---

Published as a conference paper at ICLR 2025
Algorithm 1 Sampling with ICG
Require: whcG: ICG strength
Require: y: Input condition
1: Initial value: (0.
2: for = ......,
3:
Pick random independent of (Gaussian noise or from the conditioning space).
4: Compute the ICG guided output
(2c y) D 2(z, t, ÿ) (ica(D(2z,t,,) D(z,, 9)).
5: Perform one sampling step (e.g. one step of DDPM): DeveeseeDDcc...... zt,t).
=
6: end for
7: return z0
Algorithm Sampling with TSG
Require: WTsG: TSG strength
Require: (s:: TSG hyperparameters
Require: y: Input condition (optional)
1: Initial value: I)
2: for = .,,....1
3:
Perturb the time-step embedding temb to get femb:
femb
Lemb st,,, where (O.
4: Compute the TSG guided output at t: Disu t, y) = D(z femb. 8) + Wisa Wisa (D(z,, temb: y) D(ze, Bemb y)).
5: Perform one sampling step (e.g. one step of DDPM): t).
2,-1
6: end for
7: return 20
Table 10: Hyperparameters used for the ICG experiments.
Model
ICG mode
ICG scale
CFG scale
D/T-XL/2
Random class
1.4
1.5
Stable Diffusion
Random text
3.0
4.0
Pose-to-Image
Gaussian noise
3.0
4.0
MDM
Gaussian noise
2.5
2.5
EDM
Random class
1.05
1.1
EDM2
Random class
1.25
1.25
Table 11: Hyperparameters used for the TSG experiments.
Model
Mode
TSG function
TSG scale
TSG parameters
DIT-XL/2
Unconditional
seattaant_cchulleee schedule
5.0
T. NIN 200, T 800.
DIT-XL/2
Conditional
power schedule
2.5
T_MIX 0, TAM 1000, 1,
Stable Diffusion
Unconditional
constant_ schedule
3.0
T.NIN 100. MAX 900, 1.25
Stable Diffusion
Conditional
pover_achedule
4.0
T_MIN = 400, T. MAX = 1000, = 3.0. a: 0.25
20

---

Published as a conference paper at ICLR 2025
def net_raadomm_o classs)__sssss)
2
"Random class labels.
y_random torch. randint (0, NUM_CHASSES, (BATCH_811M)
return y_random
def get random text ():
"*"Random text tokens. "**
random_idx corch.randin (0, NUM_TOKENS, (BATCH_SIZE, MAX_LENGTH)
randon tokens text_encoder
encoder (random idx, attent ion mask=None) [0]
return random_tokens
def get__aasssan_noise nww.seeeeeediingggneeoeoeeeeegge (embsddings):
"*"Random embedding based on Gaussian noise.
14
noise_embedding = forch.randd_liiee (embeddings) * embeddings.. std ()
15
return woise_embeddingg
16
def Bacb_gaassiaa_mee image (image_cond)
"*"Random condition for image-conditional models. "*"
noise noise_embedding Norch.rando. 1 like (image_cond) SCALE
return poiss_embedding
Figure 11: Implementation details for ICG. The figure presents pseudocode for implementing the random class, random text, and Gaussian noise embedding for the unconditional component in ICG.
def get_ sereetaantschuuleeeeeeeeee (t_emb, std__calisn ing_scaling-rue)) """Applies TSG for portion of sampling (t in [T_MIN, T_MAX] if < T MIN or t MAX: return t_emb
noise_scale if staiicaingg:
noise_scale = t_emb.atd()
that_eni emb = emb + torch. like (t emb) * scale
10
return that_emb
11
def ftt__power_schheeee (t_emb, t, std_scaling-Truee """Applies TSG according the power schedule, """ if M_MIN t MAX;
return emm
noise_acale - $ ** (ALPHA)
18
if std_scaling:
19
noise_scale noise_scale t_emb. ()
that_emb t_emb torch.rande like (t_emb) noise_scale
return that_emb
Figure 12: Implementation details for TSG. We provide two scheduling techniques for perturbing the time-step embedding. We empirically found that both methods perform similarly.
21

---

Published as a conference paper at ICLR 2025
CFG
ICG (Ours)
(a) Stable Diffusion
CFG
ICG (Ours)
.1.3:10
a 38-8
(b) DİT-XL/2
Figure 13: More comparisons between ICG and CFG for (a) text-to-image generation with Stable Diffusion (Rombach et al., 2022) and (b) class-conditions generation with DiT-XL/2 (Peebles & Xie, 2022).
22

---

Published as a conference paper at ICLR 2025
w/o guidance
TSG (Ours)
(a) Stable Diffusion unconditional
w/o guidance
TSG (Ours)
(b) Stable Diffusion conditional
Figure 14: More comparisons on the effectiveness of TSG for improving the quality of both uncondi-
tional and conditional generation for Stable Diffusion (Rombach al., 2022).
23

---

Published as a conference paper at ICLR 2025
w/o guidance
TSG (Ours)
(a) DiT-XL/2 unconditional
w/o guidance
TSG (Ours)
ES
(b) DIT-XL/2 conditional
Figure 15: More comparisons on the effectiveness of TSG for improving the quality of both uncondi-
tional and conditional generation for DiT-XL/2 (Peebles & Xie, 2022).
24

---

Published as a conference paper at ICLR 2025
"A photorealistic image of a car"
wo guidance
TSG
"A golden retriever puppy"
w/o guidance
TSG
"A photorealistic image of a piano"
w/o guidance
TSG
Figure 16: Showcasing the effectiveness of TSG in improving the quality of generations compared to sampling without guidance based on SDXL (Podell et al., 2023).
25

---

Published as a conference paper at ICLR 2025
w/o guidance
with TSG
(a) FFHQ dataset
w/o guidance
with TSG
(b) Bedroom dataset
w/o guidance
with TSG
(c) Church dataset
Figure 17: Visual comparison between unguided sampling and sampling with TSG for several unconditional latent diffusion models from Rombach et al. (2022). We observe that TSG consistently improves the quality of all models compared to the baseline sampling.
26


Reviews:
[{"review_id": "Ki99hsga8y", "reviewer": "Reviewer_8vmw", "summary": "This paper proposes two methods, independent condition guidance (ICG) and time-step guidance (TSG), for conditional and unconditional sampling from a diffusion model, which are summarized below. \n\n**ICG**\n\nThey propose ICG in place of classifier-free guidance (CFG) such that instead of learning two separate models or having two different tasks, one conditional and one conditional, they learn a single conditional model with only a conditional task without making use of null tokens. They do based on the insight that:\n\n1. A conditional model can be turned into an unconditional model by noting that:\n    1. the marginal distribution $p(x_t)$ is equal to the expectation $E_{y \\sim p(y)}[p(x|y)]$ which can be estimated using sampling a single independent sample $y \\sim p_{data}(y)$, which implies that $s_\\theta(x_t) \\approx s_\\theta(x_t, y)$. \n\nThey propose ICG to “streamline” the training of conditional model and improve efficiency (line 534) and as an alternative to CFG, which requires either separate models or a single model with a null token, representing unconditional generation. With ICG, the authors show that training a conditional model is sufficient and the CFG update can be replaced by:\n\n$D_{ICG} = D(z_t, t, \\widehat{y}) + w_{ICG}(D(z_t, t, y) - D(z_t, t, \\widehat{y}))$ \n\nwhere $\\widehat{y}$ is a Gaussian vector or a class sampled independent of $y$ at each time-step $t$.\n\n**TSG**\n\nTSG is based on the insight that score model architectures add the class-conditioning embedding into the time-step embedding. Therefore, similar to ICG, they perturb the time-step $t$ by $\\Delta t \\sim N(0, \\sigma)$ TSG then uses a linear combination of $D_\\theta(x_t, t)$ and $D_\\theta(x_t, t + \\Delta t)$ as the “score” network update. The justification of TSG is based on a connection to Langevin dynamics. \n\nIn practice, they do not perturb the time, but similar to ICG they perturb the time-step embedding vector by $s t^{\\alpha} n$, where $s, t, \\alpha$ selected to match the mean and scale of the “real” time-step embedding vectors and $n \\sim N(0, 1)$, and they propose the following update instead of CFG:\n\n$D_{TSG} = D(z_t, \\tilde{t}, y) + w_{TSG}(D(z_t, t, y) - D(z_t, \\tilde{t}, y))$ where $\\tilde{t}$ is the perturbed embedding.", "questions": "The authors present an estimate of the marginal model $p(x_t)$, where instead of marginalizing over all possible $y$ by learning a score model with a null token, they use a Monte Carlo estimate. However, the variance of the estimate of the marginal score and it’s impact on sampling has not been addressed in the paper. \n\n1. Supports of $p(x_t | y=1)$ and $p(x_t | y=0)$ are different: In case of two classes and when $p(x_t | \\widehat{y})$ is low, in such a case the ICG term is the score of $p(x_t | \\widehat{y})^{1 - w_{ICG}} p(x_t | y)^{w_{ICG}}$, if $p(x_t | \\widehat{y})$ is low then its score’s magnitude is high and would dominate the ICG update, potentially pushing in the direction of $p(x_0 | \\widehat{y})$\n    1. will more steps be required for ICG sampling compared to CFG? \n    2. What will happen when the support for $x_t$ all $t \\in [0, T]$ is disjoint for the two classes, such that $p(x_t | \\widehat{y}) = 0$ when $\\widehat{y}$ is the wrong class? For instance, if $p_{data} = 0.5  N(-5, 1) + 0.5 N(5, 1)$, and the model prior is a centered Gaussian.\n2. If the classes are imbalanced then sampling from the majority class can bias the sampling process when the user wishes to sample from the minority class. For instance, if $p(y = 0 | x_0) = 0.99$ and $p(y = 1 | x_0)= 0.01$,  and ICG samples $\\widehat{y}$ with an equal probability. The authors should consider modifying the algorithm to sample from $p(y)$, when available, and not a uniform distribution. \n\nSome simple theoretical guidance with/or toy examples should suffice for an explanation. Potentially with $p(x_0)$  as a mixture of Gaussians in 2d and the authors could vary the guidance scale while sampling.\n\n1. For text to image sampling with ICG, do the authors recommend sampling random text prompts to estimate the marginal model? \n2. For the text conditional experiments in Table 3, can the authors also include CFG performance as a baseline?\n3. For the ControlNet experiment, I believe the authors describe training with an empty string for 50% of their text prompts. See section 3.3 in the ControlNet paper. Do the authors use a random text and/or image prompt for the ControlNet model to define a marginal model? \n4. In line 053, the authors claim that there is no clear extension of CFG to unconditional generation:\n    1. However, CFG requires training a marginal model, which can be sampled easily. Moreover, one could randomly sample a label y to do generation as well.  \n    2. Can the authors provide details about unconditional sampling with ICG? The update defined in algorithm 1, requires a condition $y$ as input. is it just randomly sampling a label and then running ICG?\n    \n    $D_{ICG} = D(z_t, t, \\widehat{y}) + w_{ICG}(D(z_t, t, y) - D(z_t, t, \\widehat{y}))$", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and the authors perform extensive experiments and provide algorithmic and experimental details to reproduce their experiments. \n\n**Experiments**\n\nThe experiments show that ICG and CFG perform identically and ICG “simulates the behavior of CFG across several conditional models” (line 249). They also run a number of ablations, for instance, they compare using an independent Gaussian vector versus sampling $y \\sim p(y)$ in ICG, providing stronger experimental evidence.", "weaknesses": "The main question left un-answered by this work is whether a marginal model, either trained separately or with a null token, is necessary for a sampling method like CFG:\n1. An analysis of the variance of the estimate $s_\\theta(x_t \\mid y)$ is missing from the paper. \n2. An analysis of the impact of the variance on sampling is also missing from the paper\n3. Is there a trade-off between the slower convergence of training with a null token with estimation of the marginal model?\n\nSee the questions section.\n\nThe authors could improve the paper by providing a proof (or an empirical analysis) for the sampling procedure for a toy distribution where the $p_{data} = 0.5 N(-a, 1) + 0.5 N(a, 1)$. For instance, with a mixture of Gaussians as the data distribution, the marginal and conditional score functions are computable in closed form and a toy experiment, where various trade-offs can be examined, could make the paper stronger. \n\nTSG\nFor connecting TSG to Langevin dynamics, the authors show that eq 9 resembles a Langevin dynamics step. However, by using the perturbed TSG denoising model $\\widehat{D}$  to define the score $\\nabla_{z_t} \\log \\widehat{p}(z_t)$, the connection seems circular. Could the authors clarify what the connection to Langevin dynamics is and the relevance of that connection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PyYz4woMM2", "reviewer": "Reviewer_ryvA", "summary": "Classifier-free guidance (CFG) is a common method for improving the quality of samples generated from diffusion models. One limitation of CFG is the added training cost due to the requirement for both a conditional and an unconditional model. The authors propose Independent Condition Guidance (ICG), claiming that it provides the same advantages of CFG on image generation quality without additional training costs. They also propose Time-Step Guidance (TSG), a guidance algorithm based on a mixture of score estimates given both perturbed and unperturbed time-step values, which overcomes CFG’s reliance on the availability of class labels. TSG is based on a mixture of scores pertaining to a perturbed and unperturbed time-step value. The authors empirically validate their methods against CFG and un-guided baselines, demonstrating improvements in image quality over unguided sampling that are comparable in magnitude to CFG, while avoiding its limitations.", "questions": "1. Please correct or clarify the theoretical justification behind ICG.\n2. Are there not other perturbation-based guidance methods, such as the ones discussed in the Related Works section (in the same spirit as SAG), that TSG could be compared to?\n3. Why not include a comparison between ICG and CADS alone (Appendix B only includes ICG+CADS)? It seems like it shares many similarities with ICG. If this is the case, why is this comparison not included in the main paper?\n4. The connection between TSG and Langevin dynamics is interesting, but it is not clear how the hypothesized benefits of TSG relate to, or differ from, the theoretical benefits underlying the Langevin diffusion SDE that is implicit in the SDE corresponding to the forward diffusion process.\n5. Can the authors better justify the need for two hyperparameters $s$ and $\\alpha$ in TSG, and how they tuned these hyperparameters (and on which dataset)?\n6. The conditional and unconditional samples from the ICG method seem to have slightly more contrast and are more saturated than the CFG baseline. If the authors agree with my impression, can they hypothesize as to why this should the case?\n7. While the authors may be able to justify why ICG could theoretically perform as well as CFG, I don't understand why ICG would be expected to perform better than CFG, as concluded from the plots in Figure 3. Even if the authors can justify this, it would be helpful to understand how sensitive is this result is senstive to hyperparameter choices and random seeds.\n8. Figure 5: I did not understand the rationale for testing ICG against the unguided baseline as opposed to ICG against CFG on the text condition. Can the authors clarify?\n\nMinor:\n9. Can the authors revisit Section 3,4, and 5, and ensure all variables and functions are properly defined? E.g. $\\epsilon$ is undefined. This would help improve the readability.\n10. It seems misleading to say that Equation (1) is \"equivalent to\" Equation (2), and that the denoiser $D_\\theta$ \"approximates\" the score function.\n11. It would be helpful to refer to the appendix sections in relevant sections of the main text.\n\n\n---Post-rebuttal: The authors have addressed the majority of my questions and concerns in their responses to my review and in their global comment. I have therefore increased my rating.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well written and easy to follow.\n2. The introduction and related works sections provided a good justification for the limitations of CFG and the benefits of solutions like ICG and TSG. There is certainly a potential for this paper to provide a meaningful contribution to the field of diffusion-guidance, if my concerns about the theoretical claims and empirical validation are adequately addressed by the authors.\n3. The experiments are for the most part appropriate and the ablation study is useful to understand the effect of some of the hyperparameter choices.", "weaknesses": "1. A central claim is that ICG provides the same benefits of CFG without the need for additional training requirements. This hinges on the validity of the theoretical claim in Section 4, which avoids the need for a potentially expensive marginalization over classes. However, the theoretical justification appears insufficient. It seems incorrect to state that sampling a class label $\\hat{y}$ independently from $z_t$ at time-step $t$ implies the equivalence $\\nabla_{z_t} \\log p_t(\\hat{y} \\mid z_t) = \\nabla_{z_t} \\log p_t(z_t)$, since conditional independencies are a property of the diffusion model parametrized by $\\theta$, irrespective of what distribution $y$ was drawn from to compute the sum in $(6)$. Given the conditional score estimate parametrized by $\\theta$, $z_t$ should be dependent on $y$. The authors later allude to a procedure to \"bootstrap the score\", but this is not reflected in their Algorithm 1. Moreover the authors describe the correct equivalence in Appendix A, which involves marginalizing over $y$, but they don't explain how this relates to the seemingly incorrect equivalence relation (7).\n2. Empirical evidence supporting TSG is limited. There are no baseline comparisons other than an unguided model. The baseline comparisons for ICG are also limited given that other methods have been published with similar motivations as ICG.\n3. The choice of distribution from which $\\hat{y}$ is sampled seems non-trivial, and although the authors attempted to shed some light on this in their ablation study, it is unclear how such a distribution should be chosen in general, and how sensitive are their results to the choice of distribution.\n4. Several hyperparameters are introduced in the paper, but the description of the hyperparameter tuning procedure seems incomplete. Importantly, on which dataset were hyperparemeters tuned, and using what evaluation criterion?\n5. Code is not made publicly available, which diminishes the paper's impact and impairs reproducibility. The provided pseudocode does not seem to entirely capture the design choices that the authors made as part of their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7snFmj8X9", "reviewer": "Reviewer_c3pF", "summary": "Reviewing the classifier-free guidance (CFG), this paper found that the unconditional score in CFG can be derived by a random label vector y_hat (termed ICG) without training the unconditional model by setting the y_null label. Furthermore, the authors stimulate the guidance by introducing a perturbated time-step embedding based on the original t embedding (termed TSG). Experiments show that ICG achieves similar results to CFG, and TSG improves the FID for conditional and unconditional tasks.", "questions": "The authors mention that there are two methods to decide the random vector $\\hat{y}$: draw from Gaussian distribution or a random class label, so that $\\nabla_{z_t} log p_t (z_t | \\hat{y}) = \\nabla_{z_t} log p_t (z_t)$. I wonder how much would the conditional score $\\nabla_{z_t} log p_t (z_t | \\hat{y})$ change in practice given different samples $\\hat{y}_1, \\hat{y}_2, \\hat{y}_3 ...$ and the same $z_t$\n\nI would increase my rating if some of my doubts and concerns were resolved.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of ICG is novel and theoretically simple to implement. ICG provides a new perspective on CFG.\n\n2. The presentation is good and the paper is easy to follow.\n\n3. Extensive experiments show consistent effectiveness of the proposed two methods", "weaknesses": "1. My major concern lies in the proposed TSG:\n- the authors explain TSG from its connection to Langevin dynamics, but the connection is built up on Taylor approximation where $\\tilde{t} = t + \\Delta{t}$, and $\\Delta{t}$ should be sufficiently small. But in practice, the authors use $\\Delta{t}=st^\\alpha n$ where $n \\sim N(0, I), s=2, \\alpha=1$ and $t$ is the time-step, therefore I do think $\\Delta{T}$ is sufficiently small anymore.\n- I suspect the effectiveness of TSG comes from the error correction, rather than the guidance perspective, according to the results shown in Table 4.\n- Will TSG work in the diffusion models that do not embed class labels into timestep (for example, UViT)?\n\n2. please provide the NFE and samplers used to report FID throughout the paper. \n\n3. some writing issues in section 3:\n- According to eq 6 of the CFG paper [1], eq 4 in this paper looks wrong, eq 4 should be $\\hat{D} = D_\\theta (z_t, t, y) + w(D_\\theta(z_t, t, y) - D_\\theta(z_t, t, y_null))$.\n- undefined notation $\\beta_t$ in eq 2\n\n\n[1] Ho, Jonathan, and Tim Salimans. \"Classifier-free diffusion guidance.\" arXiv preprint arXiv:2207.12598 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)