Guiding a Diffusion Model with a Bad Version of Itself
Authors: Tero Karras
Miika Aittala
Tuomas Kynkäänniemi
Jaakko Lehtinen
Timo Aila
Samuli Laine
Venue: NeurIPS 2024 oral (2024)
Keywords: diffusion models
classifier-free guidance
guidance
Primary area: diffusion_based_models
TL;DR: Guiding a diffusion model with a smaller, less-trained version of itself leads to significantly improved sample and distribution quality.

Abstract: The primary axes of interest in image-generating diffusion models are image quality, the amount of variation in the results, and how well the results align with a given condition, e.g., a class label or a text prompt. The popular classifier-free guidance approach uses an unconditional model to guide a conditional model, leading to simultaneously better prompt alignment and higher-quality images at the cost of reduced variation. These effects seem inherently entangled, and thus hard to control. We make the surprising observation that it is possible to obtain disentangled control over image quality without compromising the amount of variation by guiding generation using a smaller, less-trained version of the model itself rather than an unconditional model. This leads to significant improvements in ImageNet generation, setting record FIDs of 1.01 for 64x64 and 1.25 for 512x512, using publicly available networks. Furthermore, the method is also applicable to unconditional diffusion models, drastically improving their quality.

Guiding a Diffusion Model with a Bad Version of Itself
Tero Karras NVIDIA
Miika Aittala NVIDIA
Tuomas Kynkäääniiei Aalto University
Jaakko Lehtinen NVIDIA, Aalto University
Timo Aila
Samuli Laine NVIDIA
NVIDIA
Abstract
The primary axes of interest in image- generating diffusion models image quality, the amount of variation in the results, and how well the results align with given condition, e.g,, class label or text prompt. The popular classifier-free guidance approach uses an unconditional model to guide conditional model, leading to simultaneously better prompt alignment and higher-quality images the cost of reduced variation. These effects seem inherently entangled, and thus hard to control. We make the surprising observation that it is possible obtain disentangled control over image quality without compromising the amount of variation by guiding generation using a smaller, less-trained version of the model itself rather than an unconditional model. This leads to significant improvements in ImageNet generation. setting record FIDs of 1.01 for 64x and 1.25 for 512x512, using publicly available networks. Furthermore. the method is also applicable to unconditional diffusion models, drastically improving their quality.
Introduction
generate synthetic images by reversing a stochastic corruption process. Essentially, an image is revealed from pure noise by denoising it little by little in successive steps. A neural network that implements the denoiser (equivalently [53]. the score function 20) is central design element, and various architectures have been proposed (e.g.. 21 24 38) ). Equally important are the details of the multi-step denoising process that corresponds or stochastic 44 differential equation, for which
Denoising diffusion models 15 46
47 48
9
19
mathematically solving an ordinary 35 47
many different parameterizations, solvers, and step schedules have been evaluated [22 23 30 43 57]. To control the output image, the denoiser typically conditioned
class label, an embedding of
text prompt, or some other form of conditioning input [36 41
45 56
The training objective of diffusion model aims to the entire (conditional) data distribution. This causes problems in low-probability regions: The model gets heavily penalized for not representing them, but it does not have enough data learn to generate good images corresponding them. Classifier-free guidance (CFG) (16) has become the standard method for "lowering the sampling temperature", i.e., focusing the generation on well-learned high-probability regions. By training denoiser network to operate in both conditional and unconditional setting, the sampling process can be steered away from the unconditional result in effect, the unconditional generation task specifies a result to avoid. This results in better prompt alignment and improved image quality, where the former effect is due to CFG implicitly raising the conditional part of the probability density to power greater than one 100
However, CFG has drawbacks that limit its usage general low-temperature sampling method. First, it is applicable only for conditional generation, as the guidance signal is based on the difference between conditional and unconditional denoising results. Second, because the unconditional and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

conditional denoisers are trained to solve a different task, the sampling trajectory can overshoot the desired conditional distribution, which leads to skewed and often overly simplified image compo- sitions [27 Finally, the prompt alignment and quality improvement effects cannot be controlled separately, and it remains unclear how exactly they relate to each other.
In this paper, we provide new insights into why CFG improves image quality and show how this effect can be separated out into a novel method that we call autoguidance. Our method does not suffer from the task discrepancy problem because we use an inferior version of the main model itself as the guiding model, with unchanged conditioning. This guiding model can be obtained by simply limiting. e.g., model capacity and/or training time. We validate the effectiveness of autoguidance in various synthetic test cases as well as practical image synthesis class-conditionn and text-conditional settings. In addition, our method enables guidance for unconditional synthesis. In quantitative tests, the generated image distributions are improved considerably when measured using FID [33] and FDDINOv2 [51 metrics, setting new records in ImageNet-512 and ImageNet-64 generation.
Our implementation and pre-trained models are available at
2 Background
Denoising diffusion. Denoising diffusion generates samples from a distribution Ptam(()] by it- eratively denoising sample of pure white noise, such that a noise-free random data sample is gradually revealed [15]. The idea is to consider heat diffusion of Pxata(x) into sequence of in- creasingly smoothed densities p(x; o) Paam (x) *N(x; 0, 0²) For large enough max. we have p(x; max) ~ Nx; 0, o from which we can trivially sample by drawing normally distributed white noise. The resulting sample is then evolved backward towards low noise levels by a probability
flow ODE 223 47 49
dx, -a Vx, log ((x;;o) do
(1)
that maintains the property x, p(x::o) o) for every E [0, mux]] Upon reaching 0, we obtain Xo ~ p(xo; 0) VVtma(xo) desired.
In practice, the ODE is solved numerically by stepping along the trajectory defined by Equation T This requires evaluating the so-called score function 20) Vx log p(x; σ) for a given sample x and noise level each step. Rather surprisingly, we can approximate this vector using a neural network Da(x; o) parameterized by weights 0 trained for the denoising task
= =ar ming
D + n;o) - y12.
where Puain controls the noise level distribution during training. Given Dg. we can estimate Vx Volog p(x; σ) (Dx(x; o) - x)/o². up approximation errors due to, c.g., finite capacity or training time 23 53 As such, we are free to interpret the network as predicting either a denoised sample or score vector, whichever is more convenient for the analysis hand. Many reparameteri- zations and practical ODE solvers are possible, as enumerated by Karras et al. 23 We follow their recommendations, including the schedule (t) = that lets us parameterize the ODE directly via noise level a instead of a separate time variable t.
In most applications, each data sample is associated with a label c, representing, e.g., a class index or a text prompt. At generation time, we control the outcome by choosing a label e and seeking a sample from the conditional distribution p(x|c; o)) with o 0. In practice. this is achieved by training a denoiser network Do x: a, c) that accepts e as an additional conditioning input.
Classifier-free guidance. For complex visual datasets, the generated images often fail to reproduce the clarity of the training images due to approximation errors made by finite-capacity networks. A broadly used trick called classifier-free guidance (CFG) (16] pushes the samples towards higher likelihood of the class label, sacrificing variety for "more canonical" images that the network appears to be better capable of handling.
In a general setting, guidance in a diffusion model involves two denoiser networks Do(x; o, c) and D2 (x; a, c). The guiding effect is achieved by extrapolating between the two denoising results by factor w:
(x;ox, (x; + (1 w) Do(xx,),
(3)
2

---

Trivially, setting w 0 or to = recovers the output of Do and D1, respectively, while choosing w > 1 over-emphasizes the output of Dj. Recalling the equivalence of denoisers and scores [53], we can write
pi(x(c; po(x)c;
(x(x; xs x V log (o(()cc;))
(4)
xc (x)c
Thus, guidance grants us access to the score of the density (x(ex(;; implied in the parentheses. This score can be further written TO T6
Vx log Pwo (x(c;)) Vx (olopi((x((;)) + (w 1)V log (x
(5)
pu(xic;o)
Substituting this expression into the ODE of Equation I this yields the standard evolution for generating images from pl. plus a perturbation that increases (for w 1) the ratio of pi and pn as evaluated at the sample. The latter can be interpreted as increasing the likelihood that a hypothetical classifier would attribute for the sample having come from density pi rather than po-
In CFG, we train an auxiliary unconditional denoiser (x 0) to denoise the distribution p(x: o) marginalized over c, and use this Do. In practice, this is typically [16] done using the same network Do with an empty conditioning label, setting Do == Dx(x; ( o, 0) and D2 == Do c). By Bayes' rule, the extrapolated score vector becomes V log p(x|c; During sampling, this guides the image to more strongly align with the specified class c.
(w log p(c[x;
It would be tempting to conclude that solving the diffusion ODE with the score function of Equation S produces samples from the data distribution specified by P.. c; 0). Unfortunately this not the case, because Pao (x|c; ar) does not represent valid heat diffusion of Pre [cc; 0) (58) Therefore, solving the ODE does not, in fact, follow the density. Instead, the samples are blindly pushed towards higher values of the implied density at each noise level during sampling, This can lead to distorted sampling trajectories, greatly exaggerated truncation, and mode dropping in the results [27], as well as over-saturation of colors ES). Nonetheless, the improvement in image quality is often remarkable, and high guidance values are commonly used despite the drawbacks (e.g.. 14
45
Why does CFG improve image quality?
We begin by identifying the mechanism by which CFG improves image quality instead of only affecting prompt alignment. To illustrate why unguided diffusion models often produce unsatisfactory images, and how CFG remedies the problem, study 2D toy example where small-scale denoiser network is trained to perform conditional diffusion in a synthetic dataset (Figure []]. The dataset is designed to exhibit low local dimensionality (i.e., highly anisotropic and narrow support) and hierarchical emergence of local detail upon noise removal. These are both properties that can be For details of the setup, Appendix C
expected from the actual manifold of realistic images 5 40
Score matching leads to outliers. Compared to sampling directly from the underlying distribution (Figure Tap. the unguided diffusion in Figure Tb] produces large number of extremely unlikely samples outside the bulk of the distribution. In the image generation setting, these would correspond to unrealistic and broken images.
We argue that the outliers stem from the limited capability of the score network combined with the score matching objective. It is well known that maximum likelihood (MI.) estimation leads to "con- servative" fit of the data distribution [3] in the sense that the model attempts to cover all training sam- ples. This is because the underlying Kullback-eeblerr divergence incurs extreme penalties if the model severely underestimates the likelihood of any training sample. While score matching is generally not equal to ML estimation, they are closely related [15] 32 49 and appear to exhibit broadly similar behavior. For example, it is known that for a multivariate Gaussian model, the optimal score matching fit coincides with the ML estimate [20). Figures 2a and 2b show the learned score field and implied density in our toy example for two models of different capacity at an intermediate noise level. The stronger model envelops the data more tightly, while the weaker model's density is more spread out.
From the perspective of image generation, a tendency to cover the entire training data becomes a problem: The model ends up producing strange and unlikely images from the data distribution's 's
3

---

(a) Ground truth
(b) No guidance
Classifier-free guidance
(d) Naive truncation
Autoguidance (ours)
Figure 1: A fractal-like 2D distribution with classes indicated with gray and orange regions. Approximately 99% of the probability mass is inside the shown contours. (a) Ground truth samples drawn directly from the orange class distribution. (b) Conditional sampling using small denoising diffusion model generates outliers. (c) Classifier-free guidance (w = 4) eliminates outliers but reduces diversity by over-emphasizing the class. (d) Naive truncation via lengthening the score vectors. (e) Our method concentrates samples on high-probability regions without reducing diversity.
(a) Pi (x|c: Gmad)
(b) (x(x; Tmid)
(c) Ratio P1/Po
(d) No guidance
(c) CFG with w 4
Figure 2: Closeup of the region highlighted in Figure Tc. (a) The implied learned density pi (x]c; miia) (green) at an intermediate noise level mii and its score vectors (log-gradients) plotted at represen- tative sample points. The learned density approximates the underlying ground truth p(x mii)) (orange) but fails to replicate its sharper details. (b) The weaker unconditional model learns a further spread-out density Po Po(x: miia) (red) with looser fit the data. (c) Guidance moves the points according to the gradient of the (log) ratio of the two learned densities (blue). As the higher-quality model is more sharply concentrated the data, this field tends inward towards the data distribution. The corresponding gradient is simply the difference of respective gradients in (a) and (b), illustrated at selected points. (d) Sampling trajectories taken by standard unguided diffusion following the learned score Vx log P1 (x c; a). from noise level mid to () The contours (orange) represent the ground truth noise-free density. (e) Guidance introduces an additional force shown in (c), causing the points to concentrate at the core of the data density during sampling.
extremities that not learnt accurately but included just avoid the high loss penalties. Furthermore, during training, the network has only seen real noisy images as inputs, and during sampling it may not be prepared deal with the unlikely samples it is handed down from the higher noise levels.
CFG eliminates outliers. The effect of applying classifier-free guidance during generation is demonstrated in Figure Ic As expected, the samples avoid the class boundary (i.c., there are no samples in the vicinity of the gray arca), and entire branches of the distribution also observe a second phenomenon, where the samples have been pulled towards the core of the manifold, and away from the low-probability intermediate regions. Seeing that this eliminates the unlikely outlier samples, we attribute the image quality improvement to it. However, mere boosting of the class likelihood does not explain this increased concentration.
dropped. We
We argue that this phenomenon stems from a quality difference between conditional and uncondi- tional denoiser networks. The denoiser Do faces a more difficult task of the two: It has to generate from all classes once, whereas Dj can focus on single class for any specific sample. Given the more difficult task, and typically only small slice of the training budget, the network Dg attains

---

a worse fit to the data This difference in accuracy is apparent in respective plots of the learned
densities in Figures Za and 2b
From our interpretation in Section 2 it follows that CFG is not only boosting the likelihood of the sample having come from the class c, but also that of having come from the higher-quality implied distribution. Recall that guidance boils down to an additional force (Equation 5) that pulls the samples towards higher values of log (pi (x|c; o) px)c((x(c; Plotting this ratio for our toy example in Figure 2c along with corresponding gradients that guidance contributes to the ODE vector field, we see that the ratio generally decreases with distance from the manifold due to the denominator Po representing a more spread-our distribution, and hence falling off slower than the numerator p1. Consequently, the gradients point inward towards the data manifold. Each contour of the density ratio corresponds to a specific likelihood that a hypothetical classifier would assign on sample being drawn from pi instead of po. Because the contours roughly follow the local orientation and branching of the data manifold, pushing samples deeper into the "good side" concentrates them at the manifold
Discussion. We can expect the two models to suffer from inability to fit at similar places, but to a different degree. The predictions of the denoisers will disagree more decisively in these regions. As such, CFG can be seen as a form of adaptive truncation that identifies when sample is likely to be under-fit and pushes it towards the general direction of better samples. Figures [2] and [2]shoow the effect over the course of generation: The truncation "overshoots" the correction and produces a narrower distribution than the ground truth, but in practice this does not appear to have an adverse effect on the images.
In contrast, a naive attempt at achieving this kind of truncation-inspireed by, e.g., the truncation trick in GANs [33 or lowering temperature in generative language models - would counteract the smoothing by uniformly lengthening the score vectors by a factor w 1. This is illustrated in Figure [d] where the samples are indeed concentrated in high-probability regions, but in an isotropic fashion that leaves the outer branches empty. In practice, images generated this way tend show reduced variation, oversimplified details, and monotone texture.
4 Our method
We propose to isolate the image quality improvement effect by directly guiding a high-quality model D2 with poor model Do trained on the same task, conditioning, and data distribution, but suffering from certain additional degradations, such as low capacity and/or ander-training We call this procedure autoguidance, as the model is guided with an inferior version of itself.
In the context of our 2D toy example, this turns out to work surprisingly well. Figure [e demonstrates the effect of using a smaller Do with fewer training iterations. As desired, the samples are pulled close to the distribution without systematically dropping any part of it.
To analyze why this technique works, recall that under limited model capacity, score matching tends to over-emphasize low- wwwprobability (i.e., implausible and under-trained) regions of the data distribution. Exactly where and how the problems appear depend on various factors such as network architecture, dataset, training details, etc., and cannot expect to identify and characterize the specific issues priori. However, we can expect a weaker version of the same model to make broadly similar errors in the same regions, only stronger. Autoguidance seeks identify and reduce the errors made by the stronger model by measuring its difference to the weaker model's 's prediction, and boosting it. When the two models agree, the perturbation is insignificant, but when they disagree, the difference indicates the general direction towards better samples.
As such, we can expect autoguidance to work if the two models suffer from degradations that are compatible with each other. Since any D2 can be expected to suffer from, e.g., lack of capacity and
'The visual quality difference is obvious if we simply inspect the unconditional images generated by current large-scale models. Furthermore, the unconditional case tends to work so poorly that the corresponding quantitative numbers are hardly ever reported. The EDM2-S S model 24 trained with ImageNet- 512, for example, yields FID of 2.56 in the cansitonniii setting and 11.67 in the unconditional setting.
"Discriminator guidance (25) trains an explicit classifier to discriminate between the generated samples and noisy training samples at each noise level and uses its log- ooggrrnient to guide the sampling. Our analysis is not applicable in this situation; in CFG the task is implicit and the distinction is between p1 and po.
5

---

lack of training -at least to some degree- it makes sense to choose Do so that it further exacerbates these aspects.
In practice, models that are trained separately or for a different number of iterations differ not only in accuracy of fit, but also in terms of random initialization, shuffling of the training data, etc. For guidance to be successful, the quality gap should be large enough to make the systematic spreading-out of the density outweigh these random effects.
Study on synthetic degradations. To validate our hypothesis that the two models must suffer from the same kind of degradations, we perform a controlled experiment using synthetic corruptions applied to well-trained real-world image diffusion model. We create the main and guiding networks, Di and Do. by applying different degrees of a synthetic corruption to the base model. This construction allows us to use the untouched base model as grounding when measuring the FID effect of the various combinations of corruptions applied to Dj and Do- We find that as long as the degradations are compatible, autoguidance largely undoes the damage caused by the corruptions:
Base model: As the base model, we use EDM2-S trained on ImageNet-512 without dropout (FID = 2.56).
Dropout: We construct D1 by applying 5% dropout to the base model in a post-hoc fashion (FID = 4.98), and Do by applying 10% dropout to the base model (FID 15.00). Applying autoguidance, we reach the best result (FID 2.55) with w = 2.25, matching the base model's FID.
We construct D₁ by modifying the base model to add noise to the input images so that their noise level is increased by 10% (FID 3.96). The a conditioning input of the denoiser is adjusted accordingly. The guiding model Do is constructed similarly, but with noise level increase of 20% (FID = 9.73). Applying autoguidance, we reach the best result (FID 2.56) with 2.00, again matching the base model's FID.
Input noise:
Mismatched degradations: If corrupt D1 by dropout and Du by input noise, vice versa, guidance does not improve the results at all; in these cases, the best FID is obtained by setting
1. i..., by disabling guidance and using the less corrupted D, exclusively.
While this experiment corroborates our main hypothesis, we do suggest that guiding with these synthetic degradations would be useful in practice. A realistic diffusion model will not suffer from these particular degradations, so creating a guiding model by introducing them would not yield consistent truncation towards the data manifold.
5 Results
Our primary evaluation is carried out using ImageNet (ILSVRC2012) 8 at two resolutions: 512x512 and 64. For ImageNet-512
latent diffusion 242 while ImageNet-64 works directly on
RGB pixels. We take the current state-of-the-art diffusion model EDM2 [24] as our baseline 5 We use the EDM2-S and EDM2-XXL models with default sampling parameters: 32 deterministic steps with 2nd order Heun sampler 23 For most setups, a pre-trained model is publicly available, and in the remaining cases we train the models ourselves using the official implementation (Appendix B).
We use two degradations for the guiding model: shorter training time and reduced capacity compared to the main model. We obtain the best results by having both of these enabled. With EDM2-S, for example, we use an XS-sized guiding model that receives 1 1/16th of the training iterations of the main model. We ablate the relative importance of the degradations as well as the sensitivity to these specific choices in Section 5.1 As the EDM2 networks are known to be sensitive to the guidance weight and EMA length [24]. we search the optimal values for each case using a grid search.
Table I shows that our method improves FID 13] and FDDINO 51 considerably. Using the small model (EDM2-S) in ImageNet-512, our autoguidance improves FID from 2.56 to 1.34. This beats the 1.68 achieved by the concurrently proposed CFG + Guidance Interval (27]. and is the best result reported for this dataset regardless of the model size. Using the largest model (EDM2-XXL. further improves the record to 1.25. The FDDINO,2 records are similarly improved, with the large model
htatps///github..co///..aaa

---

Method
FID
w
EMAS
EMA,
FDDINO
w
EMAp
EMA,
EDM2-S
24
2.56
0.130
68.64
0.190
+ Classifier-free guidance
24
2.23
1.40
0.025
0.025
52.32
1.90
0.085
0.085
+ Guidance interval
27
1.68
2.10
0.025
0.025
46.25
3.20
0.085
0.085
Autoguidance (XS, T/16)
Ours
1.34
2.10
0.070
0.125
36,67
2,45
0.120
0.165
Same EMA for both
1.53
1.95
0.050
0.050
40,81
2.25
0.115
0.115
512
Reduce training only
1.51
2.20
0.090
0.130
42.27
2.55
0.130
0.170
Reduce capacity only
2.13
1.80
0.120
0.160
59,89
1.90
0.140
0.085
EDM2-XXL
24
1,91
0.070
42.84
0.150
+ Classificr free guidance
24
1,81
1.20
0.015
0.015
33,09
1,70
0.015
0.015
+ Guidance interval
27
1.40
2.00
0.015
0.015
29.16
2.90
0.015
0.015
Autoguidance (M, T/3.5)
Ours
1.25
2.05
0.075
0.155
24.18
2.30
0.130
0.205
EDM2-S, unconditional
11.67
0.145
209.53
0.170
Autoguidance (XS, T/16) Ours
3.86
2.85
0.070
0.110
90.39
2.90
0.090
0.125
RIN
21
1.23
0.033
64x64
EDM2-S
24
1.58
0.075
58,52
0.160
Classifier-free guidance
1.48
1.15
0.030
0.030
41.84
1.85
0.040
0.040
Autoguidance (XS, T/8)
Ours
1.01
1.70
0.045
0.110
31.85
2.20
0.105
0.175
Table 1: Results ImageNet-512 and ImageNet-64. The parameters of autoguidance refer the capacity and amount training received by the guiding model. The latter given relative to the number of training images shown the main model (T"). The columns EMAm and EMA, indicate the length parameter of the post-hoc EMA technique [24] for the main and guiding model, respectively.
FID
FID
FID
T/32
T/16" + T/8
XXS
XS*
S
EMA
EMA,
3.5
3.5
3.0
3.0
3.0
2.5
2.5
2.5
1,47
1151
1.34
1.41
1.34
1.34
1.34
w=1.0
1.0
2.5
3.0
2.5
3.0
EMA 0.05
(a) Guidance weight training
(b) Guidance weight capacity
(c) EMA length parameters
Figure 3: Sensitivity w.r.t. autoguidance parameters, using EDM2-S ImageNet-512. The shaded regions indicate the min/max FID over 3 evaluations. (a) Sweep over guidance weight w while keeping all other parameters unchanged. The curves correspond how much the guiding model was trained relative to the number images shown to the main model. (b) Sweep over guidance weight for different guiding model capacities. configuration, denoted with in (a) and (b).
(c) Sweep over the EMA length parameters for best
lowering the record from 29.16 to 24.18. In ImageNet-64, the improvement is even larger; in this dataset, we set the new record FID and FDDINO22 1.01 and 31.85, respectively.
A particular strength of autoguidance is that it can be applied to unconditional models well. While conditional ImageNet generation may be getting close to saturation, the unconditional results remain surprisingly poor. EDM2-S achieves a FID of 1.67 in the unconditional setting, indicating that practically none of the generated images are of presentable quality. Enabling autoguidance lowers the FID substantially 3.86, and the improvement in FDDINOs2 is similarly significant.
5.1 Ablaiions
Table I further shows that beneficial allow independent EMA lengths for the main and guiding models. When both are forced to use the same EMA, FID worsens from 1.34 to 1.53 in ImageNet-512 (EDM2-S). We also measure the effect of each degradation (reduced training time, capacity) in isolation. If we set the guiding model to the same capacity as the main model and only train it for
7

---

w
3
w =
W 3
CFG
Ours
CFG
Ours
Figure 4: Example results for the Tree frog, Palace, Mushroom, Castle classes of ImageNet-512 using EDM2-S. Guidance weight increases the right; rows are classifier-free guidance and our method.
shorter time, FID worsens 1.51. If instead train the reduced-capacity guiding model for long as the main model, FID suffers lot more, to 2.1.. We can thus conclude that both degradations are beneficial and orthogonal, but majority of the improvement comes from reduced training of the guiding model. Notably, all these ablations still outperform standard CFG in terms of FID.
Figure probes the sensitivity to various hyperparameters. Our best result is obtained by training the guiding model 1/16" as much as the main model, in terms of images shown during training. Further halving the training budget is almost equally good, while doubling the amount of training starts to slowly compromise the results. The results are quite insensitive to the choice of the guidance weight. In terms of the capacity the guiding model, one step smaller (XS for EDM2-S) gave the best result. Two steps smaller (XXS) was also better than no capacity reduction (S), but started to show excessive sensitivity the guidance weight. The results also sensitive the EMA length, similarly the original EDM2. Post-hoc EMA 24] allows us to search the optimal parameters at feasible cost.
We also explored several other degradations for the guiding model but did not find them to be beneficial. First, we tried reducing the amount of training data used for the guiding model, but this did not seem improve the results over the baseline. Second, applying guidance interval (27) top of our method reduced its benefits to some extent, suggesting that autoguidance is helpful at all noise levels. Third, deriving the guiding model from the main model using synthetic degradations did not work at all. providing further evidence that the guiding model needs to exhibit the same kinds of degradations that the main model suffers from. Fourth, found that the main model had been quantized, e.g., to improve inference speed, quantizing it to an even lower precision did not yield useful guiding model.
One limitation of autoguidance the need to train a separate guiding model. That said, the additional training cost can be quite modest when using smaller model and shorter training time for the guiding model. For example, the EDM2-M model trains approximately 2.7x as fast as EDM2-XXL per iteration, and train it for 1/3.5 of iterations, so the additional cost is around +11%. For the EDM2-S/XS pair used in most of our experiments, the added training cost is only +3.6%.

---

Increasing guidance weight for our method
Increasing CFG weight
Interpolation between CFG and our method
Figure 5: Results for DeepFloyd IF (50) using the prompt "A blue jay standing on a large basket of rainbow macarons". " The rows correspond to guidance weights E {1, 2, 3, 4). The leftmost column shows results for CFG and the rightmost for autoguidance (XL-sized model guided by M-sized one). The middle columns correspond to blending between the two. See Appendix A for more examples.
5.2 Qualitative results
Figure shows examples of generated images for ImageNet-512. Both CFG and our method tend improve the perceptual quality of images, guiding the results towards clearer realizations as the guidance weight increases. However, CFG seems to have a tendency to head towards more limited number of canonical images (27) per class, while our method produces wider gamut of image compositions. An example is the atypical image of a Palace at w = 1, which CFG converts to somewhat idealized depiction as w increases. Sometimes the unguided sample contains incompatible elements of multiple possible images, such as the Castle image, which includes rough sketch of two three castles unrelated styles. In this instance, CFG apparently struggles decide what to do, whereas our method first builds the large red element into castle, and with increased guidance focuses on the red foreground object. higher number of possible output images consistent with a lower FID, implying better coverage of the training data.
In order study our method in the context of large-scale image generators, we apply DeepFloyd IF [50]. We choose this baseline because multiple differently-sized models are publicly available. Ideally we could have also used an earlier snapshot the guiding model, but those were not available. DeepFloyd IF generates images as a cascade of three diffusion models: a base model and two super- resolution stages. We apply our method the base model only, while the subsequent stages always use CFG. Figure demonstrates the effect of CFG, our method, and their various combinations. To combine autoguidance with CFG, we extend Equation to cover multiple guiding models as proposed by Liu et al. [31] and distribute the total guidance weight among them using linear interpolation (see Appendix B.2 for details). While CFG improves the image quality significantly, also simplifies the style and layout of the image towards canonical depiction. Our method similarly improves the

---

image quality, but it better preserves the image's style and visual complexity. We hope that using both guiding methods simultaneously will serve as a new, useful artistic control.
Discussion and Future work
We have shown that classifier-fres free guidance entangles several phenomena together, and that different perspective together with simple practical changes opens up an entire new design space. In addition to removing the superfluous connection conditioning, this enables significantly better results.
Potential directions for future work include formally proving the conditions that allow autoguidance to be beneficial, and deriving good rules of thumb for selecting the best guiding model. Our suggestion - an early snapshot of a smaller model - easy to satisfy in principle, but these are not available for current large-scale image generators in practice. Such generators trained in successive stages where the training data may change at some point, causing potential distribution shifts between snapshots that would violate assumptions. Various modifications to guidance [1] can be seen as inducing degradations through perturbation of attention maps or denoiser inputs. Whether these approaches could provide additional benefits in our setup remains an open question. Autoguidance also bears conceptual similarity to contrastive decoding (29] used in large language models to reduce the repetitiveness of generations, and there may be opportunities for sharing observations between the two domains.
also often
Recently, several studies 66 guidance weight noise level-dependent. A key benefit from these schedules appears to be the suppression of CFG at high noise levels, where its image quality benefit is overshadowed by the undesirable reduction in variation that is caused by large differences in the content of the differently conditioned distributions. In contrast, autoguidance is not expected to suffer from this problem at high noise levels, as both models target the same distribution. So far we have compared autoguidance only with the interval method [27], which did not find beneficial in combination. A further study on the various possible combinations, in terms of quantitative performance as well as artistic control, is a natural next step. It could also be interesting to further isolate the origin of the improvement using alternative metrics, such as precision and recall (28) Human Preference Score [55] or PickScore [26].
54] have reduced the downsides of CFG by making the
Acknowledgments
We thank David Luebke, Janne Hellsten, Ming-Yu Yu Liu, and Alex Keller for discussions and comments, and Tero Kuosmanen and Samuel Klenberg for maintaining our compute infrastructure.
References
[1] D. Ahn, H. Cho, Min, W. Jang, J. Kim, S. Kim, H. H. Park. K. H. Jin, and S. Kim. Self-rectifying difffusion sampling with perturbed-altentioon guidance. Pruc. ECCV, 2024.
[2] F. Bao, S. Nie, K. Xue, Y. Cao, C. Li, H. Su, and J. Zhu. All are worth words: A VIT backbone for diffusion models. In Proc. CVPR, 2023.
[3] C. M. Bishop. Neural networks for pattern recognition. Oxford University Press, USA, 1995.
[4] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis.
In Proc. ICLR, 2019.
[5] B. C. A. Brown, A. L. Caterini, B. L. Ross, J. C. Cresswell, and G. Loaiza- Ganem. Verifying the union of manifolds hypothesis for image data. In Proc. ICLR, 2023.
[6] H. Chang. H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang. K. Murphy, W. T.
Freeman. M. Rubinstein, Y. Li, and D. Krishnan. Muse: Text-to-image generation via masked generative transformers. In Proc. ICML, 2023.
[7] K. Crowson, S. A. Baumann, A. Birch, T. M. Abraham, D. Z. Kaplan, and E. Shippole. Scalable high-resolution pixcl-space image synthesis with hourglass diffusion transformers. In Proc. ICML, 2024, [8] Deng, W. Dong. R. Socher. L.-J. Li, K. Li, and Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. CVPR, 2009.
[9] P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. In Proc. NeurlPS, 2021. ai ai/2022/05.
[10] S. Dieleman. Guidance: A cheat code for diffuuson models. Blog post. ht
26/ le/guidance html, 2022.
[11] S. Gao, Zhou. M.-M. Cheng, and Yan. MDTv2: Masked diffusion transformer is strong image synthesizer. Proc. ICCV, 2023.
[12] D. Hendrycks and K. Gimpel. Gaussian emrrr linear units (GELUs). CoRR, abs/1600.08415. 2016.
10

---

[13] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge local Nash equilibrium. In Proc. NIPS, 2017. [14] J. Ho, W. Chan, C. Saharia. J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen Video: High definition video generation with diffusion models. CoRR, abs/2210.02303, 2022.
[15] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Proc. NeurlPS, 2020.
[16] J. Ho and T. Salimans. Classifier-free diffusion guidance. In Proc. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[17] S. Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention.
In Proc. NeurlPS, 2024.
[18] S. Hong, G. Lee, W. Jang, and S. Kim. Improving sample quality of diffusion models using self-attention
guidance. In Proc. ICCV, 2023.
[19] F. Hoogehoom, Heek, and Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In Proc. ICMI,, 2023.
[20] A. Hyvärinen. Estimation of non-normalized statistical models by score matching. JMI.R, 6(24):695-709, 2005.
[21] A. Jabri, J. Fleet, and T. Chen. Scalable adaptive computation for iterative generation. In Proc. ICMI.,
2023.
[22] A.
K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas. Gotta go fast when generating data with score-based models. CoRR, abs/2105. 11000..11000 14080, 2021.
[23] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-basse generative models. In Proc. NeurlPS, 2022.
[24] T. Karras, Aittala, J. Lehtinen, J. Hellsten, T. Aila, and S. Laine. Analyzing and improving the training dynamics of diffusion models. In Proc. CVPR, 2024.
[25] D. Kim, Y. Kim, S. J. Kwon, W. Kang, and L.-C. Moon. Refining generative process with discriminator guidance in score-based diffusion models. In Proc. ICML, 2023.
[26] Y. Kirstain, A. Polyak, U. Singer, S. Matiana. J. Penna, and O. Levy. Pick-a-Pic: An open dataset of user preferences for text-to- rext-to-image generation. In Proc. NeurlPS, 2023.
[27] T. Kynkäänniemi, M. Aittala, T. Karras, S. Laine, T. Aila, and J. Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. In Proc. NewrIPS, 2024.
[28] T. Kynkäänniemi, Karras, S. Laine, J. Lehtinen, and Aila. Improved precision and recall metric for assessing generative models. In Proc. NewrIPS, 2019.
[29] X. L. Li, A. Holtzman, D. Fried, P. Liang, Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proc. ACL, 2023.
[30] L. Liu, Y. Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds. In
Proc. ICLR, 2022.
[31] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In Proc. ECCV, 2022.
[32] S. Lyu. Interpretation and generalization of score matching. Proc. UAI, 2009.
[33] M. Marchesi. Megapixel size image creation using generative adversarial networks. CoRR, a 706.00082,
2017.
[34] P. Mishkin, L. Ahmad, M. Brundage, G. Krueger, and G. Sastry. DALL-E 2 preview - risks and limitations.
OpenAl. 2022.
[35] A. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Proc. ICML,, 2021,
[36] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, GLIDE Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. ICML., 2022.
[37] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafranice, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, El-Nouby, M. Assran, Ballas, W. Galuba, Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnacve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski.
DINOv2: Learning robust visual features without supervision. TMLR, 2024.
[38] W. Pechles and Xie. Scalable diffusion models with transformers. In Proc. ICCV, 2023.
[39] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In Proc.
ICLR, 2023.
[40] P. Pope, C. 7hu, A. Abdelkader, M. Goldblum, and T. Goldstein. The intrinsic dimension of images and its impact on learning. In Proc. ICLR, 2021.
[41] A. Ramesh. P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.061 25, 2022.
[42] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proc. CVPR, 2022.
[43] A. Sabour, S. Fidler, and K. Kreis. Align your steps: Optimizing sampling schedules in diffusion models. In Proc. ICML, 2024.
11

---

[44] S. Sadat, J. Buhmann, D. Bradley, O. Hilliges, and R. M. Weber. CADS: Unleashing the diversity of diffusion models through condition-annealed sampling. In Proc. ICLR, 2024.
[45] C. Saharia, W. Chan, S. Saxena, L. Li. J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S.
Mahdavi, R. G. Lopes, T. Salimans. J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic lext-to-image diffusion models with deep language understanding. In Proc. NeurlPS, 2022.
[46] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, In Proc. ICML. 2015.
[47] Song, C. Meng,
Ermon. Denoising diffusion implicit models. In Proc. ICLR. 2021.
[48] Y. Song and S. Frmon. Generative modeling by estimating gradients of the data distribution. In Proc.
NeurIPS, 2019.
[49] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative
modeling through stochastic differential equations. In Proc. ICLR, 2021.
[50] Stability AI. DeepPloyd IF. GitHub repository. https: sttps://github. com/deep-floyd/IF, 2023.
[51] G. Stein, J. C. Cresswell, R. Hosseinzadeh, Y. Sui, B. I.. Ross, V. Villecroze, 7. Liu, A. L.. Caterini, J. F. T. Taylor, and G. Loaiza-Ganem. Exposing flaws of generative model evaluation metries and their unfair treatment diffusion models. In Proc. NeurIPS, 2023.
[52] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and 7. Wojna. Rethinking the Inception architecture for computer vision. Proc. CVPR, 2016.
[53] P. Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(())166111677, 2011.
[54] X. Wang. N. Dufour, N. Andreou, M.-P. Cani, Abrevaya, D. Picard, and Kalegeiton. Analysis of classifier-free guidance weight schedulers. TMLR, 2024.
[55] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li. Human Preference Score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. CoRR, abs/2306.0922 093441, 2023.
[56] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control text-to-image diffusion models. In Proc. ICCV, 2023.
[57] Q. Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. In Proc. ICLR, 2023.
[58] C. Zheng and Y. Lan. Characteristic guidance: Non-lincar correction for diffusion model at large guidance scale. In Proc. ICML, 2024.
12

---

Appendices
A Additional results
Figure 6 shows additional results using DeepFloyd IF, similar to Figure
Interpolation between CFG and our method
Increasing guidance weight for our method
Increasing CFG weight
"A pirate ship sailing through clouds instead of the ocean"
Increasing guidance weight for our method
Increasing CPG weight
"A fairytale castle floating in the sky, lethered by ancient vines"
Figure 6: Additional results for DeepFloyd IF 500 The rows correspond to guidance weights E 21,2,3, 3, 4). CPG and our method (XL-sized model guided by M-sized one) on the leftmost and rightmost column, respectively. The middle columns correspond to blending between the two.
13

---

Classifer-free guidance
Classinir-ffre guidance
Guidance interval
Guidance interval
Autoguidance (ours)
Autoguidance (ours)
100
80
60
2,23
52:4
46.25
1.68
40
1.31
36.67
10m
1.0
1.5
2.0
2.5
3.0
30
1.0
1.5
2.0
2.5
3.0
(a) FID
(b) FDDINOV2
Figure 7: Sweep over guidance weight w using EDM2-S ImageNet-5122 512. The optimal EMA length was searched separately for the three methods and two metrics (FID and FDDINO.2).
CFG
Ours
Figure 8: Resulting variation for CFG and our method in Tree frog, Minibus, Mushroom classes of ImageNet-512 using EDM2-S (FDDpnoow-optimizze), In this image, we exaggerated the amount of guidance (w = 4) to make its effect on variation more clearly visible. This causes excessive saturation and other image artifacts, but clearly shows that CFG steers towards canonical templates, while our method preserves much greater variation.
Figure plots FID and FDDINOv2 as functions of guidance weight, showing that autoguidance is less sensitive to the exact choice of than CFG. Figure shows example grids that demonstrate that autoguidance retains much higher variation than CFG. DigurePprovice additional visualizations from our toy example, showing how the implied densities evolve during inference with CFG and autoguidance.
B Implementation details
We performed our main experiments on top of the publicly available EDM2 24 codebase using NVIDIA A100 GPUs, Python 3.1,.7, Py 2.2.0, CUDA 11.8, and CuDNN 8.9.7. Since method only involves using different guiding model during sampling, we were able to perform all
httpss//ggthub.ccc///aaaab
14

---

(b) CFG (x(x;o) o)
(c) Autoguidance p)(x(c;
(d) CFG ratio pi/po
(e) Autoguidance ratio p1/Po
(a) (a)c1x))))))
0.24
0. 0.00
0 0.00
0= 0.01
Figure 9: Progression implied learned densities during sampling over various setup similar to Figure 2 Contours of the corresponding ground truth distributions are also shown. (a) Main model density p1 (x|c; r). (b) Unconditional guiding model density po (o(x; o in CFG. (c) Conditional guiding model density Po (x|c; a)) in autoguidance. (d) With CFG, guidance towards higher ratio P1. Po pushes samples towards top right, especially at high a (top rows). (e) With autoguidance, this anomalous effect does not occur and samples cover the entire class
15

---

measurements using the existing command -inee scripts. We only had to modify one line of code in the sampling loop to pass the class label to the guiding network in addition the main network. Algorithm| demonstrates the steps needed to reproduce one of our results from Table I the rest can be reproduced by repeating the same steps with different models and hyperparameters, as indicated in the table. For the four cases where a pre-trained model was unavailable, we trained the models ourselves as detailed in Algorithm 2
B.1 Hyperparameter search
We optimized the autoguidance parameters for each configuration in Table and Figure 3 using automated grid search. We performed the search separately for FID and FDDINOv2 across the space of five parameters:
Model capacity: All available model capacities (EDM2-XXS, EDM2-XS, etc.) up to and including the capacity of the main model.
Training time: Number of training images in powers of two. We found that the sweet spot was always within the range {2 227 22 2 there was no need to go for lower or higher values.
Guidance weight: All values within [1.00,3 3.50) at regular intervals of 0.05.
EMA lengths: All values within [0.010, 0.250) at regular intervals of 0.005. We treated the EMA length of the main model and the guiding model as two separate parameters.
In order to reduce the computational workload, we pruned the search space by considering only a local neighborhood of parameters around the best FID or FDDINO<2 found thus far. Whenever the result improved, we placed new local grid around the corresponding parameters, resulting in gradual convergence towards the global optimum. Once reached the optimum, further re-evaluated the nearby parameter choices two more times to account for the effect of random noise. As such, each result reported in Table T and Figure represents the best of three evaluations. The typical range of random variation is indicated by the shaded regions in Figure 3
In total, the grid search resulted in roughly 30,000 metric evaluations across all of our configurations. In each metric evaluation, the main cost comes from generating 50,000 random images, which takes around 30-60 minutes using eight A100 GPUs and consumes approximately 2-5 kWh of energy, depending on model size. The overall energy consumption of our entire project was thus in ballpark of 60-150 MWh.
B.2 DeepFloyd IF experiments
In order to apply CFG and autoguidance simultaneously, extend Equation B to cover multiple guiding models as proposed by Liu et al. [31]. In this case, we have three models: the main model Dm. an unconditional model Du, and reduced-capacio conditional model D.. The guided denoising result is then defined as
Du ( c. (
Ew, (u::1)(DDmxx:00) (); -xxxx,))))
(6)
iEfu.c}
where Wa and w. correspond to the guidance weights for CFG and autoguidance, respectively. To inter- polate between the two methods, we further define wa == (1 - a (w- +1 and we == a(w 1) + 1, where w indicates the desired total amount of guidance and a E [0, a linear interpolation factor.
DeepFloyd IF (50) uses a three-stage cascade: a base model followed by two super-resolution stages. We apply our method only to the base model, while the super-resolution stages always use CFG with their default weights (4 and 9). We use their Stochastic DDPM sampler with default settings: 100. 75, 50 steps for the base model and subsequent super-resolution stages, respectively. Dynamic thresholding [45] is used for all stages.
C Details of the 2D toy example
In this section, we describe the construction of the 2D toy dataset used in the analysis of Section as well as the associated model architecture, training setup, and sampling parameters. The related code is available at https: // http://ttt....o/viii.tt www.e//ww////annnnn.ea
16

---

Algorithm Reproducing our FID result for the "Autoguidance (XS, T/16)" row in Table [
1 Bosmload EDN2 source cade.
2 git clone attps://gitbub. con/WVlabs/ aedaaas/dddd...e edm2.
cd edn2
32ecad3
5
6 Patch the sampler so that class labels are passed in to the guiding network.
7 sed -1 (x, ()/gnet(x, labels)/s" generate_inages.py
Download necessary nodels.
rclone copy --prggress --httpurr url https://nvlabs fddiiidivvidaa coneedn/
stpp:rawwsmmpphhoss/eme... ing5122// raw www.ssaashotts/cdnnnnggiii
rclone copy --progress --http-url htap:://wlabb--flcdd...e avidia..iiiii.i................. con/edm2
sttp::raa-sngpshots/elme
Reconstruct the corrosponding post-hoc EMA uadels.
python reconstruct_phem.. seadrrrra--sapphotss/ooee -ngg51222 - -ondd-oooooo-attttuadddeeeeaeeee
outpreffir-nng112 -01kking-2147483 -oustd--0.07
python reconstruct py --iddrrrunn-mapshoc//ddtt
outdir- /ttp://ouuuuueeeeeeeeeeeeeeeeeee
putprefirsing512 -outinng-134217 -outstd=0.125
20
Generate 50, 800 images using 8 GPUs.
torchrun --standalone 8 generate.images
--subdirs
watpeen/img1----21-7555555 met.aattgguiacccc/hnmm/iiiggg 0134217 125. --guidance-2..11 99.80..007...00
25
Calculate FID.
python calculate metrics.py - -
For each of the two classes c, we model the fractal-like data distribution mixture of Gaussians Me = (for). { (E.)). where di, μ4, and Σ, represent the weight, mean, and x covariance matrix of each component i, respectively. This lets calculate the ground truth scores and probability densities analytically and, consequently, to visualize them without making any additional assumptions. The probability density for given class is given by
Dataset.
Paata Paata(x|c) (x|c) = Σ dxNN(;; E.), where
SEMo
exp (- 1 [x (-2(x-1) -x
N(x /2t))2
Applying heat diffusion to Pdata (x/c), we obtain a sequence of increasingly smoothed densities p(x(c; a) parameterized by noise level o:
p(x|c;o) (xx where Σi = E, + O21
(9)
ieMe
The score function of p(x|c; 0) then given by
Vx logp((//; Eiem. d:N(x: Mi,) (E)a)) (mi Σiem. N.N (x; Ej,0)
(10)
We construct Me to represent thin tree-like structure by starting with one main "branch" and recursively subdividing into smaller ones. Each branch is represented by anisotropic Gaus- sian components and the subdivision is performed 6 times, decaying é after each subdivision and slightly randonizing the lengths and orientations of the two resulting sub-branches. This yields 127x8 1016 components per class and 1016x2 = 2032 components in total. We define the coordinate system so that the mean and standard deviation of Pasm marginalized over e, are equal to and data 0.5 along each axis, respectively, matching the recommendations by Karras et al. [23]
17

---

Algorithm 2 Training the additional EDM2 models needed in Section
1 Uncondi EDX2-5 S with Imagellet 5512, used as a main model in Table 1.
2 torchrun --nnodes=d --aproe per per_ node=8 train train.edn2.py py \
o.artdrraaroauttaacr/ttttaaaaaaa -<preset=edn2 ing5512- -duration=2848M:
-<date-darasets/aaai2222a
--cond=0
5
6 Uncondi tional EDX2-XS with Inagelett-64, used as a guiding nodel in Table 1.
7 torchrun -- nmodes=4 perroeeper_noder train_edn2-Py uncond data-ddraasttssinndd.zi c-oon-8
purddrr-ouoouuidncce/tnnat prreset-ddd2 img64-s channels-128 --12-8.8128 ddrationn2049Ki
11 Conditional EDM2-XS with Imageket-64, used as guiding model in Table 1.
torchrun -nodess= --pproc_per node=8 train.edn2.pp \
-aadddrrattouiddde///rtii/maia --datardatasets/inneee
--cond=1 presett=ddd2 ing61-5 -channels=128 -1r=8.8120 -dduratio=:11M
# Conditional FDM2-XXS xxS with InageNet-512, used as guiding model in Figure 3b.
torchrun noodes ppro__pe__node train_edn2.py
oottdrrauutooiiddcc///rraaaaaaaa -daaa-datastttiinni-d---aaa --cond-1
preset-edn2 ing512-xx -channels=6d --1r=8.0170 -duration-1121
Models. We implement the denoiser models Do and D2 as simple multi-layer perceptrons, utilizing the
design principles from EDM2 (24]. To be able to visualize the implied probability densities Figure 2 we design the model interface so that for a given noisy sample, each model outputs single scalar representing the logarithm of the corresponding unnormalized probability density, as opposed to directly outputting the denoised sample or the score vector. Concretely, let us denote the output of given model by Gø(x; a, c). The corresponding normalized probability density then given by
(x(x(c;o) = exp (Ga(x;o,c)) / exp (Go(x; dx.
(11)
By virtue of defining Go this way, we can derive the score vector, and by extension, the denoised sample, from Gg through automatic differentiation:
Vx [olgpp(()(;;o) = G=
(12)
Do(x;o,c)= = =
(13)
Besides Equation 12 also tried out the alternative formulations where the model outputs the score vector or the denoised sample directly. The results produced by all these variants were qualitatively more less identical; chose go with the formulation above purely for convenience.
To connect the above definition of Gg the raw network layers, we apply preconditioning using the same general principles as in EDM [23] Denoting the function represented by the raw network layers as Fe. we define Gg as
90 Σ Fo, x"; logo, c)² where x"
Go(x; =
(14)
02
σ
data
and the sum is taken over the output features of Fo. We scale the output of Fo by learned scaling factor go that we initialize to zero.
The goal of Equation 14 is to satisfy the following three requirements:
The input of Fo should have zero mean and unit magnitude. This is achieved through the division by /o² + of
After initialization, Go should represent the best possible first-order approximation of the
correct solution. This is achieved through the ||x*|] term, well as the fact that go after initialization.
After training, vgo Fo should have approximately unit magnitude. This is achieved through the division by on.
18

---

In practice, we use an MLP with one input layer and four hidden layers, interspersed with SiLU 12 activation functions and implemented using the magnitude- maagitude-preserving preserving primitives from EDM2 24
The input is 4-dimensional vector x x ; log σ; 1] and the output of each hidden layer has features, where n = 64 for Dj and 32 for 1
Training. Given that we have the exact score function of the ground truth distribution readily simplicity and
available (Equation 9). we train the models using exact score matching increased robustness. We thus define the loss function as
(()) po o) 0² Vx log - -Vx lo
(15)
where Purin is realized log(a) [[P Paud) [23] As alternative to exact score matching, we also experimented with the more commonly used denoising score matching, but did not observe any noticeable differences in model behavior training dynamics.
We train D₃ for 4096 iterations using a batch size of 4096 samples, and Do for 512 iterations. We P mean 2.3 and Psu 1.5, and use agd// / //max(t/tee; 1) learning rate decay schedule with Ove 0.01 and fer== 512 iterations, along with a power function EMA profile (24] with de 0.010. Overall, the setup robust with respect to the hyperparameters; the phenomena illustrated in Figures] and remain unchanged across wide range of parameter choices.
Sampling. We use the standard EDM sampler 23 with N 32 Heun steps (NFE 63), min 0.002, mmx 5, and p 7. We chose the values of N and max to be much higher than what actually needed for this dataset order avoid potential discretization errors from affecting our conclusions. In Figure T. we set w 4 for CFG and = 3 for autoguidance, and multiply the score vectors (Equation by 1.40 for naive truncation. In Figure we set w 4 and a = 0.03.
D Broader societal impact
Generative modeling, including images and videos, has significant misuse potential. can trigger negative consequences within the society in several ways. The primary concerns include various types of disinformation, but also the potential to amplify stereotypes and unwanted biases (4. Our improvements to the sample quality can make the results even more believable, even when used for disinformation. That said, do unlock any novel uses of the technology.
E
Licenses
EDM2 models
24
Creative Commons BY-NC-SA 4.0 license
DeepFloyd IF models 50
Modified MIT license
Stable Diffusion VAE model
CreativeML Open RAIL++-M license
InceptionV3 model [52
Apache 2.0 license
DINOv2 model [37
Apache 2.0 license
ImageNet dataset 8
Custom non-commercial license
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both the quantitative and qualitative claims are substantiated by our results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Primarily covered Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
20

---

Justification: Our equations build on previously known results, and highlight their practical aspects through mostly readily verifiable algebraic manipulations. We do not explicitly present all details of the derivations, and assume that the previous results are sufficiently rigorously proven in the respective literature.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendix B describes how publicly available code can be used to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
21

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Appendix describes how publicly available code can be used reproduce our results. The models that are not already publicly available, as well as the implementation of the 2D toy example, will be made available prior to the conference.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (ittpss//nnis.co cc/
wwt.://wbwwiiiiiooooooooooo....i for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.com httpps.cc/public/uuudeeee http://guiee//Coeeeeeeee http//oeeeeuiaaeee wwt.sSubmmssioooooooooooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 5 and Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Shaded regions in Figure 3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :
Answer: [Yes]
Justification: We have reviewed the Code of Ethics, and do not find concerns beyond those relating to the general topic of generative modeling of images.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: While our works falls into a high-risk category, the nature of our work (basically reusing pre-trained models) such that are not really in a position introduce new safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwtpsss.pppppsstteeeeeeo thcode www.coo///aaaaa com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "b2sGVPaSXK", "reviewer": "Reviewer_oNdT", "summary": "The paper proposes autoguidance, a new method that simulates the behavior of classifier-free guidance by using a worse version of the model itself instead of an unconditional module. The authors demonstrate that inconsistencies between the predictions from the conditional and unconditional parts of CFG are responsible for some of its shortcomings such as lower variation in generated results. By using a worse version of the same conditional model, the authors show that such inconsistencies will be reduced, and sampling trajectories will converge toward samples that are closer in distribution to the data. Therefore, the paper concludes that compared to CFG, autoguidance improves quality without sacrificing diversity.", "questions": "1. Can you provide precision/recall (PR) curves for your method vs. CFG? While FID considers both aspects, the PR curve shows the impact of guidance on quality and diversity more directly.\n\n2. In addition to improvement in quality, CFG is also heavily used to improve text-image alignment. Can you provide a more detailed experiment on how autoguidance affects this aspect? From the images, it seems that some degree of CFG is still needed for optimal text-image alignment.\n\n3. Can you provide some intuition on how to choose the guiding model besides grid-search? It seems very costly to train multiple different models just to see which one works better as the guidance model, especially if the same method is applied to text-to-image models trained on massive datasets.\n\n4. How does the method compare to algorithms designed for increasing the diversity of CFG, e.g., [1, 2, 3]? In other words, how much of the improvement comes from increased diversity and how much comes from the fact that autoguidance provides better image quality overall (due to fewer inconsistencies between the updates)? It would be great if the authors could provide a section studying this in the final version. Currently, it is only mentioned that autoguidance is beneficial at all noise scales compared to CFG, but I believe a more detailed comparison with visual examples would strengthen the paper.\n\n[1] Kynkäänniemi T, Aittala M, Karras T, Laine S, Aila T, Lehtinen J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724. 2024 Apr 11.\n\n[2] Wang X, Dufour N, Andreou N, Cani MP, Abrevaya VF, Picard D, Kalogeiton V. Analysis of Classifier-Free Guidance Weight Schedulers. arXiv preprint arXiv:2404.13040. 2024 Apr 19.\n\n[3] Sadat S, Buhmann J, Bradley D, Hilliges O, Weber RM. CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling. InThe Twelfth International Conference on Learning Representations.", "limitations": "The submission has properly discussed the limitations and social impacts.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper studies an important topic. Since CFG is widely used in current diffusion models, overcoming its shortcomings will have a noticeable impact in the future.\n\n- The method is well-motivated through controlled experiments that shed light on the behavior of CFG and how autoguidance improves it in those aspects.\n\n- The experiments are well-organized and clearly demonstrate the impact of different components in autoguidance.\n\n- The paper is well-written and enjoyable to read.", "weaknesses": "- More visual examples are needed to show how the diversity of generations changes as the guidance scale increases. In the final version, please include a batch of examples with a fixed condition and compare the sampling with CFG and autoguidance to better demonstrate the disentanglement between image quality and diversity in autoguidance.\n\n- The method is not readily applicable to pretrained diffusion models such as Stable Diffusion. This might limit the current use cases of autoguidance. However, this issue will likely not persist in the long run, as we may see the release of pretrained models compatible with autoguidance. Therefore, this weakness does not affect the long-term impact of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Regarding the concerns and questions:\n\n> More visual examples are needed to show how the diversity of generations changes as the guidance scale increases. In the final version, please include a batch of examples with a fixed condition and compare the sampling with CFG and autoguidance to better demonstrate the disentanglement between image quality and diversity in autoguidance.\n\nWe will include a grid of fixed-condition examples between CFG and autoguidance in the final revision.\n\n> Q1. Can you provide precision/recall (PR) curves for your method vs. CFG? While FID considers both aspects, the PR curve shows the impact of guidance on quality and diversity more directly.\n\nWe did not measure precision and recall, so we unfortunately don’t have the data necessary to produce such curves.\n\n> Q2. In addition to improvement in quality, CFG is also heavily used to improve text-image alignment. Can you provide a more detailed experiment on how autoguidance affects this aspect? From the images, it seems that some degree of CFG is still needed for optimal text-image alignment.\n\nWe did not run any prompt alignment metrics, so we have no quantitative data about this. Intuitively, it seems probable that the effect of autoguidance on prompt alignment is smaller than with CFG, because both models are being conditioned with the text prompt. However, as the guiding model is smaller and/or less trained, it probably responds to the condition less strongly than the main model, and thus the prompt is probably emphasized to some degree as the guidance weight is increased.\n\nIn the paper, we advocate mixing autoguidance with CFG for further creative control and provide a simple method for doing so (Appendix B.2).\n\n> Q3. Can you provide some intuition on how to choose the guiding model besides grid-search? It seems very costly to train multiple different models just to see which one works better as the guidance model, especially if the same method is applied to text-to-image models trained on massive datasets.\n\nBased on our experiments, a model of a third to a half the size of the main model is a good starting point, and the evaluations should begin around 1/16 of training iterations, or perhaps even earlier for very small models. As seen in Figure 3(a, b), doubling or halving the capacity or training time doesn’t result in any sort of catastrophic quality drop, so these parameters are not overly sensitive. That said, we do not have enough data at this point to establish proper scaling laws.\n\n> Q4. How does the method compare to algorithms designed for increasing the diversity of CFG, e.g., [1, 2, 3]? ...\n\nSo far we have compared autoguidance only with the interval method [1], which we did not find benecifial in combination. A key benefit from these schedules appears to be the suppression of CFG at high noise levels, where its image quality benefit is overshadowed by the undesirable reduction in variation that is caused by large differences in the content of the differently conditioned distributions. In contrast, autoguidance is not expected to suffer from this problem at high noise levels, as both models target the same distribution. Nevertheless, exploring further options would be a natural topic for a follow-up paper; we shall include this in the future work section", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uob6ZZcXVd", "reviewer": "Reviewer_m4Uw", "summary": "This paper presents a novel perspective on classifier-free guidance (CFG). It improves the generation quality by directing the generative model towards high-probability regions. The authors identify that this improvement stems from the quality difference between the conditional and unconditional components in CFG. Building on this insight, the paper introduces autoguidance, a new sampling algorithm that utilizes both the diffusion model and a bad version of it. Experimental results show the superiority of this method.", "questions": "1. I am still not quite clear about the necessity of the similar degradation of $p_1(x|c;\\sigma)$ and $p_0(x|c;\\sigma)$ empirically. (I am convinced by your synthetic experiment). Even if $p_1(x|c;\\sigma)$ and $p_0(x|c;\\sigma)$ suffer from different kinds of degradation, empirically the ratio of them is still possible to pull the sampling trajectory towards the high-likelihood region. If possible, could you give some toy mathematical examples about this?\n\n2. For Figure 1(e), the author applies autoguidance to the toy model. Could you also visualize the $p_0$, $p_1$ and $p_1/p_0$ in the autoguidance setting? This will help us better visualize what the similar degradation looks like.", "limitations": "The paper covers limitations and societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper employs an intuitive toy model to support its empirical findings. The specific long-tail, tree-shaped mixture of Gaussian distributions used in this model could be beneficial for future research on generative models.\n\n2. The empirical observations and proposed methods are coherent. The paper uncovers a new mechanism within CFG and enhances it through the proposed method.\n\n3. The proposed method is both simple and powerful, significantly improving the SOTA generation quality on the ImageNet dataset. This new approach has the potential to inspire further related research.", "weaknesses": "1. For the experiment, the paper lacks some quantitative comparison for their proposed method of text-to-image diffusion model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Regarding the questions:\n\n> Q1. I am still not quite clear about the necessity of the similar degradation of $p_1(x|c;\\sigma)$ and $p_0(x|c;\\sigma)$ empirically. (I am convinced by your synthetic experiment). Even if $p_1(x|c;\\sigma)$ and $p_0(x|c;\\sigma)$ suffer from different kinds of degradation, empirically the ratio of them is still possible to pull the sampling trajectory towards the high-likelihood region. If possible, could you give some toy mathematical examples about this?\n\nLet us construct a toy scenario where the ratio of two differently degraded densities would yield a misleading guidance signal. Assume that the true distribution is a unit 2D Gaussian with diagonal covariances $[1,\\ 1]$, and the two degraded versions have diagonal covariances $[1+e,\\ 1]$ and $[1,\\ 1+e]$ for some relatively small $e$. Now, guidance between these densities would push the samples inward along one axis and outward (towards lower likelihood) along the other, despite them both being centered around the correct distribution. Similarly, offsetting the means to unrelated directions would induce an overall force towards some global direction, rather than consistently towards the origin.\n\nIn a more abstract sense, the beneficial degradations appear to push and spread the densities along locally consistent directions as a function of the degradation strength, but this is ultimately an empirical finding. On the other hand, entirely different types of degradations have a lot of room for mutually inconsistent behavior.\n\n> Q2. For Figure 1(e), the author applies autoguidance to the toy model. Could you also visualize the $p_0$, $p_1$ and $p_1/p_0$ in the autoguidance setting? This will help us better visualize what the similar degradation looks like.\n\nThe probability ratio in the region shown in Figure 2 looks fairly similar with CFG and autoguidance, but we could try to construct a visualization that focuses on the regions with visible differences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLpocWc3dJ", "reviewer": "Reviewer_rSGz", "summary": "The paper introduces a novel conditioning method as an alternative to Classifier-Free Guidance (CFG), which allows for better control over image quality generation without compromising data diversity. This approach involves guiding the diffusion model generation with a lower-quality model (either less trained or with reduced capacity) instead of an unconditional model. The authors compare CFG with their guidance approach on ImageNet.", "questions": "* In the CFG paper, a complete table of FID and IS values at various omega settings is provided (omega={1,1.5,2,2.5,3,3.5,4...}). I would like to see a similar comparison between the auto-guidance approach and CFG.\n* I am interested in seeing the use of a quantized model as the low-quality model for auto-guidance.\n* Additional examples of generated images or access to the code would be beneficial for comparing the auto-guidance method with CFG.", "limitations": "The method requires training two diffusion models as opposed to just one with CFG. This difference is critical when scaling up the training of foundation models.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and clear.\n* There is a thorough analysis of CFG behavior and its limitations. The toy example comparing CFG with the authors' method is particularly insightful.", "weaknesses": "* The results on images do not clearly demonstrate the distribution coverage shown in the toy example. It appears that the low-quality model provides low-frequency guidance during generation, while the high-quality model focuses more on details. This approach results in sacrificing diversity for quality, contrary to what is depicted in Fig. 1. Additionally, Table 1 should include the Inception score in addition to the FID. Peculiar also the choice of omegas in Table 1; I wonder how the authors choose these values. A fair comparison between CFG and their method would be sufficient with omega={1,1.5,2,2.5,...}\n* In CFG, only one model needs to be trained to achieve conditioning. In this new approach, achieving greater diversity requires training two distinct models. In line 174, the authors mention: \"such as low capacity and/or under-training.\" I would expect that the low-capacity model could function in this auto-guiding setup, but an under-trained approach might face significant generalization issues. The key question is: \"How can we determine the training data size needed for the lower quality version to enhance a well-trained model?\" Considering model degradation, model quantization could be a viable solution, potentially eliminating the need for an extra model. Demonstrating how an f16 model could enhance the diversity of an f32 model using this auto-guiding method would be an interesting experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Regarding the concerns and questions:\n\n> The results on images do not clearly demonstrate the distribution coverage shown in the toy example. It appears that the low-quality model provides low-frequency guidance during generation, while the high-quality model focuses more on details. This approach results in sacrificing diversity for quality, contrary to what is depicted in Fig. 1. Additionally, Table 1 should include the Inception score in addition to the FID. Peculiar also the choice of omegas in Table 1; I wonder how the authors choose these values. A fair comparison between CFG and their method would be sufficient with omega={1,1.5,2,2.5,...}\n\nAs far as we know, there is no simple relationship between the frequency bands of the sampled image and the roles of the main and guiding models. Given that FID is very sensitive to diversity [1], our lower FIDs are a strong indication that diversity is not lost. We did not measure IS separately, as it is known to be largely consistent with FID (see Figs. 5a, 11b of the EDM2 paper [2]), at least with the EDM2 models that we use in the quantitative measurements.\n\nGuidance weights in Table 1 are the ones that gave the best results according to the hyperparameter search outlined in Appendix B.1. We are happy to add a table or a plot with a range of guidance weights in the appendix.\n\n_[1] Kynkäänniemi et al.: Improved precision and recall metric for assessing generative models. In Proc. NeurIPS, 2019._  \n_[2] Karras et al.: Analyzing and improving the training dynamics of diffusion models. In Proc. CVPR, 2024._\n\n> In CFG, only one model needs to be trained to achieve conditioning. In this new approach, achieving greater diversity requires training two distinct models. In line 174, the authors mention: \"such as low capacity and/or under-training.\" I would expect that the low-capacity model could function in this auto-guiding setup, but an under-trained approach might face significant generalization issues. ...\n\nWith autoguidance, the majority of the benefits can be obtained by using an earlier training snapshot of the main model as the guiding model (Table 1, row “reduce training only”, also Section 5.1), in which case no additional training is required. Under-training is therefore a practical approach for creating effective guiding models.\n\nIn contrast, reducing the amount of training data for the guiding model did not seem to yield a benefit (see end of Section 5.1). We did not consider reducing the amount of training data for the guiding model as a goal in itself, as the full dataset is used for training the main high-quality model in any case. That said, it may be possible to reduce the data at least somewhat when training the low-quality model without ill effects.\n\n> Q1. In the CFG paper, a complete table of FID and IS values at various omega settings is provided (omega={1,1.5,2,2.5,3,3.5,4...}). I would like to see a similar comparison between the auto-guidance approach and CFG.\n\nWe are happy to add a table or a plot comparing the FID of autoguidance and CFG across a range of guidance weights in the final version.\n\n> Q2. I am interested in seeing the use of a quantized model as the low-quality model for auto-guidance.\n\nAccording to our initial tests, increased quantization does not yield a model that could be used as the low-quality guiding model (see end of Section 5.1).\n\n> Q3. Additional examples of generated images or access to the code would be beneficial for comparing the auto-guidance method with CFG.\n\nThe code will be released after the review cycle.\n\n> The method requires training two diffusion models as opposed to just one with CFG. This difference is critical when scaling up the training of foundation models.\n\nHaving to train a separate guiding model in order to obtain full benefits of our method is indeed a limitation, but when using a smaller model and shorter training time for the guiding model, the additional training cost is modest. For example, the EDM2-M model trains approximately 2.7x as fast as EDM2-XXL per iteration, and we train it for 1/3.5 of iterations, so the additional cost is around +11% of training the main model. For the EDM2-S/XS pair used in most of our experiments, the added training cost is only +3.6%. We shall clarify this in the paper.\n\nAlso, as discussed above, using an earlier training snapshot of the main model as the guiding model yields most of the benefits of autoguidance without requiring any additional training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this work presents a novel perspective on classifier-free guidance in diffusion models, a simple and powerful new method (autoguidance), and an extremely clear presentation as well as insightful set of experiments. We believe this paper is a valuable contribution to understanding and improving guidance in diffusion models.
Decision: Accept (oral)