Elucidating the design space of classifier-guided diffusion generation
Authors: Jiajun Ma
Tianyang Hu
Wenjia Wang
Jiacheng Sun
Venue: ICLR 2024 poster (2024)
Keywords: conditional diffusion sampling
classifier guidance
Primary area: generative models
TL;DR: Through a comprehensive investigation into the design space of classifier guidance in diffusion generation, we achieved significant improvements over existing guidance schemes by leveraging off-the-shelf classifiers in a training-free fashion.

Abstract: Guidance in conditional diffusion generation is of great importance for sample quality and controllability. 
However, existing guidance schemes are to be desired. 
On one hand, mainstream methods such as classifier guidance and classifier-free guidance both require extra training with labeled data, which is time-consuming and unable to adapt to new conditions.
On the other hand, training-free methods such as universal guidance, though more flexible, have yet to demonstrate comparable performance. 
In this work, through a comprehensive investigation into the design space, we show that it is possible to achieve significant performance improvements over existing guidance schemes by leveraging off-the-shelf classifiers in a training-free fashion, enjoying the best of both worlds. 
Employing calibration as a general guideline, we propose several pre-conditioning techniques to better exploit pretrained off-the-shelf classifiers for guiding diffusion generation. 
Extensive experiments on ImageNet validate our proposed method, showing that state-of-the-art (SOTA) diffusion models (DDPM, EDM, DiT) can be further improved (up to 20\%) using off-the-shelf classifiers with barely any extra computational cost.
With the proliferation of publicly available pretrained classifiers, our proposed approach has great potential and can be readily scaled up to text-to-image generation tasks.

Published as a conference paper at ICLR 2024
ELUCIDATING THE DESIGN SPACE OF CLASSIFIER GUIDED DIFFUSION GENERATION
Jiajun Ma Tianyang ; Wenjia Jiiccheng Sun?
Hong Kong University of Science and Technology
Hong Kong University of Science and Technology (Guangzhou)
Huawei Noah's Ark Lab
ABSTRACT
Guidance in conditional diffusion generation of great importance for sample quality and controllability. However, existing guidance schemes are be desired. On one hand, mainstream methods such as classifier guidance and classifier free guidance both require extra training with labeled data, which is time-consuming and unable to adapt to new conditions. On the other hand, training-free meth- ods such as universal guidance, though more flexible, have yet demonstrate comparable performance. In this work, through comprehensive investigation into the design space, we show that it is possible achieve significant perfor- mance improvements over existing guidance schemes by leveraging aff-the-shelf classifiers in a training-free fashion, enjoying the best of both worlds. Em ploying calibration as general guideline, we propose several pre-conditioning techniques to better exploit pretrained off-the-shelf classifiers for guiding dif- fusion generation. Extensive experiments on ImageNet validate proposed method, showing that state-of-the-art (SOTA) diffusion models (DDPM, EDM, DiT) can be further improved (up to 20%) using off-the-shelf classifiers with barely any extra computational cost. With the proliferation publicly avail- able pretrained classifiers, proposed approach has great potential and be readily scaled up to text-to-image generation tasks. The code is available at Ctttss///githubb.cmm//..cee http:///wwo.....oeeeeeeeeeeeee/e
INTRODUCTION
Diffusion Probabilistic Model (DPM) (Sohl-Dickstein al. 2015 Ho
2020
Song al. 2020b
is powerful generative model that employs a forward ditlusionn process to gradually add noise to data and generate new data from noise through reversed process. DPM's exceptional sample quality and scalability have significantly contributed to the success of Artificial Intelligence Generated
Content (AIGC) in various domains, including images (Saharia et al. 2022 Ramesh et al. 2022
2021
Rombach et al.) 2022), videos (Ho et al., 2022b Singer et al. 2022 Ho et al.) 2022a Molad
ct al. 2023). and 3D objects (Poole et al. 2022 Lin et al. 2023 Wang et al. (023).
Conditional generation is one of the core tasks of AIGC. With the diffusion formulation, condition injection, especially the classical class condition, becomes more transparent as it can be modeled an extra term during the reverse process. To align with the diffusion process, Dhariwal & classifier and demonstrated significant quality improvement over the unguided baseline. [Ho & Salimans (2022) later proposed classifier-free guidance (CFG) to implicitly implement the classifier gradient with the score function difference and achieved superior performance in the classical class-conditional image generation. However, both CG and CFG require extra training with labeled data, which is not only time-consuming but also practically cumbersome, especially when adapting new conditions. To reduce computational costs, training-fic free guidance methods have been proposed (Bansal et al.) 2023) that take advantage of pretrained discriminative models. Despite the improved flexibility, they have not demonstrated convincing performance compared to CG & CFG in formal quantitative evaluation of guiding diffusion generation. There seems to be an irreconcilable trade-off between performance and flexibility and the current guidance schemes are still to be desired.
Nichol (02T)) proposed classifier guidance (CG) to train a hime/noise-dependee
"Correspondence Tianyang Hu (hutianyang. wwwpguuuuuouuuo.oooo outlook.com)

---

Published as a conference paper at ICLR 2024
In this work, focus the classical dlass-conditional diffusion generation setting and investigate the ideal method for guiding the diffusion generation, considering the following criteria: (1) Efficiency with training- free effort: (2) Superior performance in formal evaluation of guided conditional diffusion generation; and (3) Flexibility and adaptability to various new conditions. To this end, we delve into the properties of classifiers and rethink the design space of classifier guidance for diffusion generation. Through comprehensive investigation both empirically and theoretically, we reveal that: (a) Trained/finetuned dimedeepenennt classifiers have limitations; (b) Off-the-shelff classifiers' potential is far from realized.
While existing methods primarily emphasize classifier accuracy, an ideal classifier should not only provide precise label predictions but also accurate estimations of the gradient of the logarithm of the conditional probability. Given the challenges in efficiently estimating the gradient, classifier calibration emerges as promising alternative, which quantifies how well classifier recovers the ground truth conditional probability. We show that under certain smoothness conditions, a smaller calibration error leads to better estimation of the classifier propose the integral calibration error (ECE) to assess classifier guidance throughout the diffusion reverse process and subsequently, effective pre-conditioning techniques better prepare the classifier for guidance. Interestingly, our experiments reveal that trained/fine- -uned classifiers (Dhariwal & Nichol 202T) are less calibrated than off-the- ones when the noise level is high (Figure D.
4.1 Accordingly, we
Beyond a good probability estimation, an ideal classifier guidance should also integrate seamlessly with the conditional diffusion generation process. Our investigation reveals that the naive imple- mentation of classifier guidance will fade as the diffusion denoising step progresses, resulting in ineffective utilization of the classifier (Figure D. To address this newly discovered issue, we propose a simple weighing strategy to balance the joint and conditional guidance. which significantly corrects the guidance direction and results in significantly improved sample quality.
To sum up, this work aims to elucidate the design space of classifier dlassifier-uiieed diffusion generation. We carry out a comprehensive analysis of the classifier guidance, considering calibration, smoothness, guidance direction, and scheduling. Accordingly, we propose accessible and universal designs that significantly enhance guided sampling performance. Extensive experiments on ImageNet with various DPMs (DDPM, EDM, and DiT) validate our proposed method, showcasing that using off-the-shelf classifiers can consistently outperform both CG and CFG. Additionally, our method can be applied in CFG and further enhance its generation quality. We also demonstrate the scalability and universality of our method in text-to-image scenarios by incorporating CLIP (Radford et al. [202T) guidance with our design. We point out that the operation of increasing recurrent guidance (Bansal et al. 2023 does not fully exploit the potential and comes at the expense of increasing sampling time.
RELATED WORK
Diffusion models have gained considerable attention due to their potential. Ho al. (0020); Nichol
& Dhariwal (202T); Song et al. (2020a); Peebles & Xie (2022); Karras et al. (2022) demonstrated
diffusion models' capacity in generating high-quality samples. Dhariwal & Nichol ( (202) introduced
fine-tuned time-dependent U-Net (Ronneberger et al., 2015) classifiers to guide diffusion model
sampling, resulting in significant improvements. [H & Salimans (2022) introduced classifier-free
diffusion, which has been widely accepted (Rombach et al., 2022; Peebles & Xie 2022, Ramesh
et al. 2022 for generating high-quality samples using both conditional and unconditional diffusion models for inference. In our work, we demonstrate that our off- the-shelf classifier -guided conditional diffusion not only significantly outperforms classifier-free diffusion but also enhances the performance of the classifier-free diffusion model (DiT (Peebles & Xie 2022)). For guidance in other modalities, Nichol et al. (202T) proposed GLIDE, which utilizes fine-tuned noised CLIP for text-conditioned diffusion sampling. However, it requires the fine-tuning of CLIP on carefully selected noisy data.
In addition, research has explored using off-the-shelf checkpoints to guide diffusion sampling. For example, Wallace et al. (2023) examined the plug-and-play of the classifier guidance, demanding the diffusion architecture to be invertible. Epstein et al. (2023) introduced -afidguidence constraints for CLIP Aul--guided sampling in objects' editing. Bansal al. 2023) applied recurrent guidance operation to universal pretrained models, but only provided demo figures without quantitative evaluation. However, our experiments in Table 1 reveal that increasing the recurrent guidance steps does not significantly
2

---

Published as a conference paper at ICLR 2024
improve the generation quality. In contrast, our calibrated off aff-the-shelf ResNe significantly enhances the sampling quality (lower Fréchet Inception Distance (FID) (Heusel et al. 2017)) without additional time, highlighting the effectiveness of proposed guidance scheme. In Graikos et al. (0022) diverse types of regularization constraints are implemented in the diffusion sampling process. They focused on demonstrating the flexibility of diffusion models but the performance limits of each category are not thoroughly tested. Go et al. (2023) propose a multi-expert strategy that involves training multiple classifiers for specific noise value ranges guide the diffusion sampling process at corresponding time steps. However, this approach requires substantial additional computation, and the
experiments in Go al. 2023) do not provide conclusive evidence that multi- multi-expert- expert-guided sampling can outperform fine-tuned classifier-uuided diffusion (Dhariwal & Nichol 2021). In contrast, our off-the-shelf guided sampling eliminates the need for further training and demonstrates significantly higher sampling performance compared to fine-tuned classifier-guiie
diffusion (Dhariwal & Nichol
2021) and classifier free diffusion (Ho & Salimans 2022) in formal evaluation.
Table 1: Evaluation of off-the-shelf ResNet with recurrent guidance operation and our calibrated
designs in the guided sampling. The ResNet is the official Pytorch ResNet checkpoint; the diffusion model is from Dhariwal & Nichol (202I). We generate 10,000 ImageNet 128x 128 samples with 250
DDPM steps for evaluation. Sampling time is recorded as GPU hours on NVIDIA V100.
Classifier type
recurrent steps
FID
Time (hour)
ResNet
7.17
14.1
ResNet
2
7.06
16.0
ResNet
3
7.14
18.0
ResNet (Our-Calibrated)
5.19
14.1
PRELIMINARIES
Diffusion model contains a series of time-dependent model components that apply the forward and reverse processes (Ho et al. 2020 Sohl-Dickstein et al. 2015). Forward process refers to the gradual increment of noise on the original sample Ty: q( (gg) = Nax; v vārzo, (1 - ā.)) where 3, denotes forward process variance, a₂ gradually generating clean samples from noisy samples: ( = ( (o at), where no (æt, t) is derived from removing the diffusion estimated ca (fte, t) from noisy samples ăz: po ( () Vag
$e, ā II == Qgg. The reverse process refers to
1 (+. Vi-a,
8. _ c ()) and 0 denotes the reverse process variance.
Classifier guidance (Dhariwal & Nichol 2021) can be applied in the reverse process for improving
generation quality. For conditional diffusion classifier guidance and class y. the guided reverse process is adding pa with the gradient of the logarithm of the conditional probability: (( 1i po () + sa, Va, log(p(y) (t)))))))) o.). Specifically, the gradient of the logarithm of classifier f logit in softmax operation Va. 2, (of(softmaxxff (ft))) is used for Va, log(p(y/ (...).
Classifier-free guidance (Ho & Salimans 2022) uses the difference between conditional and unconditional noise (score) to represent the conditional probability guidance during the sam- pling. Va, log(p(y)āzz) x co y. t) co (ăz, 0, t). The classifier-free free guided sampling becomes 4 (2, y. t) + (s (c( t. ca (ā,, 0, t)), where s > 1 is the classifier-free scale. The unconditional (f., 0, () is trained by randomly replacing the class with null class 0.
DESIGN SPACE OF CLASSIFIER GUIDANCE
4.1 CLASSIFIERS: FINE-TUNED VS THE-TEE-SHELF
According to Dhariwal& & Nichol 2021). a classifier to be used for guiding diffusion generation requires dedicated training or fine-tuning adapt to noisy images different time steps during the diffusion process. To implement the dependeenn,, the classifier usually employs the downsam- pling component of U-Net (Ronneberger et al., 2015), and fine-tuning is performed on noisy samples I, for every t along the forward diffusion process. Such a training procedure is time-consuming (200+ GPU hours for ImageNet 128x.28).
Pyttorc ResNet checkpoints:
3

---

Published as a conference paper at ICLR 2024
In comparison, "off-the-shelf" classifiers refer to the publicly available checkpoints that can be directly deployed without further fine-tuning. The collection of such pretrained classifiers is becoming increasingly more powerful and diverse. There is line of research, under the umbrella of "model zoo",, that specifically studies how to explore and exploit pretrained models for various downstream tasks (Shu al. Z02T; Dong et al. 2022 Chen et al. 2023 Luo et al., 2022)). However, when it comes to guiding diffusion generation, "off-the-shelf" classifiers tend to be not robust against Gaussian noise and not adaptable to denedeepndentt diffusion architectures. Successfully exploiting their knowledge for diffusion models is non-trivial. In our work, we use the official Pytorch ResNet
checkpoints as the aff-the-shelf classifiers.
As stated earlier, while an ideal classifier for guiding diffusion should provide accurate estimations of V log Pyy xr), calibration error a promising alternative criterion due to the challenges in efficient gradient estimation. Proposition 4 suggests that if a classifier satisfies certain smoothness conditions, a small calibration error indicates good estimation of the gradient of the log conditional probability.
Let p E H (S2) be the underlying true density function, where H. (St) is the Sobolev space with smoothness k > 1 defined on a compact and convex region Q. Assume that there exist constants 0,, C2 > 0 such that 02 2 p(æ) 2 C1, V.,V QQ Suppose pn an estimate of p. such that (et)) I C for some constant C not depending on n, where n is the sample size. If llpm- P0))) = (1), we have log p V log Pre (2(0) = o1(1).
Proposition 4.1.
To assess the calibration of a classifier and its compatibility with the diffusion model, we propose the integral calibration error ECE (Eq 1b as estimation of ECE, where ECE, (Expected Calibration Error (Naeini er al., 2015)) measures the sample average of the difference between the classifier's accuracy and probability confidence within bins B based on the reverse process sample ft time t.
Bml ECE,, where ECE, Σ n
BCE-
Jaco(B..BB))))
K:
Figure [ depicts the ECE, curves of the two classifiers at different time stages (250 DDPM steps in total). From time 0 to 50, the fine-tuned classifier exhibits lower ECE, values compared to ResNet, demonstrating its robustness to Gaussian noise when the images are less noisy. However, as the time steps progress and the noise magnitude increases, the off-the-shelf ResNet achieves lower ECE, values than the fine-tuned classifier. This suggests that training on highly noisy or low signal-to-noise, samples does not contribute to the fine-tuned classifier guidance. This observation forms the basis that off-the-shelf guidance has the potential not only match but also surpass the performance of fine-tuned classifiers in our subsequent experiments.
ECE, at each sampling step
033
022
fine-tuned U-Net classifier
off-the-shelf ResNet
Sampling step
Figure 1: The ECE, of the fine-tuned and the off-the- the-shelf classifiers throughout sampling step.
In the subsequent sections, we explore the design space of classifier-guided diffusion generation to enhance the quality of the classifier gradients along the diffusion process and have better synergy with the diffusion score function: 1. Classifier inputs: to facilitate calibration with the diffusion reverse process, we provide the off-the-shelf classifier with the predicted denoised sample (t()) during reverse sampling. 2. Smooth guidance: building Proposition 4.1 we enhance the smoothness of the classifier, enabling calibration to result in improved guidance(gradiem estimation). 3. Guidance direction: we uncover the classifier guidance diminishes as the diffusion denoising step advances
4

---

Published as a conference paper at ICLR 2024
To address this, we balance the joint and conditional guidance direction and result in optimal guidance direction. 4. Guidance schedule: we propose a simple yet effective Sine guidance schedule that better aligns with the calibration error curve (Figure I of the off-the-shelff classifier.
in Figure 4
4.2 PREDICTED DENOISED SAMPLES
During the classifier-guided reverse process, there two types of diffusion-generraeed ple: the reverse processing sample i and the predicted denoised sample Tn (å, (Vl-aa)eo(fr, t))/ / (Song al., 2020a Bansal et al. 2023). Considering off-the-shelf clas- sifiers are typically not robust to Gaussian noise and not time-dependent, selecting the appropriate classifier imput type is crucial to ensure the best fit. In Table we compare the calibration of two input types: the reverse processing samples 2, and the predicted denoised samples Zo(() used in the guided diffusion. Table 2 calibration when provided with the denoised sample (t()) compared to åt. This improvement in calibration enhances the guided sampling quality with smaller FID.
demonstrates that the off off-the-shelf ResNet classifier achieves better
Table 2: Comparison of classifier inputs in guided sampling with respect to ECE and FID. Denote the guidance of classifier as Guidance(z) == V log(softmax((ff())))
GUIDANCE(&:)
GUIDANCE(RO(())
ECE
0.36
0.28
FID
8.61
7.17
4.3 SMOOTH CLASSIFIER
smooth classifier is important to guided diffusion generation. On one hand, Proposition indicates that the smoothness of the classifier is key ensuring good gradient estimation. On the other hand, gradient- based optimization also benefits from increased smoothness. For better guidance, propose to enhance the smoothness of the classifier. According to Zhu et al. (202T), the Softplus activation, Softplus, (xx) 3 log(1 + exp(3x)). is effective in smoothing the classifier gradient. As parameter approaches infinity, the Softplus function converges to the ReLU activation function. Table and Figure 2 demonstrate that as decreases, the ECE decreases as well, indicating that smoother activation benefits classifier calibration. Consequently, the well-calibrated design enhances the guided sampling performance compared the baseline (ReLU), reducing FID from 7.17 6.61.
Table 3: Ablation study of / in Softplus with respect to integral calibration error ECE and FID.
RELU
SOFTPLUS($=8)
SOFTPLUS(8=5)
SOFTPLUS(3=4)
SorTPLus($=3)
ECE
0.34
0.31
0.26
0.21
0.07
FID
7.17
6.99
6.89
6.73
6.61
ECE; at each sampling step
08
ResNet ReLU
ResNet Softplus
0.6
ResNet Softplus
ReeNet Softplus
0.5
ResNet Softplus
0.4
0.9
0.1
00
260
200
160
100
Sampling step
Figure 2: The ECE, curves of the off-the-shee ResNet with ReLU and Softplus activation functions.
5

---

Published as a conference paper at ICLR 2024
4.4 JOINT VS CONDITIONAL DIRECTION
In this section, we demonstrate that properly weighing the joint and conditional guidance can significantly improve sampling quality. In Dhariwal & Nichol (22T), the classifier guidance is defined as the gradient of the conditional probability, which can interpreted as the gradient of the joint and marginal energy (Grathwohl et al. 2019) difference, shown in Eq and 3 as
exp(Tf
N
(gg:(yyla) log
N exp(Tfi(x)) (f)(() - log exp(T == Ex, Ex)),
V lo = (x, V.
(3)
To gain a deeper understanding of the guidance direction, we conduct closed-form analysis in mixed-Gaussian scenarios. Proposition provides the derived closed-form joint and conditional directions in mixed-Gaussian scenarios: for conditional probability gradient, the guidance direction is combination of class mode differences; while the joint probability gradient directly targets the objective class mode μ4. Figure provides visual illustration of the two types of guidance directions.
Joint vs Conditional guidance illustration
Cass1
Class2
Class4
Joint direction
Conditional diraction
Figure Hixed-Gaussian settings of joint and conditional probability guidance toward Class3.
Proposition 4.2. Let X ~ P be a random variable defined on Rd, with the density function f(x) Σh bafa (x), where fu(Ø) is normal density function with mean μa and covariance matrix Σ₁, and by > with >K by 1. Let 7 {T,.... K} be a random variable satisfying P(Z = 1, x = bz.(( Then have
P,P 1 = =x)
e (w=)))) T2 (w0-Mu) (E, m1) =x -mx)).
V V.P 1, 2)
(mm w).
Proposition 4.2 reveals that all Σg are identity matrices, the gradient of the joint distribution is simply μ4 xr, directing towards the mode of density fi(x);; while the gradient of the conditional distribution is Σ bye (#- (e-p)))(e)))() (=) - pi). which may point to low-density region.
To strengthen the joint fy (xr) guidance (joint energy E,,, (x. v)), we reduce the value of marginal temperature 72 relative to the joint temperature T1, shown in Eq The ablation study in Table
validates the improvement in the sampling quality by weighing the joint & marginal guidance.
Ve log Po. (u|x) = (2(1ff(2)) - log exp(12fi( = V. (x. y) + V Er,
In addition to quantitative metrics, we visually demonstrate the impact of enhancing joint guidar(c) on classifier-guided sampling. Figure g displays the intermediate sampling images and the classifier gradient figures over 250 DDPM steps. Figure (a) represents the traditional conditional probability settings (πi 1, 72 1): the classifier gradient figure gradually fades from t=50 to 0, indicating loss dog depiction guidance during the sampling. contrast, Figure X (b) showcases strengthened joint guidance (7)
0.5): the classifier gradient figure increasingly highlights the dog outline,

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study of marginal logit temperature 72 with respect FID. 71 is fixed 1.
72
1.0
0.8
0.7
0.5
0.3
FID
6.20
5.62
5.45
5.27
5.30
providing consistent and accurate guidance direction throughout the entire sampling process. It aligns with Proposition [22 highlighting that the gradient of the conditional distribution may point to low-density region, while the jointly amplified gradient targets the mode of density more precisely.
Images
Classifiee Gradient
<= 200
1= 150
x=
100
=
r=0
[= 200
f- 150
1= 100
50
Classifior logit temperstue::
1,79
(b) Classifier logit temperature:
1,
0.5
Figure 4: The visualization of intermediate sampling images and classifier gradient figures under conditional probability and joint-strengthened guidance 250 DDPM steps.
4.5 GUIDANCE SCHEDULE
In Dhariwal & Nichol (021) the classifier guidance scale schedule employs a linear timely-decay variance ae 3e (Ho et 2020). To fully leverage benefits well-calibrated classifier, we introduce sin factor to the guidance schedule: Yt o, + y0r sin(nt/T)) where o, denotes the variance at time This design choice motivated by the observation in Figure ! where the off-the-shelf classifier exhibits consistently lower and more stable ECE, values during the large noise period (from the beginning to the middle of the reverse process). Consequently, we can amplify the guidance scale during this period to better exploit its effectiveness. The parameter y is used for controlling the magnitude of the guidance amplifying: the bigger the parameter y, the greater the time-dependent sin value added to the guidance schedule. Figure D.I in Appendix demonstratess the original schedule and the 7 added schedule. The impact of the sin factor is examined in the ablation study presented in Table A.3 of Appendix
EXPERIMENTS
5.1 OFF-THE-SHEEE SHELF GUIDANCE FOR DDPM
Guided diffusion (Dhariwal & Nichol 202T) demonstrates that incorporating a fine-tuned U-Net classifier can significantly enhance image generation quality. However, the classifier is demanded to be dime-dependent U-Net, and the fine-tuning process is time-consuming (200+ GPU hours for ImageNet1 128 classifier fine-tuning). our approach, we utilize off-the-shelf Py PyTorch ResNet-50 and ResNet- 101 checkpoints with our calibrated design to directly guide the diffusion sampling. Table confirms that our calibrated off-the-shelf ResNet-50 (FID: 2.36) and ResNet- 101 (FID: 2.19) not only improve the diffusion baseline quality (FID: 5.91) but also outperforms the fine-tuned classifier
guided diffusion (FID: 2.97) and the classifier- free diffusion (Ho & Salimans 2022) (FID: 2.43) by significant margin. By leveraging off- ffftth--sellf classifiers, conditional diffusion models, surpassing existing approaches. The guided sampling algorithm is outlined in Algorithm I and the hyper-parameter settings can be found in Appendix
integrate external knowledge into
7

---

Published as a conference paper at ICLR 2024
Table 5: The comparison of the baseline DDPM diffusion, the fine-tuned classifier-guided, classifier- free diffusion, and the off-the-shelf ResNet guided sampling, All models generate 50,000 ImageNet 128x 128 samples with 250 DDPM steps.
IMAGENET 128x 128
CLASSIFIER
FID
DIFFUSION BASELINE (DHARIWAL & NICHOL
2021
5.91
DIFFUSION FINETUNE GUIDED (DHRRWAL & NICHOL 2021
FINE-TUNE
182.69
2.97
CLASSIFIER-REE DIFFUSION Ho & SALIMANS 2022
158.47
2.43
DIFFUSION RESNET50 GUIDED (OURS)
OFF-THE-SHEE TER-THE-SHELL
183.51
2.36
DIFFUSION RESNETI01 GUIDED (OURS)
187.83
2.19
Algorithm 1 DDPM off-the-shelf classifier guided sampling.
Parameter: SoftPlus activation B. joint logit temperature 7I, marginal logit temperature 72-
classifier guidance scale
Diffusion model Dg, variance schedule ot class label y, reverse process sample St. predicted denoised sample fo reverse process noise eg y, t), classifier logit of input class y: (a).
Required:
Fo sampled from (O, for t E (T,...,I 1 do D,($e.y,11)
Mr e0 (1, y,
t get predicted denoised sample g (e( (f (2 exp(ro classifier gradient guidance
(l) - (ft - (vl (nea 0))) Voz
f--1 sample from (V + 19. 01)
Nom.N(pp
end for return dn
5.2 OFF-THE- THE-SEEE GUIDANCE FOR EDM
In this section, we demonstrate the effectiveness of off-the-shelf classifier guidance in fewer sampling steps based on EDM (Krraas et 2022) model. EDM utilizes sampling trajectory based on the sampling curvature d, = dr/ dt, enabling efficient and high-quality image generation. Our EDM samplig algorithm outlined Algorithm of Appendix C.2 where the normalized sample then the gradient
it/l|e;lle is used as the classifier guidance inputs y V log (sofsoox////// ( g is normalized to align with the sample 2; and curvature d,: - + Lt + (t, 1 14-1)d, + 4% (g/lllllllll / In our experiments, on ImageNet 64x64 in Table 6, with the sampling time recorded as GPU hours. The results 256 steps of SDE sampling (Kingma & Gao 2023) can be found in Table C.I of Appendix
present the results of fff-the-shelf classifier guidance results of ODE sampling
C.4
Table 6: EDM baseline and the off-the-shelf ResNet guided EDM sampling. Sampled for mutiple ODE steps. We generate 50,000 ImageNet 64x64 samples for evaluation.
IMAGENET 64x64
CLASSIFIER
FID
STEPS
TIME(HOUR)
EDM BASELINE
2.35
36
8.0
EDM RES101 GUIDED
OFF-THE-SHELF
2.22
EDM BASELINE
2.54
18
4.0
EDM RES101 GUIDED
OFF-THE-SHER
2.35
18
4.1
EDM BASELINE
3.64
10
2.1
EDM RES101 GUIDED
OFF-THE- OHE-SHE-SHELF
3.38
10
5.3 OFF-THE-SHER GUIDANCE FOR DIT
In this part, we showcase the applicability of off-the-shelf guidance in enhancing latent-spaced classifier-fiee free diffusion models, specifically DiT (Peelles & Xie 2022). DiT stands out in several aspects. Firstly, it utilizes transformer architecture instead of U-Net. Secondly, DiT operates in the latent space, which is encoded by a variational autoencoder (VAE) (Kingma & Welling (013) from Stable Diffusion (Rombach al. 2022 Lastly, DiT is trained in classifier-free setup (Ho Salimans
8

---

Published as a conference paper at ICLR 2024
and the guided performance is presented in Table / Unlike Wallace et al. (202)), ) our off-the-shelf classifier
2022). Our DiT sampliigg algorithm is outlined in Algorithm of Appendix C12
guidance does not require retraining latent-space-basee classifier and imposes requirements the diffusion architectures. Notably, there are two key designs in the Algorithm R
1. To integrate the pixel-spaced classifier into latent sampling, consider the guidance g the gradient of composite functions, specifically the VAE decoder VD within the classifier f. It can be expressed as: ! (1) log(softmax/ (E))))))))
2. To incorporate classifier guidance y into classifier- free sampling, we normalize the guidance and add the normalized ! to the conditional and unconditional noise difference. The formula is follows: Eo E0(2,, t) + (s (ee &t. t) c
&,. 0, + mg))
Table 7: DiT baseline and the off off-the-shelf ResNet guided DiT sampling, sampled for 250 DDPM steps. We generate 50,000 ImageNet 256x256 samples for evaluation.
IMAGENET 256x256
CLASSIFIE
FID
PRECISION
RECALL
DIT BASELINE
2.27
0.828
0.57
DIT RESNETIO1 GUIDED
OFF-THE-SHELF
2.12
0.817
0.59
5.4 CLIP GUIDED DIFFUSION
this section, we utilize off off-the-shelf CLIP (Rafford et al. 2021) to guide the conditional diffusion model (Dhariwal & Nichol 202T) in generating images based a given prompt (see Eq D.I in Appendix DD. Compared to approaches in Bansal al. (2023); Wallace et al. (003)) our text-to-image sampling method achieves
efficient and high-quality samples. Our method does not require recurrent guidance iteration within single step. resulting in a sampling speed that is approximately 5 times faster than the methods in Bansal al. (2023) In terms of generation quality, proposed design leads to significant improvements in CLIP scores in Figure D.6 of Appendix D Refer to
Figure and Figure D in Appendix D for more demonstration figures.
"Party hat on corgi oil painting" "Van Gogh Style Cat"
"Icc Frog"
"Bird in the universe"
Figure 5: CLIP guided figures
6 DISCUSSION
In this work, we elucidate the design space of off-the-shelf classifier guidance in diffusion genera- tion. Using our training-fre free and accessible designs, off-the-shelf classifiers can effectively guide conditional diffusion, achieving state-or of the-art performance ImageNet 128x 128. Our approach is applicable to various diffusion models such as DDPM, EDM, and DIT, well text-to-image scenarios. We believe our work contributes significantly to the investigation of the ideal guidance method for diffusion models that may greatly benefit the booming AIGC industry.
There are multiple directions to extend this work. First, we primarily investigated classifier guidance in diffusion generation while there are more sophisticated discriminative models, e.g., detection models and visual Question Answering models in other domains. Second, we only considered image generative models, and extending language models or other modalíties would also be promising direction. We believe that our proposed designs and calibration methodology hold potential for diverse modalities and we leave this for future work.

---

Published as a conference paper at ICLR 2024
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. -42-852, 2023.
Haim Brezis and Petru Mironescu. Where Sobolev interacts with Cagliarlo Journal of Functional Analysis, 277(8):2839-288 2864, 2019.
Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma. Explore and exploit the diverse knowledge in model zoo for domain generalization. In International Conference on Machine Learning, pp. 4623-4640. PMLR, 2023.
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 11:878087994 2021.
Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung- Ho Bae, and Zhenguo Li. Zood: Exploiting model zoo for out-of-distribution generalization. Advances in Neural Information Processing Systems Volume 35, 2022.
Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. arXiv preprint arXiv:2306.0098, 2023.
Hyojun Go, Yunsung Lee, Jin- Young Kim, Seunghyun Lee, Myeongho Jeong, Hyun Seung Lee, and Seungtaek Choi. Towards practical plug-and-play diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1962-1971, 2023.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. Advances Neural Information Processing Systems, 35:14715-14728, 2022.
Will Grathwohl, Kuan-Chich Wang. and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263. 2019.
Jacobsen, David Duvenaud, Mohammad Norouzi,
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update nule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-665 -851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv.2204.03458, 2022b.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurIPS, 2022.
Diederik P. Kingma and Ruiqi Gao. Variational diffusion models 2.0: Understanding diffusion model objectives as the elbo with simple data augmentation, 2023.
Diederik P Kingma and Max Welling. arXiv: 1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
10

---

Published as a conference paper at ICLR 2024
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. arXiv preprint arXhv:2305.18455, 2023.
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Y. Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models general video editors. ArXiv, abs/2302.01329, 2023.
Mahdi Pakdaman Nacini. Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilitics using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorcalistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.1498 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831, PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 061004 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention-MICC 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. / 2022.
Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Zoo-tuning: Adaptive transfer from a zoo of models. In International Conference on Machine Learning, pp. PMLR, 2021.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. aiieeo-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2266-2265. PMLR, 2015.
11

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020b.
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimiza- tion improves classifier guidance. arXiv preprint arXiv:2303.13703, 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023.
Yao Zhu, Jiacheng Sun, and Zhenguo Li. Rethinking adversarial transferability from a data distribution perspective. In International Conference on Learning Representations, 2021.
12


Reviews:
[{"review_id": "5UgaWXTT30", "reviewer": "Reviewer_6jri", "summary": "The paper investigates improving sample quality in conditional diffusion generation by leveraging off-the-shelf classifiers in a training-free context. The authors propose pre-conditioning techniques based on classifier calibration, significantly enhancing diffusion models' performance with minimal computational overhead, verified through experiments on ImageNet. They introduce a novel metric, integral calibration error (ECE), to evaluate the effectiveness of classifier guidance and find that off-the-shelf classifiers outperform trained classifiers under high noise. To combat the diminishing influence of classifier guidance in later diffusion stages, a new weighing strategy is suggested, yielding better sample quality. The paper highlights the potential of their methods in text-to-image generation and points out the limitations of current guidance enhancement methods in terms of sampling efficiency.", "questions": "- Based on the various considerations and discoveries made by the authors, I am curious whether it is possible to extend off-the-shelf guidance beyond class conditions to various modalities of conditional generation. It seemed that the CLIP experiment was intended to demonstrate such a possibility; however, as mentioned in the weaknesses, it did not feel like an experiment based on a new methodology reflecting the authors' considerations and findings.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper convincingly establishes the research context. In particular, it is effective in demonstrating the robustness differences across time step intervals between off-the-shelf classifiers and fine-tuned classifiers using the ECE metric.\n- The division of the design space for diffusion guidance appears appropriate, and the empirical process of selecting among the various options seems justified.\n- The authors' exploration reveals that guidance using off-the-shelf models exhibits performance comparable to or surpassing previous methods, which required significant computational costs.\n- The experiments with guidance through CLIP demonstrate the potential for extending guidance models beyond classifiers, indicating scalability in the approach.\n- The theoretical explanations provided lend solid persuasive strength to the authors' design choices.", "weaknesses": "- While I rate this study highly in general, it falls short in comparing with previous research utilizing off-the-shelf models.\n- Specifically, the omission of closely related prior works, [1, 2] , is significant. PPAP[2] explores guidance using a wide range of modalities of off-the-shelf models in an efficient tuning and plug-and-play manner without significant computational costs, which is directly relevant to this paper's discussion. Observations such as the varying contributions of tuning guidance models across different time steps could also reinforce the evidence between the two studies.\n- Overall, I highly appreciate the paper's contribution to the completeness of the discussion on diffusion models' guidance. However, to properly highlight this paper's direct contributions, a comparison with prior studies attempting off-the-shelf guidance is crucial. Although the authors conducted experiments comparing various design choices of off-the-shelf guidance, it is not clearly presented how these relate to previous research, and there is no direct comparison of the final FID scores with prior methodologies. The lack of such comparisons has inclined my evaluation towards rejection.\n- In the CLIP guidance experiments, it seems that the authors did not apply the same level of guidance as they did with off-the-shelf classifiers. The CLIP guidance experiment does not appear to reflect the authors' contributions with a new methodology for off-the-shelf guidance.\n\n[1]: Graikos, Alexandros, et al. \"Diffusion models as plug-and-play priors.\" Advances in Neural Information Processing Systems 35 (2022): 14715-14728.\n\n[2]: Go, Hyojun, et al. \"Towards practical plug-and-play diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OqzW268P83", "reviewer": "Reviewer_r6Zr", "summary": "In this paper, the authors analyze off-the-shelf classifier guidance diffusion through multiple perspectives including calibration, smoothness, score decomposition and guidance scale scheduling. They use calibration error as a new metric to assess the performance of classifier guidance diffusion and propose several techniques to improve this task. They conduct experiments on multiple diffusion models with pre-trained ResNet classifiers and show consistent improvement with their proposed method compared to baselines.", "questions": "1. The formula for $\\text{ECE}_t$ is with respect to $\\hat{x_t}$ but in Section 4.2 the authors talked about using $\\hat{x_0}(t)$ will provide better calibration, so which sample did the authors use when providing the ECE results for the rest of the experiments?\n\n2. It is unclear to me how did the authors incorporate the Softplus activation function into the pre-trained classifiers, did they just replace all the activation functions in the pre-trained models? Or is there anything else that they did?\n\n3. How did the authors calculate the marginal likelihood for CLIP guidance generation?\n\n4. What is the recurrent step in Table 1? Is it the same as the “backward guidance” in “Universal Guidance for Diffusion Model” paper? And why is the inference time not changed significantly with the calibrated method given there is extra marginalization required for all classes?\nWhat type of GPU did the authors use in their experiments?\n\nI am happy to raise my score if the authors address my concerns and questions in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper provides several interesting techniques to improve off-the-shelf classifier guidance diffusion. These techniques are practical, training-free and fairly easy to be incorporated into the current diffusion frameworks. Their analysis also provides interesting insight into what happens in the process of classifier guidance diffusion from multiple different perspectives.", "weaknesses": "1. I don’t really see the direct connection between Proposition 4.1 and Eq. 1. There seems to be a gap between ECE and $\\|p_n-p\\|$. Also to make the paper self contained, “bins”, “acc”, “conf” should be defined before mentioning.\n\n2. It is unclear to me how Proposition 1 and ECE inform the design choices of the method. For Section 4.2, the same conclusion can be drawn with only FID. For Section 4.3, Proposition 1 only says “with smoothness $k>1$”, but it doesn’t claim better calibration/score prediction with higher smoothness. Section 4.4 and 4.5 seems to be completely irrelevant to ECE.\n\n3. Relating to Weakness 2, I think the story line of this paper is a little bit scattered. There are many components in the story and it is difficult to tell which one is the main contribution of this paper. The components can also be better connected.\n\n4. There are four components to the proposed method: (1) use $ \\nabla_{\\hat{x_0}(t)} \\log p(y|\\hat{x_0}(t))$ instead of $\\nabla_{\\hat{x_t}} \\log p(y|\\hat{x_t})$ (2) use Softplus activation to increase the smoothness (3) use a second temperature to adjust the “ratio” of joint and marginal guidance (4) sin factor guidance schedule. Since there is no ablation study that involves all four components conducted, it is very difficult to tell which one is actually effective. It would be great if the authors can include experiments that gradually exclude these components one-by-one to see which one is the most effective.\n\nMinor suggestions:\n\n1. According to the official style files provided by ICLR call for papers, the appendix should be included in the same PDF file as the main text and the bibliography.\n\n2. Eq. 1 $k$ notation conflicts with the smoothness $k$ in Proposition 4.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hh3JHqNI70", "reviewer": "Reviewer_rssc", "summary": "The paper presents a novel method to improve guidance in conditional diffusion generation without additional training by utilizing off-the-shelf classifiers. The authors introduce pre-conditioning techniques that enhance the performance of existing state-of-the-art diffusion models by up to 20% on ImageNet. Their approach is efficient, scalable, and leverages the widespread availability of pretrained classifiers, promising advancements especially in text-to-image generation tasks.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper adeptly integrates elements from both classifier-free and classifier-based diffusion approaches. Additionally, it employs off-the-shelf classifiers to enhance performance while maintaining efficiency. The experimental results presented confirm the effectiveness of these methods.", "weaknesses": "1. Sec 4.2 \"PREDICTED DENOISED SAMPLES\" seems trivial. This is already explored by the CVPR paper \" Universal guidance for diffusion models\". The authors should not make this part a separate subsection if this is not the author's original work.\n\n2. Sec 4.3 \"SMOOTH CLASSIFIER\" seems trivial. I think there are already many works which use Softplus as activation function and explore its difference/advantage with ReLU. If I understand it correctly, the only contribution here is to replace ReLU with Softplus. The novelty point is not enough for an ICLR paper.\n\n3. Sec 4.4 \"JOINT VS CONDITIONAL DIRECTION\", the author mention \"we reduce the value of marginal temperature\", which seems kind of manual tuning parameters. Is there some validation metric the authors use to determine the optimal temperature?\n\n4. Sec 4.5 \"GUIDANCE SCHEDULE\" seems trivial. If I understand it correctly, the only contribution here is introducing a sin factor. This seems more like a trick instead of some research contribution. For it to be a research contribution, the author should first discuss what kind of guidance is good and why the author choose the sin factor here. \n\n5. Sec 5.3 \"OFF-THE-SHELF GUIDANCE FOR DIT\", the authors propose to incorporate classifier guidance g into classifier-free sampling. The idea is straightforward but the same question the authors introduce a parameter gamma_t here. How do the authors choose a proper gamma_t for a specific case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YcysWy8tS", "reviewer": "Reviewer_wqgP", "summary": "The manuscript analyzes the limitations of current classifier guidance in terms of flexibility, calibration error as well as smoothness. Based on the analysis, the authors propose to use an off-the-shelf classifier instead of a noise-aware classifier for guidance which shows some encouraging results.", "questions": "See the weaknesses. The work is interesting and potential, yet there are a number of concerns as well as writting. Will raise the score if all concerns are solved.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is very interesting, the method is simple yet effective. The current approach requires finetuning classifiers as well as training diffusion models to be a joint model. This work removes this limitation and shows that the off-the-shelf classifier has the potential to outperform noise-aware models.", "weaknesses": "Although the work is very interesting, there are still some questions:\n\n1. The comparison is not so comprehensive. Since the main baseline should be the noise-aware classifier, the work provides little comparison in Table 5. More metrics such as IS, sFID, Precision and Recall should be included. \n2. In Table 5, results for IMN64x64 as well as IMN256x256 for DDPM from Dhariwal and Nichol (2021) is missing. Whether or not the method can perform well on other resolutions.\n3. Does off-the-shelf classifier guidance provide conditional information for the unconditional diffusion model? In my belief, the main objective of the guidance should be providing conditional information for the unconditional diffusion model rather than just improving the generated image quality. In Table 5, the Diffusion model on ImageNet128x128 is conditionally trained. Thus, the conditional information from an off-the-shelf classifier is not important. Results for combining off-the-shelf classifiers with unconditional models should be included.\n4. How off-the-shelf models Resnet can be used with ImageNet64x64 or ImageNet128x128 although these models are trained on ImageNet224x224? Besides, the generated images are clip to be in range of [-1; 1], how does it fit with model trained with input images normalized by ImageNet datasets?\n5. There is a fatal gap in the modeling part in Algorithm 1, in the sampling equation $\\hat{x}_{t-1} \\sim \\mathcal{N}(\\mu + \\gamma _t g, \\sigma _t)$. In the original paper by Dhariwal and Nichol (2021), the sampling resulted from the $\\log(p _{ \\theta }(x_t|x _{t+1}) p _{\\phi}(y|x_t))$. However, given the structure of Algorithm1, this equation is no longer valid since the gradient is taken regarding $x _0 (t)$ instead of $x_t$. In order to apply the same sampling process, even when $x_0(t)$ is forward through the classifier, the gradient should be taken regarding $x_t$. \n6. I guess from the point (5), this is the main reason why the method can not be applied to DDIM?\n7. Ablation study is missing, it is quite vague to understand which proposed scheme is the main course for the improvement. From my understanding, there are three main differences from normal classifier guidance which are:\n* Gradients via $x_0(t)$ instead of $x_t$\n* Guidance schedule\n* $\\tau _2$ temperature\n\nHowever, discussion on the contribution of each of them is missing\n\n8. Given three contributions as in (7), does the performance of the noise-aware classifier guidance also get improvements?\n9. Besides Resnet, how are other architectures such as DenseNet, Transformers? Do they also work with this scheme?\n10. CLIP guidance should be compared against the noise-aware CLIP guidance\n11. The connection from 4.1, 4.2, 4.3 and 4.4, 4.5 as well as the design of the algorithm lacks some connections.\n\nIt seems that the paper is written in hurry so that the format of the paper is not really good as well as some errors in equations:\n1. Equation (2) should be $\\log exp(\\tau f _{y(x)})$? Check format of the equation (2) also.\n2. Equation (3) should be $\\log exp(\\tau _1 f _{y(x)})$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mvSDm2vrXN", "reviewer": "Reviewer_VZ9L", "summary": "The paper focuses on a very interesting question of guiding the diffusion modes by leveraging off-the-shelf classifiers in a training-free fashion. Specifically, the authors first provide a simple analysis to demonstrate that the off-the-shelf classification can achieve better accuracy than the fine-tuned classifier when the noise level is high, which may have been ignored in the previous works. Then, the authors turn to exploit the pre-trained classifier for guiding diffusion generation with comprehensive consideration of the detailed settings.", "questions": "My major commons lie in the experimental evaluation. Please provide more experimental analysis or discussions to validate the effectiveness and robustness of the proposed methods.\n\nHow about the performance in terms of CLIP score and Human-level metrics?\n\nHow about the performance for the conditional generation with the general text prompts?\n\nPlease provide more discussions about Table 4 and Figure 4.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tIn my opinion, the major contribution of this paper lies in Section 4.1, which provides an empirical analysis by evaluating the calibration of both fine-tuned and off-the-shelf classifiers.  The results reveal that fine-tuned classifiers are less calibrated than off-the-shelf ones when the noise level is high, which indicates that Off-the-shelf classifiers’ potential is far from realized.\n2.\tTo optimize the design of the proposed methods, the authors have considered multiple aspects, including the classifier inputs, smoothing guidance, guidance direction, and guidance direction, and designed the corresponding strategies to improve the overall performance.", "weaknesses": "1.\t The paper focuses on the classic class-conditional diffusion generation, which is a simple case in the text-to-image generation. How about the performance of the proposed methods for the general case with text prompts as conditions? In particular, in section 5.4, the authors also provide a simple analysis of this case with the CLIP model. However, the results are not good.\n2.\tHuman-level metrics should be involved for clearer comparisons. It is well-known that some quantitative metrics like FID, may be problematic in some cases. In particular, the FID metric, used in ablation analysis in section 4 and experiments in section 5, cannot measure conditional adherence which is important for conditional generation. CLIP score should also be considered to evaluate the performance.\n3.\tIt is weird that the final images are merely the same as each other in Figure 4 with different settings of logit temperature, while the FID score varies with different settings in Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)