Rethinking Conditional Diffusion Sampling with Progressive Guidance
Authors: Anh-Dung Dinh
Daochang Liu
Chang Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion model
conditional generative model
guidance diffusion
generative models
classifier guidance
TL;DR: The work analyses and solves the problem adversarial effects and lack of diversity by a simple progressive guidance

Abstract: This paper tackles two critical challenges encountered in classifier guidance for diffusion generative models, i.e., the lack of diversity and the presence of adversarial effects. These issues often result in a scarcity of diverse samples or the generation of non-robust features. The underlying cause lies in the mechanism of classifier guidance, where discriminative gradients push samples to be recognized as conditions aggressively. This inadvertently suppresses information with common features among relevant classes, resulting in a limited pool of features with less diversity or the absence of robust features for image construction.	We propose a generalized classifier guidance method called Progressive Guidance, which mitigates the problems by allowing relevant classes' gradients to contribute to shared information construction when the image is noisy in early sampling steps. In the later sampling stage, we progressively enhance gradients to refine the details in the image toward the primary condition. This helps to attain a high level of diversity and robustness compared to the vanilla classifier guidance. Experimental results demonstrate that our proposed method further improves the image quality while offering a significant level of diversity as well as robust features.

Rethinking Conditional Diffusion Sampling with Progressive Guidance
Anh-Dung Dinh
Daochang Liu
School of Computer Science The University of Sydney di dinhanhdung 9968gmai1 com
School of Computer Science The University of Sydney daochang liu@sydney, edu
Chang Xu
School of Computer Science
The University of Sydney c. xu@sydney. edu. au
Abstract
This paper quantifies and tackles two critical challenges encountered in classifier guidance for diffusion generative models, i.e., the lack of diversity and the presence of adversarial effects. These issues often result in a scarcity of diverse samples or the generation of non-robust features. The underlying cause lies the mechanism of classifier guidance, where discriminative gradients push samples to be recog- nized as conditions aggressively. This inadvertently suppresses information with common features among relevant classes, resulting in a limited pool of features with less diversity or the absence of robust features for image construction. We propose generalized classifier guidance method called Progressive Guidance, which miti- gates the problems by allowing relevant classes' gradients to contribute to shared information construction when the image is noisy in early sampling steps. In the later sampling stage, we progressively enhance gradients to refine the details in the image toward the primary condition. This helps to attain a high level of diversity and robustness compared to the vanilla classifier guidance. Experimental results demonstrate that proposed method further improves image quality while offer- ing significant level diversity as well as robust features. Source code is available httts::///// htt///gtttttttttt///////ee/etttt com heggthub.co//uuugddaaaaa///aaa/a dit.://aaaaanagg.giieeeddddaaaaa -ww.gdddddddddfddiio..
Introduction
has emerged state-of-the-art technique for generating high-fidelity images without relying on ad adversarial training techniques like GANs 6 g 10]. This approach has gained popularity among researchers because avoids mode collapse and training instability. For diffusion models, the generated samples can be further enhanced with improved quality and controllable attributes using guidance techniques for sampling. Classifier guidance 12 13 is
Diffusion model 1234 4 5
one of these techniques which offers numerous benefits. Conditional information can be provided to unconditional diffusion models using noise-aware classifiers' gradients to guide towards pre-defined conditions during sampling. This technique is able to not only improve image quality significantly but also trade-off between diversity and conditional information fit with the application design. More critically, the flexible nature of classifier guidance allows easy extensions of off-the-shelf diffusion models the sampling stage without any expensive re-training.
"Corresponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Adversarial Effect
(b) Diversity Supparession
(c) Progressive Guidance
Vlog POl (H)
Vlog (Oralt)
Slng POt
Plogr I+)
Vlog (Csal)))
Vlog pOcal)))
Vlog PO
Vlog Ie)
0.95
0.96
Vlog POsall))
Vlog (Ouals))
Vlog POual))
X
Xx1
Xx2
X
0.89
0.91
0.86
Patteery Sasssii
Vieg (()))))
Vleg P(Gl+)))))
Slog POLalk
Flog
Vlog (raall))
Vloweot
Plog
0.85
0.85
0.73
cat
leopard
tiger
Generated (Linyy to classtty)
Suppressd clasity)
Vanilla Guidkncee
Figure I: (a) Samples generated by vanilla classifier guidance achieve high confidence in the condition class but have suspicious features. (b) Images generated by vanilla guidance (left) show a lack of diversity where front-face face features are ever-exploited. Features shared among breed classes are often lacked, such as action pose and background details depicted in the ground truth images (right). The vanilla guidance has exploited most distinguishable features to satisfy the classifier while suppressing common features. (c) Our proposed guidance scheme allows gradients from relevant classes to join the sampling process. is the condition. The darkness of the gradients represents the gradients' weights (information degree) during sampling. More examples can be found in Suppiementary. Dataset: ImageNe164x64
In spite of all of its benefits, the classifier guidance still suffers from two critical flaws. The first flaw is the adversarial effect where the model fails to produce samples belonging to the target condition. Guiding the model with the gradient and maximizing the probability of the target condition may lead to shortcut learning that generates samples attacking the classifier. Examples can be found in Figure Iaa,, where samples achieve very high confidence in the conditional class but result in very poor features. The second flaw is the diversity suppression caused by the conflict between the diffusion model's generative nature and the classifier's discriminative nature. The optimization towards the condition overly encourages the model to generate only image features that can be easily recognized by the classifier while neglecting other features relevant to the condition, which can reduce the diversity of the generation. Figure [b) shows that common features among classes are suppressed since they are harder to classify, resulting in lack of diversity. These two problems of the classifier guidance come from its underlying guidance mechanism. Specifically, this approach assumes that the objective of the generative task the same as that of the discriminative task. By utilizing the gradient v log p(y = (r), the guidance aims to create an image with features that can be distinguished by the classifier as belonging to the label ye- However, when this approach maximizes the probability of the condition class, it also undesirably suppresses important information relevant but in other classes, causing the lack of diversity and the adversarial effect, as examples mentioned.
This paper proposes a new guidance approach for diffusion models, which generalizes the classifier guidance tackle the above problems while maintaining its flexibility and efficiency. Our core idea is to alter the aggressive gradients towards a single target in the naive classifier guidance by the gradients that are progressive along two dimensions, i.e. the class dimension and the temporal dimension as illustrated in Figure [c). Along the class dimension, our method tolerates the gradients from other classes relevant to the condition of the generative task. Along the temporal dimension, we expect the gradients to construct shared features among relevant classes when the image is noisy, and the gradients focus on features more specific to the condition after the image becomes clearer. This scheme allows larger feature pool, which circumvents information suppression and lack of diversity. Besides, the help from relevant information and the less aggressiveness toward one condition also results in more robust features in the samples.
In summary, our contributions are as follows: (1) Proposing a generalized classifier guidance method with progressive gradients that consider relevant information associated with multiple classes depending on the noise level at each timestep. (2) In-depth analysis on reducing the adversarial effects and the diversity suppression of classifier guidance sampling quantitatively and qualitatively. (3) Improvement on image generation task to reach State-of-the-art (SOTA) on different metries with different datasets.
2 Background
DDPM: The Denoising Diffusion Probabilistic Model (DDPM) is represented by the probability distribution function px(xx) ;= (o(xoo))xxx11) where X1, X2, ... XT are latent variables having
2

---

the same dimensionality as the data Xo ~ ((xo). The variable Xy follows the distribution p(XT) N(x:; 0, I). The process of pa (Xp:T) is known as the reverse process and a the Markovian chain: Po == p(x)) lI Po(x where po po(x;-1/x/ == (x H4o (Te, t), Eo Ex(ettt) t)). This process also known as the diffusion model's sampling process or inference process.
In contrast to the reverse process, the forward process corrupts the original data Xo to x with a predefined schedule of Gaussian noise. This process is fixed Markovian chain: ((x1:r Xo) ==
1, 3,1). is the fixed variance sched- IT== (xxxx((x_))) where g(x{|x{_)):==)) := v1
uled for both forward and reverse process.
There number of ways to define the output of the network 0 such noise predictor co t).
mean predictor Po (x,: t) [T] The most common method is the noise predictor c,(x,, t). After the trained to match eg (x1, 1) with e, we have the sampling equation:
1 1- -+
X
(x+
(X,, (i,,,)) t)) + 0,2,
V1-ai
with z as the random noise, 0N is the variance of the of po (x(x and Eæ is the predicted noise.
The guidance aims to provide conditional information during the sampling process so that the output image satisfies the predefined conditions. From Eq. [ we denote
Classifier Guidance:
14 = 1 (x, vi 1- cx t)). Given condition ye. lo xx) is the log of conditional you distribution of the class ye through observing x,. The sampling equation for x;-1 given x, with
guidance scale s is as below:
X N N + so xx. log (B σz).
(2)
Where @ the parameters of the pretrained classifier.
3 Methodology
We first present scheme allowing relevant information gradients to join the sampling process. After that, the progressive schedule is proposed adapt the relevant gradients taking the noise level in samples into account. From the Eq. [ the short form of sampling with guidance is as below:
X Pa to + so? Vx, log (V))
(3)
The expected output of x is to maximize (x.))xx.). Due to the probability property of sesip(ysl xx) = 1, the increase in p(ye(x)) will result in the reduction in other probability for other classes p(y (())) Vi o e with is the condition, and C is the total number of classes. This results in the suppression of other classes information which contributes to harmful effects, as mentioned in Figure [a) and (b). We propose generalize the Eq. ass follows:
X #t + o. * w Σ Vv (v)lx)))
(4)
1=1
with s; 2 is the degree of information injected into the diffusion sampling process by class y/- We further constraint Σ s; 1 and > 8,, Vi a Note that the Eq. Biss special case of Eq. where 8; = 0, Vi * c with is the condition for the sampling process of diffusion models. Similar to
14 15
guidance scale is added to balance denoising signals and classification gradients.
Remark 3.1 By incorporating gradients from other classes during sampling, the conflict between the sampling and discriminative objectives can be decreased, resulting diverse generated samples.
Explain. The use of only the condition class's gradient limits the feature pool's 's space to some specific features that are the most distinguishable from other classes. This results in the lack of common features shared among relevant classes, which causes a trade-off in the diversity of generated samples. We hypothesize that solely pushing condition gradient suppresses other classes' information since we have Σ; p(ys (x+) and the increase in [x,) result the reduction from others. As a result, the inclusion of other classes' gradients avoids this pitfall.
3

---

Remark 3.2 Using other classes' gradients beyond the conditional class gradient helps avoid adversarial effects.
Explain. substantial number of generated images achieve high confidence in the condition class; however, they possess peculiar features unrecognizable to the human eye. Relying solely con- ditional class gradients for information construction presents several robustness problems. Firstly, the gradients associated with the conditional class may contain noise that is specific to that class. Depending solely on this information opens a shortcut for the sampling process to exploit the model by utilizing the noise rather than satisfying semantic features. Secondly, suppressing information from other classes has detrimental effects on robust features. Datasets often consist of numerous classes that share common features that play vital role in constructing meaningful images. When these features are suppressed, the image becomes unrecognizable to the human eye.
Using gradients of other classes, in terms of formulation, first helps avoid adversarial attack-like generation through exploiting the sum of gradients >. V log ply (xx). Furthermore, the use of relevant class information forces the model to generate features that satisfy many classes at the same time. This helps avoid specific noise in a class and can utilize the common features shared among classes to avoid the failure of constructing key features. In-depth discussion about the Remark 3.1 and are shown in section The rest of this section will discuss the techniques and algorithms to bring the Eq. innto the practical sampling process.
3.1 Information Degree vector
The information degree s = {si, 82, ..,. sc } is the set of weights associated with the gradients from classes that contribute to the information construction of the image. In order to balance between gradients, we set constraint Σ = 1. Given e is a main condition, s, > S; V1 < i C.
From the Eq. 4 for a given condition c, the aim is to provide the most relevant features that help avoid information suppression. However, not all labels in the C classes help construct the condition Instead, for given class c, if the 7th class has higher relevance to class e than jth class, we expect to have information degree 9; > sj. There are several ways to satisfy the requirement to have correlation between labels' features and their degree of information
{51, $2; sc), such as Label Enhancement [16] or Partial Multi-label [17 [8]. However, simple technique, which does not require any training, is preferred due to concerns about the complexity of the whole sampling process. Inspired by the prompt engineering section in CLIP [19]. we propose to model the correlation between labels via the description text of labels. Instead of using fixed templates similar to CLIP, we need more details about each class to have more generative features. In this work, we utilize ChatGPT
20 to generate text deseriptions for each class. After removing stop words and other preprocessing text, we use a CLIP model to obtain each class's embedding information uy. The embedding information is then utilized to calculate the similarity based on cosine similarity, 0u Given the conditional label c, we have the information degree for each label
i... sim... Te, ||l
VI 5 < C. We denote s. { S ..., } as the vector of information degree for sampling the image from condition c. To avoid complication in formulation, later in the manuscript, we will only denote s = {81, 82, 3 for the information degree vector for the condition e without losing generality.
as 8 Σソ sim...?
sim..1
3.2 Progressive Information Degree
The subsection describes dessccrbess combination of gradients among classes. In the diffusion sampling, the image X i is less chaotic and more meaningful than the sample x,. Therefore, we also pro- gressively adapt the s at every timestep to be less chaotic. We index the s at each timestep as 8, = {51,0: 82,4, ... &c YO < l S T. A simple heuristic schedule is proposed to reduce the chaos of at each step. At every timestep, for the condition c, the degree of information $c,t is increased by a small amount Ase,t- At the same time, the Ase,t is distributedly deducted from information
degrees 8i, of other classes, ensuring condition D
81, 1. As, is formulated as:
7
8i,)): Ei Asc ES
if i c
ASi,
(5)
if # c

---

Construct from the most distinguishable features of Brittany Spaniel
Ours Vanilla
Inject shared features from English Springer and Wälsh Springer Spaniel
Figure 2: Diversity Suppression Alleviation. Brittany Spaniel the condition. Conventional guidance approach often collapses generated samples to primarily exhibit the facial features of Brittany Spaniel. However, by incorporating features from other classes, such as English Springer and Welsh Sprinter Spaniel, the sampling can capture common characteristics shared between classes such as backgrounds, poses and actions.
with y as the updating rate. After obtaining As, update the following St $t S; As..
From the above equations, if 7 1, the sampling process is primarily similar to the vanilla guidance diffusion sampling as in Eq. where only features from class e joii the sampling process.
Next, we use an entropy perspective to interpret the mechanism our method. With the sampling generalize the sampling with Kullback- Leibler divergence as:
equation E
+ z ; DKI. D (slp
When s, is fixed, the optimization of Dxz Pp (y x) totally depends on log (3 x,). How- ever, when S, varying, the minimization of KL divergence term will take the full form as DE Aelllpayyeer xx)) E St,, log $t,; st, log (v( (x,). Therefore, the process of transforming the vector of information degree can be formulated as below:
min
$ log $1,4,
so that (1) Se, St,,, St,, ff (2) 2 = (3) ( Si < <1,1 <is C, and (4) st si,l 1 L C. With s as the optimal solution, we have St-1 s. Because hard optimize the objective and the DKL (sel dellPo( (y xx)) at the same time, the constraint (4) is added with an upperbound to avoid the mitigation of the minimization of Dx (sllp x,)) regarding Using the proposed schedule, the upperbound becomes / 7 C_ع.
Throughout the diffusion model's sampling process, the information degree vector will gradually converge near to one-hot vector, leading to the minimization of objective 7 From this point of view, we have formulated the sampling as the problem of matching two distributions between conditional probability p(y xx) with information degree vector s,, The schedule aims to do the reverse entropy
regularization on target 8t, which is opposite the classic entropy regularization 21 22 The formal discussion about reverse entropy regularization is in Supplementary.
4 Analysis
The previous section has shown our method to guide image generation through a progressive scheme. We now provide further analysis visualize how our algorithm solves the problems in detail. We set up this analysis on ImageNet64x64 using ADM [1] with or without Progressive Guidance (ProG) to investigate the diversity suppression and the adversarial effect.
Diversity suppression alleviation: relevant classes of the conditions into the sampling process is crucial to overcome the issue of information suppression among labels, as illustrated in Figure 2 The Brittany Spaniel is the main condition in this example. We demonstrate that by introducing gradients solely from the Brittany
process commences, incorporating information from
5

---

Spaniel, the process converges towards the features that benefit the pretrained classifier the most, focusing primarily on simple and easily classifiable characteristics - the front face features of the dog. In contrast, by providing information about the English Springer and Welsh Springer Spaniel, we encourage the model to sample images with lower confidence in identifying them as Brittany Spaniels and to tolerate certain degree of confidence in them belonging to other breeds during the initial stage of the sampling process. This scheme allows the process to search and sample in the pool of common features between classes, facilitating the diverse samples. We further do an analysis of this feature collapse problem on different types of breeds in section B. of the Supplementary. in Figure 2 the vanilla guidance
We found out that besides Front-face features over-exploitation scheme also suffers from Front stretching pose and Green grass background problems. Similar to Figure our proposed ProG scheme shows that it can solve these problems.
a visual example of how support from gradients of other classes can effectively reduce non-robust features. In the example, the naive classifier guidance suppresses the representation of tiger features, resulting in a significantly degraded image quality due to the lack of facial features of the leopard. In contrast, our proposed method promotes the emergence of tiger, panther and leopard features during the early stages of the sampling process. It gradually diminishes tiger and panther towards the end, ultimately achieving the desired leopard features. Additional instances are also demonstrated where our proposed methods effectively address failure cases due to non-robustness features associated with the conventional classifier guidance. More examples with high-resolution images can be found in section C of the Supplementary.
Robustness features construction:
Vanillla
(lep((appa))
10eoparst 1.00
Steoporstt00 1.00
Steepards 1.00
Vanilla
Jog((tiger(a))
Scuper,1000 1.66 e
Saper:000 X
Supera 1x
Oun
v logr(ponther(ss)
Spantheri1oop 1.4 x
Spanther 9 9
Spanthera 1 1oce-1
logg(toopavd)s)) 91opard,1000
x e
Stoopords00 0.36
Stoopordt 0.99
Figure 3: Robustness Features Construction. The vanilla guidance fails at conditioning the sample leopard (top-left). The tiger, panther and leopard helps to influence the image content in the initial state to form the critical features of the leopard to overcome the failure (bottom-left), where clear tiger features are observed in the early images. The right part shows more failure cases corrected using proposed ProG. The darkness of the gradients shows their associated information degree values.
5 Related works
The field of generative models has seen significant surge in the use of diffusion models. Specifically, the Denoising Diffusion Probabilistic Model (DDPM) and its variants [
3]
2
23
24
11 25
have emerged prominent models for generative tasks which provides better stability than GANs
26 27 28
29
30
9 31 32).
Not only for generative tasks, but discriminative tasks also benefit from
diffusion scheme as a theoretical counterpart to the DDPM series. Despite having different optimization objectives and motivations, these two types of models are closely related [5). In addition to diffusion models, conditional generative models have gained significant attention in the generative model's 's research community 30 DGMs T 3 offer a conditional version by connecting a classification head to the diffusion model. These models benefit from conditional information [1] [6]. Recent research has focused on achieving controllable samplers/generators without retraining DGMs. Classifier guidance [1] proposes to use classifier gradient signals for guidance, which has been further generalized by [ to adapt different types of modality. Classifier-frd guidance [14 has shown that guidance properties can also be achieved without a classifier. CompDiffusion (37 proposes to guide the diffusion model by combining different conditions given by a pretrained model. However, most guidance works suffer from the trade-off between sample quality, diversity, and conditional
33 34 In addition the DDPM series, score-based models 4 35 are developed

---

information. [12] has handled this problem by including scoring model during the training of noise-aware classifier, causing an increase in the running time. Discriminator Guidance [38] finetunes the sampling process by training Discriminator, which also faces the problem of large training time similar [2]. contrast, our work addresses this problem without retraining the diffusion or noise-aware classifier models. [39] solves the problems of conflict during the sampling
process diffusion models, yet this work ignores the problem of robustness features construction.
Adversarial white-box attack [40] 41 42
44] shares common technique with classifier guidance to exploit the gradient ply yo X) to construct the features, leading to concerns the community
43
about the use of V log p(ye X)) in classifier guidance that causes adversarial effects. One of the main targets of our work is to solve this concern.
6 Experiments
Setup. Extensive experiments conducted on CIFARIO, ImageNet (64x64, 128x128, 256x256). We denote Progressive Guidance (ProG) as our proposed method, which is first evaluated on ADM IT and IDDPM 3 verify our claims on improving the performance of the vanilla guidance method. Subsequently. ProG is combined with the EDS [45]. an advanced guidance method to solve the gradient vanishing problem, to achieve the state-of saute-of-the-ars art, Other baselines are taken from BigGAN[[) ADM [II], EDS[45) IDDPM (3, VAQ-VAE-2 (46). LOGAN (28), DCTransformers
47 and DiT 48 (latent diffusion). The ADM with a conditional pretrained diffusion model denoted as CADM. We utilize five standard scores, which are IS, FID/sFII (26), Precision [49], and
Recall 49.. evaluate the image quality and diversity of the generated samples.
Table 1: ProG helps to achieve better IS/FID/sFID in general. The improvement is significant both IDDPM 3 and CADM/ADM IT (both unconditionally or conditionally trained). "-G" postfix attached to diffusion model stands for vanilla classifier guidance. Bold values are the performance our proposed ProG helps achieve better than vanilla guidance. The underlined values are the cases where diffusion models without guidance achieve the best value. is denoted for the score evaluated by the samples provided by the paper. means the values are directly used from the papers due the unavailability of the pretrained model.
MODEL
IS (t)
FID (1)
SFID (4)
PREC (t)
REC (f)
IMAGENET 64x64
ADM
25.64
9.95
6.58
0.60
0.65
ADM-G
46.90
6.40
9.67
0.65
0.54
ADM-G PROG
46.88
5.16
6.72
0.72
0.56
IDDPM
16.02
18.35
55.08
0.60
0,57
IDDPM-G
18.89
13.62
4.43
0.55
IDDPM-G + PROG
21.60
11.12
4.25
0.67
0.55
CADM
53.79
2.07
4.35
0.73
0.63
CADM-G
66.52
2.02
4.62
0.78
0.59
CADM-G PROG
65.65
1.87
4.33
0.77
0.60
IMAGENET 128x128
CADM
92.53
6.14
4.96
0.69
0.65
CADM-G
141.55
2.98
5.10
0.77
0,59
CADM-G +PROG
157.24
2.77
5.09
0.80
0.59
IMAGENET 256x256
ADM
39.7
26.21
6.35
0.61
0.63
ADM-G
96.15
11.96
10.28
0.75
0,45
ADM-G PROG
99.45
11.21
8.67
0.76
0.46
CÁDM
100.98
10.94
6.02
0.69
0,63
CADM-G
188.91
5.21
0.81
0.52
CADM-G PROG
222.09
4.53
5.08
0.85
0.49
CIFAR10 32x32
ADM
9.55
2.87
4.36
0.69
0.60
ADM-G
9.58
2.85
4.30
0.68
0.60
ADM-G PROG
9.45
2.81
4.28
0.69
0.60
7

---

0.01
063
0.60
Prod
0.39
Fred
0:3
0.72
70.2
C. S.
0..00
0,42
0142
PrD
0:00
PHot
0.35
10
Glocharee scaLe
GuuGecce scale
glidmnce stale
guicboe
(e) Uncondttinaa difiuioon
tbl Conditional dificiuo
Figure 4: FID and Recall trend of guidance sampling with (a) ImageNe164x64 unconditional ADM (b) ImageNe164x64 conditional ADM. For vanilla guidance, becomes evident that the guidance scale increase leads to a rapid decline in both Recall value and FID, indicating degradation in the diversity of the samples. Conversely, our method sustains a stable trend associated with increase.
Robustness metric: The issue of quantifying the adversarial effects caused by the classifier guidance method has not been adequately addressed in previous works [[] since it challenging due to the trade-off between diversity, feature quality, and robustness. We propose to leverage bank of off-the- shelf pretrained classification models to evaluate the robustness features based on the assumption that it is much more difficult for the non-robust image to trick set of separately pretrained classifiers than only a single guidance classifier. Specifically, the robustness features are quantified by averaging Top-1 accuracy using pretrained models ResNet34, ResNet50, ResNet151 (50). DenseNet169. DenseNet201 [51], SqueezeNet, SqueezeNet [52). To ensure fair comparison between the two generative schemes, we adjust the guidance scale to achieve similar diversity and feature quality, measured by similarity in the Fréchet Inception Distance (FID).
6.1 Improvement over Classifier Guidance
In this section, we show that applying the ProG method helps achieve better performance quantitatively over vanilla guidance regarding image quality, diversity, and robustness. Table [ shows a clear improvement in all the guidance sampling models on image quality and diversity. ProG achieve lower recall values in some datasets than the ADM baseline without guidance. However, this is expected as conditional generative schemes limit the search space resulting in less freedom sampling features.
Table w are adjusted between CADM-G and CADM-G ProG to achieve similar FID to evaluate robustness. The ProG significantly
Diversity trend when increasing w: By increas- ing guidance scale w. the FID scores and diversity scores such as Recall are less sacrificed in ProG to achieve higher IS score and conditional infor- mation compared to vanilla guidance (Figure 4 When increasing the guidance scale, proposed method mostly has a slower degeneration rate in FID and Recall than the vanilla guidance.
improves the robustmess of the samples.
MODEL
ROBUSTNESS (1) FID
IMAGENET 256X256
CADM-G CADM-G PROG
79.65
4.58
85.04
4.56
Robustness improvement We show that utilizing the ProG method can achieve more robust features than vanilla guidance, with the robustness performance shown in Table [ We first adjust w that the two generative processes (vanilla and our proposed) have the same FID scores. After that, we calculate the robustness score as described previously.
Key takeaway: The ProG helps to improve the guidance to achieve three targets. Firstly, we achieve better image quality and diversity than the vanilla guidance scheme, which can be observed through IS, FID, sFID, Precision, and Recall. Secondly, we alleviate the diversity suppression qualitatively and quantitatively even when the guidance scale is set be very large. Finally, we achieved better robustness than the original guidance scheme quantitatively and qualitatively.
6.2 State-of-the-Art Comparision
To compare with other state-of oatee--f-hheeeee methods, we combine our proposed method with EDS 451 While our method aims to solve information suppression, the EDS will help amplify the guidance signals at the end of the sampling process. The result shows favorable outcomes in Table Our ProG achieves the state-of-the-art IS/FID on ImageNet64x64 and ImageNet1 28x128. 128. On ImageNer256x256 the ProG helps to improve DiT-G slightly but consistently over metrics. Furthermore, we can achieve comparable level of robustness to classifier-fret free guidance in Table 4 which is a method

---

that does not share common techniques with adversarial attacks while also exhibiting a significantly improved diversity trend compared vanilla classifier guidance, as shown in Table 4 It worth noting that in the diffusion ADM model, classifier- freee guidance outperforms classifirr guidance; however, this comes the cost of nearly twice the computational burden due to the need for forwarding to diffusion twice at each timestep. Moreover, classifier-free guidance lacks the flexibility offered by the classifier guidance method. We also discuss in in-depth the benefits of using classifier guidance with ProG compared to classifier-free guidance in section E in Supplementary. Section F in the Supplementary will discuss several techniques to extend ProG into Text-to- Image Generation. The section D in Supplementary will provide the settings for all experiments.
Table 3: Combining with the EDS [45] to solve the gradient vanishing problem, our proposed method helps achieve state-of- ootte-of-hhe oftthe-rr IMAGENET64x66 and IMAGENET12&// On latent diffusson models such as DiT, we can achieve slightly better performance to reach SOTA on this dataset.
MODEL
IS (1)
FID (4)
SFID (4)
PREC (t)
REC (1)
IMAGENET 64x64
BIGGAN
44.99
4.06
3.96
0.79
0.48
IDDPM'
46.31
2.90
3.78
0.73
0.62
CADM + CLET
63.39
1.93
4.49
0.77
0,60
CADM-G EDS
61.91
1.85
4.36
0.76
0.61
CADM-G EDS PROG
65.89
1.77
4.25
0.77
0.61
IMAGENET 128x128
BIGGANt
145.93
6.02
7.18
0.86
0.35
LOGAN CADM-G EDS
148.2
3.36
[60.2
2.38
4.92
0.78
0.53
CADM-G EDS PROG
175.31
2.47
4.92
0.80
0.58
IMAGENET 256x256
BIGGAN
202.77
7.03
7.29
0.87
0.27
DCTRANS
36.51
8.24
0.36
0.67
VQ- VAE-2
31.11
17.38
0.36
0.57
IDDPM
12.26
5.42
0.70
0.62
CADM CLs-FREE
191.31
3.76
4.87
0.80
CADM-G + EDS
212.51
3.96
5.0
0.82
0.52
CADM-G EDS PROG
232.86
3.84
5.0
0.83
0.51
DIT DIFFUSTON) DiFF
122.62
9.62
6.89
0.66
0.670
DIT-G
274.69
2.27
4.58
0.82
0.58
DIT-G + PROG
278.77
2.25
4.56
0.82
0.58
Table 4: FID and Recall trend for guidance sampling with ImageNet256x256 conditional diffusion. ProG not only helps achieve better FID but also higher robustness score. Besides. the diversity trend is also better preserved than vanilla guidance when the guidance scale is increased.
MODEL
ROBUSTNESS@ (1)
FID
IMAGENET 256X256
9.40
CADM-G EDS
83.31
3.96
CADM-G EDS PROG
86.60
3.90
PreG
2.39
Pro
CADM CLS-FREE
87.14
3.95
2.59
Pr.S
Surdhnc reale
6.3 Ablation Study
ProG schedule relies on a single hyperparameter to determine the speed of convergence towards the one-hot vector. larger y leads faster convergence, while smaller ? results in slower convergence. Table debons]demosttrrtes the sensitivity of showing that varying it does not significantly impact the results. Results obtained with y values of 0.04 or 0.06 are similar. When becomes larger, the model is more likely to achieve a higher Inception Score (IS), but trade-off with FID/sFID scores, indicating decrease in sample quality. It worth noting that without the schedule (y 0), the guidance method fails, underscoring the importance of the proposed scheduler.
Effects of schedule:

---

Correlations between labels: Fixing the same schedule, we change the initial vector of information degree ST via different ways. We denote our ChatGPT-generated text as CGT. Based on CGT, we use CLIP[[9] or Verd2 to obtain the similarity matrix mentioned in section 3 We compare with the uniform label smoothing (Uni. L.S), where the Sy uniformly distributed. We also provide the performance of Label Smoothing without our progressive schedule noted as LS (NO SCHE.) to have a comprehensive understanding of the method. Table66ssove shows that the CLIP + CGT performs the best among different schemes. This shows the importance of the initial state of the ST, which helps to decide the relevant information for the main condition.
Table 5: " sensitivity comparision.
Table 6: Different approaches for ST Eq
MODEL
IS (t)
FID (L)
sFID (L)
MODEL
15 (1)
FID (4)
SFID (L)
IMAGENKT 64x64
IMAGENET 64x64
7 0.04
46.88
5.16
6.72
CLIP CGT
46.88
5.16
6.72
7 0.06
48.56
5.4
7.31
W2V + CGT
55.69
6.72
8.79
0.1
56.84
7.28
9.58
7 0.2
57.75
8.67
12.34
26.66
16.39
13.13
7 (NO SCHE.)
7.97
73.10
19.77
LS (NO SCHE.)
25.32
9.98
6.62
Computational cost: Considering the comparable FID/sFID results between ProG and classifier-fre free guidance, we uncovered several advantages of ProG over classifier-free guidance, as detailed in Section of the Supplementary. Notably, one of these ad- vantages pertains to computational efficiency. Table 7 illustrates the GPU hours required for generating 50,000 images at resolution of 256x256. When ProG is incorporated into the standard classifier guid- ance framework, the computational cost remains on par with vanilla classifier guidance. In contrast, uti- lizing classifier-free guidance substantially escalates computational expenses. It's worth noting that the computational cost of classifier guidance is predominantly influenced by the architecture of the noise-aware classifier, which is typically simpler than the network architectures used in generative models. Consequently, the observations Table are broadly applicable in most scenarios.
Table 7: Computational cost to generate
50000 images with ImageNer256x2566
MODEL
GPU HOURS
IMAGENET256X256
ADM
236
CADM-G + EDS
341
CADM-G + EDS + PROG
341
CADM CL-S-FREE
487
Conclusion
This work quantifies two problems of the classifier guidance method: dramatic diversity suppression and non-robustness feature construction. The results show better robustness features than the classifier guidance baseline and similar level to the classifier-free guidance method. Besides, we also achieve significantly better diversity trend when increasing the guidance scale quantitatively and qualitatively. We can achieve the SOTA on ImageNet64x64 and ImageNet128: 28x on FID/sFID, Our method can be combined with DiT-G 48 to achieve new SOTA for ImageNer256x256. Our current research demonstrates successful solutions to address the adversarial effects, particularly in cases where high-confidence generated samples exhibit minimal features relevant to the given condition. However, we acknowledge that we have not yet resolved the scenario where diffusion signals dominate and conflict with the classification signals. The resulting images have low confidence and fail to contain any discernible features related to the specified condition. is worth noting that these features can no longer be considered adversarial due to low confidence and often be wrongly classified by the pretrained classifier used in the sampling process. Resolving the conflict between diffusion and classification gradients remains open challenge and area for future investigation.
Acknowledgement
This work was supported in part by the Australian Research Council under Projects DP210101 859 and FT230100549. Besides, the AI training platform supporting this work was provided by High-Flyer AI (Hangzhou High-Flyer AI Fundamental Research Co., Ltd.)
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Advances in Neural Information Procexsing Systems, 33:6840-681,, 2020.
[2] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.02502, 2020.
[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference Machine Learning, pages
PMLR, 2021.
[4] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[5] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 13345 2020.
[6] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pages 2642-2651. PMLR, 2017.
[7] Minguk Kang. Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. Advances in Neural Information Processing Systems, 34:23505 23518, 2021.
[8] Liang Hou, Qi Cao, Huawei Shen, Siyuan Pan, Xiaoshuang Li, and Xueqi Cheng. Conditional gans with auxiliary discriminative classifier. In International Conference on Machine Learning, pages 8888-8902. PMI.R, 2022.
[9] Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1096, 2018.
[10] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jankko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8110-8119 8119, 2020.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8788 0794088774222 2021.
[12] Chen-Hao Chao, Wei-Fang Sun, Bun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Yi Yi Lee. Denoising likelihood score matching for conditional score-based data generation. arXiv preprint arXiv:2203. 14000 2022.
[13] Xihui Liu, Dong Huk Park, Samanch Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 289-299, 2023.
[14] Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
[15] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Boh McGrew, Ilya Sutskever. and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusson models. arXiv preprint arXiv:2112.10741 2021.
[16] Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. International Conference on Machine Learning, pages 10597-10606. PMLR, 2020.
[17] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In proceedings of the AAAI conference artificial intelligence, volume 34, pages
2020.
[18] Xiuwen Gong, Dong Yuan, and Wei Bao. Understanding partial multi-label learning via mutual information. Advances in Neural Information Processing Systems, 34:4147-4156 2021.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.

---

[20] Yiheng Liu, Tianle Han, Siyuan Ma. Jiayue Zhang, Yuanyuan Yang, Jiamiigg Tian, Hao He. Antong Li. Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang. Dingang Shen, Tianming Liu, and Bao Ge. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models, 2023.
[21] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 170100548, 2017.
[22] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 281 18-2826, 2016.
[23] Yunke Wang, Xiyu Wang, Anh- Dang Dinh, Bo Du, and Charles Xu. Learning to schedule in diffusion probabilistic models. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2478-2488, 2023.
[24] Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. arXiv preprint arXiv:2203. 13508, 2022.
[25] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint 2Xxx:2201.06502 2022.
[26] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances in neural information processing systems, 31, 2018.
[27] Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[28] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisa- tion for generative adversarial networks. arXiv preprint arXiv:1912.00953 2019.
[29] Dinh Anh Dung and Huynh Thi Thanh Binh. Gdegan: Graphical discriminative embedding gan for tabular data. In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pages 1-11. IEEE, 2022.
[30] Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. Advances in neural information processing systems, 32, 2019.
[31] Yunke Wang. Chang Xu, Bo Du, and Honglak Lee. Learning weight imperfect demonstrations. In International Conference on Machine Learning, pages
PMLR, 2021.
[32] Chengbin Du, Yanxi Li, Zhongwei Qiu, and Chang Xu. Stable diffusion is unstable. arXiv preprint arXiv:2306.02558 2023.
[33] Daochang Liu, Qiyue Li. Anh- auh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10139-10149, 2023.
[34] Shoufa Chen, Peize Sun. Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEEJCVF International Conference on Computer Vision, pages 19830-19843, 2023.
[35] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 15:12488-12445 2020.
[36] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10622 2022.
[37] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint arXiv:2206.017/4 2022.
[38] Dongjun Kim, Yeongmin Kim. Se Jung Kwon, Wanmo Kang, and 11-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models, 2023.
[39] Anh- aah-Dung Dinh, Daochang Liu, and Chang Xu. PixelAsParam: A gradient view on diffusion sampling with guidance. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 8120-88 8137. PMLR, 23 29-29 Jul 2023.
12

---

[40] Ye Liu, Yaya Cheng, Lianli Gao, Xianglong Liu, Qilong Zhang, and Jingkuan Song. Practical evaluation of adversarial robustness via adaptive auto attack. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15105-15114, 2022.
[41] lan Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and hameessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
[42] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706 06083, 2017.
[43] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. International conference on machine learning, pages 2206-2216. PMLR. 2020.
[44] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 59. Ieee, 2017.
[45] Guangeong Zheng, Shengming Li, Hui Wang, Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li. Entropy-driven sampling and training scheme for conditional diffusion generation. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXII, pages 754-769. Springer, 2022.
[46] Ali Razavi. Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[47] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021.
[48] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[49] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[51] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4700-4708, 2017.
[52] Forrest landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 2016.
[53] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 131.31130 2013.
13


Reviews:
[{"review_id": "LLPcMGXJ2T", "reviewer": "Reviewer_zai6", "summary": "In this work authors propose to address challenges of classifier guidance Diffusion models and propose progressive-guidance where during sampling/reverse diffusion process. Initial iterations of reverse process receives classifier gradient from multiple relevant classes instead of just target class so that more relevant features can be retained. Authors also illustrate this enables generated images to be more diverse as features in intial iterations need not 'ONLY' be purely discriminative for current target class of interest.", "questions": "In terms of Suppressed samples illustration and sample diversity, empirical metrics demonstrate on-par or worse diversity except ImageNet 64 x 64? So it is unclear in what setting is proposed method effective and what are current limitations of proposed approach. \n            Also, it might be worthwhile to consider evaluation for few interesting sub-sets rather than dataset level, as it might be useful to evaluate in settings  where categories have large sub-classes which needs to capture fine-grained details vs not many sub-classes? Does Pro-G have sensitivity \n\nWhat is sensitivity of proposed progressive guidance method based on accuracy of noise-aware classifier at different noise-levels? Analyzing such sensitivity and how it effects FID, Precision/Recall w.r.t generation would be informative. \n               Also, how challenging is it to train noise-aware classifier? As except few of initial works there aren't many follow up works which use classifier guidance as pointed out by authors on few of challenges. I understand authors used pre-trained checkpoints but it would be informative for community towards better interpretation of applicability of proposed method and classifier guidance more generally.", "limitations": "Proposed method is simple and effective and its encouraging to see that classifier guidance is on-par with classifier free guidance in terms of sample quality and Diversity, but the quality boost is marginal.\n\nI am not sure if this is valuable enough contribution in terms of novelty and insights for NeurIPS, as proposed method is straight forward and does not provide extensive novel insights either from Diffusion models behavior or empirical properties of classifier guidance.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Authors propose a simple and useful method to improve sample diversity of Diffusion models within classifier guidance setting and in appendix they show sample diversity is on-par with classifier-free guidance. \n\nProgressive Guidance makes sense more generally from generation perspective too, as within generative paradigm we first sample higher level semantics which is not very fine-grained then conditioned on that we sample latent variables/features relevant to fine grained details so it does make sense that we don't want to hyper-focus on one-particular class but it probably depends on complexity of class-taxonomy of particular dataset.", "weaknesses": "Core idea in paper is easy to follow but lack novel insights or significant contributions. \n\nFew suggestions in terms of details and writing: \n\nThough this paper follows classifier guidance from previous literature, it might be useful to summarize noise-aware classifier performance at difference noise-levels, especially from what iteration is classifier gradient incorporated in sampling process.\n\nWhat is value of gamma at different sampling steps w.r.t number of inference steps and schedulers to easily interpret results and setting.\n\nWhat is actual guidance scale value across sampling steps/reverse diffusion, more specifically at \\gamma = 0.04 at what stage of sampling is classifier guidance focusing on one-hot vector. If you can explicitly state that, it might be useful to the reader to easily interpret setting.\n\t As in later iterations of sampling process much of semantics or high-level features are inferred and guidance might not play such a vital role?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness discussion\n\n### Weakness 1: Noise-aware classifier performance at different noise-level and the sensitivity associated with $\\\\gamma$.\n\nIn the ablation study in section 6.3, we have already discussed the sensitivity of $\\\\gamma$ value that has affected the generated image quality. Table 5 in the main paper shows the sensitivity that affects the IS/FID and sFID.\n\nWe further follow the suggestion of the reviewer to explore the sensitivity of $\\\\gamma$ with the performance of noise-aware at different timesteps with image quality as in Table R3:\n\n| $\\\\gamma$ | FID  | Acc@25 | Acc@75 | Acc@150 | Acc@200 | Acc@250 |\n|----------|------|--------|--------|---------|---------|---------|\n| 0.04     | 5.16 | 00.00  | 0.31   | 20.00   | 78.42   | 100     |\n| 0.06     | 5.4  | 00.00  | 0.31   | 20.31   | 79.06   | 100     |\n| 0.1      | 7.28 | 00.00  | 0.31   | 21.87   | 78.75   | 99.68   |\n| 0.2      | 8.67 | 00.00  | 0.31   | 20.62   | 79.37   | 100     |\n\nTable R3: Sensitivity of $\\gamma$ regarding FID and the noise-aware classifier accuracy. Acc@25 means the classifier's accuracy at the $25^{th}$ timestep.\n\nAs we can observe, FID is very sensitive with $\\gamma$, which means the generated image quality is heavily affected by $\\\\gamma$. However, the classifier performance at different noise levels has little sensitivity regarding $\\gamma$ and has little correlation with the image quality.\n\n### Weakness 2: Value of $\\\\gamma$ at different timestep\n\nThe value of $\\gamma$ does not change following the timesteps. This value represents how fast the information degree should converge to a one-hot vector and is kept constant throughout the sampling process. \n\n### Weakness 3: Trend of guidance scale value for the primary class\n\nGiven $\\\\gamma = 0.04$, the information degree vector often converges into a one-hot vector at around $50^{th}$ timestep. We can observe the trend in Figure R1 in the attached pdf file in the joint rebuttal. In our experience, guidance plays a significant role in fine-tuning class-specific details at the end of the sampling process.\n\n## Question discussion\n\n### Question 1: Clarification on the diversity performance\n\nDiversity improvement is not only available for ImageNet64x64. For each resolution, we have:\n\n**ImageNet64x64**: Figure 4(a), (b) in the main paper \n\n**ImageNet128x128**: Figure 9 (left), (right) in Appendix\n\n**ImageNet256x256**: Table 4 (right Figure) in the main paper, and Figure 10, Figure 16 in Appendix.\n\nAll the figures indicate a clear superiority of the proposed ProG compared to vanilla classifier guidance.\n\n*Why do some Recall values in Tables 1 and 3 have similar values between ProG and vanilla guidance?*\n\nBecause we keep the same $w$ values as in the original papers. Except ImageNet64x64 ($w=4$) and ImageNet256x256 with unconditional diffusion model ($w=10$), other resolutions such as on ImageNet128, the gradient scale $w = 0.5$, ImageNet256(with EDS) $w= 0.5$, ImageNet256 (conditional diffusion) $w=0.7$. With very small $w$, vanilla guidance can achieve good IS/FID/sFID/Recall value but sacrifice the conditional information, meaning that the generated images do not have class information. Due to this small amount of gradients in the sampling process, the application of ProG in these cases did not bring such significant results in diversity. In this work, we show that, by increasing $w$ to achieve conditional information, ProG helps to avoid sacrificing diversity.\n\n### Question 2: Limitations of the proposed approach\n\nAs discussed in lines 294 to 299 in the main paper, although our proposed ProG successfully alleviates the adversarial effects where the images have high conditional confidence but has many suspicious features, it currently can not solve the case where we achieve low conditional confidence for the image. That is the case where classification information is ignored during the sampling process. \n\n\n### Question 3: Clarification about the extensive novel insights for the classifier guidance\n\nWe INDEED incorporated **7 insights** for the behavior of the vanilla classifier guidance for the diffusion model. (all images after Figure 4 are in Appendix due to the page limitation)\n\nFor **diversity suppression**, we have done on the **subset** of many types of breeds in ImageNet:\n\n+ [**INS1**] Problem of front-face features collapsing of vanilla classifier guidance in Figure 5 (for Brittany Spaniel), Figure 6 (for English Springer), and Figure 7(for Welsh Springer Spaniel).\n+ [**INS2**] Problem of collapsing all images into single pose features in Figure 5 (for Brittany Spaniel)\n+ [**INS3**] Problem of green grass background in vanilla guidance for some types of dog Figure 7(for Welsh Sprinter Spaniel)\n\nFor **non-robust features construction**, we did mainly on some types of breeds and leopard class of ImageNet  :\n\n+ [**INS4**] Problem of losing background in vanilla guidance in Figure 8.\n+ [**INS5**] Problem of non-robust features construction of vanilla guidance in Figure 11 (for ImageNet64x64) and Figure 12, 13, 14 (For ImageNet256x256)\n\nFor intuition for solving the **two problems**, which are adversarial effects and diversity suppressions:\n\n+ [**INS6**] Figure 2 provides the intuition of how the information from other classes helps to avoid the lack of diversity.\n+ [**INS7**] Figure 3 provides the intuition on how the information from the tiger can help construct the leopard's robust features.\n\n### Question 5: The training of the noise-aware classifier\n\nThe training of a noise-aware classifier is mostly the same as that of a standard classifier. The only difference is the data augmentation step; in the noise-aware classifier, random noise is added to the image before training. As a result, training a noise-aware classifier is much more straightforward than training a diffusion model. We have tried training the models several times and have had no difficulty during training. The training details and hyperparameters are on page 27 in [10].\n"}, {"review_id": "l1KUqyLOpF", "reviewer": "Reviewer_zHLG", "summary": "This work points out that diffusion models with classifier guidance only focus on the given category but ignore the other relevant category information. Thus, this work proposes the Progressive Guidance (PG) method to address two problems, i.e., lack of diversity and the adversarial effect (samples have high scores but poor visual quality). The proposed method uses progressive gradients from the class dimension and the diffusion temporal dimension to change the gradient of the classifier guidance on a single condition. In terms of the class dimension, PG allows gradient-assisted conditions to be generated for other class information related to a given class. In terms of diffusion temporal dimension, the weight of the gradient also changes over time. The experimental results show that PG improves image quality, sample diversity, and robustness compared with the competing methods.", "questions": "Please see my comments above.", "limitations": "Please see my comments above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow.\n\n2. This work proposes a novel way to improve the classifier guidance method.\n\n3. The proposed dramatic diversity and non-robustness feature construction approaches show desirable robustness than the commonly used baseline method.\n\n4. The proposed method can be combined with powerful backbone networks to achieve favorable performance.", "weaknesses": "1. Though the theoretical analyses are convincing, the experimental results show that the proposed method sometimes underperforms the classifier-free guidance. The authors should explain this point to verify the effectiveness of their proposed method.\n\n2. In Section 3, the analyses could be more convincing if more evidence is provided. In addition, the presentation of Sections 3.1 and 3.2 should be improved to make it clear.\n\n3. The computational costs should be clarified, considering that the diffusion models are usually expensive to implement for producing high-resolution images.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Question 1: Clarify the performance of classifier guidance in some cases.\n\nSometimes, the proposed classifier guidance performs poorly than classifier-free guidance, such as FID/sFID and Robustness score on ImageNet256x256. However, the two methods can be considered comparable on this case due to several reasons:\n* The gap between FID/sFID between the two methods is not significant (3.84 vs. 3.76) (~0.08)\n* The gap between Robustness score between the two methods is not so significant ( 86.60 vs 87.14) (~0.54%)\n* ProG has a much **better level of diversity** than classifier-free guidance, as shown in Figure  16(right) in the Appendix. FID and Recall values are much higher than classifier-free guidance when $w$ is large.\n* ProG has a much **lower computational cost** compared to classifier-free guidance, as shown in Table 8 (Appendix). Classifier guidance only needs 341 GPU hours to generate 50000 images compared to 487 for classifier-free guidance.\n\nBesides the comparability in the performance between the two methods, classifier guidance has several advantages over classifier-free guidance in terms of application:\n1. **Training flexibility**:  Classifier-free guidance used the information from the conditional diffusion model. As a result, when the condition is modified, or a new condition is available, there would be no light-cost solution but to train the whole expensive diffusion model. On the other hand, since classifier guidance allows separate classifiers. The update in condition allows updating on the classifier only without re-training expensive diffusion models.\n2. **Sampling flexibility**: We can apply the classifier-free technique only when we have both unconditional and conditional diffusion models simultaneously (can be separated or joined). However, classifier guidance can be used given solely unconditional diffusion, solely conditional diffusion, or both.\n\n### Question 2: Analyse in section 3 should be improved, and more evidence should be provided.\n\nDue to the page limit, most of the evidence from section 3 is put into the Appendix. We will return this evidence to the main paper for a better reading experience. In detail, we have:\n\nFor **diversity suppression**, we have done on many types of breeds in ImageNet:\n\n+ [**EVD1**] Problem of front-face features collapsing of vanilla classifier guidance in Figure 5 (for Brittany Spaniel), Figure 6 (for English Springer), and Figure 7(for Welsh Springer Spaniel).\n+ [**EVD2**] Problem of collapsing all images into single pose features in Figure 5 (for Brittany Spaniel)\n+ [**EVD3**] Problem of green grass background in vanilla guidance for some types of dog Figure 7(for Welsh Sprinter Spaniel)\n\nFor **non-robust features construction**, we did mainly on some types of breeds and leopard class of ImageNet  :\n\n+ [**EVD4**] Problem of losing background in vanilla guidance in Figure 8.\n+ [**EVD5**] Problem of non-robust features construction of vanilla guidance in Figure 11 (for ImageNet64x64) and Figure 12, 13, 14 (For ImageNet256x256)\n\n### Question 3: The computational cost\n\nWe have mentioned the computational cost in Table 8 (Appendix). To clarify the computational cost, we will revise Table 8 into the below table, where Diffusion computational cost and vanilla guidance cost will be detailed.\n\n|             Model        | Computational cost (GPU hours) |\n|:------------------------:|:------------------:|\n| Diffusion                |         236        |\n| Vanilla guidance    |         341        |\n| ProG guidance    |         341        |\n| Classifier-free guidance |         487        |\n\nTable R4: Computational cost to generate 50000 images with 256x256 resolution."}, {"review_id": "S7bqnqLvHc", "reviewer": "Reviewer_cJ7t", "summary": "This paper proposes to inject the graident of other clasess to improve the diversity for conditional sampling of diffusion model.", "questions": "as above", "limitations": "as above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The general idea is simple, the gradient of the classifier tends to use the most discriminative feature and thus hurts the performance, so we use the gradient of other classes in early phases to improve the diversity.\n\n2. The paper also provides some entropy arguments, which partly justifies the method.\n\n3.Extensive experiments are conducted and the performance gain is clear.", "weaknesses": "1. Missing details. How many steps are used for the eq 4?. \"In the later sampling stage, we progressively enhance gradients to refine the details 13 in the image toward the primary condition\" Do you just use classifier guidance after some steps?\n\n2. Authors use CLIP embedding to compute the similarity between different classes. Can we use clip gradient to guide the generation directly (insert the graident of CLIP similarity between the generated image and target class, say \"dog\" )  in DDIM steps? Will that also improve the diversity as CLIP has seen many different dog images. Some comparions are needed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\n### Question 1: Elaborating on details\n\n1. For equation 4, the number of timesteps is 1000. However, similar to [10], we adapt the respace to 250 timesteps (skip four iterations each time). All other hyperparameters are detailed in Table 10 (Appendix)\n2. \"*Do you just use classifier guidance after some steps*\"? -> No, we use classifier guidance from the start to the end.\n3. \"*In the later sampling stage, we progressively enhance gradients to refine the details in the image toward the primary condition*.\" We mean that when we start, the gradients from other classes are noisy. Following the iterations, the gradients from different categories are reduced, and the gradient from the primary class is emphasized (more weight).  \n\n### Question 2: Use CLIP embedding for guidance\n\nThis would be an exciting idea since it allows using off-the-shelf information to improve generated images from diffusion models. We set up the experiments as below:\n1. Use pretrained CLIP from GLIDE [11] (an extension of classifier guidance for text-to-image guidance)\n2. Use the pretrained Diffusion model (ADM) from [10] as a model to generate ImageNet\n\nThe method is named CLIP guidance. We evaluate on ImageNet64x64. The results are shown in Table R5:\n\n\n|                  | FID | sFID | Recall |\n|:----------------:|:---:|------|--------|\n| Vanilla guidance | 6.40  | 9.67    | 0.54   \t     |\n| ProG guidance    | **5.16**  | **6.72**    | 0.56   \t |\n| CLIP guidance    | 8.18  | 9.4     | **0.59**   \t |\n| Diffusion w/o guidance| 9.95| 6.58| 0.65|\n\nTable R5: ImageNet64x64. Comparison between Vanilla guidance, ProG guidance, and CLIP guidance.\n\nThe results show that the ProG achieves the best FID/sFID among the guidance schemes. However, the Recall value of CLIP guidance is the highest one. This indicates the ProG provides the diversity, but in the original data distribution, while the CLIP guidance could help to achieve better variety outside of the data distribution. It must be noted that the FID obtained from the CLIP guidance is better than the original diffusion model without guidance. This is concrete evidence that the guidance from off-the-shelf information is working. However, it might need a lot of experiments to observe the fundamental problems inside. We like the idea and would be interested in further developing this idea in future work. \n\n"}, {"review_id": "C4hhPddCuU", "reviewer": "Reviewer_xoqA", "summary": "To tackle the generative issue of low diversity and artifacts of classier guidance for conditional diffusion generation, this paper proposes an entropy view for calculating the conditional score gradient. The proposed method proposes two modifications for classifier guidance, 1) exchanging one-hot class labels with soft-labels based on class similarity. 2) progressive score weights for different time steps. And the proposed achieves better results than the baseline method and classifier-free guidance method.", "questions": "Please refer to the weakness.", "limitations": "This paper does not discuss the limitations. However, the proposed method might not be general for conditional generation with diverse modalities of conditional labels (etc. text, segmentation map).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple and straightforward yet effective to achieve better generative results.\n2. The proposed method suggests an entropy view to review the classifier guidance method is interesting. In the real dataset, we should consider the real conditional distribution of the label given image, which is neglected by the vanilla one.", "weaknesses": "1. Although the proposed method is simple and straightforward, the flexibility would be restricted for different modalities of conditions such as (text and segmentation map), which can be easily addressed via classifier-free guidance. Is there any solution for classifier guidance with flexible conditional labels (different modalities)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\n### Question 1: Extend the proposed method into Text-to-image guidance.\nGLIDE[11] is an easy extension of classifier guidance for text-to-image guidance. The sampling equation for GLIDE is shown below:\n\n$$x_{t-1} = \\mu _t + \\sigma _t * \\mathbf{z} + s \\sigma_t^2 \\nabla _{x_t} (f(x_t) . g(c))\\quad \\quad _{(r1)}$$\n\n$f(x_t)$ is the image embedding vector and $g(c)$ is the text or description embedding vector. Equation (r1) is mostly similar to equation (3) in our main paper; the only difference is the gradient term resulted from the similarity between two embedding vectors instead of the classification gradient as in the main paper.\n\nWe add new experiments to apply ProG for GLIDE in equation (r1) following two scenarios:\n1. Given one caption, we will utilize a random set of 1000, 5000, or 10000 captions to act as relevant information during sampling. we have: $$g(c) = \\sum_{i=0}^{N+1} s_i g(c_i)\\quad\\quad_{(r2)}  $$ with $i = 0$ is the index of the primary caption, and $i \\neq 0$ represents other captions. The initial  values of $s_i$ are set as: $$s_i = \\frac{g(c_0) . g(c_i)}{\\sum^{N+1}_{j=0} g(c_0) .g(c_j)} \\quad \\quad _{(r3)}$$ The value of $s_i$ is progressively updated during sampling as in section 3.2 in the main paper. This scheme is named **GLIDE-ProG**\n2. Given one caption, we use four other captions that have the same meaning as the original caption but different words. Since four other captions, all have the same meaning; we have different strategies to set the $s_i$ values: $$s_i = \\begin{cases}a, \\quad \\text{if } i = 0\\\\\\ \\frac{1-a}{4}, \\quad \\text{otherwise} \\end{cases}$$, with $0\\leq a \\leq 1$ is hyperparameter, we try with $a=0.3$. This method is named **GLIDE-ProGsim**\n\nWe set up an evaluation like GLIDE [11] to evaluate zero-shot FID on MS-CoCo. Note: 4 additional equivalent captions of GLIDE-ProGsim are taken from the set of captions available for each image in MS-Coco captions. 1k, 5k, and 10k captions are randomly sampled from the MSCoco training set. Table R1 and Table R2 show the evaluation results:\n\n| | zero-shot FID | computational cost (GPU hours)|\n|:----:|:------:|--------|\n| GLIDE |    24.80  | 34.27  |\n| GLIDE-ProG w N=1k |  23.47| 34.66 |\n| GLIDE-ProG w N=5k |23.50|\t34.83 |\n| GLIDE-ProG w N=10k|\t **23.31**|34.83|\n| GLIDE-ProGsim | 23.87 |34.84 |\n\nTable R1: MSCoCo64x64 zero-shot FID evaluation where 30000 captions are sampled from the MSCoco validation set.\n\n\n\n\n|               \t| zero-shot FID | computational cost (GPU hours) |\n|:----------------:|:-------------:|--------------------|\n| GLIDE         \t|      34.80    |      38.45              |\n| GLIDE-ProG w N=1k |      32.55    |    45.50                |\n| GLIDE-ProG w N=5k |\t   32.37\t|\t\t\t\t45.80\t |\n| GLIDE-ProG w N=10k|\t   32.28\t|\t\t\t\t46.10\t |\n| GLIDE-ProGsim \t|      **31.91**    |   46.23                 |\n\nTable R2: MSCoCo256x256 zero-shot FID evaluation where 30000 captions are sampled from the MSCoco validation set.\n\n**Conclusion**: From Table R1 and Table R2, the ProG scheme helps significantly improve the performance of text-to-image guidance in different scenarios with low additional computational costs. Given the many captions available, we can use the first scenario to improve the generated images. Otherwise, the second scenario is also very easy to implement. The additional captions can be gathered from Large Language Models (LLMs) to generate images.\n\n### Question 2: The flexibility of classifier guidance.\n\nLike classifier-free guidance, classifier guidance can extend to different modalities. More than that, classifier guidance also has many advantages:\n\n* Training: Classifier-free needs complete retraining of diffusion models for new conditions, while classifier guidance only requires classifier updates.\n* Sampling: Classifier guidance works with unconditional or conditional diffusion, unlike classifier-free, that requires both.\n* Computational cost: Classifier-free is computationally expensive (Table R6).\n* Extendability: Both support or can extend to various conditions, e.g., Text to image.\n\n| Model| Sampling cost (GPU hours) |\n|:--:|:----:|\n| Diffusion  |  236 |\n| Vanilla guidance |  341  |\n| ProG guidance | 341|\n| Classifier-free guidance | 487|\n\nTable R6: Computational cost to generate 50000 images with 256x256 resolution.\n\nWe summarize the features of each guidance method in Table R7.\n\n| | **Training flex.** | **Sampling flex.** | **Low cost** | **Robustness** | **Diversity** | **Extendibility** |\n|--|:-----:|:--:|:--:|:---:|:--:|:----:|\n| Vanilla guidance   | yes | yes | yes | no | no | yes  |\n| Classifier-free guidance | no  | no   | no  | yes  | no | yes|\n| ProG   | yes  | yes | yes  | yes | yes | yes|\n\nTable R7: As we can see, the main reason for the popularity of classifier-free guidance is its robust features. However, ProG can combine all the advantages of Vanilla and Classifier-free guidance in one unified scheme.\n"}, {"review_id": "QDTl3rvg2h", "reviewer": "Reviewer_vxij", "summary": "In this paper, the authors proposed a new classifier guidance technique for diffusion models named Progressive Guidance (PROG). PROG is an extended version of the classifier guidance method and incorporates the relevant classes' information (beyond the target class alone) to determine the guidance direction, particularly in the presence of noisy images during the early sampling stage. Through extensive experiments on various datasets and diffusion models, the authors demonstrate the effectiveness of PROG in comparison to the standard classifier guidance technique.", "questions": "see above", "limitations": "As mentioned in the weaknesses section, I think that the performance improvement of the proposed method over the standard classifier guidance is marginal, and the limited applicability of classifier guidance is a significant limitation of this paper. To strengthen the submission, I recommend that the authors emphasize the strong advantages of their approach compared to standard method (classifier guidance). By highlighting these unique aspects, the authors can make their submission more compelling and is able to address the aforementioned limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "[1] I think the paper is well written and easy to understand.", "weaknesses": "[1] I think the technical novelty of this paper may not meet the standards required for acceptance at NeurIPS.\n\n[2] In my opinion, the proposed PROG technique is only marginally better or comparable to the standard classifier guidance method (according to Tables 1 and 3).\n\n[3] Furthermore, I am not sure about whether the proposed classifier guidance (PROG) is superior to classifier-free guidance in terms of performance and applicability. The image generation community has recently witnessed a rapid shift from class conditional image synthesis to text-to-image synthesis. However, it appears that the proposed classifier guidance method may not be applicable to text-to-image synthesis tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Question 1: The novelty of the work\nSince the reviewer does not mention the reason why our work is not novel, we summarize our three main novelties below:\n\n[**Nov1**] : Detects and justifies **diversity suppression** problem due to classification gradient. As far as we know, this problem has **NOT been investigated** before. In GANs, the mode collapse problem that causes the lack of diversity has a different essence than the diversity suppression problem. The problem can be observed in Figure 2 (main paper) and Figures 5, 6, and 7 (Appendix). => **novel**\n\n[**Nov2**]: Quantifies the problem of **adversarial effect** due to classification gradient as observed in Figure 3 (main paper), Figure 8, 11, 12, 13, 14 (Appendix). As far as we know,  this problem has **NOT been investigated** before. Although in [1], the authors have mentioned the adversarial effect of the classifier guidance, this problem has never been justified and solved. => **novel**\n\n[**Nov3**]: Develop an intuitive approach addressing both [**Nov1**] and [**Nov2**] simultaneously. First, mitigate adversarial effects by utilizing gradients from diverse classes, minimizing noise associated with one class. This is exemplified in Figure 3. Second, incorporate information from other classes to prevent diversity suppression and overemphasis on a single class, as illustrated in Figure 2. Lastly, introduce progressive guidance to enhance primary conditions toward sampling completion. Due to its unique philosophy and absence in prior literature, our proposal stands as a **novel** technical contribution.\n\n### Question 2: The significance of the proposed method compared to vanilla classifier guidance.\n\nIn the main paper and Appendix, we offer **5 significant improvements** compared to vanilla guidance:\n\n[**SIG1**] *Diversity quantitative* improvement: in Figure 4 (up to **45% improvement FID**, and **35% on Recall**); right Figure in Table 4 (improve up to **22% on FID** and **22.5% on Recall**), Figure 9 (improve up to **28.5% on FID** and **40% on Recall**) , Figure 10 (up to **50% on FID** and **37.5% on Recall**) in Appendix . Since the gap is large, it is **significant** enough to observe.\n\n[**SIG2**] *Quantitative robust features* improvement as detailed in Table 2, 4. The improvement is clear (around **6% in Table 2** and **3% in Table 4**). Thus,  this improvement is considered **significant**.\n\n[**SIG3**] *Diversity qualitative* improvement as detailed in Figure 2, 5 (right), 6 (right), 7 (right). The improvement can be observed by human eyes. As a result, it should be **significant**.\n\n[**SIG4**] *Qualitative robust features* improvement as detailed in Figure 3(right), Figure 8, Figure 11, 12, 13, 14. The improvement can be observed by human eyes, so we believe it is **significant**.\n\n[**SIG5**] *Quantitative improvement over some traditional generative metrics* such as FID, sFID and IS. The improvement is shown by percentage as in Table R8 and R9.\n\n| | FID imp.  | sFID  imp. | IS  imp. |\n|------|--|----|--\n| ImageNet 64x64   | 19.37%/ 20% / 7% | 30.5% /  4% / 6.27% | 0% / 12.54% / 0% |\n| ImageNet 128x128 |  7% | 0.1% | 11% |\n| ImageNet 256x256 | 1% / 6%   | 2.5% /18.5% | 2.5% / 18.5%|\n| CIFAR-10| 1.4%| 0.4% | -1.35%|\n\nTable R8: (Table 1 in main paper) Except for t the CIFAR-10,  we achieved significant improvement.\n\n|  | FID  imp.| sFID   imp.| IS  imp.|\n|----|--|--|--|\n| ImageNet 64x64   | 2.5%| 4.30% | 6.42%\t     |\n| ImageNet 128x128 | 4.3% | 0% | 9.40%        |\n| ImageNet 256x256 | 3% / 0.88%| 0%| 8.77% / 1.46%|\n\nTable R9:  (Table 3 in main paper) The improvement is clear on FID and IS on all databases and sFID on ImageNet64x64.\n\nIn Table R8, Table R9, [**SIG1**], [**SIG2**], [**SIG3**], [**SIG4**], significant improvements are evident, except for CIFAR-10. The limited progress on CIFAR-10 is due to its classes having little shared information, reducing ProG's impact.\n### Question 3.1: Extend ProG to Text-to-Image problem:\n\nWe have successfully extended our proposed ProG to improve guidance for Text-to-Image. The experimental settings and results are in the joint rebuttal to all reviewers.\n\n### Question 3.2: Text-to-image (Text2Img) condition vs. class condition (ClsCon)\nThe reviewer mentions a shift from ClsCon to Text2Img, but recent works in both years (2022 and 2023) indicate continued interest in ClsCon [2-9]. Both ClsCon and Text2Img are crucial in generative models, and it's unjust to prioritize one over the other.\n\n### Question 3.3: Classifier guidance vs. Classifier-free guidance\nThe reviewer mentions the limited application of classifier guidance compared to classifier-free guidance. However, classifier guidance offers more flexibility:\n\n* Training: Classifier-free needs complete retraining of diffusion models for new conditions, while classifier guidance only requires classifier updates.\n* Sampling: Classifier guidance works with unconditional or conditional diffusion, unlike classifier-free, which requires both.\n* Computational cost: Classifier-free is computationally expensive (Table R6).\n* Extendability: Both  can extend to various conditions, e.g., Text to image.\n\n| Model| Sampling cost (GPU hours) |\n|:--:|:----:|\n| Diffusion  |  236 |\n| Vanilla guidance |  341  |\n| ProG guidance | 341|\n| Classifier-free guidance | 487|\n\nTable R6: Computational cost to generate 50000 images with 256x256 resolution.\n\nWe summarize the features of each guidance method in Table R7.\n\n| | **Training flex.** | **Sampling flex.** | **Low cost** | **Robustness** | **Diversity** | **Extendibility** |\n|--|:-----:|:--:|:--:|:---:|:--:|:----:|\n| Vanilla guidance   | yes | yes | yes | no | no | yes  |\n| Classifier-free guidance | no  | no   | no  | yes  | no | yes|\n| ProG   | yes  | yes | yes  | yes | yes | yes|\n\nTable R7: As we can see, the main reason for the popularity of classifier-free guidance is its robust features. However, ProG can combine all the advantages of Vanilla and Classifier-free guidance in one unified scheme.\n"}, {"review_id": "lHVb2osKGa", "reviewer": "Reviewer_mBBP", "summary": "This paper proposes a generalized classifier guidance method for diffusion models with progressive guidance along both the class and temporal dimensions, to handle the adversarial effect and diversity suppression problems of the vanilla classifier guidance. In experiments, the proposed method shows advantages over the vanilla classifier guidance and achieves a new state-of-the-art when combined with other methods under certain settings.", "questions": "Please address the writing issues then I would consider this paper as a good contribution to the community.", "limitations": "The limitation is briefly discussed at the end of the paper. I think the authors should elaborate on this to provide more insights.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper has clear and well-established motivations, working on two important problems in classifier guidance encountered by the community, the adversarial effect and diversity suppression. This can be a good contribution to the community.\n\n2. The proposed progressive guidance method is simple, effective, intuitive, and well-aligned with the motivations. The entropy perspective is also very interesting.\n\n3. The paper is well-written, with good clarity, nice flow, and good intuition. At the same time, detailed explanations and illustrative analysis are provided for the method for better understanding. The reading experience is good.\n\n4. The experiments are relatively comprehensive with multiple datasets and baselines. The alleviation of the adversarial effect and diversity suppression is validated. The advantage over the vanilla classifier guidance is shown. New SOTA is achieved under certain settings.\n\n5. Code is available.\n", "weaknesses": "1. In Table 1, the results on CIFAR are not as good as on ImageNet. What is the potential reason and any insight?\n\n2. The information degree is based on the description generated by ChatGPT. What are the prompts? What is the impact of different prompts?\n\n3. Although the paper is well-written in general, there are still some clarity issues:\n- The derivation and introduction of the entropy perspective in the main paper are not very easy to understand. It should be improved in a more logical way in writing.\n- What do the superscripts on methods in Table 3 mean? What does “LS (NO SCHE.)” mean in Table 6 mean?\n\n4. There are also some minor writing issues:\n- In Figure 1c, the meaning of darkness is not stated.\n- In Figure 1c, it should be stated that “c1” is the condition.\n- In Figure 1 caption, missing space before “Dataset: ImageNet64x64”.\n- In L57, “Propose” > “Proposing”.\n- In L61, “over” > “of”.\n- In L61, “SOTA” should be full.\n- “Markov chain” > “Markovian chain”.\n- In section 2, the definition of epsilon and sigma is not mentioned. The background should be self-contained.\n- In L79, “log_phi p” should be “log p_phi”. Phi is not explicitly defined.\n- In L87-88, p should be p_phi. Make it consistent.\n- In L91, >= should be a single symbol.\n- The flow at the start of section 3.1 is too fast. Add some rationale and definition of information degree first.\n- In L131, missing space after “ChatGPT”.\n- In Eq.7, s_t,i should be s_i,t. Also check other places.\n- In Table 2, should “CADM-G+ProG” be “CADM+ProG”?\n- In L260, remove spacing after “EDS [32]”.\n- In Table 3 caption, “State” > “state”.\n- The capitalization in the reference should be corrected, such as “gans”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n### Question 1: Low improvement on CIFAR-10\nWe have analyzed this problem and discovered that the problem lies in the semantic labels of CIFAR10 itself. Compared to ImageNet, the labels in CIFAR-10 has less relevant information than each other. Most of the shared information between classes is background information. For example, an airplane and a bird only share blue background; an automobile might only share a green background or a street background with a horse. As a result, most of the supporting information from other classes quickly turns into noise during the sampling. In contrast, in ImageNet, many classes share similar features, such as a set of breeds, Brittany Spaniel, Standard poodle, keeshond, and Eskimo dog, that share a lot of backgrounds, colors, and poses. The shared information from relevant classes is beneficial for constructing the primary class.\n\n### Question 2: Prompts and its effect\n\nThe prompt I use has the form:\n```\nAdd text description. For example, \"Tench\" will turn into \"Characterized by its distinctive olive-green to golden-brown coloration, the tench has a robust and slightly elongated body with a rounded tail fin. It inhabits slow-moving or still waters such as lakes, ponds, and slow rivers across Europe and parts of Asia. Renowned for its adaptability to varying water conditions, the tench can thrive in environments with low oxygen levels due to its unique respiratory adaptations.\". Apply for the following fields:\n\n1. Goldfish, Carassius auratus\n2. Great white shark, white shark, man-eater, man-eating shark, Carharodon Zacharias\n```\n\nThe primary motivation is hinting at the type of description we want. Suppose we use different types of prompts that do not have a hint. The output is a lengthy paragraph that includes information unrelated to the description, such as its origination place or history, which is harder to do text preprocessing and less relevant to generate image features.\n\n### Question 3.1: Introduction and derivation of reverse entropy regularization\nWe will re-write this part in the final manuscripts. In detail, we will add an explanation for each constraint as below:\n1. Constraints (1), (2), (3) are to satisfy the information degree as a distribution with $s_c$ achieves the highest value. This is matched with the conditions of information degree mentioned in section 3.1.\n2. As mentioned in eq.6, when $\\mathbf{s}_t$ is varied, \n\n$$D_{KL}(\\mathbf{s} _t || p _{\\phi}(\\mathbf{y}|\\mathbf{x}_t)) = \\sum _{i=1}^C s _{t, i} \\log(s _{t, i}) - s _{t,i} \\log p _{ \\phi}(y_i | \\mathbf{x} _t)$$\nis considered in full form instead of only $\\sum _{i=1}^C -s _{t,i} \\log p _{ \\phi }(y_i | \\mathbf{x} _t)$ as equ. (4).  \n\nHowever due to the term $\\sum_{i=1}^C s_{t,i} \\log s_{t,i}$, the objective of  $\\min _{\\mathbf{s} _t} D _{KL}$ and equ. (7) in the main paper is conflict. To avoid this case, we introduce a constraint\n\n$$|s^*_{t,i} - s_{t,i}|  \\leq l, \\forall 1 \\leq i \\leq C \\quad \\quad _{(4)}$$\n\nThis constraint (4) helps to keep every value of $s_{i,t}$ can not change larger than the bound $l$ at one timestep, resulting in a minimal change toward KL divergence while minimizing the entropy objective.\n\n### Question 3.2: Clarification on Table 3 and Table 6.\n\nAs we mentioned in lines 83 - 85 Appendix, $+$ is denoted for the score evaluated by the samples provided by the paper. $++$ means the values are directly used from the main article due to the unavailability of the pretrained model. The $*$ over classifier-free guidance means that we don't have available implementation and the pretrained model from the paper. As a result, we do our implementation as well as evaluation. We will state these clearly in the final manuscript.\n\nIn Table 6, the LS (NO SCHE.) means we apply Label smoothing for information degree without a progressive schedule. This means that the values of information degree do not change during the sampling. This helps to verify that our improvement does not come from smoothing the gradient scales.\n\n### Question 4: Minor writing issues\nSince we can not upload the revision, we will edit the writing according to the reviewer's suggestion in the final manuscripts:\n1. The darkness in Figure 1c means the weight emphasized for that gradient. Darker means more weight put on that gradient.\n2. In Table 2, \"CADM-G + ProG\" is utilized consistently throughout the paper. Thus, it is correct. However, in Table 3, All \"CADM\" should be \"CADM-G\"; we will correct them in the final version.\n3. We will revise all the typos, grammar mistakes, and other space redundancy missing throughout the paper, including the reviewer's suggestion\n\n### Question 5: Clarification on the limitations of the work\n\nIn this work, we consider two problems: diversity and adversarial effects in generated images. For the adversarial effects, we characterize the problem as the generated image achieving very high confidence in the conditional class but having minimal features to belong to that class. However, we can observe many generated images that do not have high confidence in the conditional class. For these cases, our proposed method, ProG, can not solve. We hypothesize that the signals from the Diffusion model somehow have dominated the signals from the classifier under some conditions. This negates all the classifier gradients that can help to transform the images into the expected condition.\n"}]

Author rebuttal:
We extend our sincere gratitude to the esteemed reviewers for their insightful and constructive feedback, which has significantly contributed to the enhancement of our manuscript. In this joint reply, we address recurring inquiries raised by multiple reviewers and references for other rebuttals, thereby conserving space and referring to pertinent references to provide comprehensive responses. 

*Note: All the references to Table/Figure/Equation that appeared inside the rebuttal will have a mark "r" or "R" before the number to distinguish from numbering in the main paper*

### Gen. Question 1: Extend our proposed ProG for Text-to-Image guidance 

In GLIDE, [11] has proposed to extend classifier guidance for text-to-image guidance. The sampling equation for GLIDE is shown below:

$$x_{t-1} = \mu _t + \sigma _t * \mathbf{z} + s \sigma_t^2 \nabla _{x_t} (f(x_t) . g(c))\quad \quad _{(r1)}$$

$f(x_t)$ is the image embedding vector and $g(c)$ is the text or description embedding vector. Equation (r1) is mostly similar to equation (3) in our main paper; the only difference is the gradient term resulted from the similarity between two embedding vectors instead of the classification gradient as in the main paper.

Our proposed ProG is applied to GLIDE in equation (r1) following two scenarios:
1. Given one caption, we will utilize a random set of 1000, 5000, or 10000 captions to act as relevant information during sampling. we have: $$g(c) = \sum_{i=0}^{N+1} s_i g(c_i)\quad\quad_{(r2)}  $$ with $i = 0$ is the index of the primary caption, and $i \neq 0$ represents other captions. The initial  values of $s_i$ are set as: $$s_i = \frac{g(c_0) . g(c_i)}{\sum^{N+1}_{j=0} g(c_0) .g(c_j)} \quad \quad _{(r3)}$$ The value of $s_i$ is progressively updated during sampling as in section 3.2 in the main paper. This scheme is named **GLIDE-ProG**
2. Given one caption, we use four other captions that have the same meaning as the original caption but different words. Since four other captions, all have the same meaning; we have different strategies to set the $s_i$ values: $$s_i = \begin{cases}a, \quad \text{if } i = 0\\\ \frac{1-a}{4}, \quad \text{otherwise} \end{cases}$$, with $0 \leq a \leq 1$ is hyperparameter, we try with $a=0.3$. This method is named **GLIDE-ProGsim**

We set up an evaluation like GLIDE [11] to evaluate zero-shot FID on MS-CoCo. Note: 4 additional equivalent captions of GLIDE-ProGsim are taken from the set of captions available for each image in MS-Coco captions. 1k, 5k, and 10k captions are randomly sampled from the MSCoco training set. Table R1 and Table R2 show the evaluation results:

| | zero-shot FID | computational cost (GPU hours)|
|:----:|:------:|--------|
| GLIDE |    24.80  | 34.27  |
| GLIDE-ProG w N=1k |  23.47| 34.66 |
| GLIDE-ProG w N=5k |23.50|	34.83 |
| GLIDE-ProG w N=10k|	 **23.31**|34.83|
| GLIDE-ProGsim | 23.87 |34.84 |

Table R1: MSCoCo64x64 zero-shot FID evaluation where 30000 captions are sampled from the MSCoco validation set.

|| zero-shot FID | computational cost (GPU hours) |
|:---:|:--:|---|
| GLIDE  | 34.80| 38.45  |
| GLIDE-ProG w N=1k | 32.55 |    45.50 |
| GLIDE-ProG w N=5k |32.37	|45.80|
| GLIDE-ProG w N=10k|	32.28|46.10|
| GLIDE-ProGsim 	|  **31.91**  |46.23|

Table R2: MSCoCo256x256 zero-shot FID evaluation where 30000 captions are sampled from the MSCoco validation set.


**Conclusion**: From Table R1 and Table R2, the ProG scheme helps significantly improve the performance of text-to-image guidance in different scenarios with low additional computational costs. Given the many captions available, we can use the first scenario to improve the generated images. Otherwise, the second scenario is also very easy to implement. The additional captions can be gathered from Large Language Models (LLMs) to generate images.

**Reference**:

[1] Ho, J., & Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.

[2] Gao, S., Zhou, P., Cheng, M. M., & Yan, S. (2023). A masked diffusion transformer is a strong image synthesizer. arXiv preprint arXiv:2303.14389.

[3]  Kim, D., Kim, Y., Kwon, S.J., Kang, W. &amp; Moon, I.. (2023). Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models. Proceedings of the 40th International Conference on Machine Learning

[4] Hang, T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., ... & Guo, B. (2023). Efficient diffusion training via min-snr weighting strategy. arXiv preprint arXiv:2303.09556. (ICCV2023)

[5] Peebles, W., & Xie, S. (2022). Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748.

[6] Sauer, A., Schwarz, K., & Geiger, A. (2022, July). Stylegan-xl: Scaling stylegan to large, diverse datasets. In ACM SIGGRAPH 2022 conference proceedings (pp. 1-10).

[7] Singh, R., Shukla, A., & Turaga, P. (2023). Polynomial Implicit Neural Representations For Large Diverse Datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2041-2051).

[8] Ganz, Roy, and Michael Elad. "BIGRoC: Boosting Image Generation via a Robust Classifier." Transactions on Machine Learning Research (2023).

[9] Dinh, A., Liu, D. &amp; Xu, C.. (2023). PixelAsParam: A Gradient View on Diffusion Sampling with Guidance. Proceedings of the 40th International Conference on Machine Learning

[10] Dhariwal, P., & Nichol, A. (2021). Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34, 8780-8794.

[11] 
Nichol, A.Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., Sutskever, I. &amp; Chen, M.. (2022). GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.Proceedings of the 39th International Conference on Machine Learning




Meta-review: The paper received all positive reviews. Some minor technical concerns about writing and experiments raised by the reviewers were then solved by the rebuttal. Overall this work points out that previous diffusion models with classifier guidance only focus on the given category while ignoring the other relevant category information, and proposes the Progressive Guidance (PG) method to address the problems, allowing relevant classes' gradients to contribute to shared information construction.  Such method can be of large interest to the community working on classifier-guided diffusion models.
Decision: Accept (poster)