Title: Rethinking Conditional Diffusion Sampling with Progressive Guidance
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion model

conditional generative model

guidance diffusion

generative models

classifier guidance
Abstract: This paper tackles two critical challenges encountered in classifier guidance for diffusion generative models, i.e., the lack of diversity and the presence of adversarial effects. These issues often result in a scarcity of diverse samples or the generation of non-robust features. The underlying cause lies in the mechanism of classifier guidance, where discriminative gradients push samples to be recognized as conditions aggressively. This inadvertently suppresses information with common features among relevant classes, resulting in a limited pool of features with less diversity or the absence of robust features for image construction.	We propose a generalized classifier guidance method called Progressive Guidance, which mitigates the problems by allowing relevant classes' gradients to contribute to shared information construction when the image is noisy in early sampling steps. In the later sampling stage, we progressively enhance gradients to refine the details in the image toward the primary condition. This helps to attain a high level of diversity and robustness compared to the vanilla classifier guidance. Experimental results demonstrate that our proposed method further improves the image quality while offering a significant level of diversity as well as robust features.

Rethinking Conditional Diffusion Sampling with Progressive Guidance

Abstract

This paper quantifies and tackles two critical challenges encountered in classifier guidance for diffusion generative models, i.e., the lack of diversity and the presence of adversarial effects. These issues often result in a scarcity of diverse samples or the generation of non-robust features. The underlying cause lies in the mechanism of classifier guidance, where discriminative gradients push samples to be recognized as conditions aggressively. This inadvertently suppresses information with common features among relevant classes, resulting in a limited pool of features with less diversity or the absence of robust features for image construction. We propose a generalized classifier guidance method called Progressive Guidance, which mitigates the problems by allowing relevant classes' gradients to contribute to shared information construction when the image is noisy in early sampling steps. In the later sampling stage, we progressively enhance gradients to refine the details in the image toward the primary condition. This helps to attain a high level of diversity and robustness compared to the vanilla classifier guidance. Experimental results demonstrate that our proposed method further improves image quality while offering a significant level of diversity as well as robust features. Source code is available at: https://github.com/dungdinhanh/prog-guided-diffusion .

1 Introduction

Diffusion model [1, 2, 3, 4, 5] has emerged as a state-of-the-art technique for generating high-fidelity images without relying on adversarial training techniques like GANs [6, 7, 8, 9, 10]. This approach has gained popularity among researchers because it avoids mode collapse and training instability. For diffusion models, the generated samples can be further enhanced with improved quality and controllable attributes using guidance techniques for sampling. Classifier guidance [11, 12, 13] is one of these techniques which offers numerous benefits. Conditional information can be provided to unconditional diffusion models using noise-aware classifiers' gradients to guide towards pre-defined conditions during sampling. This technique is able to not only improve image quality significantly but also trade-off between diversity and conditional information to fit with the application design. More critically, the flexible nature of classifier guidance allows easy extensions of off-the-shelf diffusion models at the sampling stage without any expensive re-training.

cat

leopard

tiger

0.73

0.86

0.94

0.85

0.91

0.96

0.89

0.95

0.85

English springer

Brittany spaniel

(b) Diversity Suppression

(a) Adversarial Effect

(c) Progressive Guidance

𝑥𝑇

𝑥𝑡1

𝑥𝑡2

𝑥0

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

∇log 𝑝(𝑦𝑐1 |𝑥)

∇log 𝑝(𝑦𝑐2 |𝑥)

∇log 𝑝(𝑦𝑐3 |𝑥)

Generated

(Easy to classify)

Suppressed

(Hard to classify)

In spite of all of its benefits, the classifier guidance still suffers from two critical flaws. The first flaw is the adversarial effect where the model fails to produce samples belonging to the target condition. Guiding the model with the gradient and maximizing the probability of the target condition may lead to shortcut learning that generates samples attacking the classifier. Examples can be found in Figure 1(a), where samples achieve very high confidence in the conditional class but result in very poor features. The second flaw is the diversity suppression caused by the conflict between the diffusion model's generative nature and the classifier's discriminative nature. The optimization towards the condition overly encourages the model to generate only image features that can be easily recognized by the classifier while neglecting other features relevant to the condition, which can reduce the diversity of the generation. Figure 1(b) shows that common features among classes are suppressed since they are harder to classify, resulting in a lack of diversity. These two problems of the classifier guidance come from its underlying guidance mechanism. Specifically, this approach assumes that the objective of the generative task is the same as that of the discriminative task. By utilizing the gradient ∇ log p ( y = y c | x ) , the guidance aims to create an image with features that can be distinguished by the classifier as belonging to the label y c . However, when this approach maximizes the probability of the condition class, it also undesirably suppresses important information relevant but in other classes, causing the lack of diversity and the adversarial effect, as examples mentioned.

This paper proposes a new guidance approach for diffusion models, which generalizes the classifier guidance to tackle the above problems while maintaining its flexibility and efficiency. Our core idea is to alter the aggressive gradients towards a single target in the naive classifier guidance by the gradients that are progressive along two dimensions, i.e. the class dimension and the temporal dimension as illustrated in Figure 1(c). Along the class dimension, our method tolerates the gradients from other classes relevant to the condition of the generative task. Along the temporal dimension, we expect the gradients to construct shared features among relevant classes when the image is noisy, and the gradients focus on features more specific to the condition after the image becomes clearer. This scheme allows a larger feature pool, which circumvents information suppression and lack of diversity. Besides, the help from relevant information and the less aggressiveness toward one condition also results in more robust features in the samples.

In summary, our contributions are as follows: (1) Proposing a generalized classifier guidance method with progressive gradients that consider relevant information associated with multiple classes depending on the noise level at each timestep. (2) In-depth analysis on reducing the adversarial effects and the diversity suppression of classifier guidance sampling quantitatively and qualitatively. (3) Improvement on image generation task to reach State-of-the-art (SOTA) on different metrics with different datasets.

2 Background

DDPM : The Denoising Diffusion Probabilistic Model (DDPM) is represented by the probability distribution function p θ ( x 0 ) := ∫ p θ ( x 0: T ) d x 1: T , where x 1 , x 2 , ..., x T are latent variables having

the same dimensionality as the data x 0 ∼ q ( x 0 ) . The variable x T follows the distribution p ( x T ) = N ( x T ; 0 , I ) . The process of p θ ( x 0: T ) is known as the reverse process and is a the Markovian chain: p θ := p ( x T ) ∏ T t =1 p θ ( x t -1 | x t ) , where p θ ( x t -1 | x t ) := N ( x t -1 ; µ θ ( x t , t ) , Σ θ ( x t , t )) . This process is also known as the diffusion model's sampling process or inference process.

In contrast to the reverse process , the forward process corrupts the original data x 0 to x T with a predefined schedule of Gaussian noise. This process is a fixed Markovian chain: q ( x 1: T | x 0 ) := ∏ T t =1 q ( x t | x t -1 ) , where q ( x t | x t -1 ) := N ( x t ; √ 1 -β t x t -1 , β t I ) . β t is the fixed variance scheduled for both forward and reverse process.

There are a number of ways to define the output of the network θ such as noise predictor ϵ θ ( x t , t ) , mean predictor ˜ µ θ ( x t , t ) [1]. The most common method is the noise predictor ϵ θ ( x t , t ) . After the θ are trained to match ϵ θ ( x t , t ) with ϵ , we have the sampling equation:

with z as the random noise, σ t is the variance of the of p θ ( x t -1 | x t ) and ϵ θ is the predicted noise.

Classifier Guidance : The guidance aims to provide conditional information during the sampling process so that the output image satisfies the predefined conditions. From Eq. 1, we denote µ t := 1 √ α t ( x t -1 -α t √ 1 -¯ α t ϵ θ ( x t , t )) . Given condition y c , log p ϕ ( y c | x t ) is the log of conditional distribution of the class y c through observing x t . The sampling equation for x t -1 given x t with guidance scale s is as below:

Where ϕ is the parameters of the pretrained classifier.

3 Methodology

We first present a scheme allowing relevant information gradients to join the sampling process. After that, the progressive schedule is proposed to adapt the relevant gradients taking the noise level in samples into account. From the Eq. 2, the short form of sampling with guidance is as below:

The expected output of x t -1 is to maximize p ϕ ( y c | x t ) . Due to the probability property of ∑ C i =1 p ( y i | x t ) = 1 , the increase in p ( y c | x t ) will result in the reduction in other probability for other classes p ( y i | x t ) , ∀ i = c with c is the condition, and C is the total number of classes. This results in the suppression of other classes of information which contributes to harmful effects, as mentioned in Figure 1(a) and (b). We propose to generalize the Eq. 3 as follows:

̸

̸

with s i ≥ 0 is the degree of information injected into the diffusion sampling process by class y i . We further constraint ∑ s i = 1 and s c > s i , ∀ i = c . Note that the Eq. 3 is a special case of Eq. 4 where s i = 0 , ∀ i = c with c is the condition for the sampling process of diffusion models. Similar to [14, 15, 11], a guidance scale w is added to balance denoising signals and classification gradients.

̸

Remark 3.1 By incorporating gradients from other classes during sampling, the conflict between the sampling and discriminative objectives can be decreased, resulting in diverse generated samples.

Explain. The use of only the condition class's gradient limits the feature pool's space to some specific features that are the most distinguishable from other classes. This results in the lack of common features shared among relevant classes, which causes a trade-off in the diversity of generated samples. We hypothesize that solely pushing condition gradient suppresses other classes' information since we have ∑ i p ( y i | x t ) = 1 and the increase in p ( y c | x t ) result in the reduction from others. As a result, the inclusion of other classes' gradients avoids this pitfall.

Explain. A substantial number of generated images achieve high confidence in the condition class; however, they possess peculiar features unrecognizable to the human eye. Relying solely on conditional class gradients for information construction presents several robustness problems. Firstly, the gradients associated with the conditional class may contain noise that is specific to that class. Depending solely on this information opens a shortcut for the sampling process to exploit the model by utilizing the noise rather than satisfying semantic features. Secondly, suppressing information from other classes has detrimental effects on robust features. Datasets often consist of numerous classes that share common features that play vital role in constructing meaningful images. When these features are suppressed, the image becomes unrecognizable to the human eye.

Using gradients of other classes, in terms of formulation, first helps avoid adversarial attack-like generation through exploiting the sum of gradients ∑ i ∇ log p ( y i | x t ) . Furthermore, the use of relevant class information forces the model to generate features that satisfy many classes at the same time. This helps avoid specific noise in a class and can utilize the common features shared among classes to avoid the failure of constructing key features. In-depth discussion about the Remark 3.1 and 3.2 are shown in section 4. The rest of this section will discuss the techniques and algorithms to bring the Eq. 4 into the practical sampling process.

3.1 Information Degree vector

The information degree s = { s 1 , s 2 , ..., s C } is the set of weights associated with the gradients from classes that contribute to the information construction of the image. In order to balance between gradients, we set a constraint ∑ s i = 1 . Given c is a main condition, s c > s i ∀ 1 ≤ i ≤ C .

From the Eq. 4, for a given condition c , the aim is to provide the most relevant features that help avoid information suppression. However, not all labels in the C classes help construct the condition c . Instead, for a given class c , if the i th class has higher relevance to class c than j th class, we expect to have information degree s i > s j . There are several ways to satisfy the requirement to have a correlation between labels' features and their degree of information s = { s 1 , s 2 , ..., s C } , such as Label Enhancement [16] or Partial Multi-label [17, 18]. However, a simple technique, which does not require any training, is preferred due to concerns about the complexity of the whole sampling process. Inspired by the prompt engineering section in CLIP [19], we propose to model the correlation between labels via the description text of labels. Instead of using fixed templates similar to CLIP, we need more details about each class to have more generative features. In this work, we utilize ChatGPT [20] to generate text descriptions for each class. After removing stop words and other preprocessing text, we use a CLIP model to obtain each class's embedding information v i . The embedding information is then utilized to calculate the similarity based on cosine similarity, i.e., sim i,j = v i ∗ v j || v i |||| v j || . Given the conditional label c , we have the information degree for each label as s i = sim c,i ∑ C j =1 sim c,j , ∀ 1 ≤ i ≤ C . We denote s c = { s c 1 , s c 2 , ..., s c C } as the vector of information degree for sampling the image from condition c . To avoid complication in formulation, later in the manuscript, we will only denote s = { s 1 , s 2 , ..., s C } for the information degree vector for the condition c without losing generality.

3.2 Progressive Information Degree

The subsection 3.1 describes a combination of gradients among classes. In the diffusion sampling, the image x t -1 is less chaotic and more meaningful than the sample x t . Therefore, we also progressively adapt the s at every timestep to be less chaotic. We index the s at each timestep as s t = { s 1 ,t , s 2 ,t , ..., s C,t } , ∀ 0 ≤ t ≤ T . A simple heuristic schedule is proposed to reduce the chaos of s at each step. At every timestep, for the condition c , the degree of information s c,t is increased by a small amount ∆ s c,t . At the same time, the ∆ s c,t is distributedly deducted from information degrees s i,t of other classes, ensuring condition ∑ C i =1 s i,t = 1 . ∆ s t is formulated as:

̸

̸

Construct from the most distinguishable features of Brittany Spaniel

Inject shared features from English Springer and Welsh Springer Spaniel

Vanilla

Ours

TAA

with 0 ≤ γ ≤ 1 as the updating rate. After obtaining ∆ s t , we update the s following s t -1 = s t -∆ s t . From the above equations, if γ = 1 , the sampling process is primarily similar to the vanilla guidance diffusion sampling as in Eq. 2 where only features from class c join in the sampling process.

Next, we use an entropy perspective to interpret the mechanism of our method. With the sampling equation 4, we can generalize the sampling with Kullback-Leibler divergence as:

When s t is fixed, the optimization of D KL ( s || p ϕ ( y | x t )) totally depends on log p ϕ ( y i | x t ) . However, when s t is varying, the minimization of KL divergence term will take the full form as D KL ( s t || p ϕ ( y | x t )) = ∑ C i =1 s t,i log s t,i -s t,i log p ϕ ( y i | x t ) . Therefore, the process of transforming the vector of information degree can be formulated as below:

so that (1) s c,t > s i,t , ∀ i = c, 1 ≤ i ≤ C, (2) ∑ C i =1 s i,t = 1 , (3) 0 ≤ s i,t < 1 , ∀ 1 ≤ i ≤ C, and (4) | s ∗ i,t -s i,t | ≤ l, ∀ 1 ≤ i ≤ C . With s ∗ t as the optimal solution, we have s t -1 = s ∗ t . Because it is hard to optimize the objective 7 and the D KL ( s t || p ϕ ( y | x t )) at the same time, the constraint (4) is added with an upperbound l to avoid the mitigation of the minimization of D KL ( s || p ϕ ( y | x t )) regarding x t . Using the proposed schedule, the upperbound becomes l = γ ∗ C -1 C .

̸

Throughout the diffusion model's sampling process, the information degree vector will gradually converge near to one-hot vector, leading to the minimization of objective 7. From this point of view, we have formulated the sampling as the problem of matching two distributions between conditional probability p ( y | x t ) with information degree vector s t . The schedule aims to do the reverse entropy regularization on target s t , which is opposite to the classic entropy regularization [21, 22]. The formal discussion about reverse entropy regularization is in Supplementary.

4 Analysis

The previous section has shown our method to guide image generation through a progressive scheme. We now provide further analysis to visualize how our algorithm solves the problems in detail. We set up this analysis on ImageNet64x64 using ADM [11] with or without Progressive Guidance (ProG) to investigate the diversity suppression and the adversarial effect.

Diversity suppression alleviation : As the process commences, incorporating information from relevant classes of the conditions into the sampling process is crucial to overcome the issue of information suppression among labels, as illustrated in Figure 2. The Brittany Spaniel is the main condition in this example. We demonstrate that by introducing gradients solely from the Brittany

Spaniel, the process converges towards the features that benefit the pretrained classifier the most, focusing primarily on simple and easily classifiable characteristics - the front face features of the dog. In contrast, by providing information about the English Springer and Welsh Springer Spaniel, we encourage the model to sample images with lower confidence in identifying them as Brittany Spaniels and to tolerate a certain degree of confidence in them belonging to other breeds during the initial stage of the sampling process. This scheme allows the process to search and sample in the pool of common features between classes, facilitating the diverse samples. We further do an analysis of this feature collapse problem on different types of breeds in section B.1 of the Supplementary. We found out that besides Front-face features over-exploitation as in Figure 2, the vanilla guidance scheme also suffers from Front stretching pose and Green grass background problems. Similar to Figure 2, our proposed ProG scheme shows that it can solve these problems.

Robustness features construction : Figure 3 provides a visual example of how support from gradients of other classes can effectively reduce non-robust features. In the example, the naive classifier guidance suppresses the representation of tiger features, resulting in a significantly degraded image quality due to the lack of facial features of the leopard . In contrast, our proposed method promotes the emergence of tiger , panther and leopard features during the early stages of the sampling process. It gradually diminishes tiger and panther towards the end, ultimately achieving the desired leopard features. Additional instances are also demonstrated where our proposed methods effectively address failure cases due to non-robustness features associated with the conventional classifier guidance. More examples with high-resolution images can be found in section C of the Supplementary.

𝑠𝑡𝑖𝑔𝑒𝑟,1000

= 1.6 × 𝑒 -3

𝛻 log 𝑝(𝑡𝑖𝑔𝑒𝑟|𝑥𝑡 )

𝑠𝑝𝑎𝑛𝑡ℎ𝑒𝑟,1000

= 1.4 × 𝑒 -3

𝛻 log 𝑝(𝑝𝑎𝑛𝑡ℎ𝑒𝑟|𝑥𝑡 )

𝛻log𝑝(𝑙𝑒𝑜𝑝𝑎𝑟𝑑|𝑥𝑡)

𝑠𝑙𝑒𝑜𝑝𝑎𝑟𝑑,1000

= 2.2 × 𝑒 -3

𝑠𝑡𝑖𝑔𝑒𝑟,500

= 1 × 𝑒 -3

𝑠𝑙𝑒𝑜𝑝𝑎𝑟𝑑,500

= 0.36

𝑠𝑝𝑎𝑛𝑡ℎ𝑒𝑟,500

= 9 × 𝑒 -4

𝑠𝑡𝑖𝑔𝑒𝑟,0

= 1 × 𝑒 -8

𝑠𝑝𝑎𝑛𝑡ℎ𝑒𝑟,0

= 1 × 𝑒 -10

𝑠𝑙𝑒𝑜𝑝𝑎𝑟𝑑,0

= 0.99

𝛻log𝑝(𝑙𝑒𝑜𝑝𝑎𝑟𝑑|𝑥𝑡)

𝑠𝑙𝑒𝑜𝑝𝑎𝑟𝑑,1000

= 1.00

𝑠𝑙𝑒𝑜𝑝𝑎𝑟𝑑,500

= 1.00

𝑠𝑙𝑒𝑜𝑝𝑎𝑟𝑑,0

= 1.00

Ours

Vanilla

Vanilla

Ours

5 Related works

The field of generative models has seen a significant surge in the use of diffusion models. Specifically, the Denoising Diffusion Probabilistic Model (DDPM) and its variants [1, 3, 2, 23, 24, 11, 25] have emerged as prominent models for generative tasks which provides better stability than GANs [26, 27, 28, 29, 30, 9, 31, 32]. Not only for generative tasks, but discriminative tasks also benefit from diffusion scheme [33, 34]. In addition to the DDPM series, score-based models [4, 35] are developed as a theoretical counterpart to the DDPM series. Despite having different optimization objectives and motivations, these two types of models are closely related [5]. In addition to diffusion models, conditional generative models have gained significant attention in the generative model's research community [6, 30]. DGMs [1, 5] offer a conditional version by connecting a classification head to the diffusion model. These models benefit from conditional information [11, 36]. Recent research has focused on achieving controllable samplers/generators without retraining DGMs. Classifier guidance [11] proposes to use classifier gradient signals for guidance, which has been further generalized by [13] to adapt different types of modality. Classifier-free guidance [14] has shown that guidance properties can also be achieved without a classifier. CompDiffusion [37] proposes to guide the diffusion model by combining different conditions given by a pretrained model. However, most guidance works suffer from the trade-off between sample quality, diversity, and conditional

information. [12] has handled this problem by including a scoring model during the training of a noise-aware classifier, causing an increase in the running time. Discriminator Guidance [38] finetunes the sampling process by training a Discriminator, which also faces the problem of large training time similar to [12]. In contrast, our work addresses this problem without retraining the diffusion or noise-aware classifier models. [39] solves the problems of conflict during the sampling process of diffusion models, yet this work ignores the problem of robustness features construction. Adversarial white-box attack [40, 41, 42, 43, 44] shares common technique with classifier guidance to exploit the gradient ∇ log p ( y c | X ) to construct the features, leading to concerns in the community about the use of ∇ log p ( y c | X ) in classifier guidance that causes adversarial effects. One of the main targets of our work is to solve this concern.

6 Experiments

Setup. Extensive experiments are conducted on CIFAR10, ImageNet (64x64, 128x128, 256x256). We denote Progressive Guidance (ProG) as our proposed method, which is first evaluated on ADM [11] and IDDPM [3] to verify our claims on improving the performance of the vanilla guidance method. Subsequently, ProG is combined with the EDS [45], an advanced guidance method to solve the gradient vanishing problem, to achieve the state-of-the-art. Other baselines are taken from BigGAN[9], ADM [11], EDS[45], IDDPM [3], VAQ-VAE-2[46], LOGAN [28], DCTransformers [47] and DiT[48] (latent diffusion). The ADM with a conditional pretrained diffusion model is denoted as CADM. We utilize five standard scores, which are IS, FID/sFID [26], Precision [49], and Recall [49], to evaluate the image quality and diversity of the generated samples.

4

8

6

10

5

7

9

8

10

11

6

FID

guidance scale

10

(a) Unconditional diffusion

4

8

6

0.43

0.45

0.62

0.58

0.55

0.53

0.50

0.48

0.60

Recall

guidance scale

0.5

2

4

6

2

4

12

14

10

6

8

FID

guidance scale

Recal

l

0.35

0.50

0.55

0.45

0.60

0.40

guidance scale

0.5

2

4

6

(b) Conditional diffusion

ProG

w/oProG

ProG

w/o ProG

ProG

w/o ProG

ProG

w/o ProG

Robustness metric : The issue of quantifying the adversarial effects caused by the classifier guidance method has not been adequately addressed in previous works [1, 15] since it is challenging due to the trade-off between diversity, feature quality, and robustness. We propose to leverage a bank of off-theshelf pretrained classification models to evaluate the robustness features based on the assumption that it is much more difficult for the non-robust image to trick a set of separately pretrained classifiers than only a single guidance classifier. Specifically, the robustness features are quantified by averaging Top-1 accuracy using pretrained models ResNet34, ResNet50, ResNet151 [50], DenseNet169, DenseNet201 [51], SqueezeNet, SqueezeNet [52]. To ensure a fair comparison between the two generative schemes, we adjust the guidance scale to achieve similar diversity and feature quality, measured by similarity in the Fréchet Inception Distance (FID).

6.1 Improvement over Classifier Guidance

In this section, we show that applying the ProG method helps achieve better performance quantitatively over vanilla guidance regarding image quality, diversity, and robustness. Table 1 shows a clear improvement in all the guidance sampling models on image quality and diversity. ProG achieve lower recall values in some datasets than the ADM baseline without guidance. However, this is expected as conditional generative schemes limit the search space resulting in less freedom in sampling features.

Diversity trend when increasing w : By increasing guidance scale w , the FID scores and diversity scores such as Recall are less sacrificed in ProG to achieve a higher IS score and conditional information compared to vanilla guidance (Figure 4). When increasing the guidance scale, our proposed method mostly has a slower degeneration rate in FID and Recall than the vanilla guidance.

Robustness improvement We show that utilizing the ProG method can achieve more robust features than vanilla guidance, with the robustness performance shown in Table 2. We first adjust w so that the two generative processes (vanilla and our proposed) have the same FID scores. After that, we calculate the robustness score as described previously.

Key takeaway : The ProG helps to improve the guidance to achieve three targets. Firstly, we achieve better image quality and diversity than the vanilla guidance scheme, which can be observed through IS, FID, sFID, Precision, and Recall. Secondly, we alleviate the diversity suppression qualitatively and quantitatively even when the guidance scale is set to be very large. Finally, we achieved better robustness than the original guidance scheme quantitatively and qualitatively.

6.2 State-of-the-Art Comparision

To compare with other state-of-the-art methods, we combine our proposed method with EDS [45]. While our method aims to solve information suppression, the EDS will help amplify the guidance signals at the end of the sampling process. The result shows favorable outcomes in Table 3. Our ProG achieves the state-of-the-art IS/FID on ImageNet64x64 and ImageNet128x128. On ImageNet256x256, the ProG helps to improve DiT-G slightly but consistently over all metrics. Furthermore, we can achieve a comparable level of robustness to classifier-free guidance in Table 4, which is a method

that does not share common techniques with adversarial attacks while also exhibiting a significantly improved diversity trend compared to vanilla classifier guidance, as shown in Table 4. It is worth noting that in the diffusion ADM model, classifier-free guidance outperforms classifier guidance; however, this comes at the cost of nearly twice the computational burden due to the need for forwarding to diffusion twice at each timestep. Moreover, classifier-free guidance lacks the flexibility offered by the classifier guidance method. We also discuss in-depth the benefits of using classifier guidance with ProG compared to classifier-free guidance in section E in Supplementary. Section F in the Supplementary will discuss several techniques to extend ProG into Text-to-Image Generation. The section D in Supplementary will provide the settings for all experiments.

MODEL

ROBUSTNESS@1 (

↑

)

FID

IMAGENET 256X256

CADM-G + EDS

83.31

3.96

CADM-G + EDS + PROG

86.60

3.90

87.14

3.95

2

4

6

8

10

Guidance scale

5

6

7

8

9

FID

ProG

w/o ProG

2

4

6

8

10

Guidance scale

0.33

0.35

0.38

0.40

0.43

0.45

0.48

Recall

ProG

w/o ProG

6.3 Ablation Study

Effects of schedule : ProG schedule relies on a single hyperparameter γ to determine the speed of convergence towards the one-hot vector. A larger γ leads to faster convergence, while a smaller γ results in slower convergence. Table 5 demonstrates the sensitivity of γ , showing that varying it does not significantly impact the results. Results obtained with γ values of 0 . 04 or 0 . 06 are similar. When γ becomes larger, the model is more likely to achieve a higher Inception Score (IS), but trade-off with FID/sFID scores, indicating a decrease in sample quality. It is worth noting that without the schedule ( γ = 0 ), the guidance method fails, underscoring the importance of the proposed scheduler.

Correlations between labels : Fixing the same schedule, we change the initial vector of information degree s T via different ways. We denote our ChatGPT-generated text as CGT. Based on CGT, we use CLIP[19] or Word2Vec [53]to obtain the similarity matrix mentioned in section 3. We compare with the uniform label smoothing (Uni. LS), where the s T is uniformly distributed. We also provide the performance of Label Smoothing without our progressive schedule noted as LS (NO SCHE.) to have a comprehensive understanding of the method. Table 6 shows that the CLIP + CGT performs the best among different schemes. This shows the importance of the initial state of the s T , which helps to decide the relevant information for the main condition.

Computational cost : Considering the comparable FID/sFID results between ProG and classifier-free guidance, we uncovered several advantages of ProG over classifier-free guidance, as detailed in Section E of the Supplementary. Notably, one of these advantages pertains to computational efficiency. Table 7 illustrates the GPU hours required for generating 50,000 images at a resolution of 256x256. When ProG is incorporated into the standard classifier guidance framework, the computational cost remains on par with vanilla classifier guidance. In contrast, uti-

lizing classifier-free guidance substantially escalates computational expenses. It's worth noting that the computational cost of classifier guidance is predominantly influenced by the architecture of the noise-aware classifier, which is typically simpler than the network architectures used in generative models. Consequently, the observations in Table 7 are broadly applicable in most scenarios.

7 Conclusion

This work quantifies two problems of the classifier guidance method: dramatic diversity suppression and non-robustness feature construction. The results show better robustness features than the classifier guidance baseline and a similar level to the classifier-free guidance method. Besides, we also achieve a significantly better diversity trend when increasing the guidance scale quantitatively and qualitatively. We can achieve the SOTA on ImageNet64x64 and ImageNet128x128 on FID/sFID. Our method can be combined with DiT-G [48] to achieve a new SOTA for ImageNet256x256. Our current research demonstrates successful solutions to address the adversarial effects, particularly in cases where high-confidence generated samples exhibit minimal features relevant to the given condition. However, we acknowledge that we have not yet resolved the scenario where diffusion signals dominate and conflict with the classification signals. The resulting images have low confidence and fail to contain any discernible features related to the specified condition. It is worth noting that these features can no longer be considered adversarial due to low confidence and often be wrongly classified by the pretrained classifier used in the sampling process. Resolving the conflict between diffusion and classification gradients remains an open challenge and an area for future investigation.

References

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning , pages 2642-2651. PMLR, 2017.

Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. Advances in Neural Information Processing Systems , 34:23505-23518, 2021.

Liang Hou, Qi Cao, Huawei Shen, Siyuan Pan, Xiaoshuang Li, and Xueqi Cheng. Conditional gans with auxiliary discriminative classifier. In International Conference on Machine Learning , pages 8888-8902. PMLR, 2022.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Yi Lee. Denoising likelihood score matching for conditional score-based data generation. arXiv preprint arXiv:2203.14206 , 2022.

Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 289-299, 2023.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In International Conference on Machine Learning , pages 10597-10606. PMLR, 2020.

Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6575-6582, 2020.

Xiuwen Gong, Dong Yuan, and Wei Bao. Understanding partial multi-label learning via mutual information. Advances in Neural Information Processing Systems , 34:4147-4156, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models, 2023.

Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 , 2017.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Yunke Wang, Xiyu Wang, Anh-Dung Dinh, Bo Du, and Charles Xu. Learning to schedule in diffusion probabilistic models. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2478-2488, 2023.

Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. arXiv preprint arXiv:2203.13508 , 2022.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503 , 2022.

Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances in neural information processing systems , 31, 2018.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139144, 2020.

Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.00953 , 2019.

Dinh Anh Dung and Huynh Thi Thanh Binh. Gdegan: Graphical discriminative embedding gan for tabular data. In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , pages 1-11. IEEE, 2022.

Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. Advances in neural information processing systems , 32, 2019.

Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In International Conference on Machine Learning , pages 10961-10970. PMLR, 2021.

Chengbin Du, Yanxi Li, Zhongwei Qiu, and Chang Xu. Stable diffusion is unstable. arXiv preprint arXiv:2306.02583 , 2023.

Daochang Liu, Qiyue Li, Anh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10139-10149, 2023.

Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19830-19843, 2023.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems , 33:12438-12448, 2020.

Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10619-10629, 2022.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint arXiv:2206.01714 , 2022.

Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models, 2023.

Anh-Dung Dinh, Daochang Liu, and Chang Xu. PixelAsParam: A gradient view on diffusion sampling with guidance. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 8120-8137. PMLR, 23-29 Jul 2023.

Ye Liu, Yaya Cheng, Lianli Gao, Xianglong Liu, Qilong Zhang, and Jingkuan Song. Practical evaluation of adversarial robustness via adaptive auto attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15105-15114, 2022.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pages 2206-2216. PMLR, 2020.

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pages 39-57. Ieee, 2017.

Guangcong Zheng, Shengming Li, Hui Wang, Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li. Entropy-driven sampling and training scheme for conditional diffusion generation. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII , pages 754-769. Springer, 2022.

Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019.

Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841 , 2021.

William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748 , 2022.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360 , 2016.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalized classifier guidance method for diffusion models with progressive guidance along both the class and temporal dimensions, to handle the adversarial effect and diversity suppression problems of the vanilla classifier guidance. In experiments, the proposed method shows advantages over the vanilla classifier guidance and achieves a new state-of-the-art when combined with other methods under certain settings.\n\nSTRENGTHS:\n1. This paper has clear and well-established motivations, working on two important problems in classifier guidance encountered by the community, the adversarial effect and diversity suppression. This can be a good contribution to the community.\n\n2. The proposed progressive guidance method is simple, effective, intuitive, and well-aligned with the motivations. The entropy perspective is also very interesting.\n\n3. The paper is well-written, with good clarity, nice flow, and good intuition. At the same time, detailed explanations and illustrative analysis are provided for the method for better understanding. The reading experience is good.\n\n4. The experiments are relatively comprehensive with multiple datasets and baselines. The alleviation of the adversarial effect and diversity suppression is validated. The advantage over the vanilla classifier guidance is shown. New SOTA is achieved under certain settings.\n\n5. Code is available.\n\nWEAKNESSES:\n1. In Table 1, the results on CIFAR are not as good as on ImageNet. What is the potential reason and any insight?\n\n2. The information degree is based on the description generated by ChatGPT. What are the prompts? What is the impact of different prompts?\n\n3. Although the paper is well-written in general, there are still some clarity issues:\n- The derivation and introduction of the entropy perspective in the main paper are not very easy to understand. It should be improved in a more logical way in writing.\n- What do the superscripts on methods in Table 3 mean? What does “LS (NO SCHE.)” mean in Table 6 mean?\n\n4. There are also some minor writing issues:\n- In Figure 1c, the meaning of darkness is not stated.\n- In Figure 1c, it should be stated that “c1” is the condition.\n- In Figure 1 caption, missing space before “Dataset: ImageNet64x64”.\n- In L57, “Propose” > “Proposing”.\n- In L61, “over” > “of”.\n- In L61, “SOTA” should be full.\n- “Markov chain” > “Markovian chain”.\n- In section 2, the definition of epsilon and sigma is not mentioned. The background should be self-contained.\n- In L79, “log_phi p” should be “log p_phi”. Phi is not explicitly defined.\n- In L87-88, p should be p_phi. Make it consistent.\n- In L91, >= should be a single symbol.\n- The flow at the start of section 3.1 is too fast. Add some rationale and definition of information degree first.\n- In L131, missing space after “ChatGPT”.\n- In Eq.7, s_t,i should be s_i,t. Also check other places.\n- In Table 2, should “CADM-G+ProG” be “CADM+ProG”?\n- In L260, remove spacing after “EDS [32]”.\n- In Table 3 caption, “State” > “state”.\n- The capitalization in the reference should be corrected, such as “gans”.\n\nQUESTIONS:\nPlease address the writing issues then I would consider this paper as a good contribution to the community.\n\nLIMITATIONS:\nThe limitation is briefly discussed at the end of the paper. I think the authors should elaborate on this to provide more insights.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a new classifier guidance technique for diffusion models named Progressive Guidance (PROG). PROG is an extended version of the classifier guidance method and incorporates the relevant classes' information (beyond the target class alone) to determine the guidance direction, particularly in the presence of noisy images during the early sampling stage. Through extensive experiments on various datasets and diffusion models, the authors demonstrate the effectiveness of PROG in comparison to the standard classifier guidance technique.\n\nSTRENGTHS:\n[1] I think the paper is well written and easy to understand.\n\nWEAKNESSES:\n[1] I think the technical novelty of this paper may not meet the standards required for acceptance at NeurIPS.\n\n[2] In my opinion, the proposed PROG technique is only marginally better or comparable to the standard classifier guidance method (according to Tables 1 and 3).\n\n[3] Furthermore, I am not sure about whether the proposed classifier guidance (PROG) is superior to classifier-free guidance in terms of performance and applicability. The image generation community has recently witnessed a rapid shift from class conditional image synthesis to text-to-image synthesis. However, it appears that the proposed classifier guidance method may not be applicable to text-to-image synthesis tasks.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nAs mentioned in the weaknesses section, I think that the performance improvement of the proposed method over the standard classifier guidance is marginal, and the limited applicability of classifier guidance is a significant limitation of this paper. To strengthen the submission, I recommend that the authors emphasize the strong advantages of their approach compared to standard method (classifier guidance). By highlighting these unique aspects, the authors can make their submission more compelling and is able to address the aforementioned limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
To tackle the generative issue of low diversity and artifacts of classier guidance for conditional diffusion generation, this paper proposes an entropy view for calculating the conditional score gradient. The proposed method proposes two modifications for classifier guidance, 1) exchanging one-hot class labels with soft-labels based on class similarity. 2) progressive score weights for different time steps. And the proposed achieves better results than the baseline method and classifier-free guidance method.

STRENGTHS:
1. The proposed method is simple and straightforward yet effective to achieve better generative results.
2. The proposed method suggests an entropy view to review the classifier guidance method is interesting. In the real dataset, we should consider the real conditional distribution of the label given image, which is neglected by the vanilla one.

WEAKNESSES:
1. Although the proposed method is simple and straightforward, the flexibility would be restricted for different modalities of conditions such as (text and segmentation map), which can be easily addressed via classifier-free guidance. Is there any solution for classifier guidance with flexible conditional labels (different modalities)?

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
This paper does not discuss the limitations. However, the proposed method might not be general for conditional generation with diverse modalities of conditional labels (etc. text, segmentation map).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes to inject the graident of other clasess to improve the diversity for conditional sampling of diffusion model.

STRENGTHS:
1. The general idea is simple, the gradient of the classifier tends to use the most discriminative feature and thus hurts the performance, so we use the gradient of other classes in early phases to improve the diversity.

2. The paper also provides some entropy arguments, which partly justifies the method.

3.Extensive experiments are conducted and the performance gain is clear.

WEAKNESSES:
1. Missing details. How many steps are used for the eq 4?. "In the later sampling stage, we progressively enhance gradients to refine the details 13 in the image toward the primary condition" Do you just use classifier guidance after some steps?

2. Authors use CLIP embedding to compute the similarity between different classes. Can we use clip gradient to guide the generation directly (insert the graident of CLIP similarity between the generated image and target class, say "dog" )  in DDIM steps? Will that also improve the diversity as CLIP has seen many different dog images. Some comparions are needed.

QUESTIONS:
as above

LIMITATIONS:
as above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work points out that diffusion models with classifier guidance only focus on the given category but ignore the other relevant category information. Thus, this work proposes the Progressive Guidance (PG) method to address two problems, i.e., lack of diversity and the adversarial effect (samples have high scores but poor visual quality). The proposed method uses progressive gradients from the class dimension and the diffusion temporal dimension to change the gradient of the classifier guidance on a single condition. In terms of the class dimension, PG allows gradient-assisted conditions to be generated for other class information related to a given class. In terms of diffusion temporal dimension, the weight of the gradient also changes over time. The experimental results show that PG improves image quality, sample diversity, and robustness compared with the competing methods.\n\nSTRENGTHS:\n1. The paper is well-organized and easy to follow.\n\n2. This work proposes a novel way to improve the classifier guidance method.\n\n3. The proposed dramatic diversity and non-robustness feature construction approaches show desirable robustness than the commonly used baseline method.\n\n4. The proposed method can be combined with powerful backbone networks to achieve favorable performance.\n\nWEAKNESSES:\n1. Though the theoretical analyses are convincing, the experimental results show that the proposed method sometimes underperforms the classifier-free guidance. The authors should explain this point to verify the effectiveness of their proposed method.\n\n2. In Section 3, the analyses could be more convincing if more evidence is provided. In addition, the presentation of Sections 3.1 and 3.2 should be improved to make it clear.\n\n3. The computational costs should be clarified, considering that the diffusion models are usually expensive to implement for producing high-resolution images.\n\nQUESTIONS:\nPlease see my comments above.\n\nLIMITATIONS:\nPlease see my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work authors propose to address challenges of classifier guidance Diffusion models and propose progressive-guidance where during sampling/reverse diffusion process. Initial iterations of reverse process receives classifier gradient from multiple relevant classes instead of just target class so that more relevant features can be retained. Authors also illustrate this enables generated images to be more diverse as features in intial iterations need not 'ONLY' be purely discriminative for current target class of interest.\n\nSTRENGTHS:\nAuthors propose a simple and useful method to improve sample diversity of Diffusion models within classifier guidance setting and in appendix they show sample diversity is on-par with classifier-free guidance. \n\nProgressive Guidance makes sense more generally from generation perspective too, as within generative paradigm we first sample higher level semantics which is not very fine-grained then conditioned on that we sample latent variables/features relevant to fine grained details so it does make sense that we don't want to hyper-focus on one-particular class but it probably depends on complexity of class-taxonomy of particular dataset.\n\nWEAKNESSES:\nCore idea in paper is easy to follow but lack novel insights or significant contributions. \n\nFew suggestions in terms of details and writing: \n\nThough this paper follows classifier guidance from previous literature, it might be useful to summarize noise-aware classifier performance at difference noise-levels, especially from what iteration is classifier gradient incorporated in sampling process.\n\nWhat is value of gamma at different sampling steps w.r.t number of inference steps and schedulers to easily interpret results and setting.\n\nWhat is actual guidance scale value across sampling steps/reverse diffusion, more specifically at \\gamma = 0.04 at what stage of sampling is classifier guidance focusing on one-hot vector. If you can explicitly state that, it might be useful to the reader to easily interpret setting.\n\t As in later iterations of sampling process much of semantics or high-level features are inferred and guidance might not play such a vital role?\n\nQUESTIONS:\nIn terms of Suppressed samples illustration and sample diversity, empirical metrics demonstrate on-par or worse diversity except ImageNet 64 x 64? So it is unclear in what setting is proposed method effective and what are current limitations of proposed approach. \n            Also, it might be worthwhile to consider evaluation for few interesting sub-sets rather than dataset level, as it might be useful to evaluate in settings  where categories have large sub-classes which needs to capture fine-grained details vs not many sub-classes? Does Pro-G have sensitivity \n\nWhat is sensitivity of proposed progressive guidance method based on accuracy of noise-aware classifier at different noise-levels? Analyzing such sensitivity and how it effects FID, Precision/Recall w.r.t generation would be informative. \n               Also, how challenging is it to train noise-aware classifier? As except few of initial works there aren't many follow up works which use classifier guidance as pointed out by authors on few of challenges. I understand authors used pre-trained checkpoints but it would be informative for community towards better interpretation of applicability of proposed method and classifier guidance more generally.\n\nLIMITATIONS:\nProposed method is simple and effective and its encouraging to see that classifier guidance is on-par with classifier free guidance in terms of sample quality and Diversity, but the quality boost is marginal.\n\nI am not sure if this is valuable enough contribution in terms of novelty and insights for NeurIPS, as proposed method is straight forward and does not provide extensive novel insights either from Diffusion models behavior or empirical properties of classifier guidance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}