SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anh-Dung Dinh
dinhanhdung1996@gmail.com
daochang.liu@sydney.edu.au
Chang Xu *
c.xu@sydney.edu.au
This paper quantifies and tackles two critical challenges encountered in classifier guidance for diffusion generative models, i.e., the lack of diversity and the presence of adversarial effects. These issues often result in a scarcity of diverse samples or the generation of non-robust features. The underlying cause lies in the mechanism of classifier guidance, where discriminative gradients push samples to be recognized as conditions aggressively. This inadvertently suppresses information with common features among relevant classes, resulting in a limited pool of features with less diversity or the absence of robust features for image construction. We propose a generalized classifier guidance method called Progressive Guidance, which mitigates the problems by allowing relevant classes' gradients to contribute to shared information construction when the image is noisy in early sampling steps. In the later sampling stage, we progressively enhance gradients to refine the details in the image toward the primary condition. This helps to attain a high level of diversity and robustness compared to the vanilla classifier guidance. Experimental results demonstrate that our proposed method further improves image quality while offering a significant level of diversity as well as robust features. Source code is available at: https://github.com/dungdinhanh/prog-guided-diffusion.
Diffusion model [1, 2, 3, 4, 5] has emerged as a state-of-the-art technique for generating high-fidelity images without relying on adversarial training techniques like GANs [6, 7, 8, 9, 10]. This approach has gained popularity among researchers because it avoids mode collapse and training instability. For diffusion models, the generated samples can be further enhanced with improved quality and controllable attributes using guidance techniques for sampling. Classifier guidance [11, 12, 13] is one of these techniques which offers numerous benefits. Conditional information can be provided to unconditional diffusion models using noise-aware classifiers' gradients to guide towards pre-defined conditions during sampling. This technique is able to not only improve image quality significantly but also trade-off between diversity and conditional information to fit with the application design. More critically, the flexible nature of classifier guidance allows easy extensions of off-the-shelf diffusion models at the sampling stage without any expensive re-training.
(a) Adversarial Effect
0.95
(b) Diversity Suppression
Generated Easy to classify
Figure 1: (a) Samples generated by vanilla classifier guidance achieve high confidence in the condition class but have suspicious features. (b) Images generated by vanilla guidance (left) show a lack of diversity where front-face features are over-exploited. Features shared among breed classes are often lacked, such as action pose and background details depicted in the ground truth images (right). The vanilla guidance has exploited the most distinguishable features to satisfy the classifier while suppressing common features. (c) Our proposed guidance scheme allows gradients from relevant classes to join the sampling process.  $c_{1}$  is the condition. The darkness of the gradients represents the gradients' weights (information degree) during sampling. More examples can be found in Supplementary. Dataset: ImageNet64x64
Suppressed (Hard to classify)
$\nabla \log p(y_{c1}|x)$
Vlogp(vx)
(c) Progressive Guidance
$\rho (y_{c1}|x)$
$\nabla \log p(y_{c2}|x)$
$(y_{c2}|x)$
$\nabla \log p(y_{c3}|x)$
(yc3|x)
$\nabla \log p(y_{c1}|x)$
$\nabla \log p(y_{c2}|x)$
g p(y>c|x)
g.  $p(y_{-3}|x)$
Vanilla Guidance
$|x$  ）
$\nabla \log p(y_{c1}|x)$
$\nabla \log p(y_{c2}|x)$
$\nabla \log p(y_{c3}|x)$
Vlo
Vlo
#
$\nabla \log p(y_{c1}|x)$
$|x$  ）
Vlo
（x）
（x）
In spite of all of its benefits, the classifier guidance still suffers from two critical flaws. The first flaw is the adversarial effect where the model fails to produce samples belonging to the target condition. Guiding the model with the gradient and maximizing the probability of the target condition may lead to shortcut learning that generates samples attacking the classifier. Examples can be found in Figure 1(a), where samples achieve very high confidence in the conditional class but result in very poor features. The second flaw is the diversity suppression caused by the conflict between the diffusion model's generative nature and the classifier's discriminative nature. The optimization towards the condition overly encourages the model to generate only image features that can be easily recognized by the classifier while neglecting other features relevant to the condition, which can reduce the diversity of the generation. Figure 1(b) shows that common features among classes are suppressed since they are harder to classify, resulting in a lack of diversity. These two problems of the classifier guidance come from its underlying guidance mechanism. Specifically, this approach assumes that the objective of the generative task is the same as that of the discriminative task. By utilizing the gradient  $\nabla \log p(y = y_c|x)$ , the guidance aims to create an image with features that can be distinguished by the classifier as belonging to the label  $y_{c}$ . However, when this approach maximizes the probability of the condition class, it also undesirably suppresses important information relevant but in other classes, causing the lack of diversity and the adversarial effect, as examples mentioned.
This paper proposes a new guidance approach for diffusion models, which generalizes the classifier guidance to tackle the above problems while maintaining its flexibility and efficiency. Our core idea is to alter the aggressive gradients towards a single target in the naive classifier guidance by the gradients that are progressive along two dimensions, i.e. the class dimension and the temporal dimension as illustrated in Figure 1(c). Along the class dimension, our method tolerates the gradients from other classes relevant to the condition of the generative task. Along the temporal dimension, we expect the gradients to construct shared features among relevant classes when the image is noisy, and the gradients focus on features more specific to the condition after the image becomes clearer. This scheme allows a larger feature pool, which circumvents information suppression and lack of diversity. Besides, the help from relevant information and the less aggressiveness toward one condition also results in more robust features in the samples.
In summary, our contributions are as follows: (1) Proposing a generalized classifier guidance method with progressive gradients that consider relevant information associated with multiple classes depending on the noise level at each timestep. (2) In-depth analysis on reducing the adversarial effects and the diversity suppression of classifier guidance sampling quantitatively and qualitatively. (3) Improvement on image generation task to reach State-of-the-art (SOTA) on different metrics with different datasets.
DDPM: The Denoising Diffusion Probabilistic Model (DDPM) is represented by the probability distribution function  $p_{\theta}(\mathbf{x}_0) \coloneqq \int p_{\theta}(\mathbf{x}_{0:T}) d\mathbf{x}_{1:T}$ , where  $\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T$  are latent variables having
the same dimensionality as the data  $\mathbf{x}_0\sim q(\mathbf{x}_0)$ . The variable  $\mathbf{x}_T$  follows the distribution  $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T;\mathbf{0},\mathbf{I})$ . The process of  $p_{\theta}(\mathbf{x}_{0:T})$  is known as the reverse process and is a the Markovian chain:  $p_{\theta}\coloneqq p(\mathbf{x}_T)\prod_{t = 1}^T p_\theta (\mathbf{x}_{t - 1}|\mathbf{x}_t)$ , where  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)\coloneqq \mathcal{N}(\mathbf{x}_{t - 1};\mu_{\theta}(x_t,t),\Sigma_{\theta}(x_t,t))$ . This process is also known as the diffusion model's sampling process or inference process.
In contrast to the reverse process, the forward process corrupts the original data  $\mathbf{x}_0$  to  $\mathbf{x}_T$  with a predefined schedule of Gaussian noise. This process is a fixed Markovian chain:  $q(\mathbf{x}_{1:T}|\mathbf{x}_0)\coloneqq \prod_{t = 1}^{T}q(\mathbf{x}_t|\mathbf{x}_{t - 1})$ , where  $q(\mathbf{x}_t|\mathbf{x}_{t - 1})\coloneqq \mathcal{N}(\mathbf{x}_t;\sqrt{1 - \beta_t}\mathbf{x}_{t - 1},\beta_t\mathbf{I})$ .  $\beta_{t}$  is the fixed variance scheduled for both forward and reverse process.
There are a number of ways to define the output of the network  $\theta$  such as noise predictor  $\epsilon_{\theta}(\mathbf{x}_t,t)$  mean predictor  $\tilde{\mu}_{\theta}(\mathbf{x}_t,t)$  [1]. The most common method is the noise predictor  $\epsilon_{\theta}(\mathbf{x}_t,t)$ . After the  $\theta$  are trained to match  $\epsilon_{\theta}(\mathbf{x}_t,t)$  with  $\epsilon$ , we have the sampling equation:
$$
\mathbf {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathbf {x} _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right) + \sigma_ {t} \mathbf {z}, \tag {1}
$$
with  $\mathbf{z}$  as the random noise,  $\sigma_t$  is the variance of the of  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  and  $\epsilon_{\theta}$  is the predicted noise.
Classifier Guidance: The guidance aims to provide conditional information during the sampling process so that the output image satisfies the predefined conditions. From Eq. 1, we denote  $\mu_t := \frac{1}{\sqrt{\alpha_t}} (\mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \epsilon_\theta (\mathbf{x}_t, t))$ . Given condition  $y_c$ ,  $\log p_{\phi}(y_c | \mathbf{x}_t)$  is the log of conditional distribution of the class  $y_c$  through observing  $\mathbf{x}_t$ . The sampling equation for  $\mathbf{x}_{t-1}$  given  $\mathbf{x}_t$  with guidance scale  $s$  is as below:
$$
\mathbf {x} _ {t - 1} \sim \mathcal {N} \left(\mu_ {t} + s \sigma_ {t} ^ {2} \nabla_ {\mathbf {x} _ {t}} \log p _ {\phi} \left(y _ {c} \mid \mathbf {x} _ {t}\right), \sigma_ {t}\right). \tag {2}
$$
Where  $\phi$  is the parameters of the pretrained classifier.
We first present a scheme allowing relevant information gradients to join the sampling process. After that, the progressive schedule is proposed to adapt the relevant gradients taking the noise level in samples into account. From the Eq. 2, the short form of sampling with guidance is as below:
$$
\mathbf {x} _ {t - 1} = \mu_ {t} + \sigma_ {t} * \mathbf {z} + s \sigma_ {t} ^ {2} \nabla_ {\mathbf {x} _ {t}} \log p _ {\phi} \left(y _ {c} | \mathbf {x} _ {t}\right). \tag {3}
$$
The expected output of  $\mathbf{x}_{t-1}$  is to maximize  $p_{\phi}(y_c|\mathbf{x}_t)$ . Due to the probability property of  $\sum_{i=1}^{C} p(y_i|\mathbf{x}_t) = 1$ , the increase in  $p(y_c|\mathbf{x}_t)$  will result in the reduction in other probability for other classes  $p(y_i|\mathbf{x}_t), \forall i \neq c$  with  $c$  is the condition, and  $C$  is the total number of classes. This results in the suppression of other classes of information which contributes to harmful effects, as mentioned in Figure 1(a) and (b). We propose to generalize the Eq. 3 as follows:
$$
\mathbf {x} _ {t - 1} = \mu_ {t} + \sigma_ {t} * \mathbf {z} + w \sum_ {i = 1} ^ {C} s _ {i} \sigma_ {t} ^ {2} \nabla_ {\mathbf {x} _ {t}} \log p _ {\phi} \left(y _ {i} | \mathbf {x} _ {t}\right), \tag {4}
$$
with  $s_i \geq 0$  is the degree of information injected into the diffusion sampling process by class  $y_i$ . We further constraint  $\sum s_i = 1$  and  $s_c > s_i, \forall i \neq c$ . Note that the Eq. 3 is a special case of Eq. 4 where  $s_i = 0, \forall i \neq c$  with  $c$  is the condition for the sampling process of diffusion models. Similar to [14, 15, 11], a guidance scale  $w$  is added to balance denoising signals and classification gradients.
Remark 3.1 By incorporating gradients from other classes during sampling, the conflict between the sampling and discriminative objectives can be decreased, resulting in diverse generated samples.
Explain. The use of only the condition class's gradient limits the feature pool's space to some specific features that are the most distinguishable from other classes. This results in the lack of common features shared among relevant classes, which causes a trade-off in the diversity of generated samples. We hypothesize that solely pushing condition gradient suppresses other classes' information since we have  $\sum_{i}p(y_{i}|\mathbf{x}_{t}) = 1$  and the increase in  $p(y_c|\mathbf{x}_t)$  result in the reduction from others. As a result, the inclusion of other classes' gradients avoids this pitfall.
Remark 3.2 Using other classes' gradients beyond the conditional class gradient helps avoid adversarial effects.
Explain. A substantial number of generated images achieve high confidence in the condition class; however, they possess peculiar features unrecognizable to the human eye. Relying solely on conditional class gradients for information construction presents several robustness problems. Firstly, the gradients associated with the conditional class may contain noise that is specific to that class. Depending solely on this information opens a shortcut for the sampling process to exploit the model by utilizing the noise rather than satisfying semantic features. Secondly, suppressing information from other classes has detrimental effects on robust features. Datasets often consist of numerous classes that share common features that play vital role in constructing meaningful images. When these features are suppressed, the image becomes unrecognizable to the human eye.
Using gradients of other classes, in terms of formulation, first helps avoid adversarial attack-like generation through exploiting the sum of gradients  $\sum_{i}\nabla \log p(y_{i}|\mathbf{x}_{t})$ . Furthermore, the use of relevant class information forces the model to generate features that satisfy many classes at the same time. This helps avoid specific noise in a class and can utilize the common features shared among classes to avoid the failure of constructing key features. In-depth discussion about the Remark 3.1 and 3.2 are shown in section 4. The rest of this section will discuss the techniques and algorithms to bring the Eq. 4 into the practical sampling process.
3.1 Information Degree vector
The information degree  $\mathbf{s} = \{s_1, s_2, \dots, s_C\}$  is the set of weights associated with the gradients from classes that contribute to the information construction of the image. In order to balance between gradients, we set a constraint  $\sum s_i = 1$ . Given  $c$  is a main condition,  $s_c > s_i \forall 1 \leq i \leq C$ .
From the Eq. 4, for a given condition  $c$ , the aim is to provide the most relevant features that help avoid information suppression. However, not all labels in the  $C$  classes help construct the condition  $c$ . Instead, for a given class  $c$ , if the  $i^{th}$  class has higher relevance to class  $c$  than  $j^{th}$  class, we expect to have information degree  $s_i > s_j$ . There are several ways to satisfy the requirement to have a correlation between labels' features and their degree of information  $\mathbf{s} = \{s_1, s_2, \dots, s_C\}$ , such as Label Enhancement [16] or Partial Multi-label [17, 18]. However, a simple technique, which does not require any training, is preferred due to concerns about the complexity of the whole sampling process. Inspired by the prompt engineering section in CLIP [19], we propose to model the correlation between labels via the description text of labels. Instead of using fixed templates similar to CLIP, we need more details about each class to have more generative features. In this work, we utilize ChatGPT [20] to generate text descriptions for each class. After removing stop words and other preprocessing text, we use a CLIP model to obtain each class's embedding information  $v_i$ . The embedding information is then utilized to calculate the similarity based on cosine similarity, i.e.,  $\mathrm{sim}_{i,j} = \frac{v_i \times v_j}{||v_i|| ||v_j||}$ . Given the conditional label  $c$ , we have the information degree for each label as  $s_i = \frac{\mathrm{sim}_{c,i}}{\sum_{j=1}^C \mathrm{sim}_{c,j}}$ ,  $\forall 1 \leq i \leq C$ . We denote  $\mathbf{s}_c = \{s_1^c, s_2^c, \dots, s_C^c\}$  as the vector of information degree for sampling the image from condition  $c$ . To avoid complication in formulation, later in the manuscript, we will only denote  $\mathbf{s} = \{s_1, s_2, \dots, s_C\}$  for the information degree vector for the condition  $c$  without losing generality.
3.2 Progressive Information Degree
The subsection 3.1 describes a combination of gradients among classes. In the diffusion sampling, the image  $\mathbf{x}_{t-1}$  is less chaotic and more meaningful than the sample  $\mathbf{x}_t$ . Therefore, we also progressively adapt the s at every timestep to be less chaotic. We index the s at each timestep as  $\mathbf{s}_t = \{s_{1,t}, s_{2,t}, \dots, s_{C,t}\}, \forall 0 \leq t \leq T$ . A simple heuristic schedule is proposed to reduce the chaos of s at each step. At every timestep, for the condition  $c$ , the degree of information  $s_{c,t}$  is increased by a small amount  $\Delta s_{c,t}$ . At the same time, the  $\Delta s_{c,t}$  is distributedly deducted from information degrees  $s_{i,t}$  of other classes, ensuring condition  $\sum_{i=1}^{C} s_{i,t} = 1$ .  $\Delta \mathbf{s}_t$  is formulated as:
$$
\Delta s _ {i, t} = \left\{ \begin{array}{l l} - \gamma * (1 - s _ {i, t}), & \text {i f} i = c \\ - \Delta s _ {c} * \frac {s _ {i , t}}{\sum_ {j = 1 , j \neq c} ^ {C} s _ {j , t}}, & \text {i f} i \neq c \end{array} , \right. \tag {5}
$$
Figure 2: Diversity Suppression Alleviation. Brittany Spaniel is the condition. Conventional guidance approach often collapses generated samples to primarily exhibit the facial features of Brittany Spaniel. However, by incorporating features from other classes, such as English Springer and Welsh Sprinter Spaniel, the sampling can capture common characteristics shared between classes such as backgrounds, poses and actions.
with  $0 \leq \gamma \leq 1$  as the updating rate. After obtaining  $\Delta \mathbf{s}_t$ , we update the  $\mathbf{s}$  following  $\mathbf{s}_{t-1} = \mathbf{s}_t - \Delta \mathbf{s}_t$ . From the above equations, if  $\gamma = 1$ , the sampling process is primarily similar to the vanilla guidance diffusion sampling as in Eq. 2 where only features from class  $c$  join in the sampling process.
Next, we use an entropy perspective to interpret the mechanism of our method. With the sampling equation 4, we can generalize the sampling with Kullback-Leibler divergence as:
$$
\mathbf {x} _ {t - 1} = \mu_ {t} + \sigma_ {t} * \mathbf {z} - w \sigma_ {t} ^ {2} \nabla_ {\mathbf {x} _ {t}} D _ {K L} (\mathbf {s} | | p _ {\phi} (\mathbf {y} | \mathbf {x} _ {t})). \tag {6}
$$
When  $\mathbf{s}_t$  is fixed, the optimization of  $D_{KL}(\mathbf{s}||p_{\phi}(\mathbf{y}|\mathbf{x}_t))$  totally depends on  $\log p_{\phi}(y_i|\mathbf{x}_t)$ . However, when  $\mathbf{s}_t$  is varying, the minimization of KL divergence term will take the full form as  $D_{KL}(\mathbf{s}_t||p_{\phi}(\mathbf{y}|\mathbf{x}_t)) = \sum_{i=1}^{C} s_{t,i} \log s_{t,i} - s_{t,i} \log p_{\phi}(y_i|\mathbf{x}_t)$ . Therefore, the process of transforming the vector of information degree can be formulated as below:
$$
\min  _ {\mathbf {s} _ {t}} - \sum_ {i = 1} ^ {C} s _ {i, t} \log s _ {i, t}, \tag {7}
$$
so that (1)  $s_{c,t} > s_{i,t}, \forall i \neq c, 1 \leq i \leq C$ , (2)  $\sum_{i=1}^{C} s_{i,t} = 1$ , (3)  $0 \leq s_{i,t} < 1, \forall 1 \leq i \leq C$ , and (4)  $|s_{i,t}^* - s_{i,t}| \leq l, \forall 1 \leq i \leq C$ . With  $\mathbf{s}_t^*$  as the optimal solution, we have  $\mathbf{s}_{t-1} = \mathbf{s}_t^*$ . Because it is hard to optimize the objective 7 and the  $D_{KL}(\mathbf{s}_t||p_\phi(\mathbf{y}|\mathbf{x}_t))$  at the same time, the constraint (4) is added with an upperbound  $l$  to avoid the mitigation of the minimization of  $D_{KL}(\mathbf{s}||p_\phi(\mathbf{y}|\mathbf{x}_t))$  regarding  $\mathbf{x}_t$ . Using the proposed schedule, the upperbound becomes  $l = \gamma * \frac{C-1}{C}$ .
Throughout the diffusion model's sampling process, the information degree vector will gradually converge near to one-hot vector, leading to the minimization of objective 7. From this point of view, we have formulated the sampling as the problem of matching two distributions between conditional probability  $p(\mathbf{y}|\mathbf{x}_t)$  with information degree vector  $\mathbf{s}_t$ . The schedule aims to do the reverse entropy regularization on target  $\mathbf{s}_t$ , which is opposite to the classic entropy regularization [21, 22]. The formal discussion about reverse entropy regularization is in Supplementary.
The previous section has shown our method to guide image generation through a progressive scheme. We now provide further analysis to visualize how our algorithm solves the problems in detail. We set up this analysis on ImageNet64x64 using ADM [11] with or without Progressive Guidance (ProG) to investigate the diversity suppression and the adversarial effect.
Diversity suppression alleviation: As the process commences, incorporating information from relevant classes of the conditions into the sampling process is crucial to overcome the issue of information suppression among labels, as illustrated in Figure 2. The Brittany Spaniel is the main condition in this example. We demonstrate that by introducing gradients solely from the Brittany
Spaniel, the process converges towards the features that benefit the pretrained classifier the most, focusing primarily on simple and easily classifiable characteristics — the front face features of the dog. In contrast, by providing information about the English Springer and Welsh Springer Spaniel, we encourage the model to sample images with lower confidence in identifying them as Brittany Spaniels and to tolerate a certain degree of confidence in them belonging to other breeds during the initial stage of the sampling process. This scheme allows the process to search and sample in the pool of common features between classes, facilitating the diverse samples. We further do an analysis of this feature collapse problem on different types of breeds in section B.1 of the Supplementary. We found out that besides Front-face features over-exploitation as in Figure 2, the vanilla guidance scheme also suffers from Front stretching pose and Green grass background problems. Similar to Figure 2, our proposed ProG scheme shows that it can solve these problems.
Robustness features construction: Figure 3 provides a visual example of how support from gradients of other classes can effectively reduce non-robust features. In the example, the naive classifier guidance suppresses the representation of tiger features, resulting in a significantly degraded image quality due to the lack of facial features of the leopard. In contrast, our proposed method promotes the emergence of tiger, panther and leopard features during the early stages of the sampling process. It gradually diminishes tiger and panther towards the end, ultimately achieving the desired leopard features. Additional instances are also demonstrated where our proposed methods effectively address failure cases due to non-robustness features associated with the conventional classifier guidance. More examples with high-resolution images can be found in section C of the Supplementary.
Figure 3: Robustness Features Construction. The vanilla guidance fails at conditioning the sample as leopard (top-left). The tiger, panther and leopard helps to influence the image content in the initial state to form the critical features of the leopard to overcome the failure (bottom-left), where clear tiger features are observed in the early images. The right part shows more failure cases corrected using our proposed ProG. The darkness of the gradients shows their associated information degree values.