SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The field of generative models has seen a significant surge in the use of diffusion models. Specifically, the Denoising Diffusion Probabilistic Model (DDPM) and its variants [1, 3, 2, 23, 24, 11, 25] have emerged as prominent models for generative tasks which provides better stability than GANs [26, 27, 28, 29, 30, 9, 31, 32]. Not only for generative tasks, but discriminative tasks also benefit from diffusion scheme [33, 34]. In addition to the DDPM series, score-based models [4, 35] are developed as a theoretical counterpart to the DDPM series. Despite having different optimization objectives and motivations, these two types of models are closely related [5]. In addition to diffusion models, conditional generative models have gained significant attention in the generative model's research community [6, 30]. DGMs [1, 5] offer a conditional version by connecting a classification head to the diffusion model. These models benefit from conditional information [11, 36]. Recent research has focused on achieving controllable samplers/generators without retraining DGMs. Classifier guidance [11] proposes to use classifier gradient signals for guidance, which has been further generalized by [13] to adapt different types of modality. Classifier-free guidance [14] has shown that guidance properties can also be achieved without a classifier. CompDiffusion [37] proposes to guide the diffusion model by combining different conditions given by a pretrained model. However, most guidance works suffer from the trade-off between sample quality, diversity, and conditional
information. [12] has handled this problem by including a scoring model during the training of a noise-aware classifier, causing an increase in the running time. Discriminator Guidance [38] finetunes the sampling process by training a Discriminator, which also faces the problem of large training time similar to [12]. In contrast, our work addresses this problem without retraining the diffusion or noise-aware classifier models. [39] solves the problems of conflict during the sampling process of diffusion models, yet this work ignores the problem of robustness features construction. Adversarial white-box attack [40, 41, 42, 43, 44] shares common technique with classifier guidance to exploit the gradient  $\nabla \log p(y_c|X)$  to construct the features, leading to concerns in the community about the use of  $\nabla \log p(y_c|X)$  in classifier guidance that causes adversarial effects. One of the main targets of our work is to solve this concern.
Setup. Extensive experiments are conducted on CIFAR10, ImageNet (64x64, 128x128, 256x256). We denote Progressive Guidance (ProG) as our proposed method, which is first evaluated on ADM [11] and IDDPM [3] to verify our claims on improving the performance of the vanilla guidance method. Subsequently, ProG is combined with the EDS [45], an advanced guidance method to solve the gradient vanishing problem, to achieve the state-of-the-art. Other baselines are taken from BigGAN[9], ADM [11], EDS[45], IDDPM [3], VAQ-VAE-2[46], LOGAN [28], DCTransformers [47] and DiT[48] (latent diffusion). The ADM with a conditional pretrained diffusion model is denoted as CADM. We utilize five standard scores, which are IS, FID/sFID [26], Precision [49], and Recall [49], to evaluate the image quality and diversity of the generated samples.
Table 1: ProG helps to achieve better IS/FID/sFID in general. The improvement is significant on both IDDPM [3] and CADM/ADM [11] (both unconditionally or conditionally trained). "G" postfix attached to diffusion model stands for vanilla classifier guidance. Bold values are the performance our proposed ProG helps achieve better than vanilla guidance. The underlined values are the cases where diffusion models without guidance achieve the best value.  $\dagger$  is denoted for the score evaluated by the samples provided by the paper.  $\ddagger$  means the values are directly used from the papers due to the unavailability of the pretrained model.
MODEL	IS (↑)	FID (↓)	SFID (↓)	PREC (↑)	REC (↑)
IMAGENET 64x64
ADM	25.64	9.95	6.58	0.60	0.65
ADM-G	46.90	6.40	9.67	0.65	0.54
ADM-G + PROG	46.88	5.16	6.72	0.72	0.56
IDDPM	16.02	18.35	5.08	0.60	0.57
IDDPM-G	18.89	13.62	4.43	0.63	0.55
IDDPM-G + PROG	21.60	11.12	4.25	0.67	0.55
CADM	53.79	2.07	4.35	0.73	0.63
CADM-G	66.52	2.02	4.62	0.78	0.59
CADM-G + PROG	65.65	1.87	4.33	0.77	0.60
IMAGENET 128x128
CADM	92.53	6.14	4.96	0.69	0.65
CADM-G	141.55	2.98	5.10	0.77	0.59
CADM-G + PROG	157.24	2.77	5.09	0.80	0.59
IMAGENET 256x256
ADM	39.7	26.21	6.35	0.61	0.63
ADM-G	96.15	11.96	10.28	0.75	0.45
ADM-G + PROG	99.45	11.21	8.67	0.76	0.46
CADM	100.98	10.94	6.02	0.69	0.63
CADM-G	188.91	4.58	5.21	0.81	0.52
CADM-G + PROG	222.09	4.53	5.08	0.85	0.49
CIFAR10 32x32
ADM	9.55	2.87	4.36	0.69	0.60
ADM-G	9.58	2.85	4.30	0.68	0.60
ADM-G + PROG	9.45	2.81	4.28	0.69	0.60
Figure 4: FID and Recall trend of guidance sampling with (a) ImageNet64x64 unconditional ADM (b) ImageNet64x64 conditional ADM. For vanilla guidance, it becomes evident that the guidance scale increase leads to a rapid decline in both Recall value and FID, indicating a degradation in the diversity of the samples. Conversely, our method sustains a stable trend associated with  $w$  increase.
Robustness metric: The issue of quantifying the adversarial effects caused by the classifier guidance method has not been adequately addressed in previous works [1, 15] since it is challenging due to the trade-off between diversity, feature quality, and robustness. We propose to leverage a bank of off-the-shelf pretrained classification models to evaluate the robustness features based on the assumption that it is much more difficult for the non-robust image to trick a set of separately pretrained classifiers than only a single guidance classifier. Specifically, the robustness features are quantified by averaging Top-1 accuracy using pretrained models ResNet34, ResNet50, ResNet151 [50], DenseNet169, DenseNet201 [51], SqueezeNet, SqueezeNet [52]. To ensure a fair comparison between the two generative schemes, we adjust the guidance scale to achieve similar diversity and feature quality, measured by similarity in the Fréchet Inception Distance (FID).
6.1 Improvement over Classifier Guidance
In this section, we show that applying the ProG method helps achieve better performance quantitatively over vanilla guidance regarding image quality, diversity, and robustness. Table 1 shows a clear improvement in all the guidance sampling models on image quality and diversity. ProG achieve lower recall values in some datasets than the ADM baseline without guidance. However, this is expected as conditional generative schemes limit the search space resulting in less freedom in sampling features.
Table 2:  $w$  are adjusted between CADM-G and CADM-G + ProG to achieve similar FID to evaluate robustness. The ProG significantly improves the robustness of the samples.
MODEL	ROBUSTNESS (↑)	FID
IMAGENET 256x256
CADM-G	79.65	4.58
CADM-G + PROG	85.04	4.56
Diversity trend when increasing  $w$ : By increasing guidance scale  $w$ , the FID scores and diversity scores such as Recall are less sacrificed in ProG to achieve a higher IS score and conditional information compared to vanilla guidance (Figure 4). When increasing the guidance scale, our proposed method mostly has a slower degeneration rate in FID and Recall than the vanilla guidance.
Robustness improvement We show that utilizing
the ProG method can achieve more robust features than vanilla guidance, with the robustness performance shown in Table 2. We first adjust  $w$  so that the two generative processes (vanilla and our proposed) have the same FID scores. After that, we calculate the robustness score as described previously.
Key takeaway: The ProG helps to improve the guidance to achieve three targets. Firstly, we achieve better image quality and diversity than the vanilla guidance scheme, which can be observed through IS, FID, sFID, Precision, and Recall. Secondly, we alleviate the diversity suppression qualitatively and quantitatively even when the guidance scale is set to be very large. Finally, we achieved better robustness than the original guidance scheme quantitatively and qualitatively.
6.2 State-of-the-Art Comparison
To compare with other state-of-the-art methods, we combine our proposed method with EDS [45]. While our method aims to solve information suppression, the EDS will help amplify the guidance signals at the end of the sampling process. The result shows favorable outcomes in Table 3. Our ProG achieves the state-of-the-art IS/FID on ImageNet64x64 and ImageNet128x128. On ImageNet256x256, the ProG helps to improve DiT-G slightly but consistently over all metrics. Furthermore, we can achieve a comparable level of robustness to classifier-free guidance in Table 4, which is a method
that does not share common techniques with adversarial attacks while also exhibiting a significantly improved diversity trend compared to vanilla classifier guidance, as shown in Table 4. It is worth noting that in the diffusion ADM model, classifier-free guidance outperforms classifier guidance; however, this comes at the cost of nearly twice the computational burden due to the need for forwarding to diffusion twice at each timestep. Moreover, classifier-free guidance lacks the flexibility offered by the classifier guidance method. We also discuss in-depth the benefits of using classifier guidance with ProG compared to classifier-free guidance in section E in Supplementary. Section F in the Supplementary will discuss several techniques to extend ProG into Text-to-Image Generation. The section D in Supplementary will provide the settings for all experiments.
Table 3: Combining with the EDS [45] to solve the gradient vanishing problem, our proposed method helps achieve state-of-the-art on IMAGENET64x64 and IMAGENET128x128. On latent diffusion models such as DiT, we can achieve slightly better performance to reach SOTA on this dataset.
MODEL	IS (↑)	FID (↓)	SFID (↓)	PREC (↑)	REC (↑)
IMAGENET 64x64
BIGGAN†	44.99	4.06	3.96	0.79	0.48
IDDPM*	46.31	2.90	3.78	0.73	0.62
CADM + CLS-FREE*	63.39	1.93	4.49	0.77	0.60
CADM-G + EDS	61.91	1.85	4.36	0.76	0.61
CADM-G + EDS + PROG	65.89	1.77	4.25	0.77	0.61
IMAGENET 128x128
BIGGAN†	145.93	6.02	7.18	0.86	0.35
LOGAN‡	148.2	3.36
CADM-G + EDS	160.2	2.58	4.92	0.78	0.58
CADM-G + EDS + PROG	175.31	2.47	4.92	0.80	0.58
IMAGENET 256x256
BIGGAN†	202.77	7.03	7.29	0.87	0.27
DCTRANS‡	-	36.51	8.24	0.36	0.67
VQ-VAE-2‡	-	31.11	17.38	0.36	0.57
IDDPM‡	-	12.26	5.42	0.70	0.62
CADM + CLS-FREE	191.31	3.76	4.87	0.80	0.55
CADM-G + EDS	212.51	3.96	5.0	0.82	0.52
CADM-G + EDS + PROG	232.86	3.84	5.0	0.83	0.51
DIT (LATENT DIFFUSION)	122.62	9.62	6.89	0.66	0.670
DIT-G	274.69	2.27	4.58	0.82	0.58
DIT-G + PROG	278.77	2.25	4.56	0.82	0.58
Table 4: FID and Recall trend for guidance sampling with ImageNet256x256 conditional diffusion. ProG not only helps to achieve better FID but also a higher robustness score. Besides, the diversity trend is also better preserved than vanilla guidance when the guidance scale is increased.
MODEL
IMAGENET 256X256	ROBUSTNESS@1 (↑)	FID
CADM-G + EDS	83.31	3.96
CADM-G + EDS + PROG	86.60	3.90
CADM + CLS-FREE	87.14	3.95
Effects of schedule: ProG schedule relies on a single hyperparameter  $\gamma$  to determine the speed of convergence towards the one-hot vector. A larger  $\gamma$  leads to faster convergence, while a smaller  $\gamma$  results in slower convergence. Table 5 demonstrates the sensitivity of  $\gamma$ , showing that varying it does not significantly impact the results. Results obtained with  $\gamma$  values of 0.04 or 0.06 are similar. When  $\gamma$  becomes larger, the model is more likely to achieve a higher Inception Score (IS), but trade-off with FID/sFID scores, indicating a decrease in sample quality. It is worth noting that without the schedule ( $\gamma = 0$ ), the guidance method fails, underscoring the importance of the proposed scheduler.
Correlations between labels: Fixing the same schedule, we change the initial vector of information degree  $\mathbf{s}_T$  via different ways. We denote our ChatGPT-generated text as CGT. Based on CGT, we use CLIP[19] or Word2Vec [53] to obtain the similarity matrix mentioned in section 3. We compare with the uniform label smoothing (Uni. LS), where the  $\mathbf{s}_T$  is uniformly distributed. We also provide the performance of Label Smoothing without our progressive schedule noted as LS (NO SCHE.) to have a comprehensive understanding of the method. Table 6 shows that the CLIP + CGT performs the best among different schemes. This shows the importance of the initial state of the  $\mathbf{s}_T$ , which helps to decide the relevant information for the main condition.
Table 5:  $\gamma$  sensitivity comparison.
MODEL
IMAGENET 64X64	IS (↑)	FID (↓)	sFID (↓)
γ = 0.04	46.88	5.16	6.72
γ = 0.06	48.56	5.4	7.31
γ = 0.1	56.84	7.28	9.58
γ = 0.2	57.75	8.67	12.34
γ = 0.0 (NO SCHE.)	7.97	73.10	19.77
Computational cost: Considering the comparable FID/sFID results between ProG and classifier-free guidance, we uncovered several advantages of ProG over classifier-free guidance, as detailed in Section E of the Supplementary. Notably, one of these advantages pertains to computational efficiency. Table 7 illustrates the GPU hours required for generating 50,000 images at a resolution of  $256 \times 256$ . When ProG is incorporated into the standard classifier guidance framework, the computational cost remains on par with vanilla classifier guidance. In contrast, uti
lizing classifier-free guidance substantially escalates computational expenses. It's worth noting that the computational cost of classifier guidance is predominantly influenced by the architecture of the noise-aware classifier, which is typically simpler than the network architectures used in generative models. Consequently, the observations in Table 7 are broadly applicable in most scenarios.
Table 6: Different approaches for  ${\mathbf{s}}_{T}$  Eq.6
MODEL	IS (↑)	FID (↓)	SFID (↓)
IMAGENET 64x64
CLIP + CGT	46.88	5.16	6.72
W2V + CGT	55.69	6.72	8.79
UNI. LS	26.66	16.39	13.13
LS (NO SCHE.)	25.32	9.98	6.62
Table 7: Computational cost to generate 50000 images with ImageNet256x256
MODEL	GPU HOURS
IMAGENET256X256
ADM	236
CADM-G + EDS	341
CADM-G + EDS + PROG	341
CADM + CLS-FREE	487
This work quantifies two problems of the classifier guidance method: dramatic diversity suppression and non-robustness feature construction. The results show better robustness features than the classifier guidance baseline and a similar level to the classifier-free guidance method. Besides, we also achieve a significantly better diversity trend when increasing the guidance scale quantitatively and qualitatively. We can achieve the SOTA on ImageNet64x64 and ImageNet128x128 on FID/sFID. Our method can be combined with DiT-G [48] to achieve a new SOTA for ImageNet256x256. Our current research demonstrates successful solutions to address the adversarial effects, particularly in cases where high-confidence generated samples exhibit minimal features relevant to the given condition. However, we acknowledge that we have not yet resolved the scenario where diffusion signals dominate and conflict with the classification signals. The resulting images have low confidence and fail to contain any discernible features related to the specified condition. It is worth noting that these features can no longer be considered adversarial due to low confidence and often be wrongly classified by the pretrained classifier used in the sampling process. Resolving the conflict between diffusion and classification gradients remains an open challenge and an area for future investigation.
This work was supported in part by the Australian Research Council under Projects DP210101859 and FT230100549. Besides, the AI training platform supporting this work was provided by High-Flyer AI (Hangzhou High-Flyer AI Fundamental Research Co., Ltd.)
[40] Ye Liu, Yaya Cheng, Lianli Gao, Xianglong Liu, Qilong Zhang, and Jingkuan Song. Practical evaluation of adversarial robustness via adaptive auto attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15105–15114, 2022.
[41] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
[42] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
[43] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[44] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (sp), pages 39-57. IEEE, 2017.
[45] Guangcong Zheng, Shengming Li, Hui Wang, Taiping Yao, Yang Chen, Shouhong Ding, and Xi Li. Entropy-driven sampling and training scheme for conditional diffusion generation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 754-769. Springer, 2022.
[46] Ali Razavi, Aaron Van den Oord, and Oriol Vinyls. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[47] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021.
[48] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[49] Tuomas Kynkänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[51] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[52] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. SqueezeNet: Alexnet-level accuracy with 50x fewer parameters and  $< 0.5$  mb model size. arXiv preprint arXiv:1602.07360, 2016.
[53] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.