SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation metrics. We use the top-1 accuracy on the validation set to evaluate classification performance. For a comprehensive evaluation of generation performance, we first consider the Fréchet inception distance (FID) [21], sFID [88], Inception Score (IS) [22], precision, and recall [89] on 50K generated samples. We calculate all generation metrics based on the implementation of ADM [4]. We also add the metric  $\mathrm{FID}_{\mathrm{CLIP}}$ , which operates similarly to FID but substitutes the Inception-V3 feature spaces with CLIP features, to eliminate confusion that FID can be artificially reduced by aligning the histograms of Top-N classifications without the actual improvement of image quality [90].
Implementation. DPT is easy to understand and implement. In particular, it only requires several lines of code based on the implementation of the classifier and conditional diffusion model. We provide the pseudocode of DPT in the style of PyTorch in Appendix B.
The choice of  $K$  and CFG. We conduct detailed ablation experiments on the number of augmented pseudo images per class (i.e.,  $K$ ) and the classifier-free guidance scale (i.e., CFG) in Appendix G and find that the optimal  $K$  value is 128 and the optimal CFG values for different ImageNet resolutions are 0.8 for  $128 \times 128$ , 0.4 for  $256 \times 256$ , and 0.7 for  $512 \times 512$ .
The choice of resolution and number of labels. We were primarily driven by the task of ImageNet  $256 \times 256$  generation to systematically compare with a large family of baselines. In this context, we conducted detailed experiments, including settings with one, two, five labels per class, and  $1\%$  labels. We find that the performance of DPT with five labels per class is comparable to the supervised baseline, leading us to use this setting as the default in our other tasks such as ImageNet  $128 \times 128$  and ImageNet  $512 \times 512$  generation.
5.2 Image Generation with Few Labels
We show that diffusion models with a few labels can generate realistic and semantically accurate images. In particular, DPT achieves better results than semi-supervised methods on ImageNet  $128 \times 128$  and comparable results to supervised methods on both ImageNet  $256 \times 256$  and  $512 \times 512$ .
Table 1: Image generation results on ImageNet  $128 \times 128$ .† labels the results taken from the corresponding references and  $\star$  labels baseline achieved by us. We bold the best result under the corresponding setting. With  $<0.1\%$  labels, DPT outperforms strong semi-supervised generative models  $S^3$ GAN [12].
Method	Model	Label fraction (# labels/class)	FID-50K ↓	IS ↑
U-ViT-Huge(supervised baseline)*	Diff.	100%	4.53	219.8
S3GAN [12]†	GAN	5%	10.4	59.6
S3GAN [12]†	GAN	10%	8.0	78.7
S3GAN [12]†	GAN	20%	7.7	83.1
DPT (ours, with U-ViT-Huge and MSN)	Diff.	< 0.1%(1)	4.59	153.6
DPT (ours, with U-ViT-Huge and MSN)	Diff.	< 0.4%(5)	4.58	210.9
Table 2: Image generation results on ImageNet  ${256} \times  {256}$  . † labels the results taken from the corresponding references and * labels baselines achieved by us. DPT and the corresponding baselines employ the same model architectures [5]. With < 0.4% labels, DPT outperforms strong conditional generative models with full labels, including CDM [24], ADM [4] and LDM [25]. We bold the best result achieved with full labels and underline the best result achieved with few labels. For a fair comparison, we also list the parameters of the diffusion model, including its auxiliary components.
Method	Model	Label fraction (# labels/class)	FID ↓	FIDCLIP ↓	sFID ↓	IS ↑	Precision ↑	Recall ↑	# Params
IC-GAN [67]†	GAN	0%	15.6	-	59.0	-	-	-	-
BigGAN-deep [91]†	GAN	100%	6.95	-	7.36	171.4	0.87	0.28	-
StyleGAN-XL [92]†	GAN	100%	2.30	-	4.02	265.12	0.78	0.53	-
IDDPM [23]†	Diff.	100%	12.26	-	5.42	-	0.70	0.62	550M
CDM [24]†	Diff.	100%	4.88	-	-	158.71	-	-	-
ADM [4]†	Diff.	100%	3.94	-	6.14	215.84	0.83	0.53	673M
LDM-4-G [25]†	Diff.	100%	3.60	-	-	247.67	0.87	0.48	455M
DiT-XL/2-G [7]†	Diff.	100%	2.27	-	4.60	278.24	0.83	0.57	675M
U-ViT-Large [5]†	Diff.	100%	3.40	-	6.63	219.94	0.83	0.52	371M
With U-ViT-Large
Supervised baseline*	Diff.	100%	3.31	2.39	6.68	221.61	0.83	0.53	371M
Unsupervised baseline*	Diff.	0%	27.99	5.40	7.03	33.86	0.60	0.62	371M
DPT (ours, with MSN)	Diff.	< 0.1%(1)	4.34	2.57	6.68	162.96	0.80	0.53	371M
DPT (ours, with MSN)	Diff.	< 0.2%(2)	3.44	2.37	6.58	199.74	0.82	0.53	371M
DPT (ours, with MSN)	Diff.	< 0.4%(5)	3.37	2.35	6.71	217.53	0.83	0.52	371M
DPT (ours, with MSN)	Diff.	1%(≈12)	3.35	2.34	6.66	223.09	0.83	0.52	371M
With U-ViT-Huge
Supervised baseline†	Diff.	100%	2.29	1.75	5.68	263.88	0.82	0.57	585M
DPT (ours, with MSN)	Diff.	< 0.1%(1)	3.08	1.84	5.56	201.68	0.80	0.58	585M
DPT (ours, with MSN)	Diff.	< 0.2%(2)	2.52	1.81	5.49	230.34	0.81	0.57	585M
DPT (ours, with MSN)	Diff.	< 0.4%(5)	2.50	1.82	5.54	243.10	0.83	0.55	585M
DPT (ours, with Semi-ViT)	Diff.	1%(≈12)	2.42	1.77	5.48	259.93	0.82	0.56	585M
We evaluate semi-supervised generation performance of DPT on ImageNet  $128 \times 128$ , as shown in Tab. 1. In particular, DPT with only  $< 0.1\%$  labels outperforms the SOTA semi-supervised generative model S $^3$ GAN [12] with  $20\%$  labels (FID 4.59 vs. 7.7), suggesting DPT has superior label efficiency.
In Tab. 2, we compare DPT with state-of-the-art generative models on ImageNet  $256 \times 256$ . We construct highly competitive baselines based on diffusion models with U-ViT-Large [5]. According to Tab. 2, our supervised and unsupervised baselines achieve an FID of 3.31 and 27.99, respectively. Leveraging the pseudo-labels predicted by the strong semi-supervised learner [17], DPT with few labels improves the unconditional baseline significantly and is even comparable to the supervised baseline under all metrics. In particular, with only two labels per class, DPT improves the FID of the unsupervised baseline by 24.55 and is comparable to the supervised baseline with a gap of 0.13. Moreover, we also construct more competitive baselines based on U-ViT-Huge to advance DPT. With one (i.e.,  $< 0.1\%$ ) label per class, our more powerful DPT achieves an FID of 3.08, outperforming strong supervised diffusion models including IDDPM [23], CDM [24], ADM [4] and LDM [25]. Additionally, with  $1\%$  labels, DPT achieves an FID of 2.42, comparable to the
Table 3: Image generation results on ImageNet  ${512} \times  {512}$  . † labels the results taken from the corresponding references. We bold the best result under the corresponding setting.
Method	Model	Label fraction (# labels/class)	FID-50K ↓	IS ↑
BigGAN-deep [91]†	GAN	100%	8.43	177.90
StyleGAN-XL [92]†	GAN	100%	2.41	267.75
ADM [4]†	Diff.	100%	3.85	221.72
DiT-XL/2-G [7]†	Diff.	100%	3.04	240.82
U-ViT-Huge (supervised baseline)†	Diff.	100%	4.05	263.79
DPT (ours, with U-ViT-Huge and MSN)	Diff.	< 0.4%(5)	4.05	252.08
Table 4: Top-1 accuracy on the ImageNet validation set with few labels.  $\dagger$  labels the results taken from corresponding references,  $\ddagger$  labels the results taken from Assran et al. [17] and  $\star$  labels the baselines reproduced by us. DPT and the corresponding baseline employ exactly the same classifier architectures. With one, two, five labels per class and  $1\%$  labels, DPT improves the state-of-the-art semi-supervised learner [17, 16] consistently and substantially. We bold the best result under the corresponding setting and underline the second-best result.
Method	Architecture	Top-1 accuracy ↑ given # labels per class (label fraction)
1(< 0.1%)	2(< 0.2%)	5(< 0.5%)	≈ 12(1%)
EMAN [57]†	ResNet-50	-	-	-	63.0
PAWS [51]†	ResNet-50	-	-	-	66.5
BYOL [28]†	ResNet-200	-	-	-	71.2
SimCLRv2 [18]†	ResNet-152	-	-	-	76.6
Semi-ViT [16]†	ViT-Huge	-	-	-	80.0
iBOT [93]‡	ViT-B/16	46.1 ± 0.3	56.2 ± 0.7	64.7 ± 0.3	-
DINO [94]‡	ViT-B/8	45.8 ± 0.5	55.9 ± 0.6	64.6 ± 0.2	-
MAE [29]‡	ViT-H/14	11.6 ± 0.4	18.6 ± 0.2	32.8 ± 0.2	-
MSN [17]†	ViT-B/4	54.3 ± 0.4	64.6 ± 0.7	72.4 ± 0.3	75.7
MSN [17]†	ViT-L/7	57.1 ± 0.6	66.4 ± 0.6	72.1 ± 0.2	75.1
MSN (baseline)*	ViT-B/4	52.9	64.9	72.4	-
DPT (ours)	ViT-B/4	58.6	69.5	74.4	-
MSN (baseline)*	ViT-L/7	56.2	66.5	72.0	-
DPT (ours)	ViT-L/7	58.9	69.2	73.4	-
Semi-ViT (baseline)*	ViT-Huge	-	-	-	79.4
DPT (ours)	ViT-Huge	-	-	-	80.2
state-of-the-art supervised diffusion model [7]. Lastly, DPT with few labels performs comparably to the fully supervised baseline under the  $\mathrm{FID}_{\mathrm{CLIP}}$  metric, which suggests that DPT can generate high-quality samples and does not achieve a lower FID solely due to better Top-N alignment.
We also conduct an experiment on higher resolution (i.e.,  $512 \times 512$ ) in Tab. 3, with five (i.e.,  $< 0.4\%$ ) labels, DPT achieves an FID of 4.05, which is the same as that of the supervised baseline. The above quantitative results demonstrate that DPT can achieve excellent generation performance and label efficiency at diverse resolutions. Qualitatively, as presented in Fig. 1, DPT can generate realistic, diverse, and semantically correct images even with a single label, which agrees with the quantitative results in Tab. 2 and Tab. 3. We provide more samples and failure cases in Appendix F.1 and a detailed class-wise analysis to show how classification helps generation in Appendix H.
Besides, Tab. 5 in Appendix A compares DPT with state-of-the-art generative models on CIFAR-10. DPT achieves competitive performance using only  $0.08\%$  labels with EDM [6], which relies on full labels (FID 1.81 vs. 1.79). This result demonstrates the generalizability of DPT on different datasets.
5.3 Image Classification with Few Labels
We demonstrate that generative augmentation remains a useful approach for semi-supervised classification aided by diffusion models. In particular, DPT achieves state-of-the-art semi-supervised classification performance on ImageNet datasets across various settings and the second-best results on CIFAR-10.
Tab. 4 compares DPT with state-of-the-art semi-supervised classifiers on the ImageNet validation set with few labels. Specifically, DPT outperforms strong semi-supervised baselines [17, 16] consistently and substantially with one, two, five labels per class and  $1\%$  labels and achieves state-of-the-art top-1 accuracies of 59.0, 69.5, 74.4 and 80.2, respectively. In particular, with two labels per class, DPT leverages the pseudo images generated by the diffusion model and improves MSN with ViT-B/4 by an accuracy of  $4.6\%$ . Besides, we compare the performance of DPT with that of SOTA fully supervised models (as shown in Tab. 12 in Appendix F.2) and find that DPT performs comparably to Inception-v4 [95], using only  $1\%$  labels.
Moreover, Tab. 6 in Appendix A compares DPT with state-of-the-art semi-supervised classifiers on CIFAR-10. DPT with four labels per class achieves the second-best error rate of  $4.68 \pm 0.17\%$ .
This paper presents a simple yet effective training strategy called DPT for conditional image generation and classification in semi-supervised learning. Empirically, we demonstrate that DPT can achieve SOTA semi-supervised generation and classification performance on ImageNet datasets across various settings. DPT probably inspires future work in diffusion models and semi-supervised learning.
Limitation. One limitation of DPT is directly using the pseudo images to improve the performance of DPT for its simplicity and effectiveness while we could use pre-trained models like CLIP to filter out noisy image-label pairs that images do not semantically align well with the label. Another limitation pertains to the direct use of pseudo labels. Given our use of classifier-free guidance, we have the flexibility to assign low-confidence pseudo labels to the null token with a high probability, which aids in filtering out noisy pseudo labels.
Social impact. We believe that DPT can benefit real-world applications with few labels (e.g., medical analysis). However, the proposed semi-supervised diffusion models may aggravate social issues such as "DeepFakes". The problem can be relieved by automatic detection with machine learning, which is an active research area.
This work was supported by NSF of China (Nos. 62076145); Beijing Outstanding Young Scientist Program (No. BJWZYJH012019100020098); Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (No. 22XNKJ13). C. Li was also sponsored by Beijing Nova Program (No. 20220484044).
[1] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in Proceedings of the 32nd International Conference on Machine Learning, 2015.
[2] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems, 2020.
[3] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," in 9th International Conference on Learning Representations, 2021.
[4] P. Dhariwal and A. Nichol, "Diffusion models beat gans on image synthesis," Advances in Neural Information Processing Systems, vol. 34, pp. 8780-8794, 2021.
[110] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
[111] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132-7141.
[112] M. Tan and Q. Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," in International conference on machine learning. PMLR, 2019, pp. 6105-6114.
[113] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in International Conference on Machine Learning. PMLR, 2021, pp. 10347-10357.
[114] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012-10022.