SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zebin You $^{1,2*}$ , Yong Zhong $^{1,2*}$ , Fan Bao $^{3}$ , Jiacheng Sun $^{4}$ , Chongxuan Li $^{1,2†}$ , Jun Zhu $^{3}$
<sup>1</sup> Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
$^{2}$  Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China
$^{3}$  Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University
zebin@ruc.edu.cn; yongzhong@ruc.edu.cn; bf19@mails.tsinghua.edu.cn;
sunjiacheng1@huawei.com; chongxuanli@ruc.edu.cn; dcszj@tsinghua.edu.cn
In an effort to further advance semi-supervised generative and classification tasks, we propose a simple yet effective training strategy called dual pseudo training (DPT), built upon strong semi-supervised learners and diffusion models. DPT operates in three stages: training a classifier on partially labeled data to predict pseudo-labels; training a conditional generative model using these pseudo-labels to generate pseudo images; and retraining the classifier with a mix of real and pseudo images. Empirically, DPT consistently achieves SOTA performance of semi-supervised generation and classification across various settings. In particular, with one or two labels per class, DPT achieves a Fréchet Inception Distance (FID) score of 3.08 or 2.52 on ImageNet  $256 \times 256$ . Besides, DPT outperforms competitive semi-supervised baselines substantially on ImageNet classification tasks, achieving top-1 accuracies of  $59.0(+2.8)$ ,  $69.5(+3.0)$ , and  $74.4(+2.0)$  with one, two, or five labels per class, respectively. Notably, our results demonstrate that diffusion can generate realistic images with only a few labels (e.g.,  $<0.1\%$ ) and generative augmentation remains viable for semi-supervised classification. Our code is available at https://github.com/ML-GSAI/DPT.
Diffusion probabilistic models [1, 2, 3, 4, 5, 6, 7] have achieved excellent performance in image generation. However, empirical evidence has shown that labeled data is indispensable for training such models [8, 4]. Indeed, lacking labeled data leads to much lower performance of the generative model. For instance, the representative work (i.e., ADM) [4] achieves an FID of 10.94 on fully labeled ImageNet  $256 \times 256$ , while an FID of 26.21 without labels.
To improve the performance of diffusion models without utilizing labeled data, prior work [8, 9] initially conducts clustering and subsequently trains diffusion models conditioned on the cluster indices. Although these methods can, in some instances, exhibit superior performance over supervised models on low-resolution data, such phenomena have not yet been observed on high-resolution data (e.g., on ImageNet  $256 \times 256$ , an FID of 5.19, compared to an FID of 3.31 achieved by supervised models, see Appendix C). Besides, cluster indices may not always align with ground truth labels, making it hard to control semantics in samples. Compared to unsupervised methods, semi-supervised generative models [10, 11, 12] often perform much better and provide the same way to control the
† Correspondence to Chongxuan Li.
Figure 1: Selected samples from DPT. Top row:  $512 \times 512$  samples from DPT trained with five  $(<0.4\%)$  labels per class. Bottom rows:  $256 \times 256$  samples from DPT trained with one  $(<0.1\%)$  label per class (Left: "Ostrich"; Mid: "King penguin"; Right: "Indigo bunting").
semantics of samples as the supervised ones by using a small number of labels. However, to our knowledge, although it is attractive, little work in the literature has investigated semi-supervised diffusion models. This leads us to a key question: can diffusion models generate high-fidelity images with controllable semantics given only a few (e.g.,  $< 0.1\%$ ) labels?
On the other hand, while it is natural to use images sampled from generative models for semi-supervised classification [10, 11], discriminative methods [13, 14, 15] dominate the area recently. In particular, self-supervised based learners [16, 17, 18] have demonstrated state-of-the-art performance on ImageNet. However, generative models have rarely been considered for semi-supervised classification recently. Therefore another key question arises: can generative augmentation be a useful approach for such strong semi-supervised classifiers, with the aid of advanced diffusion models?
To answer the above two key and pressing questions, we propose a simple but effective training strategy called dual pseudo training (DPT), built upon strong diffusion models and semi-supervised classifiers. DPT is three-staged (see Fig. 3). First, a classifier is trained on partially labeled data and used to predict pseudo-labels for all data. Second, a conditional generative model is trained on all data with pseudo-labels and used to generate pseudo images given labels. Finally, the classifier is trained on real data augmented by pseudo images with labels. Intuitively, in DPT, the two opposite conditional models (i.e. diffusion model and classifier) provide complementary learning signals to each other and benefit mutually (see a detailed discussion in Appendix E).
We evaluate the effectiveness of DPT through diverse experiments on multi-scale and multi-resolution benchmarks, including CIFAR-10 [19] and ImageNet [20] at resolutions of  $128 \times 128$ ,  $256 \times 256$ , and  $512 \times 512$ . Quantitatively, DPT obtains SOTA semi-supervised generation results on two common metrics, including FID [21] and IS [22], in all settings. In particular, in the highly appealing task, i.e. ImageNet  $256 \times 256$  generation, DPT with one (i.e.,  $< 0.1\%$ ) labels per class achieves an FID of 3.08, outperforming strong supervised diffusion models including IDDPM [23], CDM [24], ADM [4] and LDM [25] (see Fig. 2 (a)). It is worth noting that the comparison with previous models here is meant to illustrate that DPT maintains good performance even with minimal labels, rather than directly comparing it to these previous models (direct comparison is unfair as different diffusion models were used). Furthermore, DPT with two (i.e.,  $< 0.2\%$ ) labels per class is comparable to supervised baseline
(a) DPT vs. supervised diffusion models.
(b) DPT vs. semi-supervised classifiers.
Figure 2: Generation and classification results of DPT on ImageNet with few labels. (a) DPT with  $< 0.1\%$  labels outperforms strong supervised diffusion models [4, 24, 25]. (b) DPT substantially improves SOTA semi-supervised learners [17].
U-ViT [5] (FID 2.52 vs. 2.29). Moreover, on ImageNet  $128 \times 128$  generation, DPT with one (i.e.,  $< 0.1\%$  ) labels per class outperforms SOTA semi-supervised generative models  $\mathrm{S}^3\mathrm{GAN}$  [12] with  $20\%$  labels (FID 4.59 vs. 7.7). Qualitatively, DPT can generate realistic, diverse, and semantically correct images with very few labels, as shown in Fig 1. We also explore why classifiers can benefit generative models through class-level visualization and analysis in Appendix H.
As for semi-supervised classification, DPT achieves state-of-the-art (SOTA) performance in various settings, including ImageNet with one, two, five labels per class and  $1\%$  labels. On the smaller dataset, namely CIFAR-10, DPT with four labels per class achieves the second-best error rate of  $4.68 \pm 0.17\%$ . Besides, on ImageNet classification benchmarks with one, two, five labels per class and  $1\%$  labels, DPT outperforms competitive semi-supervised baselines [17, 16], achieving state-of-the-art top-1 accuracy of  $59.0(+2.8)$ ,  $69.5(+3.0)$ ,  $74.4(+2.0)$  and  $80.2(+0.8)$  respectively (see Fig. 2 (b)). Similarly to generation tasks, we also investigate why generative models can benefit classifiers via class-level visualization and analysis in Appendix I.
In summary, our novelty and key contributions are as follows:
- We present Dual Pseudo Training (DPT), a straightforward yet effective strategy designed to advance the frontiers of semi-supervised diffusion models and classifiers.
- We achieve SOTA semi-supervised generation performance on CIFAR-10 and ImageNet datasets across various settings. Moreover, we demonstrate that diffusion models with a few labels (e.g.,  $< 0.1\%$  ) can generate realistic, diverse, and semantically accurate images, as depicted in Fig 1.
- We achieve SOTA semi-supervised classification performance on ImageNet datasets across various settings and the second-best results on CIFAR-10. Besides, we demonstrate that aided by diffusion models, generative augmentation remains a viable approach for semi-supervised classification.
- We explore why diffusion models and semi-supervised learners benefit mutually with few labels via class-level visualization and analysis, as showcased in Appendix H and Appendix I.
2 Settings and Preliminaries
We present settings and preliminaries on two representative self-supervised based learners for semi-supervised learning [17] [16] in Sec. 2.1 and conditional diffusion probabilistic models [2, 5, 26] in Sec. 2.2, respectively. We consider image generation and classification in semi-supervised learning, where the training set consists of  $N$  labeled images  $\mathcal{S} = \{(\mathbf{x}_i^l,y_i^l)\}_{i = 1}^N$  and  $M$  unlabeled images  $\mathcal{D} = \{\mathbf{x}_i^u\}_{i = 1}^M$ . We assume  $N\ll M$ . For convenience, we denote the set of all real images as  $\mathcal{X} = \{\mathbf{x}_i^u\}_{i = 1}^M\cup \{\mathbf{x}_i^l\}_{i = 1}^N$ , and the set of all possible classes as  $\mathcal{Y}$ .
Figure 3: An overview of DPT. First, a (semi-supervised) classifier is trained on partially labeled data and used to predict pseudo-labels for all data. Second, a conditional generative model is trained on all data with pseudo-labels and used to generate pseudo images given random labels. Finally, the classifier is trained or fine-tuned on real data augmented by pseudo images with labels.
2.1 Semi-Supervised Classifier
Masked Siamese Networks (MSN) [17] employ a ViT-based [27] anchor encoder  $f_{\theta}(\cdot)$  and a target encoder  $f_{\bar{\theta}}(\cdot)$ , where  $\bar{\theta}$  is the exponential moving average (EMA) [28] of parameters  $\theta$ . For a real image  $\mathbf{x}_i \in \mathcal{X}, 1 \leq i \leq M + N$ , MSN obtains  $H + 1$  random augmented images, denoted as  $\mathbf{x}_{i,h}, 1 \leq h \leq H + 1$ . MSN then applies either a random mask or a focal mask to the first  $H$  augmented images and obtain  $\mathrm{mask}(\mathbf{x}_{i,h}), 1 \leq h \leq H$ . MSN optimizes  $\theta$  and a learnable matrix of prototypes  $\mathbf{q}$  by the following objective function:
$$
\frac {1}{H (M + N)} \sum_ {i = 1} ^ {M + N} \sum_ {h = 1} ^ {H} \mathrm {C E} \left(\mathbf {p} _ {i, h}, \mathbf {p} _ {i, H + 1}\right) - \lambda \mathrm {H} (\bar {\mathbf {p}}), \tag {1}
$$
where CE and H are cross entropy and entropy respectively,  $\mathbf{p}_{i,h} = \mathrm{softmax}((f_{\theta}(\mathrm{mask}(\mathbf{x}_{i,h}))\cdot \mathbf{q} / \tau)$ ,  $\bar{\mathbf{p}}$  is the mean of  $\mathbf{p}_{i,h}$ ,  $\mathbf{p}_{i,H + 1} = \mathrm{softmax}(f_{\bar{\theta}}(\mathbf{x}_{i,H + 1})\cdot \mathbf{q} / \tau')$ ,  $\tau, \tau'$  and  $\lambda$  are hyper-parameters, and  $\cdot$  denotes cosine similarity. MSN is an efficient semi-supervised approach by extracting features for all labeled images in  $S$  and training a linear classifier on top of the features using L2-regularized logistic regression. When a self-supervised pre-trained model is available, MSN demonstrates high efficiency in training a semi-supervised classifier on a single CPU core.
Semi-ViT [16] is three-staged. First, it trains a ViT-based encoder  $f_{\theta}(\cdot)$  on all images in  $\mathcal{X}$  via self-supervised methods such as MAE [29]. Second,  $f_{\theta}(\cdot)$  is merely fine-tuned on  $S$  in a supervised manner. Let  $\bar{\theta}$  be the EMA of  $\theta$ , and  $\mathbf{x}_i^{u,s}$  and  $\mathbf{x}_i^{u,w}$  denote the strong and weak augmented versions of  $\mathbf{x}_i^u$  respectively. Finally, Semi-ViT optimizes a weighted sum of two cross-entropy losses:
$$
\mathcal {L} = \mathcal {L} _ {l} + \mu \mathcal {L} _ {u} = \frac {1}{N} \sum_ {j = 1} ^ {N} \operatorname {C E} \left(f _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {j} ^ {l}\right), \operatorname {v e c} \left(y _ {j} ^ {l}\right)\right) + \tag {2}
$$
$$
\frac {\mu}{M} \sum_ {i = 1} ^ {M} \mathbb {I} [ f _ {\bar {\theta}} (\mathbf {x} _ {i} ^ {u, w}) _ {\hat {y} _ {i}} \geq \tau ] \mathrm {C E} (f _ {\boldsymbol {\theta}} (\mathbf {x} _ {i} ^ {u, s}), \mathrm {v e c} (\hat {y} _ {i})),
$$
where  $f_{\overline{\theta}}(\mathbf{x})_y$  is the logit of  $f_{\overline{\theta}}(\mathbf{x})$  indexed by  $y$ ,  $\hat{y}_i = \arg \max_y f_{\overline{\theta}}(\mathbf{x}_i^{u,w})_y$  is the pseudo-label,  $\mathrm{vec}(\cdot)$  returns the one-hot representation, and  $\tau$  and  $\mu$  are hyper-parameters.
2.2 Conditional Diffusion Probabilistic Models
Denoising Diffusion Probabilistic Model (DDPM) [2] gradually adds noise  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  to data  $\mathbf{x}_0$  from time  $t = 0$  to  $t = T$  in the forward process, and progressively removes noise to recover data starting at  $\mathbf{x}_T\sim \mathcal{N}(\mathbf{0},\mathbf{I})$  in the reverse process. It trains a predictor  $\epsilon_{\theta}$  to predict the noise  $\epsilon$  by the following objective:
$$
\mathcal {L} = \mathbb {E} _ {t, \mathbf {x} _ {0}, \boldsymbol {\epsilon}} [ \| \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\mathbf {x} _ {t}, \mathbf {c}, t) - \boldsymbol {\epsilon} \| _ {2} ^ {2} ], \tag {3}
$$
where  $\mathbf{c}$  indicates conditions such as classes and texts.
Classifier-Free Guidance (CFG) [26] leverages a conditional noise predictor  $\epsilon_{\theta}(\mathbf{x}_t,\mathbf{c},t)$  and an unconditional noise predictor  $\epsilon_{\theta}(\mathbf{x}_t,t)$  in inference to improve sample quality and enhance semantics. Formally, CFG iterates the following equation starting at  $\mathbf{x}_T$ :
$$
\mathbf {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \tilde {\boldsymbol {\epsilon}} _ {t}\right) + \sigma_ {t} ^ {2} \mathbf {z}, \tag {4}
$$
where  $\tilde{\epsilon}_t = (1 + \omega)\pmb {\epsilon}_\theta (\mathbf{x}_t,\mathbf{c},t) - \omega \pmb {\epsilon}_\theta (\mathbf{x}_t,t),\omega$  is the guidance strength,  $\mathbf{z}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$  , and  $\alpha_{t},\beta_{t}$ $\bar{\alpha}_{t}$  and  $\sigma_t$  are constants w.r.t. the time  $t$
U-ViT [5] is a ViT-based backbone for diffusion probabilistic models, which achieves excellent performance in conditional sampling on large-scale datasets.
we propose a three-stage strategy called dual pseudo training (DPT) to advance semi-supervised generation and classification tasks, illustrated in Fig. 3 and detailed as follows.
3.1 First Stage: Train Classifier
DPT trains a semi-supervised classifier on partially labeled data  $S \cup \mathcal{D}$ , predicts a pseudo-label  $\hat{y}$  of any image  $\mathbf{x} \in \mathcal{X}$  by the classifier, and constructs a dataset consisting of all images with pseudo-labels, i.e.  $S_{1} = \{(\mathbf{x},\hat{y})|\mathbf{x} \in \mathcal{X}\}^{3}$ . Notably, here we treat the classifier as a black box without modifying the training strategy or any hyperparameter. Therefore, any well-trained classifier can be adopted in DPT in a plug-and-play manner. Indeed, we use recent advances in self-supervised based learners for semi-supervised learning, i.e. MSN [17], and Semi-ViT [16]. These two classifiers both provide the generative model with accurate, low-noise labels of high quality.
3.2 Second Stage: Classifier Benefits Generative Model
DPT trains a conditional generative model on all real images with pseudo-labels  $S_{1}$ , samples  $K$  pseudo images for any class label  $y$  after training, and constructs a dataset consisting of pseudo images with uniform labels<sup>4</sup>. We denote the dataset as  $S_{2} = \cup_{y\in \mathcal{Y}}\{(\hat{\mathbf{x}}_{i,y},y)\}_{i = 1}^{K}$ , where  $\hat{\mathbf{x}}_{i,y}$  is the  $i$ -th pseudo image for class  $y$ . Similarly to the classifier, DPT also treats the conditional generative model as a black box. Inspired by the impressive image generation results of diffusion probabilistic models, we take a U-ViT-based [5] denoise diffusion probabilistic model [2] with classifier-free guidance [26] as the conditional generative model. Everything remains the same as the original work (see Sec. 2.2) except that the set of all real images with pseudo-labels  $S_{1}$  is used for training.
We emphasize that  $S_{1}$  obtained by the first stage is necessary. In fact,  $S$  is of small size (e.g., one label per class) and not sufficient to train conditional diffusion models. Besides, it is unclear how to leverage unlabeled data to train such models. Built upon efficient and strong semi-supervised approaches [17, 16],  $S_{1}$  provides useful learning signals (with relatively small noise) to train conditional diffusion models. We present quantitative and qualitative empirical evidence in Fig. 2 (a) and Fig. 1 respectively to affirmatively answer the first key question, namely, diffusion models with a few labels (e.g.,  $< 0.1\%$ ) can generate realistic and semantically accurate images.
For simplicity, we also use pseudo-labels instead of the ground truth for real labeled data, which are rare and have a small zero-one training loss, making no significant difference.
4The prior distribution of  $y$  can be estimated on  $\bar{S}$
3.3 Third Stage: Generative Model Benefits Classifier
MSN based DPT. We train the classifier employed in the first stage on real data augmented by  $S_{2}$  to boost classification performance. For simplicity and efficiency, we freeze the models pre-trained by Eq. (1) in the first stage and replace  $S$  with  $S \cup S_{2}$  to train a linear probe in MSN [17]. DPT substantially boosts the classification performance as presented in Fig. 2 (b).
Semi-ViT based DPT. We freeze the models, which are pre-trained in a self-supervised manner in the first stage of Semi-ViT, and replace  $S$  with  $S \cup S_2$  to train a classifier in the third stage of Semi-ViT. We argue that pseudo images can be used in different stages of Semi-ViT and can both boost the classification performance. (see Appendix F.2).
Both consistent improvements provide a positive answer to the second key question, namely, generative augmentation remains a useful approach for semi-supervised classification. Besides, we can leverage the classifier in the third stage to refine the pseudo-labels and train the generative model with one more stage. Although we observe an improvement empirically (see results in Appendix F.3), we focus on the three-stage strategy in the main paper for simplicity and efficiency.
Semi-Supervised Classification and Generation. The two tasks are often studied independently. For semi-supervised classification, classical work includes generative approaches based on VAE [10, 30, 31] and GAN [32, 22, 33, 34, 35, 36], and discriminative approaches with confidence regularization [37, 38, 39, 40], consistency regularization [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 13, 14, 51, 52] and other approaches [53, 54, 55, 56]. Recently, large-scale self-supervised based approaches [18, 28, 57, 17, 16] have made remarkable progress in semi-supervised learning. Besides, semi-supervised conditional image generation is challenging because generative modeling is more complex than prediction. In addition, it is highly nontrivial to design proper regularization when the input label is missing. Existing work is based on VAE [10] or GAN [11, 12], which are limited to low-resolution data (i.e.,  $\leq 128 \times 128$ ) and require  $10\%$  labels or so to achieve comparable results to supervised baselines.
In comparison, DPT handles both classification and generation tasks in extreme settings with very few labels (e.g., one label per class,  $< 0.1\%$  labels). Built upon recent advances in semi-supervised learners and diffusion models, DPT substantially improves the state-of-the-art results in both tasks.
Pseudo Data and Labels. We mention additional empirical work on generating pseudo data for supervised learning [58], adversarial robust learning [59, 60], contrastive representation learning [61] and zero-shot learning [62, 63]. Regarding theory, in the context of supervised classification, Zheng et al. [64] have mentioned that when the training dataset size is small, generative data augmentation can improve the learning guarantee at a constant level. This finding can be extended to semi-supervised classification, which is left as future work.
Besides, prior work [65, 66, 67, 8] uses cluster index or instance index as pseudo-labels to improve unsupervised generation results, which are not directly comparable to DPT. With additional few labels, DPT can generate images of much higher quality and directly control the semantics of images with class labels.
Diffusion Models. Recently, diffusion probabilistic models [32, 2, 3, 6] achieve remarkable progress in image generation [4, 25, 24, 8, 26, 7], text-to-image generation [68, 69, 70, 25, 71], 3D scene generation [72], image-editing [73, 74, 75], molecular design [76, 77], and semi-supervised medical science [78, 79]. There are learning-free methods [80, 81, 82, 83, 84] and learning-based ones [85, 86] to speed up the sampling process of diffusion models. In particular, we adopt third-order DPM-solver [84], which is a recent learning-free method, for fast sampling. As for the architecture, most diffusion models rely on variants of the U-Net architecture introduced in score-based models [87] while recent work [5] proposes a promising vision transformer for diffusion models, as employed in DPT.
To the best of our knowledge, there has been little research on semi-supervised conditional diffusion models and diffusion-based semi-supervised classification, which are the focus of this paper.
We present the main experimental settings in Sec. 5.1. For more details, please refer to Appendix C. To evaluate the performance of DPT, we compare it with state-of-the-art conditional diffusion models and semi-supervised learners in Sec. 5.2 and Sec. 5.3 respectively. We also visualize and analyze the interaction between the stages to explain the excellent performance of DPT (see Appendix I, H).
Dataset. We evaluate DPT on the ImageNet [20] dataset, which consists of 1,281,167 training and 50,000 validation images. In the first and third stages, we use the same pre-processing protocol for real images as the baselines [17, 16]. For instance, in MSN, the real data are resized to  $256 \times 256$  and then center-cropped to  $224 \times 224$ . In the second stage, real images are center-cropped to the target resolution following [5]. In the third stage, we consider pseudo images at resolution  $256 \times 256$  and center-crop them to  $224 \times 224$ . For semi-supervised classification, we consider the challenging settings with one, two, five labels per class and  $1\%$  labels. The labeled and unlabeled data split is the same as that of corresponding methods [17, 16]. We also evaluate DPT on CIFAR-10 (see detailed experiments in Appendix A).
Baselines. For semi-supervised classification, we consider state-of-the-art semi-supervised approaches [17, 16] in the setting of low-shot (e.g., one, two, five labels per class and  $1\%$  labels) as baselines. For conditional generation, we consider the state-of-the-art diffusion models with a U-ViT architecture [5] as the baseline.
Model Architectures and Hyperparameters. For a fair comparison, we use the exact same architectures and hyperparameters as the baselines [17, 16, 5]. In particular, for MSN based DPT, we use a ViT B/4 (or a ViT L/7) model [17] for classification and a U-ViT-Large (or a U-ViT-Huge) model [5] for conditional generation. As for Semi-ViT based DPT, we use a ViT-Huge model [16] for classification and a U-ViT-Huge model [5] for conditional generation. More details are provided in Appendix C for reference.