Title: Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

semi-supervised generation

semi-supervised diffusion models

semi-supervised classification

image generation.
Abstract: In an effort to further advance semi-supervised generative and classification tasks, we propose a simple yet effective training strategy called *dual pseudo training* (DPT), built upon strong semi-supervised learners and diffusion models. DPT operates in three stages: training a classifier on partially labeled data to predict pseudo-labels; training a conditional generative model using these pseudo-labels to generate pseudo images; and retraining the classifier with a mix of real and pseudo images.  Empirically, DPT consistently achieves SOTA performance of semi-supervised generation and classification across various settings. In particular, with one or two labels per class, DPT achieves a Fréchet Inception Distance (FID) score of 3.08 or 2.52 on ImageNet $256\times256$. Besides, DPT outperforms competitive semi-supervised baselines substantially on  ImageNet classification tasks, *achieving top-1 accuracies of 59.0 (+2.8), 69.5 (+3.0), and 74.4 (+2.0)* with one, two, or five labels per class, respectively. Notably, our results demonstrate that diffusion can generate realistic images with only a few labels (e.g., $<0.1$%) and generative augmentation remains viable for semi-supervised classification. Our code is available at *https://github.com/ML-GSAI/DPT*.

Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels

Abstract

In an effort to further advance semi-supervised generative and classification tasks, we propose a simple yet effective training strategy called dual pseudo training (DPT), built upon strong semi-supervised learners and diffusion models. DPT operates in three stages: training a classifier on partially labeled data to predict pseudo-labels; training a conditional generative model using these pseudo-labels to generate pseudo images; and retraining the classifier with a mix of real and pseudo images. Empirically, DPT consistently achieves SOTA performance of semi-supervised generation and classification across various settings. In particular, with one or two labels per class, DPT achieves a Fréchet Inception Distance (FID) score of 3.08 or 2.52 on ImageNet 256 × 256 . Besides, DPT outperforms competitive semi-supervised baselines substantially on ImageNet classification tasks, achieving top-1 accuracies of 59.0 (+2.8), 69.5 (+3.0), and 74.4 (+2.0) with one, two, or five labels per class, respectively. Notably, our results demonstrate that diffusion can generate realistic images with only a few labels (e.g., < 0 . 1% ) and generative augmentation remains viable for semi-supervised classification. Our code is available at https://github.com/ML-GSAI/DPT .

1 Introduction

Diffusion probabilistic models [1, 2, 3, 4, 5, 6, 7] have achieved excellent performance in image generation. However, empirical evidence has shown that labeled data is indispensable for training such models [8, 4]. Indeed, lacking labeled data leads to much lower performance of the generative model. For instance, the representative work (i.e., ADM) [4] achieves an FID of 10.94 on fully labeled ImageNet 256 × 256 , while an FID of 26.21 without labels.

To improve the performance of diffusion models without utilizing labeled data, prior work [8, 9] initially conducts clustering and subsequently trains diffusion models conditioned on the cluster indices. Although these methods can, in some instances, exhibit superior performance over supervised models on low-resolution data, such phenomena have not yet been observed on high-resolution data (e.g., on ImageNet 256 × 256 , an FID of 5.19, compared to an FID of 3.31 achieved by supervised models, see Appendix C). Besides, cluster indices may not always align with ground truth labels, making it hard to control semantics in samples. Compared to unsupervised methods, semi-supervised generative models [10, 11, 12] often perform much better and provide the same way to control the

semantics of samples as the supervised ones by using a small number of labels. However, to our knowledge, although it is attractive, little work in the literature has investigated semi-supervised diffusion models. This leads us to a key question: can diffusion models generate high-fidelity images with controllable semantics given only a few (e.g., < 0 . 1% ) labels?

On the other hand, while it is natural to use images sampled from generative models for semisupervised classification [10, 11], discriminative methods [13, 14, 15] dominant the area recently. In particular, self-supervised based learners [16, 17, 18] have demonstrated state-of-the-art performance on ImageNet. However, generative models have rarely been considered for semi-supervised classification recently. Therefore another key question arises: can generative augmentation be a useful approach for such strong semi-supervised classifiers, with the aid of advanced diffusion models?

To answer the above two key and pressing questions, we propose a simple but effective training strategy called dual pseudo training (DPT), built upon strong diffusion models and semi-supervised classifiers. DPT is three-staged (see Fig. 3). First, a classifier is trained on partially labeled data and used to predict pseudo-labels for all data. Second, a conditional generative model is trained on all data with pseudo-labels and used to generate pseudo images given labels. Finally, the classifier is trained on real data augmented by pseudo images with labels. Intuitively, in DPT, the two opposite conditional models (i.e. diffusion model and classifier) provide complementary learning signals to each other and benefit mutually (see a detailed discussion in Appendix E).

We evaluate the effectiveness of DPT through diverse experiments on multi-scale and multi-resolution benchmarks, including CIFAR-10 [19] and ImageNet [20] at resolutions of 128 × 128 , 256 × 256 , and 512 × 512 . Quantitatively, DPT obtains SOTA semi-supervised generation results on two common metrics, including FID [21] and IS [22], in all settings. In particular, in the highly appealing task, i.e. ImageNet 256 × 256 generation, DPT with one (i.e., < 0.1%) labels per class achieves an FID of 3.08, outperforming strong supervised diffusion models including IDDPM [23], CDM [24], ADM [4] and LDM [25] (see Fig. 2 (a)). It is worth noting that the comparison with previous models here is meant to illustrate that DPT maintains good performance even with minimal labels, rather than directly comparing it to these previous models (direct comparison is unfair as different diffusion models were used). Furthermore, DPT with two (i.e., < 0.2%) labels per class is comparable to supervised baseline

6

FID-50K - ImageNet 256x256

CDM (100%)

4

ADM (100%)

DPT (ours, <0.1%)

LDM (100%)

3

DPT (ours, <0.2%)

2

DPT (ours, 1%)s

Supervised Baseline,SOTA (100%)

(a) DPT vs. supervised diffusion models.

76

SOTA Semi-Supervised Learners

DPT

74

2108

DPT (ours)

SOTA

DPT

８64０

SOTA

60

DPT

58

SOTA

56

54

<0.1% (1)

<0.2% (2)

<0.4% (5)

(b) DPT vs. semi-supervised classifiers.

U-ViT [5] (FID 2.52 vs. 2.29). Moreover, on ImageNet 128 × 128 generation, DPT with one (i.e., < 0.1 %) labels per class outperforms SOTA semi-supervised generative models S 3 GAN [12] with 20 % labels (FID 4.59 vs. 7.7). Qualitatively, DPT can generate realistic, diverse, and semantically correct images with very few labels, as shown in Fig 1. We also explore why classifiers can benefit generative models through class-level visualization and analysis in Appendix H.

As for semi-supervised classification, DPT achieves state-of-the-art (SOTA) performance in various settings, including ImageNet with one, two, five labels per class and 1 % labels. On the smaller dataset, namely CIFAR-10, DPT with four labels per class achieves the second-best error rate of 4.68 ± 0.17%. Besides, on ImageNet classification benchmarks with one, two, five labels per class and 1 % labels, DPT outperforms competitive semi-supervised baselines [17, 16], achieving state-of-theart top-1 accuracy of 59.0 (+2.8), 69.5 (+3.0), 74.4 (+2.0) and 80.2 (+0.8) respectively (see Fig. 2 (b)). Similarly to generation tasks, we also investigate why generative models can benefit classifiers via class-level visualization and analysis in Appendix I.

In summary, our novelty and key contributions are as follows:

We present Dual Pseudo Training (DPT), a straightforward yet effective strategy designed to advance the frontiers of semi-supervised diffusion models and classifiers.

We achieve SOTA semi-supervised generation performance on CIFAR-10 and ImageNet datasets across various settings. Moreover, we demonstrate that diffusion models with a few labels (e.g., < 0 . 1% ) can generate realistic, diverse, and semantically accurate images, as depicted in Fig 1.

Weachieve SOTA semi-supervised classification performance on ImageNet datasets across various settings and the second-best results on CIFAR-10. Besides, we demonstrate that aided by diffusion models, generative augmentation remains a viable approach for semi-supervised classification.

We explore why diffusion models and semi-supervised learners benefit mutually with few labels via class-level visualization and analysis, as showcased in Appendix H and Appendix I.

2 Settings and Preliminaries

We present settings and preliminaries on two representative self-supervised based learners for semisupervised learning [17] [16] in Sec. 2.1 and conditional diffusion probabilistic models [2, 5, 26] in Sec. 2.2, respectively. We consider image generation and classification in semi-supervised learning, where the training set consists of N labeled images S = { ( x l i , y l i ) } N i =1 and M unlabeled images D = { x u i } i M =1 . We assume N ≪ M . For convenience, we denote the set of all real images as X = { x u i } i M =1 ∪ { x l i } N i =1 , and the set of all possible classes as Y .

Conditional Generative Model

Classifier

Classifier

Partially Labeled Real Images

All Real Images with Pseudo Labels

Pseudo Labels for All Real Images

Pseudo Images with Uniform Labels

Real Data Augmented by Pseudo Ones

"bird" "panda"

Training

Predicting

"bird" "panda"

1

2

3

Sampling

"bird" "panda"

"bird"

"panda"

2.1 Semi-Supervised Classifier

Masked Siamese Networks (MSN) [17] employ a ViT-based [27] anchor encoder f θ ( · ) and a target encoder f ¯ θ ( · ) , where ¯ θ is the exponential moving average (EMA) [28] of parameters θ . For a real image x i ∈ X , 1 ≤ i ≤ M + N , MSN obtains H + 1 random augmented images, denoted as x i,h , 1 ≤ h ≤ H + 1 . MSN then applies either a random mask or a focal mask to the first H augmented images and obtain mask ( x i,h ) , 1 ≤ h ≤ H . MSN optimizes θ and a learnable matrix of prototypes q by the following objective function:

where CE and H are cross entropy and entropy respectively, p i,h = softmax (( f θ ( mask ( x i,h )) · q /τ ) , ¯ p is the mean of p i,h , p i,H +1 = softmax ( f ¯ θ ( x i,H +1 ) · q /τ ′ ) , τ , τ ′ and λ are hyper-parameters, and · denotes cosine similarity. MSN is an efficient semi-supervised approach by extracting features for all labeled images in S and training a linear classifier on top of the features using L 2 -regularized logistic regression. When a self-supervised pre-trained model is available, MSN demonstrates high efficiency in training a semi-supervised classifier on a single CPU core.

Semi-ViT [16] is three-staged. First, it trains a ViT-based encoder f θ ( · ) on all images in X via self-supervised methods such as MAE [29]. Second, f θ ( · ) is merely fine-tuned on S in a supervised manner. Let ¯ θ be the EMA of θ , and x u,s i and x u,w i denote the strong and weak augmentated versions of x u i respectively. Finally, Semi-ViT optimizes a weighted sum of two cross-entropy losses:

where f ¯ θ ( x ) y is the logit of f ¯ θ ( x ) indexed by y , ˆ y i = arg max y f ¯ θ ( x u,w i ) y is the pseudo-label, vec ( · ) returns the one-hot representation, and τ and µ are hyper-parameters.

2.2 Conditional Diffusion Probabilistic Models

Denoising Diffusion Probabilistic Model (DDPM) [2] gradually adds noise ϵ ∼ N ( 0 , I ) to data x 0 from time t = 0 to t = T in the forward process, and progressively removes noise to recover data starting at x T ∼ N ( 0 , I ) in the reverse process. It trains a predictor ϵ θ to predict the noise ϵ by the following objective:

where c indicates conditions such as classes and texts.

Classifier-Free Guidance (CFG) [26] leverages a conditional noise predictor ϵ θ ( x t , c , t ) and an unconditional noise predictor ϵ θ ( x t , t ) in inference to improve sample quality and enhance semantics. Formally, CFG iterates the following equation starting at x T :

where ˜ ϵ t = (1 + ω ) ϵ θ ( x t , c , t ) -ω ϵ θ ( x t , t ) , ω is the guidance strength, z ∼ N ( 0 , I ) , and α t , β t , ¯ α t and σ t are constants w.r.t. the time t .

U-ViT [5] is a ViT-based backbone for diffusion probabilistic models, which achieves excellent performance in conditional sampling on large-scale datasets.

3 Method

we propose a three-stage strategy called dual pseudo training (DPT) to advance semi-supervised generation and classification tasks, illustrated in Fig. 3 and detailed as follows.

3.1 First Stage: Train Classifier

DPT trains a semi-supervised classifier on partially labeled data S ∪ D , predicts a pseudo-label ˆ y of any image x ∈ X by the classifier, and constructs a dataset consisting of all images with pseudo-labels, i.e. S 1 = { ( x , ˆ y ) | x ∈ X} 3 . Notably, here we treat the classifier as a black box without modifying the training strategy or any hyperparameter. Therefore, any well-trained classifier can be adopted in DPT in a plug-and-play manner. Indeed, we use recent advances in self-supervised based learners for semi-supervised learning, i.e. MSN [17], and Semi-ViT [16]. These two classifiers both provide the generative model with accurate, low-noise labels of high quality.

3.2 Second Stage: Classifier Benefits Generative Model

DPT trains a conditional generative model on all real images with pseudo-labels S 1 , samples K pseudo images for any class label y after training, and constructs a dataset consisting of pseudo images with uniform labels 4 . We denote the dataset as S 2 = ∪ y ∈Y { (ˆ x i,y , y ) } K i =1 , where ˆ x i,y is the i -th pseudo image for class y . Similarly to the classifier, DPT also treats the conditional generative model as a black box. Inspired by the impressive image generation results of diffusion probabilistic models, we take a U-ViT-based [5] denoise diffusion probabilistic model [2] with classifier-free guidance [26] as the conditional generative model. Everything remains the same as the original work (see Sec. 2.2) except that the set of all real images with pseudo-labels S 1 is used for training.

We emphasize that S 1 obtained by the first stage is necessary. In fact, S is of small size (e.g., one label per class) and not sufficient to train conditional diffusion models. Besides, it is unclear how to leverage unlabeled data to train such models. Built upon efficient and strong semi-supervised approaches [17, 16], S 1 provides useful learning signals (with relatively small noise) to train conditional diffusion models. We present quantitative and qualitative empirical evidence in Fig. 2 (a) and Fig. 1 respectively to affirmatively answer the first key question, namely, diffusion models with a few labels (e.g., < 0 . 1% ) can generate realistic and semantically accurate images.

3.3 Third Stage: Generative Model Benefits Classifier

MSN based DPT. We train the classifier employed in the first stage on real data augmented by S 2 to boost classification performance. For simplicity and efficiency, we freeze the models pre-trained by Eq. (1) in the first stage and replace S with S ∪ S 2 to train a linear probe in MSN [17]. DPT substantially boosts the classification performance as presented in Fig. 2 (b).

Semi-ViT based DPT. We freeze the models, which are pre-trained in a self-supervised manner in the first stage of Semi-ViT, and replace S with S ∪ S 2 to train a classifier in the third stage of Semi-ViT. We argue that pseudo images can be used in different stages of Semi-ViT and can both boost the classification performance. (see Appendix F.2).

Both consistent improvements provide a positive answer to the second key question, namely, generative augmentation remains a useful approach for semi-supervised classification. Besides, we can leverage the classifier in the third stage to refine the pseudo-labels and train the generative model with one more stage. Although we observe an improvement empirically (see results in Appendix F.3), we focus on the three-stage strategy in the main paper for simplicity and efficiency.

4 Related Work

Semi-Supervised Classification and Generation. The two tasks are often studied independently. For semi-supervised classification, classical work includes generative approaches based on VAE [10, 30, 31] and GAN [32, 22, 33, 34, 35, 36], and discriminative approaches with confidence regularization [37, 38, 39, 40], consistency regularization [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 13, 14, 51, 52] and other approaches [53, 54, 55, 56]. Recently, large-scale self-supervised based approaches [18, 28, 57, 17, 16] have made remarkable progress in semi-supervised learning. Besides, semi-supervised conditional image generation is challenging because generative modeling is more complex than prediction. In addition, it is highly nontrivial to design proper regularization when the input label is missing. Existing work is based on V AE [10] or GAN [11, 12], which are limited to low-resolution data (i.e., ≤ 128 × 128) and require 10% labels or so to achieve comparable results to supervised baselines.

In comparison, DPT handles both classification and generation tasks in extreme settings with very few labels (e.g., one label per class, < 0 . 1% labels). Built upon recent advances in semi-supervised learners and diffusion models, DPT substantially improves the state-of-the-art results in both tasks.

Pseudo Data and Labels. We mention additional empirical work on generating pseudo data for supervised learning [58], adversarial robust learning [59, 60], contrastive representation learning [61] and zero-shot learning [62, 63]. Regarding theory, in the context of supervised classification, Zheng et al. [64] have mentioned that when the training dataset size is small, generative data augmentation can improve the learning guarantee at a constant level. This finding can be extended to semi-supervised classification, which is left as future work.

Besides, prior work [65, 66, 67, 8] uses cluster index or instance index as pseudo-labels to improve unsupervised generation results, which are not directly comparable to DPT. With additional few labels, DPT can generate images of much higher quality and directly control the semantics of images with class labels.

Diffusion Models. Recently, diffusion probabilistic models [32, 2, 3, 6] achieve remarkable progress in image generation [4, 25, 24, 8, 26, 7], text-to-image generation [68, 69, 70, 25, 71], 3D scene generation [72], image-editing [73, 74, 75], molecular design [76, 77], and semi-supervised medical science [78, 79]. There are learning-free methods [80, 81, 82, 83, 84] and learning-based ones [85, 86] to speed up the sampling process of diffusion models. In particular, we adopt third-order DPMsolver [84], which is a recent learning-free method, for fast sampling. As for the architecture, most diffusion models rely on variants of the U-Net architecture introduced in score-based models [87] while recent work [5] proposes a promising vision transformer for diffusion models, as employed in DPT.

To the best of our knowledge, there has been little research on semi-supervised conditional diffusion models and diffusion-based semi-supervised classification, which are the focus of this paper.

5 Experiment

We present the main experimental settings in Sec. 5.1. For more details, please refer to Appendix C. To evaluate the performance of DPT, we compare it with state-of-the-art conditional diffusion models and semi-supervised learners in Sec. 5.2 and Sec. 5.3 respectively. We also visualize and analyze the interaction between the stages to explain the excellent performance of DPT (see Appendix I, H).

5.1 Experimental Settings

Dataset. We evaluate DPT on the ImageNet [20] dataset, which consists of 1,281,167 training and 50,000 validation images. In the first and third stages, we use the same pre-processing protocol for real images as the baselines [17, 16]. For instance, in MSN, the real data are resized to 256 × 256 and then center-cropped to 224 × 224. In the second stage, real images are center-cropped to the target resolution following [5]. In the third stage, we consider pseudo images at resolution 256 × 256 and center-crop them to 224 × 224. For semi-supervised classification, we consider the challenging settings with one, two, five labels per class and 1% labels. The labeled and unlabeled data split is the same as that of corresponding methods [17, 16]. We also evaluate DPT on CIFAR-10 (see detailed experiments in Appendix A).

Baselines. For semi-supervised classification, we consider state-of-the-art semi-supervised approaches [17, 16] in the setting of low-shot (e.g., one, two, five labels per class and 1 % labels) as baselines. For conditional generation, we consider the state-of-the-art diffusion models with a U-ViT architecture [5] as the baseline.

Model Architectures and Hyperparameters. For a fair comparison, we use the exact same architectures and hyperparameters as the baselines [17, 16, 5]. In particular, for MSN based DPT, we use a ViT B/4 (or a ViT L/7) model [17] for classification and a U-ViT-Large (or a U-ViT-Huge) model [5] for conditional generation. As for Semi-ViT based DPT, we use a ViT-Huge model [16] for classification and a U-ViT-Huge model [5] for conditional generation. More details are provided in Appendix C for reference.

Evaluation metrics. We use the top-1 accuracy on the validation set to evaluate classification performance. For a comprehensive evaluation of generation performance, we first consider the Fréchet inception distance (FID) [21], sFID [88], Inception Score (IS) [22], precision, and recall [89] on 50K generated samples. We calculate all generation metrics based on the implementation of ADM [4]. We also add the metric FIDCLIP, which operates similarly to FID but substitutes the Inception-V3 feature spaces with CLIP features, to eliminate confusion that FID can be artificially reduced by aligning the histograms of Top-N classifications without the actual improvement of image quality [90].

Implementation. DPT is easy to understand and implement. In particular, it only requires several lines of code based on the implementation of the classifier and conditional diffusion model. We provide the pseudocode of DPT in the style of PyTorch in Appendix B.

The choice of K and CFG . We conduct detailed ablation experiments on the number of augmented pseudo images per class (i.e., K ) and the classifier-free guidance scale (i.e., CFG ) in Appendix G and find that the optimal K value is 128 and the optimal CFG values for different ImageNet resolutions are 0 . 8 for 128 × 128 , 0 . 4 for 256 × 256 , and 0 . 7 for 512 × 512 .

The choice of resolution and number of labels. We were primarily driven by the task of ImageNet 256×256 generation to systematically compare with a large family of baselines. In this context, we conducted detailed experiments, including settings with one, two, five labels per class, and 1% labels. We find that the performance of DPT with five labels per class is comparable to the supervised baseline, leading us to use this setting as the default in our other tasks such as ImageNet 128×128 and ImageNet 512×512 generation.

5.2 Image Generation with Few Labels

We show that diffusion models with a few labels can generate realistic and semantically accurate images. In particular, DPT achieves better results than semi-supervised methods on ImageNet 128 × 128 and comparable results to supervised methods on both ImageNet 256 × 256 and 512 × 512 .

We evaluate semi-supervised generation performance of DPT on ImageNet 128 × 128 , as shown in Tab. 1. In particular, DPT with only < 0 . 1% labels outperforms the SOTA semi-supervised generative model S 3 GAN [12] with 20 % labels (FID 4.59 vs. 7.7), suggesting DPT has superior label efficiency.

In Tab. 2, we compare DPT with state-of-the-art generative models on ImageNet 256 × 256 . We construct highly competitive baselines based on diffusion models with U-ViT-Large [5]. According to Tab. 2, our supervised and unsupervised baselines achieve an FID of 3.31 and 27.99, respectively. Leveraging the pseudo-labels predicted by the strong semi-supervised learner [17], DPT with few labels improves the unconditional baseline significantly and is even comparable to the supervised baseline under all metrics. In particular, with only two labels per class, DPT improves the FID of the unsupervised baseline by 24.55 and is comparable to the supervised baseline with a gap of 0.13. Moreover, we also construct more competitive baselines based on U-ViT-Huge to advance DPT. With one (i.e., < 0.1%) label per class , our more powerful DPT achieves an FID of 3.08, outperforming strong supervised diffusion models including IDDPM [23], CDM [24], ADM [4] and LDM [25]. Additionally, with 1% labels, DPT achieves an FID of 2.42, comparable to the

state-of-the-art supervised diffusion model [7]. Lastly, DPT with few labels performs comparably to the fully supervised baseline under the FIDCLIP metric, which suggests that DPT can generate high-quality samples and does not achieve a lower FID solely due to better Top-N alignment.

We also conduct an experiment on higher resolution (i.e., 512 × 512 ) in Tab. 3, with five (i.e., < 0.4%) labels , DPT achieves an FID of 4.05, which is the same as that of the supervised baseline. The above quantitative results demonstrate that DPT can achieve excellent generation performance and label efficiency at diverse resolutions. Qualitatively, as presented in Fig. 1, DPT can generate realistic, diverse, and semantically correct images even with a single label, which agrees with the quantitative results in Tab. 2 and Tab. 3. We provide more samples and failure cases in Appendix F.1 and a detailed class-wise analysis to show how classification helps generation in Appendix H.

Besides, Tab. 5 in Appendix A compares DPT with state-of-the-art generative models on CIFAR-10. DPT achieves competitive performance using only 0 . 08% labels with EDM [6], which relies on full labels (FID 1.81 vs. 1.79). This result demonstrates the generalizability of DPT on different datasets.

5.3 Image Classification with Few Labels

We demonstrate that generative augmentation remains a useful approach for semi-supervised classification aided by diffusion models. In particular, DPT achieves state-of-the-art semi-supervised classification performance on ImageNet datasets across various settings and the second-best results on CIFAR-10.

Tab. 4 compares DPT with state-of-the-art semi-supervised classifiers on the ImageNet validation set with few labels. Specifically, DPT outperforms strong semi-supervised baselines [17, 16] consistently and substantially with one, two, five labels per class and 1 % labels and achieves state-of-the-art top-1 accuracies of 59.0, 69.5, 74.4 and 80.2, respectively. In particular, with two labels per class, DPT leverages the pseudo images generated by the diffusion model and improves MSN with ViT-B/4 by an accuracy of 4.6%. Besides, we compare the performance of DPT with that of SOTA fully supervised models (as shown in Tab. 12 in Appendix F.2) and find that DPT performs comparably to Inception-v4 [95], using only 1 % labels.

Moreover, Tab. 6 in Appendix A compares DPT with state-of-the-art semi-supervised classifiers on CIFAR-10. DPT with four labels per class achieves the second-best error rate of 4.68 ± 0.17%.

6 Conclusions

This paper presents a simple yet effective training strategy called DPT for conditional image generation and classification in semi-supervised learning. Empirically, we demonstrate that DPT can achieve SOTA semi-supervised generation and classification performance on ImageNet datasets across various settings. DPT probably inspires future work in diffusion models and semi-supervised learning.

Limitation. One limitation of DPT is directly using the pseudo images to improve the performance of DPT for its simplicity and effectiveness while we could use pre-trained models like CLIP to filter out noisy image-label pairs that images do not semantically align well with the label. Another limitation pertains to the direct use of pseudo labels. Given our use of classifier-free guidance, we have the flexibility to assign low-confidence pseudo labels to the null token with a high probability, which aids in filtering out noisy pseudo labels.

Social impact. We believe that DPT can benefit real-world applications with few labels (e.g., medical analysis). However, the proposed semi-supervised diffusion models may aggravate social issues such as 'DeepFakes'. The problem can be relieved by automatic detection with machine learning, which is an active research area.

References

J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, 'Deep unsupervised learning using nonequilibrium thermodynamics,' in Proceedings of the 32nd International Conference on Machine Learning , 2015.

J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' in Advances in Neural Information Processing Systems , 2020.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, 'Score-based generative modeling through stochastic differential equations,' in 9th International Conference on Learning Representations , 2021.

P. Dhariwal and A. Nichol, 'Diffusion models beat gans on image synthesis,' Advances in Neural Information Processing Systems , vol. 34, pp. 8780-8794, 2021.

F. Bao, C. Li, Y. Cao, and J. Zhu, 'All are worth words: a vit backbone for score-based diffusion models,' arXiv preprint arXiv:2209.12152 , 2022.

T. Karras, M. Aittala, T. Aila, and S. Laine, 'Elucidating the design space of diffusion-based generative models,' in Proc. NeurIPS , 2022.

W. Peebles and S. Xie, 'Scalable diffusion models with transformers,' arXiv preprint arXiv:2212.09748 , 2022.

F. Bao, C. Li, J. Sun, and J. Zhu, 'Why are conditional generative models better than unconditional ones?' in NeurIPS 2022 Workshop on Score-Based Methods , 2022.

V. T. Hu, D. W. Zhang, Y. M. Asano, G. J. Burghouts, and C. G. Snoek, 'Self-guided diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 18 413-18 422.

D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, 'Semi-supervised learning with deep generative models,' in Advances in Neural Information Processing Systems , 2014.

C. Li, K. Xu, J. Zhu, and B. Zhang, 'Triple generative adversarial nets,' in Advances in Neural Information Processing Systems , 2017, pp. 4088-4098.

M. Luˇ ci´ c, M. Tschannen, M. Ritter, X. Zhai, O. Bachem, and S. Gelly, 'High-fidelity image generation with fewer labels,' in International conference on machine learning . PMLR, 2019, pp. 4183-4192.

K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, 'Fixmatch: Simplifying semi-supervised learning with consistency and confidence,' Advances in neural information processing systems , vol. 33, pp. 596-608, 2020.

B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, 'Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling,' Advances in Neural Information Processing Systems , vol. 34, pp. 18 408-18 419, 2021.

Y. Wang, H. Chen, Q. Heng, W. Hou, M. Savvides, T. Shinozaki, B. Raj, Z. Wu, and J. Wang, 'Freematch: Self-adaptive thresholding for semi-supervised learning,' arXiv preprint arXiv:2205.07246 , 2022.

Z. Cai, A. Ravichandran, P. Favaro, M. Wang, D. Modolo, R. Bhotika, Z. Tu, and S. Soatto, 'Semi-supervised vision transformers at scale,' in NeurIPS , 2022.

M. Assran, M. Caron, I. Misra, P. Bojanowski, F. Bordes, P. Vincent, A. Joulin, M. Rabbat, and N. Ballas, 'Masked siamese networks for label-efficient learning,' in European Conference on Computer Vision . Springer, 2022, pp. 456-473.

T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, 'Big self-supervised models are strong semi-supervised learners,' Advances in neural information processing systems , vol. 33, pp. 22 243-22 255, 2020.

A. Krizhevsky and G. Hinton, 'Learning multiple layers of features from tiny images,' Citeseer , 2009.

J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2009, pp. 248-255.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, 'Gans trained by a two time-scale update rule converge to a local nash equilibrium,' in Advances in Neural Information Processing Systems , 2017, pp. 6626-6637.

T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, 'Improved techniques for training GANs,' in Advances in Neural Information Processing Systems , 2016.

A. Q. Nichol and P. Dhariwal, 'Improved denoising diffusion probabilistic models,' in International Conference on Machine Learning . PMLR, 2021, pp. 8162-8171.

J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, 'Cascaded diffusion models for high fidelity image generation.' J. Mach. Learn. Res. , vol. 23, pp. 47-1, 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 684-10 695.

J. Ho and T. Salimans, 'Classifier-free diffusion guidance,' arXiv preprint arXiv:2207.12598 , 2022.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, 'An image is worth 16x16 words: Transformers for image recognition at scale,' in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al. , 'Bootstrap your own latent-a new approach to self-supervised learning,' Advances in neural information processing systems , vol. 33, pp. 21 271-21 284, 2020.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. B. Girshick, 'Masked autoencoders are scalable vision learners,' in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 15 979-15 988.

L. Maaløe, C. K. Sønderby, S. K. Sønderby, and O. Winther, 'Auxiliary deep generative models,' in International conference on machine learning . PMLR, 2016, pp. 1445-1453.

C. Li, J. Zhu, and B. Zhang, 'Max-margin deep generative models for (semi-) supervised learning,' IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 11, pp. 2762-2775, 2017.

J. T. Springenberg, 'Unsupervised and semi-supervised learning with categorical generative adversarial networks,' in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , Y. Bengio and Y. LeCun, Eds., 2016.

Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, 'Good semi-supervised learning that requires a bad gan,' in Advances in Neural Information Processing Systems , 2017, pp. 6510-6520.

Z. Gan, L. Chen, W. Wang, Y. Pu, Y. Zhang, H. Liu, C. Li, and L. Carin, 'Triangle generative adversarial networks,' in Advances in Neural Information Processing Systems , 2017, pp. 5247-5256.

C. Li, K. Xu, J. Zhu, J. Liu, and B. Zhang, 'Triple generative adversarial networks,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, 'Datasetgan: Efficient labeled data factory with minimal human effort,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 10 145-10 155.

T. Joachims et al. , 'Transductive inference for text classification using support vector machines,' in ICML , vol. 99, 1999, pp. 200-209.

Y. Grandvalet and Y. Bengio, 'Semi-supervised learning by entropy minimization,' in Advances in Neural Information Processing Systems , vol. 17, 01 2004.

D.-H. Lee et al. , 'Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,' in Workshop on challenges in representation learning, ICML , vol. 3, 2013, p. 896.

A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, 'Label propagation for deep semi-supervised learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2019, pp. 5070-5079.

T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, 'Virtual adversarial training: a regularization method for supervised and semi-supervised learning,' IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 8, pp. 1979-1993, 2018.

A. Tarvainen and H. Valpola, 'Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,' in Advances in Neural Information Processing Systems , 2017, pp. 1195-1204.

S. Laine and T. Aila, 'Temporal ensembling for semi-supervised learning,' in International Conference on Learning Representations , 2017.

Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang, 'Smooth neighbors on teacher graphs for semisupervised learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 8896-8905.

B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson, 'There are many consistent explanations of unlabeled data: Why you should average,' in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019.

A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow, 'Realistic evaluation of deep semi-supervised learning algorithms,' in Advances in Neural Information Processing Systems , 2018, pp. 3235-3246.

D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, 'Mixmatch: A holistic approach to semi-supervised learning,' in Advances in Neural Information Processing Systems , 2019, pp. 5049-5059.

D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, 'Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring,' in International Conference on Learning Representations , 2019.

J. Li, C. Xiong, and S. C. Hoi, 'Comatch: Semi-supervised learning with contrastive graph regularization,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9475-9484.

M. Zheng, S. You, L. Huang, F. Wang, C. Qian, and C. Xu, 'Simmatch: Semi-supervised learning with similarity matching,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 14 471-14 481.

M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat, 'Semisupervised learning of visual features by non-parametrically predicting view assignments with support samples,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 8443-8452.

H. Tang, L. Sun, and K. Jia, 'Stochastic consensus: Enhancing semi-supervised learning with consistency of stochastic classifiers,' in European Conference on Computer Vision . Springer, 2022, pp. 330-346.

X. Wang, L. Lian, and S. X. Yu, 'Unsupervised selective labeling for more effective semisupervised learning,' in European Conference on Computer Vision . Springer, 2022, pp. 427-445.

B. Chen, J. Jiang, X. Wang, P. Wan, J. Wang, and M. Long, 'Debiased self-training for semi-supervised learning,' Advances in Neural Information Processing Systems , vol. 35, pp. 32 424-32 437, 2022.

H. Tang and K. Jia, 'Towards discovering the effectiveness of moderately confident samples for semi-supervised learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 14 658-14 667.

J. Lim, D. Um, H. J. Chang, D. U. Jo, and J. Y. Choi, 'Class-attentive diffusion network for semi-supervised classification,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 10, 2021, pp. 8601-8609.

Z. Cai, A. Ravichandran, S. Maji, C. Fowlkes, Z. Tu, and S. Soatto, 'Exponential moving average normalization for self-supervised and semi-supervised learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 194-203.

S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. J. Fleet, 'Synthetic data from diffusion models improves imagenet classification,' arXiv preprint arXiv:2304.08466 , 2023.

S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann, 'Fixing data augmentation to improve adversarial robustness,' arXiv preprint arXiv:2103.01946 , 2021.

Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, and S. Yan, 'Better diffusion models further improve adversarial training,' arXiv preprint arXiv:2302.04638 , 2023.

A. Jahanian, X. Puig, Y. Tian, and P. Isola, 'Generative models as a data source for multiview representation learning,' in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022.

R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi, 'Is synthetic data from generative models ready for image recognition?' arXiv preprint arXiv:2210.07574 , 2022.

V. Besnier, H. Jain, A. Bursuc, M. Cord, and P. Pérez, 'This dataset does not exist: training models from generated images,' in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 1-5.

C. Zheng, G. Wu, and C. Li, 'Toward understanding generative data augmentation,' arXiv preprint arXiv:2305.17476 , 2023.

M. Noroozi, 'Self-labeled conditional gans,' arXiv preprint arXiv:2012.02162 , 2020.

S. Liu, T. Wang, D. Bau, J.-Y. Zhu, and A. Torralba, 'Diverse image generation via selfconditioned gans,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 14 286-14 295.

A. Casanova, M. Careil, J. Verbeek, M. Drozdzal, and A. Romero Soriano, 'Instanceconditioned gan,' Advances in Neural Information Processing Systems , vol. 34, pp. 27 51727 529, 2021.

A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, 'GLIDE: towards photorealistic image generation and editing with text-guided diffusion models,' in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , vol. 162, 2022, pp. 16 784-16 804.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , 2022.

S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, 'Vector quantized diffusion model for text-to-image synthesis,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 696-10 706.

Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro et al. , 'ediffi: Text-to-image diffusion models with an ensemble of expert denoisers,' arXiv preprint arXiv:2211.01324 , 2022.

B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, 'Dreamfusion: Text-to-3d using 2d diffusion,' arXiv preprint arXiv:2209.14988 , 2022.

C. Meng, Y. He, Y. Song, J. Song, J. Wu, J. Zhu, and S. Ermon, 'Sdedit: Guided image synthesis and editing with stochastic differential equations,' in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022.

J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, 'Ilvr: Conditioning method for denoising diffusion probabilistic models,' in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 2021, pp. 14 347-14 356.

M. Zhao, F. Bao, C. Li, and J. Zhu, 'Egsde: Unpaired image-to-image translation via energyguided stochastic differential equations,' Advances in Neural Information Processing Systems , 2022.

E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling, 'Equivariant diffusion for molecule generation in 3d,' in International Conference on Machine Learning . PMLR, 2022, pp. 8867-8887.

F. Bao, M. Zhao, Z. Hao, P. Li, C. Li, and J. Zhu, 'Equivariant energy-guided sde for inverse molecular design,' arXiv preprint arXiv:2209.15408 , 2022.

A. Alshenoudy, B. Sabrowsky-Hirsch, S. Thumfart, M. Giretzlehner, and E. Kobler, 'Semisupervised brain tumor segmentation using diffusion models,' in IFIP International Conference on Artificial Intelligence Applications and Innovations . Springer, 2023, pp. 314-325.

S. Gong, C. Chen, Y. Gong, N. Y. Chan, W. Ma, C. H.-K. Mak, J. Abrigo, and Q. Dou, 'Diffusion model based semi-supervised learning on brain hemorrhage images for efficient midline shift quantification,' in International Conference on Information Processing in Medical Imaging . Springer, 2023, pp. 69-81.

J. Song, C. Meng, and S. Ermon, 'Denoising diffusion implicit models,' in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

F. Bao, C. Li, J. Zhu, and B. Zhang, 'Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models,' in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022.

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, 'Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models,' arXiv preprint arXiv:2211.01095 , 2022.

Q. Zhang and Y. Chen, 'Fast sampling of diffusion models with exponential integrator,' in NeurIPS 2022 Workshop on Score-Based Methods , 2022.

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, 'Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,' arXiv preprint arXiv:2206.00927 , 2022.

F. Bao, C. Li, J. Sun, J. Zhu, and B. Zhang, 'Estimating the optimal covariance with imperfect mean in diffusion probabilistic models,' in Proceedings of the 39th International Conference on Machine Learning , 2022, pp. 1555-1584.

T. Salimans and J. Ho, 'Progressive distillation for fast sampling of diffusion models,' in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022.

Y. Song and S. Ermon, 'Generative modeling by estimating gradients of the data distribution,' Advances in Neural Information Processing Systems , vol. 32, 2019.

C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia, 'Generating images with sparse representations,' in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , vol. 139, 2021, pp. 7958-7968.

T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila, 'Improved precision and recall metric for assessing generative models,' Advances in Neural Information Processing Systems , vol. 32, 2019.

T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, and J. Lehtinen, 'The role of imagenet classes in fr \ 'echet inception distance,' arXiv preprint arXiv:2203.06026 , 2022.

A. Brock, J. Donahue, and K. Simonyan, 'Large scale gan training for high fidelity natural image synthesis,' in International Conference on Learning Representations , 2018.

A. Sauer, K. Schwarz, and A. Geiger, 'Stylegan-xl: Scaling stylegan to large diverse datasets,' in ACM SIGGRAPH 2022 Conference Proceedings , 2022, pp. 1-10.

J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, 'ibot: Image bert pretraining with online tokenizer,' International Conference on Learning Representations (ICLR) , 2022.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, 'Emerging properties in self-supervised vision transformers,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9650-9660.

C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, 'Inception-v4, inception-resnet and the impact of residual connections on learning,' in Proceedings of the AAAI conference on artificial intelligence , vol. 31, no. 1, 2017.

T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, 'Training generative adversarial networks with limited data,' in Advances in Neural Information Processing Systems , 2020.

Y. Chen, X. Tan, B. Zhao, Z. Chen, R. Song, J. Liang, and X. Lu, 'Boosting semi-supervised learning by exploiting all unlabeled data,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 7548-7557.

A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, 'Semi-supervised learning with ladder networks,' in Advances in Neural Information Processing Systems , 2015.

T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, 'Virtual adversarial training: a regularization method for supervised and semi-supervised learning,' IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 8, pp. 1979-1993, 2018.

Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, 'Unsupervised data augmentation for consistency training,' Advances in Neural Information Processing Systems , vol. 33, pp. 6256-6268, 2020.

Y. Xu, L. Shang, J. Ye, Q. Qian, Y.-F. Li, B. Sun, H. Li, and R. Jin, 'Dash: Semi-supervised learning with dynamic thresholding,' in International Conference on Machine Learning . PMLR, 2021, pp. 11 525-11 536.

H. Pham, Z. Dai, Q. Xie, and Q. V. Le, 'Meta pseudo labels,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 11 557-11 568.

J. Mairal, 'Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more,' arXiv preprint arXiv:1912.08165 , 2019.

K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, 'Momentum contrast for unsupervised visual representation learning,' in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9726-9735.

T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, 'A simple framework for contrastive learning of visual representations,' in International Conference on Machine Learning , vol. 119, 2020, pp. 1597-1607.

X. Chen and K. He, 'Exploring simple siamese representation learning,' in IEEE Conference on Computer Vision and Pattern Recognition , 2021, pp. 15 750-15 758.

Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, 'Simmim: A simple framework for masked image modeling,' in International Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

R. A. Fisher, 'On the mathematical foundations of theoretical statistics,' Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character , vol. 222, no. 594-604, pp. 309-368, 1922.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, 'Rethinking the inception architecture for computer vision,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2818-2826.

J. Hu, L. Shen, and G. Sun, 'Squeeze-and-excitation networks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 7132-7141.

M. Tan and Q. Le, 'Efficientnet: Rethinking model scaling for convolutional neural networks,' in International conference on machine learning . PMLR, 2019, pp. 6105-6114.

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, 'Training dataefficient image transformers & distillation through attention,' in International Conference on Machine Learning . PMLR, 2021, pp. 10 347-10 357.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 10 012-10 022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a novel training strategy, dual pseudo training (DPT), that involves three steps: First, DPT uses state-of-the-art semi-supervised learning methods to predict pseudo-labels on a partially-labeled dataset and diffusion models. Second, DPT trains a conditional generative model with pseudo-labeled data for pseudo-images. Finally, the classifier is re-trained with a dataset which includes both real and pseudo images. DPT surpasses existing strong baseline diffusion models in terms of FID score and semi-supervised learning methods on image classification on CIFAR-10 and ImageNet.\n\nSTRENGTHS:\n* The idea of using Semi-Supervised Learning (SSL) to train diffusion models and then improve SSL is novel. This is analogous to distillation methods that sometimes lead to better student performance than the teacher network.\n* The method boosts the performance of both the diffusion model and the discriminative model, becoming the new state-of-the-art method in both fields.\n* The writing is clear. I enjoy reading the paper.\n* The ablations in the appendix are comprehensive and answered several questions that I have.\n\nWEAKNESSES:\n* The baseline diffusion method in this work that leverages full labels is inherently stronger than LDM. Since DPT does not outperform baseline with full supervision, the claim that \"with one or two labels per class, DPT ... surpassing strong diffusion models with full labels, such as IDDPM, CDM, ADM, and LDM\" is misleading to the readers that DPT, with pseudo-labels, leads to better performance than using full labels.\n* Since the authors use a different setting for diffusion method (e.g., with different backbone), the comparison with previous works (e.g., ADM and LDM) is not fair. The authors are encouraged to either use baseline settings from previous methods or migrate their setting to baseline methods to make sure the comparison is fair.\n* The discussions of computational efficiency (e.g., in terms of wall clock time) has been omitted. Diffusion models have been known to be slow in the inference process, especially when applied in the pixel space rather than the latent space. Note that this method is still valuable even if it's not particularly compute efficient, as long as the compute required is reasonable, as many semi-supervised learning methods are also less efficient compared to supervised learning.\n\nQUESTIONS:\nQuestion:\n* The method seems to be sensitive to CFG. If I understand it correctly, a CFG of 0 means no guidance. While methods such as stable diffusion typically use a large CFG for an optimal trade-off between diversity and fidelity (7.5 by default, which is equivalent to 6.5 in the notation of this paper), the method uses CFG less than 1. Why is this large discrepancy?\n\nThe authors are welcome to address the questions in the weakness section.\n\nLIMITATIONS:\nAs mentioned in the weakness section, the discussions of computational efficiency (e.g., in terms of wall clock time) has been omitted. The authors are encouraged to discuss their compute usage and wall clock time as well as the one for their baseline.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo advance semi-supervised generative and classification tasks, this paper proposes a simple yet effective training strategy termed dual pseudo training (DPT). The proposed DPT operates in three stages: training a strong semi-supervised learner to predict pseudo labels, training a strong diffusion model using both real and pseudo labels to generate pseudo images, and retraining the semi-supervised learner with a mix of real and pseudo images. Experimental results show the effectiveness of the proposed DPT on semi-supervised generation and classification across various settings.\n\nSTRENGTHS:\n1. The paper is well-written.\n2. The proposed DPT bridging strong semi-supervised learners and diffusion models by pseudo labels and images is technically sound.\n3. The idea that diffusion models and semi-supervised learners benefit mutually with few labels is interesting and of practical use.\n4. It is impressive that the proposed DPT using 1% labels yields comparable results to the supervised baseline using 100% labels (cf. Table 2).\n\nWEAKNESSES:\n1. Does this paper have any technical novelty? It should be highlighted in the abstract and introduction parts.\n\n2. The proposed DPT improves semi-supervised learning models with generative augmentation via diffusion models. Despite substantial performance improvements, the computing overhead and time budget may be big. It would be better to provide a comparative analysis between the proposed method and existing ones.\n\n3. At the second stage of the proposed DPT, pseudo labels may not always align with ground truth labels, making it hard to control semantics in samples. How does the pseudo label accuracy affect the pseudo image fidelity and semantics? Some illustrative examples are preferred.\n\n4. Would the classifier retrained with a mix of real and pseudo images be used to predict pseudo labels as the first stage? The three stages may form a positive cycle to iteratively improve the pseudo label accuracy and pseudo image semantics.\n\n5. Lines 57-58: The exploration on why classifiers can benefit generative models through class-level visualization and analysis is important for the method validation. It would be better to place it in the main text.\n\n6. In Table 1, the semi-supervised generative model S^3GAN was proposed in 2019. Haven't there been any new methods in the last few years?\n\n7. In tables of image generation results, different methods may differ in the model size, leading to unfair comparison. It would be better to show the number of model parameters in these tables.\n\n8. Many recent related works on semi-supervised learning have not been included, discussed, and compared in this paper, e.g., [a-h].\n\n[a] Zhang et al. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurIPS, 2021.\n\n[b] Tang et al. Stochastic Consensus: Enhancing Semi-Supervised Learning with Consistency of Stochastic Classifiers. ECCV, 2022.\n\n[c] Wang et al. Unsupervised Selective Labeling for More Effective Semi-Supervised Learning. ECCV, 2022.\n\n[d] Tang et al. Towards Discovering the Effectiveness of Moderately Confident Samples for Semi-Supervised Learning. CVPR, 2022.\n\n[e] Lim et al. Class-Attentive Diffusion Network for Semi-Supervised Classification. AAAI, 2021.\n\n[f] Gong, S. et al. (2023). Diffusion Model Based Semi-supervised Learning on Brain Hemorrhage Images for Efficient Midline Shift Quantification. In: Frangi, A., de Bruijne, M., Wassermann, D., Navab, N. (eds) Information Processing in Medical Imaging. IPMI 2023. Lecture Notes in Computer Science, vol 13939. \n\n[g] Alshenoudy, A., Sabrowsky-Hirsch, B., Thumfart, S., Giretzlehner, M., Kobler, E. (2023). Semi-supervised Brain Tumor Segmentation Using Diffusion Models. In: Maglogiannis, I., Iliadis, L., MacIntyre, J., Dominguez, M. (eds) Artificial Intelligence Applications and Innovations. AIAI 2023. IFIP Advances in Information and Communication Technology, vol 675.\n\n[h] Chen et al. Debiased Self-Training for Semi-Supervised Learning. NeurIPS, 2022.\n\nQUESTIONS:\n1. Does the \"retraining\" in the 3-rd stage mean fine-tuning or training from scratch?\n\n2. Why do the strong diffusion models with full labels, such as IDDPM, CDM, ADM, and LDM, perform worse than the proposed DPL with few labels? \n\nHow would the proposed DPT perform if full labels are used? It provides the uppper bound.\n\n3. On Line 54, DiT-XL/2-G [7] (FID 2.27) is the SOTA supervised baseline, rather than [5] (FID 3.40). Please carefully check and correct.\n\n5. In Fig. 2 (a), it is no need to indicate the label fraction by the bubble area, as the x-axis has already indicated the label fraction. Different shapes or colors are enough to show the advantage of the proposed DPT.\n\n6. In Table 2, the \"bold\" and \"underline\" look a little messy. Some columns of evaluation metrics have no mark while others have many. A better marking scheme should be figured out to make the table easier to understand.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an interesting interaction between a generative model and a semi-classifier. The pipeline can boost both the generative sample quality and discriminative performance of the semi-classifier in the few label settings. A three-stage pipeline is designed. First, the semi-classifier is trained on a few label data and the semi-label is acquired for all the unlabeled data. Second, the generative model is trained on both label and semi-label data to achieve comparable sample quality as a full-supervised one. At last, the semi-classifier can further improve its performance from the new samples generated by the learned generative model. DPT with two (i.e., < 0.2%) labels per class is comparable to SOTA supervised baseline (FID 2.52 vs. 2.29).\n\nSTRENGTHS:\n- Well written and easy to follow.\n- Motivation is clear and the pipeline is simple.\n- Good results. DPT with two (i.e., < 0.2%) labels per class is comparable to SOTA supervised baseline (FID 2.52 vs. 2.29). \n- Interesting interaction between semi-classifier and diffusion model.\n\nWEAKNESSES:\n- 61-63 Specifically, DPT with four labels per class achieves an error rate of 4.53% on 62 CIFAR-10, which even outperforms the performance of SOTA methods on CIFAR-10 63 with 25 labels per class. The Full-flex[1] algorithm has better performance than 4.53% for 4 labels per class on CIFAR-10, which might make this claim incorrect.\n- It's interesting to see if the framework is general to other semi-algorithm and diffusion model pairs.\n- lack of analysis of the key components of the interaction between the generative model and semi-classifier. For example, what is the key factor to make semi-classifier success for generative model in the design? And What is the key factor to make genertive model success for semi-classifier in the design? This should be discussed in the paper.\n\n\n[1] Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data\n\nQUESTIONS:\n- Is the algorithm stable for different starting samples in the first stage?\n-  what are the key components for the successful interaction between the generative model and semi-classifier?\n- How about the PSNR or other commonly used generation evaluation metrics? Since FID and IS are highly imagenet related.\n- Can it improve the upper bound of generation quality? If combining full supervised label data with this pipeline, will it surpass the 2.29 FID? \n- What about the improvement of unconditional generation?\n- The weaknesses mentioned above should be addressed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes to exploit generative diffusion models to benefit semi-supervised learning and vice versa.

Specifically, the proposed training strategy called dual pseudo training (DPT) consists of 3 stages:

(1) Train a semi-supervised learning (SSL) model on labeled and unlabeled data

(2) Train a conditional generative diffusion model using pseudo labels predicted by the SSL model in (1)

(3) Use the generative model in (2) to generate more labeled images to train a classifier

DPT achieves both SOTA performances on image generation and semi-supervised classification tasks on ImageNet and CIFAR-10, demonstrating "Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels".

STRENGTHS:
(1) The paper is well-written and easy to follow, 

(2) The proposed method is simple but effective. And it can leverage any SSL algorithms or generative models in a plug-and-play manner.

(3) The results are very impressive, both in numbers (SOTA in both classification and generation tasks) and samples (Figure 1).

WEAKNESSES:
(1) Naive combination of SSL and generative models is a straightforward idea; the novelty is limited.

QUESTIONS:
I do not have questions for this paper.

LIMITATIONS:
As stated by the author, filtering of pseudo labels in stage 1 and pseudo images in stage 2 should be investigated to reduce noise.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper empirically studies the effectiveness of combining diffusion models and semi-supervised classification. First, an image encoder is trained using a self-supervised method, then finetuned with a small fraction of the ground-truth labels. Second, the image classifier assigns pseudo-labels to all unlabeled images in the training dataset which are then used to train a label-conditioned diffusion model. Finally, the diffusion model is used to generate image-label pairs that are used to further train the image classifier from the first step. Multiple experiments on CIFAR10 and ImageNet demonstrate the effectiveness of this approach, both in improving the diffusion model and the image classifier.\n\nSTRENGTHS:\n- The paper demonstrates state-of-the-art results on both semi-supervised image classification and semi-supervised image generation\n- The presentation of both the model and the experiments is easy to follow and the text is well written\n- Exploring how diffusion models and discriminative models like image classifiers can mutually benefit from one another is interesting and relevant to both academics and practitioners\n\nWEAKNESSES:\n1. As shown in [1], the FID can be decreased by aligning the histograms of Top-N classifications without improvements to the image quality. Albeit there exists no perfect method to quantify this effect, it would be insightful to see some investigation into whether the lower FID actually translates to improved image quality or merely better alignment in terms of Top-N classifications.\n\n2. In the introduction, the paper claims that (unsupervised conditional) diffusion models trained with cluster assignments as pseudo-labels still underperform their ground-truth supervised counterparts. As shown in [2] this no longer holds true when the cluster number is set higher than the number of ground-truth classes. It would be interesting to explore how this insight could further benefit the semi-supervised setting, for example by further sub-clustering images with the same pseudo-labels. I consider the paper should be published if they fix their claim about unsupervised diffusion models.\n\n\n[1] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, J. Lehtinen, \"The Role of ImageNet Classes in Fréchet Inception Distance\", ICLR 2023.\n[2] H. Tao, D. W. Zhang, Y. Asano, G. J. Burghouts, C. G. M. Snoek, \"Self-Guided Diffusion Models\", CVPR 2023.\n\nQUESTIONS:\nThe paper reports image generation results on ImageNet for multiple resolutions and differing numbers of ground-truth labels per class. What is the motivation behind the specific choices for the resolution and number of labels combinations? Does the choice of the resolution for the generative model affect the classification results?\n\nLIMITATIONS:\nA potential negative impact of this line of work (semi-supervised generation/classification) is that it could amplify stereotypes where a few examples are taken as representative of the whole class (group). Some discussion on this would benefit the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}