Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels
Authors: Zebin You
Yong Zhong
Fan Bao
Jiacheng Sun
Chongxuan Li
Jun Zhu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: diffusion models
semi-supervised generation
semi-supervised diffusion models
semi-supervised classification
image generation.
TL;DR: Diffusion models and semi-supervised learners can benefit mutually to achieve SOTA semi-supervised classification and generation results on ImageNet-1K and CIFAR-10 with few labels.

Abstract: In an effort to further advance semi-supervised generative and classification tasks, we propose a simple yet effective training strategy called *dual pseudo training* (DPT), built upon strong semi-supervised learners and diffusion models. DPT operates in three stages: training a classifier on partially labeled data to predict pseudo-labels; training a conditional generative model using these pseudo-labels to generate pseudo images; and retraining the classifier with a mix of real and pseudo images.  Empirically, DPT consistently achieves SOTA performance of semi-supervised generation and classification across various settings. In particular, with one or two labels per class, DPT achieves a Fréchet Inception Distance (FID) score of 3.08 or 2.52 on ImageNet $256\times256$. Besides, DPT outperforms competitive semi-supervised baselines substantially on  ImageNet classification tasks, *achieving top-1 accuracies of 59.0 (+2.8), 69.5 (+3.0), and 74.4 (+2.0)* with one, two, or five labels per class, respectively. Notably, our results demonstrate that diffusion can generate realistic images with only a few labels (e.g., $<0.1$%) and generative augmentation remains viable for semi-supervised classification. Our code is available at *https://github.com/ML-GSAI/DPT*.

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China 2 Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China 3 Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch MI. Center, Tsinghua University Huawei Noah's Ark Lab edur.uuceeddee... sunjiachengl @huawei. com; chongxuanli@nn frruc. edu, en; twwngnua...eeeee edu. cn; edu. en Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels
Zebin Y:u Yong Zhong 1.2 Fan Bao Jiacheng Sun', Chongxuan Li Jun Zhu
zebin@ruc edu. en; yongzhongoruc. edu. en; bf bf19@mais. hes0mmmlss.ssinnhuu..eeeee
Abstract
In an effort further advance supi-super generative and classification tasks, we propose a simple yet effective training strategy called dual pseudo training (DPT), built upon strong semi-supervised learners and diffusion models. DPT operates in three stages: training a classifier on partially labeled data to predict pseudo-labels; training conditional generative model using these pseudo-labels to generate pseudo images; and retraining the classifier with mix of real and pseudo images. Empirically, DPT consistently achieves SOTA performance of semi-supervised generation and classification across various settings. In particular, with one or two labels per class, DPT achieves Fréchet Inception Distance (FID) score of 3.08 or 2.52 on ImageNet 256 x 256. Besides, DPT outperforms competitive semi-supervised baselines substantially on ImageNet classification tasks, achieving top-1 accuracies of 59.0 (+2.8), 69.5 (+3.0), and 74.4 (+2.0) with one, two, or five labels per class, respectively. Notably, our results demonstrate that diffusion can generate realistic images with only a few labels (e.g.. < 0.1%) and generative augmentation remains viable for semi-supervised classification. Our code is available at httt:::/ggghub.comMMGGG..
Introduction
have achieved excellent performance in image generation. However, empirical evidence has shown that labeled data is indispensable for training such models [8] 4. Indeed, lacking labeled data leads to much lower performance of the generative model. For instance, the representative work (i.e., ADM) [4] achieves an FID of 10.94 on fully labeled ImageNet 256 x 256, while an FID of 26.21 without labels.
Diffusion probabilistic models 2
4 5
6 7
To improve the performance of diffusion models without utilizing labeled data, prior work g initially conducts clustering and subsequently trains diffusion models conditioned on the cluster indices. Although these methods can, some instances, exhibit superior performance over supervised models on low-resolution data, such phenomena have not yet been observed on high-resolution data (e.g., on ImageNet 256 x 256, an FID of 5.19, compared to an FID of 3.31 achieved by supervised models, see Appendix C Besides, cluster indices may not always align with ground truth labels, making hard to control semantics in samples. Compared to unsupervised methods, cemi-supervised generative models [10 [2] often perform much better and provide the same way to control the
"qqual contribution.
Corresponneece to Chongxuan Li.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Selected samples from DPT. Top row: 512 512 samples from DPT trained with five (< 0.4%) labels per class. Bottom rows: 256 x 256 samples from DPT trained with one (< 0.1% label per class (Left: "Ostrich"; Mid: "King penguin"; Right: "Indigo bunting").
Figure I:
semantics of samples as the supervised ones by using a small number of labels. However, to our knowledge, although it is attractive, little work in the literature has investigated semi-supervised diffusion models. This leads us to a key question: can diffusion models generate high-fidelity images with controllable semanties given only few (e.g., 0.1%) labels?
On the other hand, while natural to use images sampled from generative models for semi- supervised classification [10] [T] discriminative methods [13] 14 [5] dominant the area recently. In particular, self-supervised based learners [16 17 [8 have demonstrated state-of-the-art performance on ImageNet. However, generative models have rarely been considered for superviee classi- fication recently. Therefore another key question arises: can generative augmentation a useful approach for such strong semi-supervised classifiers, with the aid of advanced diffusion models?
To answer the above two key and pressing questions, we propose simple but effective training strategy called dual pseudo training (DPT), built upon strong diffusion models and semi-supervised classifiers. DPT is three-staged (see Fig. 3) First, classifier trained on partially labeled data and used to predict pseudo-labed labels for all data. Second, conditional generative model is trained on all data with pseudo-labels and used to generate pseudo images given labels. Finally, the classifier is trained on real data augmented by pseudo images with labels. Intuitively, in DPT, the two opposite conditional models (i.e. diffusion model and classifier) provide complementary learning signals to each other and benefit mutually (see detailed discussion in Appendix E).
We evaluate the effectiveness of DPT through diverse experiments on multi-scale and multi-resolution benchmarks, including CIFAR- 10 [19] and ImageNet (20] at resolutions of 128 128, 256 x 256, and 512 512. Quantitatively, DPT obtains SOTA supesvuper generation results on two common metrics, including FID 21 and IS 222 in all settings. In particular, in the highly appealing task, i.e. ImageNet 256 256 generation, DPT with one (i.e., < 0. 0.1%) labels per class achieves an FID 3.08. outperforming strong supervised diffusion models including IDDPM 23] CDM (24). ADM 4 and LDM 255 (see Fig. (a)). It worth noting that the comparison with previous models here is meant to illustrate that DPT maintains good performance even with minimal labels, rather than directly comparing these previous models (direct comparison is unfair different diffusion models were used). Furthermore, DPT with two (i.c., 0.2%) labels per class is comparable to supervised baseline

---

76
SOTA Sanié Sageniaall
74
DPT (ous)
FID S0K ImugeNe 256x256
Top- Accunry ImageNer
CDM (100%)
(100%)
50TA
loun, (our, s0.25%
LAD (10M5))
64
D3PT
DPT (ous, 196) Supariial 50C,iis,
S0TA
+0.1%
<0.2%
<0.4% (3)
Label Frnctio
Label Fraction Labols per Class)
(a) DPT supervised diffusion models.
(h) DPT semi-supervised classifiers.
Figure 2: Generation and classification results of DPT on ImageNet with few labels. (a) DPT
with < 0.1 % labels outperforms strong supervised diffusion models 4 24 25 (b) DPT substantially improves SOTA semi-supervised learners 17
U-ViT [5] (FID 2.52 vs. 2.29). Moreover, ImageNet 128 128 generation, DPT with one (i.e., < 0.1 %) labels per class outperforms SOTA semi-supervised generative models SAGAN [12] with 20% labels (FID 4.59 vs. 7.7). Qualitatively, DPT can generate realistic, diverse, and semantically correct images with very few labels, as shown in Fig I We also explore why classifiers can benefit generative models through class-level visualization and analysis in Appendix H
As for semi-supervin classification, DPT achieves state-of-the-art (SOTA) performance in various settings, including ImageNet with one, two, five labels per class and 1% labels. On the smaller dataset, namely CIFAR-10, DPT with four labels per class achieves the second-best error rate of 4..8....7%. Besides, TeageNet classification benchmarks with one, two, five labels per class and % labels, DPT outperforms competitive semi-supervised baselines [17 achieving state-of-the- art top-1 accuracy of 59.0 (+2.8), 69.5 (+3.0), 74.4 (+2.0) and 80.2 (+0.8) respectively (see Fig. (b)). Similarly to generation tasks, we also investigate why generative models can benefit classifiers via class-level visualization and analysis in Appendix
In summary, our novelty and key contributions are as follows:
We present Dual Pseudo Training (DPT), straightforward effective strategy designed to advance the frontiers of semi-supervised diffusion models and classifiers.
We achieve SOTA semi-supervised generation performance on CIFAR-10 and ImageNet datasets across various settings. Moreover, we demonstrate that diffusion models with few labels (e.g.. 0. 0.1%) can generate realistic, diverse, and semantically accurate images, as depicted in Fig
We achieve SOTA semi-supervised classification performance ImageNet datasets across various settings and the second-best results on CIFAR-10. 10. Besides, we demonstrate that aided by diffusion models, generative augmentation remains viable approach for cemi-supervised classification.
We explore why diffusion models and semi-supervised learners benefit mutually with few labels via class-level visualization and analysis, as showcased in Appendix H and Appendix [
2 Settings and Preliminaries
We present settings and preliminaries on two representative self-supervised based learners for semi-
supervised learning [17 [66] in Sec. [.. and conditional diffusion probabilistic models 2 5 26 in Sec. [.2] respectively. We consider image generation and classification in semi-supervised learning. where the training consists of N labeled images S {(x{, N and M unlabeled images
D {x" 1==1- We assume N < M. For convenience, denote the set of all real images as
X' [x h [x and the set of all possible classes as V

---

Partially Labeled Real Images
Real Images with Pseudo L. abels
Real Data Augmented by Pseudo Ones
"panda"
"bird"
"panda"
"bird"
"panda"
Classifier
Conditional G Generative Model
C Chasifier
Training
"panda"
Predicting
Sampling
Pseudo L Labels for All Real Images
Pseudo Images with Uniform Labels
An overview of DPT. First, a (semi- sueer-supere classifier is trained on partially labeled data and used to predict pseudo-labels for all data. Second, conditional generative model is trained on all data with pscudo-labeh, and used to generate pscudo images given random labels. Finally, the classifier is trained fine-tuned real data augmented by pseudo images with labels.
Figure 3:
2.1 Sepi-Supervised Classifier
Masked Siamese Networks (MSN) 17 employ ViT-based 27 anchor encoder fo(.) and target encoder fa(.), where is the exponential moving average (EMA) (8 of parameters For a real image x; E X, < i M N, MSN obtains H random augmented images, denoted as Xi,h; 1 < < H 1. MSN then applies either random mask or a focal mask to the first H augmented images and obtain mask (x,, 1, S
H. MSN optimizes and learnable matrix of prototypes q by the following objective function:
M+N H Σ
Σ CE(PLA AH(P), h=1
(I)
H(M N)
where CE and H are cross entropy and entropy respectively, P4, softmax ( (masmma q/T) P is the mean of Pi Pi, softmax (fxxmax(ff(xx,x++1)) - and À are hyper-parameters, and denotes cosine similarity. MSN an efficient supi-vupee approach by extracting features for all labeled images in S and training linear classifier top of the features using Lg regularizze logistic regression. When a self-supervised pre-trained model is available, MSN demonstrates high efficiency in training semi-supervised classifier on single CPU core.
Simi-ViT 16 is three-staged. First, it trains ViT-based encoder fo(.) all images in X' via self-supervised methods such as MAE 22 Second, f-(() is merely fine-tuned on S a supervised manner. Let be the EMA 0, and x and x 1... denote the strong and weak augmentated versions of x respectively. Finally, Semi-Vii optimizes weighted sum of two cross-entropy losses:
L Li pLn.
CE(Jo(x)), vec(u))) +
1=1
(2)
[[ )r r|CE([o(x"))) ('))), vec((:)),
M
where fx(x)y is the logit of (x) indexed by y. û. arg max, So (x!" is the pseudo-label, vec(-) returns the one-hot representation, and and are hyper-parameters.

---

2.2 Conditional Diffusion Probabilistic Models
Denoising Diffusion Probabilistic Model (DDPM) [2] gradually adds noise e ~~~ (0, 1) to data Xo from time / = 0 to L = T in the forward process, and progressively removes noise to recover data starting at x ~ M/(0, 1) in the reverse process. It trains predictor ep to predict the noise e by the following objective:
L= Exl c,t) -c1/2,
where с indicates conditions such as classes and texts.
Classifier-Free Guidance (CFG) [26] leverages conditional noise predictor co(X,, c, t) and an unconditional noise predictor cø (x, t) in inference to improve sample quality and enhance semantics. Formally, CFG iterates the following equation starting at X7:
8
X1-1
Xx
(1) + +,2z,
Vas
Cz
where é = (1 + (X,, wee(x; t), is the guidance strength, z ~ I), and a, Sc. āg and dy are constants w.r.t the time
U-VIT [5] is a ViT-based backbone for diffusion probabilistic models, which achieves excellent performance in conditional sampling large-scale datasets.
Method
we propose a three-stage strategy called dual pseudo training (DPT) to advance semi-supervised generation and classification tasks, illustrated in Fig. and detailed as follows.
3.1 First Stage: Train Classifier
DPT trains a semi sumi-supervised classifier on partially labeled data S U D. predicts a pscudo-label ý of any image pseudo-labels, i.e. S = {xx 7) x E x) Notably, here we treat the classifier as a black box without modifying the training strategy any hyperparameter. Therefore, any well-trained classifier adopted DPT in plug-and- aud--and-lay manner. Indeed, we use recent advances in self-supervised based learners for semi-supervised learning, i.e. MSN (17], and Semi-ViT (16). These two classifiers both provide the generative model with accurate, low-noise labels of high quality.
X by the classifier, and constructs dataset consisting of all images with
be
3.2 Second Stage: Classifier Benefits Generative Model
DPT trains a conditional generative model on all real images with pseudo-labels St, samples K pseudo images for any class label y after training, and constructs dataset consisting of pseudo images with uniform label:" We denote the dataset as Sy i-th pseudo image for class y. Similarly to the classifier, DPT also treats the conditional generative model as a black box. Inspired by the impressive image generation results of diffusion probabilistic
( ) where &, is the
models, we take U-ViT-based [5] denoise diffusion probabilistic model 2 with classifier-fres guidance (26] as the conditional generative model. Everything remains the same as the original work (see Sec. 2.2 except that the set of all real images with pseudo-labels S₁ is used for training.
We emphasize that S₁ obtained by the first stage is necessary. In fact, S of small size (e.g., one label class) and not sufficient to train conditional diffusion models. Besides, it is unclear how to leverage unlabeled data to train such models. Built upon efficient and strong semi-supervised approaches [17 16), Sj provides useful learning signals (with relatively small noise) to train conditional diffusion models. We present quantitative and qualitative empirical evidence in Fig 2 (a) and Fig. | respectivvll to affirmatively answer the first key question, namely, diffusion models with few labels (c.g..
< 0.1%) can generate realistic and semantically accurate images.
} For simplicity. we also use pseudo-labely instead of the ground truth real labeled data, which are rare have a small zero-one training loss, making no significant difference. 'The prior distribution of y be estimated on S.
5

---

3.3 Third Stage: Generative Model Benefits Classifier
MSN based DPT. We train the classifier employed in the first stage real data augmented by S to boost classification performance. For simplicity and efficiency, we freeze the models pre-trained by Eq. 4 in the first stage and replace S with S U S2 to train linear probe in MSN [17]. DPT substantially boosts the classification performance as presented in Fig. (b).
ViT based DPT. We freeze the models, which are pre-trained in a self-supervised manner in the first stage of Semi-ViT, and replace S with S U U2 to train a classifier in the third stage of Semi-ViT. We argue that pseudo images can be used in different stages of Semi-ViT and can both boost the classification performance. (see Appendix F.2).
Both consistent improvements provide a positive answer to the second key question, namely, gener- ative augmentation remains useful approach for semi-supervised classification. Besides, we can leverage the classifier in the third stage to refine the pseudo-labels and train the generative model with one more stage. Although we observe an improvement empirically (see results in Appendix F3 we focus on the three-stage strategy in the main paper for simplicity and efficiency.
4 Related Work
Semi-Supervised Classification and Generation. The two tasks are often studied indepen- dently. For supervise classification, classical work includes generative approaches based and GAN 32
on VAE 10 30
31
22
36), and discriminative approaches with confi- consistency regularization [1 42
33
35|
dence regularization 337
38,
39
40
43
44
45
46
47 48 49]
50
13 approaches [8
56). Recently, large- saf--superviee based have made remarkable progress in semi-supervised learning. Besides, semi-supervised conditional image generation challenging because generative modeling is more complex than prediction. In addition, it is highly nontrivial to design proper regularization when the input label is missing. Existing work is based on VAE [10] or GAN [2]. which are limited to low-rcsolution data (i.e., 128 x 128) and require 10% labels or to achieve comparable results to supervised baselines.
14
51
52 and other approaches 53
54
55
28
57
17
16
In comparison, DPT handles both classification and generation tasks in extreme settings with very few labels (e.g., one label per class, < 0.1% labels). Built upon recent advances in semi-supervised learners and diffusion models, DPT substantially improves the state-of-the art results in both tasks.
Pseudo Data and Labels. We mention additional empirical work on generating pseudo data for supervised learning (88] adversarial robust learning [59 60), contrastive representation learning [6]] and zero-shot learning 662 63 Regarding theory, in the context of supervised classification, Zheng et al. [64] have mentioned that when the training dataset size is small, generative data augmentation can improve the learning guarantee at constant level. This finding can be extended to cupi-supervieed classification, which is left as future work.
uses cluster index or instance index pseudo-labels improve unsupervised generation results, which are not directly comparable to DPT. With additional few labels, DPT can generate images of much higher quality and directly control the semantics of images with class labels.
Besides, prior work 66 66 67 g
Diffusion Models. Recently, diffusion probabilistic models 32
2 G achieve remarkable progress
in image generation 4 255
24
8 26
71. text-to-image generation (68 69 70 25 71]. 3D scene
generation 772. image-editing 77
74 75 I molecular design 76. 77). and semi- supervispe medical
science 77 79]. There are learning- free methods 880 8
82
83
84 and learning-basse
ones [85] 86)
to speed up the sampling process of diffusion models. In particular, we adopt third-order DPM- solver 84], which is a recent learning- free method, for fast sampling. As for the architecture, most diffusion models rely variants of the U-Net architecture introduced in score-based models [87] while recent work [5] proposes a promising vision transformer for diffusion models, as employed in DPT.
To the best of our knowledge, there has been little research on semi-supervised conditional diffusion models and diffusion- sumi-super classification, which are the focus of this paper.

---

5
Experiment
We present the main experimental settings in Sec. 5.1 For more details, please refer Appendix C To evaluate the performance of DPT, we compare it with state-of-the-ar conditional diffusion models and semi-supervised learners in Sec. 5.2 and Sec. Sec.5..3 respectively. We also visualize and analyze the interaction between the stages to explain the excellent performance of DPT (see Appendix I H).
5.1 Experimental Settings
Dataset. We evaluate DPT on the ImageNet 200 dataset, which consists of 1, 1,281,167 training and 50,000 validation images. In the first and third stages, we use the same pre-processing protocol for real images as the baselines [17 [16]. For instance, in MSN, the real data are resized to 256 x 256 and then center-cropped 224 224. In the second stage, real images are center-cropped to the target resolution following [5]. In the third stage, we consider pseudo images at resolution 256 x 256 and center-crop them to 224 x 224. For semi- semi-supervised classification, we consider the challenging settings with one, two, five labels per class and 1% labels. The labeled and unlabeled data split is the same as that of corresponding methods [17 [6]. We also evaluate DPT on CIFAR-10 10 (see detailed experiments in Appendix A).
Baselines. For semi-supervised classification, we consider state-of-the-art semi-supervised ap- proaches [17 [6) in the setting of low-shot (e.g., one, two, five labels per class and 1% labels) as baselines. For conditional generation, we consider the state-of-the-art diffusion models with a U-ViT architecture (5] as the baseline.
Model Architectures and Hyperparameters. For fair comparison, architectures and hyperparameters as the baselines [17 16 5 In particular, for MSN based DPT, use VIT B/4 (or VIT L/7) model (17] for classification and U-ViT-Large (or U-ViT-Huge) model [5) for conditional generation. As for Semi-ViT based DPT, we use a ViT-Huge model [16] for classification and U-ViT-Huge model [5] for conditional generation. More details are provided
the exact same
in Appendix for reference.
Evaluation metrics. We use the top-1 accuracy the validation set to evaluate classification performance. For a comprchensive evaluation of generation performance, we first consider the Fréchet inception distance (FID) [21], sFID [88], Inception Score (IS) (22,, precision, and recall [89] on 50K generated samples. We calculate all generation metrics based on the implementation of
ADM 4.. We also add the metric FIDCLIP, which operates similarly FID but substitutes the Inception V3 feature spaces with CLIP features, to eliminate confusion that FID can be artificially
reduced by aligning the histograms of Top-N classifications without the actual improvement of image quality (90)
Implementation. DPT is easy understand and implement. In particular, it only requires several lines of code based on the implementation of the classifier and conditional diffusion model. We provide the pseudocode of DPT in the style of PyTorch in Appendix B
The choice of and CFG. We conduct detailed ablation experiments on the number of augmented pseudo images per class (i.e., K) and the classifier-free guidance scale (i.e., CFG) in Appendix G and find that the optimal K value 128 and the optimal CFG values for different ImageNet resolutions are 0.8 for 128 128, 0.4 for 256 x 256, and 0.7 for 512 x 512.
The choice of resolution and number of labels. We were primarily driven by the task of ImageNet 256x256 generation to systematically compare with large family of baselines. In this context, conducted detailed experiments, including settings with one, two, five labels per class, and 1 1% labels. We find that the performance of DPT with five labels per class is comparable to the supervised baseline, leading us use this setting as the default in our other tasks such as ImageNet 128x128 and ImageNet 512x512 generation.
5.2 Image Generation with Few Labels
We show that diffusion models with few labels can generate realistic and semantically accurate images. In particular, DPT achieves better results than semi-supervised methods on ImageNet 128 x 128 and comparable results to supervised methods on both ImageNet 256 256 and 512 x 512.
7

---

Table 1: Image generation results on ImageNet 128 x 128. i labels the results taken from the corresponding references and labels baseline achieved by us. We bold the best result under the corresponding setting. With < 0.1% labels, DPT outperforms strong semi-supervised generative models GAN [2].
Method
Model
Label fraction
FID-50K
IS
(# labels/class)
U-WI--Huggesupeervieed baseline)*
Diff.
100%
4.53
219.8
GAGAN
12
GAN
5%
10.4
59.6
S'GAN
GAN
10%
78.7
GAN 12
GAN
20%
7.7
83.1
DPT (ours, with U-ViT-Huge and MSN)
Diff. 0..%((1)
4.59
153.6
DPT (ours, with U- U-Vi MSN)
Diff.
0.4%
4.58
210,9
Table 2: Image generation results on ImageNet 256 x 256. 1 labels the results taken from the corresponding references and * labels baselines achieved by us. DPT and the corresponding baselines employ the same model architectures [5]. With 0.4% labels, DPT outperforms strong conditional generative models with full labels, including CDM [24], ADM (4) and LDM [25]. We bold the best result achieved with full labels and underline the best result achieved with few labels. For fair comparison, we also list the parameters of the diffusion model, including its auxiliary components.
Method
Model
Label fraction
FID
FIDeur
sFID 1
1St
Precision
Recall
#
Params
(# labels/class)
IC-GAN 67
GAN
0%
15.6
59.0
BigGAN-deep
GAN
100%
6.95
7.36
171.4
0.87
StyleGAN-XL.
92
GAN
100%
2,30
4.02
265.12
0.78
0.53
IDDPM 23
Dif..
100%
12.26
5.42
0.70
0.62
550M
CDM 24
Diff.
100%
4.88
158.71
ADM
100%
215.84
673M
LDM-4-
65
Diff.
100%
3.60
247.67
0.87
0.48
455M
DVT-XL/2-G
Diff.
100%
2.27
4.60
278.24
0.83
0.57
675M
Large
WILE
With 01 U-VgT-Large
Supervised baseline"
Diff.
100%
3.31
2.39
221.61
0.83
WIEE
WI
Unsupervised baseline"
DifE.
0%
27.99
5.40
7.03
33.86
0.60
0.62
WIE
DPT (ours, DPT (ours, with MSN) DPT (ours, with MSN) DPT (ours, with MSN)
Diff.
(.1%
WILE
Diff.
0.29((2)
3.44
2.37
6.58
199. 74
0.82
0.53
Diff.
(6)
3.37
2.35
217.53
0.83
0.52
371M
Diff.
196(== 12)
3.35
2.34
6.66
223.09
0.83
0.52
371M
U-VeT-Huge
Supervised buseline!
Diff.
100%
2.29
1.75
5.68
263.88
0.82
0.57
585M
DPT (ours, with MSN)
(1.1%
585M
DPT (ours, with MSN)
Diff.
< (.2%%
2.52
1.81
5.49
230.34
0.81
0.57
585M
DPT (ours, with MSN) DPT (ours, with Semi-ViT VIT)
Diff.
(5)
2.50
1.82
5.54
243.10
0.83
0.55
585M
DiI.
196(== 12)
2.42
1.2
5.48
259.93
0.82
0.56
585M
We evaluate sumi-supervised generation performance of DPT on ImageNet 128 128, as shown in Tab. [ In particular, DPT with only < 0.1% labels outperforms the SOTA semi-supervised generative model S SAN 12 with 20% labels (FID 4.59 vs. 7.7), suggesting DPT has superior label efficiency.
In Tab. we compare DPT with state-of-the-art generative models on ImageNet 256 X 256. We construct highly competitive baselines based diffusion models with U-ViT-Large [5]. According to Tab. our supervised and unsupervised baselines achieve an FID of 3.31 and 27.99, respectively. Leveraging the pseudo-labels predicted by the strong semi-supervised learner T77. DPT with few labels improves the unconditional baseline significantly and is even comparable to the supervised baseline under all metrics. In particular, with only two labels per class, DPT improves the FID of the unsupervised baseline by 24.55 and is comparable to the supervised baseline with gap of 0.13. Moreover, we also construct more competitive baselines based on U-ViT-Huge to advance DPT. With one (iee,, < 0.1%) label per class, our more powerful DPT achieves an FID of 3.08, outperforming strong supervised diffusion models including IDDPM 23,, CDM 244, ADM 4 and LDM 225.. Additionally, with 1% labels, DPT achieves an FID of 2.42. comparable to the

---

Table 3: Image generation results on ImageNet 512 x 512. f labels the results taken from the corresponding references. We bold the best result under the corresponding setting.
Method
Model
Label fraction
FID-50K
IS f
(# labels/class)
BigGAN-deep 91
GAN
100%
8.43
177.90
StyleGAN-XI. 92
GAN
100%
2.41
267.75
ADM [
Diff.
100%
3.85
221.72
DiT-XI/2-G 77
Diff.
100%
3.04
240.82
U-ViT-Huge (supervised baseline)
Diff.
100%
4.05
263.79
DPT (ours, with U-ViT-Huge and MSN)
Diff.
< 0.1%((5
4.05
252.08
Table Top-1 accuracy on the ImageNet validation set with few labels. t labels the results taken from corresponding references, labels the results taken from Assran et al. [17] and labels the baselines reproduced by us. DPT and the corresponding baseline employ exactly the same classifier architectures. With one, two, five labels per class and 1% labels, DPT improves the state-of-the-ar semi-supervised learner [17 [6] consistently and substantially. We bold the best result under corresponding setting and underline the second-best result.
Method
Architccture
Top-1 accuracy 1 given labels per class (label fraction)
1(< 0.1%) 2(< 0.2%) 5(< 0.5%)
12(1%)
EMAN
ResNet-50
63.0
PAWS 51
ResNet-50
66.5
BYOL 28
ResNet-200
71.2
SimCLRv2
18
ResNet-152
76.6
Semi-Vit
ViT-Huge
80.0
iOTT 98
VIT-B/16
46.1 0.3
56.2 0.7
64.7 0.3
DINO 94
VIT-B/8
0.5
55.9
6.2
MAE 29
VIT-H/14
11.6
0.2
32.8 0.2
MSN
ViT-B/4
54.3 0.4
64.6
72.4
75.7
MSN
ViT-L/7
57.1 0.6
66.4 = 0.6
72.1 + 0.2
75.1
MSN (baseline)*
VIT-B/4
52.9
64.9
72.4
DPT (ours)
VIT-B/4
58.6
69.5
74.4
MSN (baseline)"
VIT-1/7
56.2
66.5
72.0
DPT (ours)
VIT-1/7
58.9
69.2
73.4
Semi-VIT VIT (baseline)*
ViT-Huge ViT-Huge
79.4
DPT (ours)
80.2
of-te-of-the-art supervised diffusion model (7]. Lastly, DPT with few labels performs comparably to the fully supervised baseline under the FIDCLI metric, which suggests that DPT can generate high-quality samples and does achieve lower FID solely due to better Top-N alignment.
We also conduct an experiment on higher resolution (i.c., 512 X 512) in Tab. 3 with five (i.e., < 0.4%) labels, DPT achieves FID of 4.05, which is the same that of the supervised baseline. The above quantitative results demonstrate that DPT can achieve excellent generation performance and label efficiency at diverse resolutions. Qualitatively, realistic, diverse, and semantically correct images even with a single label, which agrees with the quantitative results in Tab. 2 and Tab. We provide more samples and failure cases in Appendix F. and a detailed class-wiss analysis to show how classification helps generation in Appendix H
presented in Fig. DPT can generate
Besides, Tab. 5 in Appendix A compares DPT with state-of-the-ard generative models on CIFAR-10 10. DPT achieves competitive performance using only 0.08% labels with EDM (6) which relies on full labels (FID 1.81 vs. 1.79). This result demonstrates the generalizability of DPT on different datasets.

---

5.3 Image Classification with Few Labels
We demonstrate that generative augmentation remains a useful approach for semi-supervee supervise clas- sification aided by diffusion models. In particular, DPT achieves state-of ofre-of-iie -o---ff-thheeee the-art remi-supervised classification performance on ImageNet datasets across various settings and the second-best results on CIFAR-10.
Tab. 4 compares DPT with state-of-the-art semi-supervised classifiers on the ImageNet validation set with few labels. Specifically, DPT outperforms strong semi-supervised baselines [17 consistently and substantially with one, two,five labels per class and 1% labels and achieves state-of-the-ar top-1 accuracies of 59.0, 69.5, 74.4 and 80.2, respectively. In particular, with two labels per class, DPT leverages the pseudo images generated by the diffusion model and improves MSN with ViT-B/4 by an accuracy of 4.6%. Besides, we compare the performance of DPT with that of SOTA fully supervised models (as shown Tab. I2 Appendix F22 and find that DPT performs comparably to Inception-yy -4 [95]. using only 1% labels.
Moreover, Tab. Appendix A compares DPT with state-of-the-art semi-supervised classifiers CIFAR-10. DPT with four labels per class achieves the second-best error rate of 0.68
Conclusions
This paper presents simple yet effective training strategy called DPT for conditional image gen- eration and classification supervupe learning. Empirically, we demonstrate that DPT can achieve SOTA semi-supervised generation and classification performance on ImageNet datasets across various settings. DPT probably inspires future work in diffusion models and semi-supervised learning.
Limitation. One limitation of DPT is directly using the pseudo images improve the performance of DPT for its simplicity and effectiveness while we could use pre- pre-trained models like CLIP to filter out noisy image-label pairs that images do not semantically align well with the label. Another limitation pertains to the direct use of pseudo labels. Given our use of classifier-fres guidance, we have the flexibility assign low-confidence pseudo labels the null token with high probability, which aids in filtering out noisy pseudo labels.
Social impact. We believe that DPT can benefit real-world applications with few labels (e.g., medical analysis). However, the proposed semi-supervised diffusion models may aggravate social issues such as "DeepFakes". The problem can be relieved by automatic detection with machine learning, which is an active research area.
Acknowledgement
This work was supported by NSF of China (Nos. 62076145); Beijing Outstanding Young Scientist Program (No.
Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (No. 22XNKJ13). C. Li was also sponsored by Beijing Nova Program (No. 20220484044).
References
[1] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics,"" in Proceedings of the 32nd International Conference on Machine Learning, 2015.
[2] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," in Advances in Neural Information Processing Systems, 2020.
[3] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score based generative modeling through stochastic differential equations," in 9th International Conference Learning Representations, 2021.
[4] P. Dhariwal and Nichol, "Diffusion models beat gans on image synthesis," Advances in Neural Information Processing Systems, vol. 34, pp. 8780-8794, 2021.
10

---

[5] F. Bao, C. Li, Y. Cao. and J. Zhu, "All are worth words: a vit backbone for score- based diffusion models," arXiv preprint MxXiv:209.121522 2022.
[6] T. Karras, M. Aittala, T. Aila, and S. Laine, "Elucidating the design space of diffusion-based generative models," in Proc. NeurIPS, 2022.
[7] W. Peebles and S. Xie, "Scalable diffusion models with transformers." arXiv preprint arXiv2212.09748, 2022.
[8] F. Bao, C. Li, J. Sun, and Zhu, "Why are conditional generative models better than uncondi- tional ones?" in NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[9] V. T. Hu, D. W. Zhang. Y. M. Asano, G. J. Burghouts, and C. G. Snoek, "Self--uided diffusion models," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 413-18 18-1884222 422.
[10] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, "Seemisupervised learning with deep generative models," in Advances in Neural Information Processing Systems, 2014.
[11] C. Li. Information Processing Systems, 2017, pp. 4088-4098.
Li. K. Xu, J. Zhu, and Zhang, "Triple generative adversarial nets," in Advances in Neural
[12] M. Lučić, M. Tschannen, M. Ritter, X. Zhai, O. Bachem, and S. Gelly, "High-fidelity image generation with fewer labels," in International conference on machine learning. PMLR, 2019. pp.
[13] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Ku- rakin, and C.-L. Li, "Fixmatch: Simplifying semi- superviper learning with consistency and confidence," Advances in neural information processing systems, vol. 33, pp. 596-608, 2020.
[14] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang. M. Okumura, and T. Shinozaki, "Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling," Advances in Neural Information Processing Systems, vol. 34, pp. 18 408 18 419, 2021.
[15] Y. Wang, H. Chen, Q Heng, W. Hou, M. Savvides, T. Shinozaki, B. Raj. Z. Wu, and
J. Wang, "Freematch: Self-adaptive thresholding for semi-supervised learning," arXiv preprint arXiv:2205.072266 2022.
[16] Z. Cai, A. Ravichandran, P. Favaro, M. Wang. D. Modolo, R. Bhotika, Z. Tu, and S. Soatto, "Semi-supervised vision transformers at scale," in NeurIPS, 2022.
[17] M. Assran, M. Caron, I. Misra, P. Bojanowski, F. Bordes, P. Vincent, A. Joulin, M. Rabbat, and N. Ballas, "Masked siamese networks for label-efficient learning," in European Conference on Computer Vision. Springer. 2022, pp. 456-473.
[18] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and E. Hinton, "Big self-supervised models are strong semi-supervised learners," Advances in neural information processing systems, vol. 33. pp. 22 243-22 255, 2020.
[19] Krizhevsky and Hinton, "Learning multiple layers of features from tiny images," Citeseer 2009.
[20] J. Deng, W. Dong, R. Socher, Li, K. Li, and Fei-Fei, "Imagenet: A large-scale hierarchical image database," in IEEE Computer Society Conference on Computer Vision and Puttern Recognition, 2009, pp. 248-255.
[21] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by a two time-scale update rule converge to local nash equilibrium," " in Advances in Neural Information Processing Systems, 2017, pp. 6626-6637.
[22] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, "Improved techniques for training GANs," in Advances in Neural Information Processing Systems, 2016.
[23] Q. Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," Inter- national Conference on Machine Learning. PMLR, 2021, pp. 8162-8171.

---

[24] J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, "Cascaded diffusion models for high fidelity image generation." J. Mach. Learn. Res., vol. 23, pp. 47-1, 2022.
[25] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022. pp. 10 18684-10695.
[26] J. Ho and T. Salimans, "Classifier-fres diffusion guidance," " arXiv preprint arXiv:220.1125%. 2022.
[27] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x 16 words: Transformers for image recognition at scale," in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[28] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Guo, M. Gheshlaghi Azar al., "Bootstrap your own latent-a new approach to self-supervised learning," Advances in neural information processing systems, vol. 33, pp. 21 271-21 284, 2020.
[29] He, Chen, S. Xie, Li. Dollár, and R. B. Girshick, "Masked autoencoders are scalable vision learners." in /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 979- 988.
[30] L. Maaløe, C. K. Sønderby, S. K. Sønderby, and Winther, "Auxiliary deep generative models," in International conference muchine learning. PMLR, 2016, pp. 1445-1453.
[31] C. Li, J. Zhu, and B. Zhang, "Max-margin deep generative models for (semi-) supervised learning," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 11, pp. 2762-2775,2011
[32] J. T. Springenberg, "Unsupervised and sumi-super learning with categorical generative adversarial networks," in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2016
[33] Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, "Good semi-supervised learning that requires a bad gan," in Advances in Neural Information Processing Systems, 2017, pp. 6510-6520
[34] Z. Gan, L. Chen, W. Wang, Y. Pu, Y. Zhang, H. Liu, Ci, and L. Carin, "Triangle generative adversarial networks," " in Advances Neural Information Processing Systems, 2017, pp- 5247-5256.
[35] C. Li, K. Xu, J. Zhu, J. Liu, and B. Zhang, "Triple generative adversarial networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[36] Y. Zhang, H. LLng. J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, "Datasetgan: Efficient labeled data factory with minimal human effort," in Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 145 155.
[37] T. Joachims el al., "Transductive inference for text classification using support vector machines," in ICML, vol. 99, 1999, pp. 200-209.
[38] Y. Grandvalet and Y. Bengio, "Semi-superrised learning by entropy minimization." in Ad- vances in Neural Information Processing Systems, vol. 17, 01 2004.
[39] D.-H. Lee et al., "Pseudo-label: The simple and efficient semi sumivisuereieed learning method for deep neural networks," in Workshop on challenges in representation learning, ICML, vol. 2013, p. 896.
[40] A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, "Label propagation for deep semi-supervised learning." " Proceedings of the IEEE conference on computer vision and pattern recognition, 2019, pp. 5070-5079.
12

---

[41] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, "Virtual adversarial training: regularization method for supervised and semi-supervised learning," IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 8, pp. 1979-1993, 2018.
[42] A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results," in Advances in Neural Informa- tion Processing Systems, 2017, pp. 1195-1204.
[43] S. Laine and T. Aila, "Temporal ensembling for sumi-supervised learning," "" in International Conference on Learning Represenlations, 2017.
[44] Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang, "Smooth neighbors on teacher graphs for semi- supervised learning," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8896-8905.
[45] Athiwaratkun, M. Finzi, Izmailov, and A. G. Wilson, "There are many consistent explanations of unlabeled data: Why you should average," in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[46] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow, "Realistic evaluation of deep semi-supervised learning algorithms," in Advances in Neural Information Processing Systems, 2018, pp. 3235-3246.
[47] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, "Mixmatch: A holistic approach to semi-supervised learning," in Advances in Neural Information Processing Systems, 2019, pp. 5049-5059.
[48] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, "Remix- match: Semi-supervised learning with distribution matching and augmentation anchoring," in International Conference on Learning Representaatinss 2019.
[49] J. Li, C. Xiong, and S. C. Hoi, "Comatch: Semi-supervised learning with contrastive graph regularization." in Proceedings the IEEE/CVF International Conference Computer Vision, 2021, pp. 9475-9484.
[50] M. Zheng, S. You, L. Huang, F. Wang. C. Qian, and C. Xu, "Simmatch: Semi-supervised learning with similarity matching," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 14447114488
[51] M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat, "Semi- supervised learning of visual features by non- pou-paamettiially predicting view assignments with support samples," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 8452.
[52] H. Tang, L. Sun, and K. Jia, "Stochastic consensus: Enhancing semi-supervised learning with consistency of stochastic classifiers," " in European Conference on Computer Vision Springer, 2022, pp. 330-346.
[53] X. Wang, L. Lian, and S. X. Yu, "Unsupervised selective labeling for more effective semi- supervised learning," " in European Conference on Computer Vision. Springer, 2022, pp. 427-445.
[54] Chen, J. Jiang, X. Wang, Wan, J. Wang, and M. Long, "Debiased self-training for semi-supervised learning," Advances in Neural Information Processing Systems, vol. 35, pp. 32 224424-3244 2022.
[55] H. Tang and K. Jia, "Towards discovering the effectiveness of moderately confident samples for semi-supervised learning," in Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 648--166677
[56] J. Lim, D. Um, H. J. Chang, D. U. Jo, and J. Y. Choi, "Class-attentive diffusion network for sumi-supervised classification." " Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 10, 2021, pp. 8601-8609.
13

---

[57] Z. Cai, A. Ravichandran, S. Maji, C. Fowlkes, Z. Tu, and S. Soatto, "Exponential moving average normalization for sulf-super and supervvie learning"" in Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2021, pp. 194-203.
[58] S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. J. Fleet, "Synthetic data from diffusion models improves imagenet classification," arXiv preprint arXiv:2304. 24040666404 2023.
[59] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann, "Fixing data augmentation to improve adversarial robustness," arXiv preprint arXiv:2 2021.
[60] Z. Wang. T. Pang, C. Du, M. Lin, W. Liu, and S. Yan, "Better diffusion models further improve adversarial training," " arXiv preprint
2302:0460233 2023.
[61] A. Jahanian, X. Puig, Y. Tian, and P. Isola, "Generative models as data source for multiview representation learning," in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[62] R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi, "Is synthetic data from generative models ready for image recognition?" arXiv preprint arXiv:2210.07574, 2022.
[63] V. Besnier, H. Jain, A. Bursuc, M. Cord, and P. Pérez, "This dataset does not exist: training models from generated images," " in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 1-5.
[64] Zhengg G. Wu, and Li, "Toward understanding generative data augmentation," arXiv preprint arXiv:2305. 17476, 2023.
[65] M. Noroozi, "Self-labeled conditional gans," arXiv preprint arXiv:20/2.02//63 2020.
[66] S. Liu, T. Wang, D. Bau, J.-Y. Zhu, and A. Torralba, "Diverse image generation via self- conditioned gans," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 14 1426--12955
[67] A. Casanova, M. Careil, J. Verbeek, M. Drozdzal, and A. Romero Soriano, "Instance- conditioned gan," Advances Neural Information Processing Systems, vol. 34, pp. 27 517- 27 529, 2021.
[68] A. Q. Nichol, Dhariwal, A. Ramesh, Shyam, Mishkin, McGrew, Sutskever, and M. Chen, "GLIDE: towards photorealistic image generation and editing with rext-guided diffusion models," in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol. 162, 2022, pp. 16 184-16804.
[69] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with clip latents," arXiv preprint arXiv:2204.066 2525. 2022.
[70] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, "Vector quantized diffusion model for text-to-image synthesis," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10 100666117770 706.
[71] Y. Balaji, S. Nah. X. Huang, A. Vahdat, Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro et al., "ediffi: Text-to-image diffusion models with an ensemble of expert denoisers," arXiv preprint arXiv:2211.0133 2022.
[72] B. Poole, A. Jain, J. Barron, and B. Mildenhall, "Dreamfusion: Text-to-3d using 2d diffusion." arXiv preprint arXiv:2209.1499 2022.
[73] C. Meng, Y. He, Y. Song. J. Song, J. Wu, J. Zhu, and Ermon, "Sdedit: Guided image synthe- sis and editing with stochastic differential equations," in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[74] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, "Ilvr: Conditioning method for denoising diffusion probabilistic models," 202/ IEEF/CVF International Conference on Computer Vision (ICCV), 2021, pp. 14347-14356.
14

---

[75] M. Zhao, F. Bao, C. Li, and J. Zhu, "Egsde: Unpaired image-to to-aaage translation via energy- guided stochastic differential equations." Advances in Neural Information Processing Systems, 2022.
[76] E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling, "Equivariant diffusion for molecule generation in 3d," in International Conference on Machine Learning. PMLR, 2022, pp. 8867-8887.
[77] F. Bao, M. Zhao. Z. Hao, P. Li, C. Li, and Zhu, "Equivariant energy-guided sde for inverse molecular design," arXiv preprini arXiv:2209. 15408, 2022.
[78] A. Alshenoudy, B. Sabrowsky- -iishh,, S. Thumfart, M. Giretzlehner, and E. Kobler, "Semi- supervised brain tumor segmentation using diffusion models," in IFIP International Conference on Artificial Intelligence Applications and Innovations. Springer, 2023, pp. -25.
[79] S. Gong, C. Chen, Y. Gong, N. Y. Chan, W. Ma, C. H.-K. Mak, J. Abrigo, and Q. Dou, "Diffusion model based semi-supervised learning on brain hemorrhage images for efficient midline shift quantification." in International Conference Information Processing in Medical Imaging. Springer, 2023, pp. 69-81.
[80] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[81] F. Bao, C. Li, J. Zhu, and B. Zhang, "Analytic-dpm: an analytic estimate of the optimal reverse variance diffusion probabilistic models," in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[82] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, "Dpm-solvert+: Fast solver for guided sampling of diffusion probabilistic models," arXiv preprint arXiv:2211.01095, 2022.
[83] Q. Zhang and Y. Chen, "Fast sampling of diffusion models with exponential integrator," in NeurlPS 2022 Workshop on Score-Based Methods, 2022.
[84] C. Lu, Y. Zhou, F, Bao, J. Chen, C. Li, and J. Zhu, "Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps," arXiv preprint 20X0::2206.00927, 2022.
[85] F. Bao, C. J. J. Sun, J. Zhu, and B. Zhang, "Estimating the optimal covariance with imperfect mean in diffusion probabilistic models," " Proceedings of the 39th International Conference Machine Learning, 2022, pp. 1555-1584.
[86] T. Salimans and J. Ho, "Progressive distillation for fast sampling of diffusion models," in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[87] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution." Advances in Neural Information Processing Systems, vol. 32, 2019.
[88] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia, "Generating images with sparse representations." " in Proceedings the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol. 139, 2021, pp. 7958-7968.
[89] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila, "Improved precision and recall metric for assessing generative models," Advances in Neural Information Processing Systems, vol. 32, 2019.
[90] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, and Lehtinen, "The role of imagenet classes in fr) 'echet inception distance," " arXiv preprint arXiv:2203.0660 2022.
[91] A. Brock, J.
Donahue, and K. Simonyan, "Large scale gan training for high fidelity natural image synthesis," " in International Conference on Learning Representations, 2018
[92] A. Sauer, K. Schwarz, and A. Geiger, "Stylegan-xl: Scaling stylegan to large diverse datasets." in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1-10.
15

---

[93] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, "ibot: Image bert pre- training with online tokenizer," International Conference on Learning Representations (ICLR), 2022.
[94] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin, "Emerging properties in sule-sue vision transformers," in Proceedings of the IEEF/CVE Interna- tional Conference on Computer Vision, 2021, pp. 9650-9660.
[95] C. Szegedy, S. loffe, V. Vanhoucke, and A. Alemi, "Inception-v4, inception-resnet and the impact of residual connections on learning," in Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017.
[96] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, "Training generative adversarial networks with limited data," in Advances in Neural Information Processing Systems, 2020.
[97] Y. Chen, X. Tan, B. Zhao, Z. Chen, R. Song. J. Liang, and X. Lu, "Boosting semi-supervised learning by exploiting all unlabeled data," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1548-7557,
[98] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, "Semiisuperrised learning with ladder networks," in Advances in Neural Information Processing Systems, 2015.
[99] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, "Virtual adversarial training: regularization method for supervised and semi-supervised learning." IEEE transactions pattern analysis and machine intelligence, vol. 41, no. 8, pp. 1779-1993, 2018.
[100] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, "Unsupervised data augmentation for consistency training." " Advances in Neural Information Processing Systems, vol. 33, pp. 6256662666600 66666268, 2020.
[101] Y. Xu, L. Shang, Ye, Q. Qian, Y.-F. Li, B. Sun, H. Li, and R. Jin, "Dash: Semi-supervised learning with dynamic thresholding," in International Conference on Machine Learning. PMLR, 2021, pp. 525-11 536.
[102] H. Pham, Z. Dai, Q Xie, and Q. Le,,, "Meta pseudo labels," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 11 557-11 568.
[103] J. Mairal, "Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more," arXiv preprint arXiv: (wt.:ii:1122.8811... 2019.
[104] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, "Momentum contrast for unsupervised visual representation learning," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9726-9753
[105] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, "A simple framework for contrastive learning of visual representationss" in International Conference on Machine Learning. vol. 119, 2020, pp. 1597-1607.
[106] X. Chen and K. He, "Exploring simple siamese representation learning," in IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 750-15
[107] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, "Simmim: A simple framework for masked image modeling," in International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[108] A. Fisher, "On the mathematical foundations of theoretical statistics," Philosophical trans- actions of the Royal Society of London, Series A, containing papers of a mathematical or physical character, vol. 222, no. 594-604, pp. 309-368, 1922.
[109] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp- 770-778.
16

---

[110] C. Szegedy, V. Vanhoucke, S. loffe. J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818-2824
[111] J. Hu, L. Shen, and G. Sun, Sgqueeze-and-exiiieee networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132-7141.
[112] M. Tan and Q. Le, "Efficientne:: Rethinking model scaling for convolutional neural networks," in International conference machine learning. PMLR, 2019, pp. 6105-6114.
[113] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, "Training data- efficient image transformers & distillation through attention," in International Conference on Machine Learning. PMLR, 2021, pp. 10 347-10357.
[114] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021, pp. 10012-10022
17


Reviews:
[{"review_id": "YPUlOzw6Mo", "reviewer": "Reviewer_kGQr", "summary": "This paper empirically studies the effectiveness of combining diffusion models and semi-supervised classification. First, an image encoder is trained using a self-supervised method, then finetuned with a small fraction of the ground-truth labels. Second, the image classifier assigns pseudo-labels to all unlabeled images in the training dataset which are then used to train a label-conditioned diffusion model. Finally, the diffusion model is used to generate image-label pairs that are used to further train the image classifier from the first step. Multiple experiments on CIFAR10 and ImageNet demonstrate the effectiveness of this approach, both in improving the diffusion model and the image classifier.", "questions": "The paper reports image generation results on ImageNet for multiple resolutions and differing numbers of ground-truth labels per class. What is the motivation behind the specific choices for the resolution and number of labels combinations? Does the choice of the resolution for the generative model affect the classification results?", "limitations": "A potential negative impact of this line of work (semi-supervised generation/classification) is that it could amplify stereotypes where a few examples are taken as representative of the whole class (group). Some discussion on this would benefit the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper demonstrates state-of-the-art results on both semi-supervised image classification and semi-supervised image generation\n- The presentation of both the model and the experiments is easy to follow and the text is well written\n- Exploring how diffusion models and discriminative models like image classifiers can mutually benefit from one another is interesting and relevant to both academics and practitioners", "weaknesses": "1. As shown in [1], the FID can be decreased by aligning the histograms of Top-N classifications without improvements to the image quality. Albeit there exists no perfect method to quantify this effect, it would be insightful to see some investigation into whether the lower FID actually translates to improved image quality or merely better alignment in terms of Top-N classifications.\n\n2. In the introduction, the paper claims that (unsupervised conditional) diffusion models trained with cluster assignments as pseudo-labels still underperform their ground-truth supervised counterparts. As shown in [2] this no longer holds true when the cluster number is set higher than the number of ground-truth classes. It would be interesting to explore how this insight could further benefit the semi-supervised setting, for example by further sub-clustering images with the same pseudo-labels. I consider the paper should be published if they fix their claim about unsupervised diffusion models.\n\n\n[1] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, J. Lehtinen, \"The Role of ImageNet Classes in Fréchet Inception Distance\", ICLR 2023.\n[2] H. Tao, D. W. Zhang, Y. Asano, G. J. Burghouts, C. G. M. Snoek, \"Self-Guided Diffusion Models\", CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer kGQr for the acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.\n\n## Weakness1: Lower FID translates to image quality improvement or better Top-N alignment?\n\nThank you for your insightful suggestion and pointing out the work [a]. As shown in [a], the $FID_{CLIP}$ metric can avoid such issues and we **have added the $FID_{CLIP}$ results to Tab.1 in our response pdf** following your suggestion. DPT with few labels performs comparably to the fully supervised baseline under the $FID_{CLIP}$ metric, which suggests that DPT can generate high-quality samples, and does NOT just achieve a lower FID due to better Top-N alignment. Your feedback is highly valuable in helping us provide a more comprehensive evaluation of our model. We will add the $FID_{CLIP}$ metric in Tab. 2 of the final version. \n\n[a] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, J. Lehtinen, \"The Role of ImageNet Classes in Fréchet Inception Distance\", ICLR 2023. \n\n## Weakness2: The claim about unsupervised diffusion models.\n\nThank you for your insightful suggestion and pointing out the work [b]. We will fix the claim about unsupervised diffusion models following your suggestion. We discuss the work [b] in details (which will be added to the introduction and related work of our final version) as follows:\n\n(1) As shown in [b], unsupervised diffusion models can outperform supervised baselines when the number of clusters is set higher than the number of ground-truth classes. However, this effect hasn't been confirmed in the context of ImageNet with high resolution (i.e., $\\geq 128 \\times 128$). \n\n(2) The idea of combining the unsupervised conditional diffusion model approach outlined in [b] with semi-supervised methods is interesting and we believe that this could further improve the performance and it is a promising future work.\n\n[b] H. Tao, D. W. Zhang, Y. Asano, G. J. Burghouts, C. G. M. Snoek, \"Self-Guided Diffusion Models\", CVPR 2023.\n\n## Q1: Motivation for choices of resolution and number of labels; resolution for generative model impact on classification\n\nThanks for the valuable question. \n\n**As for the choice of resolution and number of labels**, we were primarily driven by the task of ImageNet $256 \\times 256$ generation to systematically compare with a large family of baselines. In this context, we conducted detailed experiments, including settings with one, two, five labels per class, and 1% labels. We find that the performance of DPT with five labels per class is comparable to the supervised baseline, leading us to use this setting as the default in our other tasks such as ImageNet $128 \\times 128$ and ImageNet $512 \\times 512$ generation. \n\n**Regarding the impact of resolution choice on the classification results**, we haven't directly investigated this yet. It mainly depends on the quality of the pseudo images sampled from stage 2. Intuitively, a higher resolution may lead to a worse generative performance given a few labels. However, empirically, DPT can generate high quality samples on resolution of 128, 256 and 512. \n\nWe will add the above discussion in the experimental section of our final version.\n\n## Limitations: Potential negative impact\n\nThanks for your suggestion. We will follow your suggestion and add this discussion to the social impact of our final version."}, {"review_id": "FiU2d4FXR3", "reviewer": "Reviewer_2c3F", "summary": "This paper proposes to exploit generative diffusion models to benefit semi-supervised learning and vice versa.\n\nSpecifically, the proposed training strategy called dual pseudo training (DPT) consists of 3 stages:\n\n(1) Train a semi-supervised learning (SSL) model on labeled and unlabeled data\n\n(2) Train a conditional generative diffusion model using pseudo labels predicted by the SSL model in (1)\n\n(3) Use the generative model in (2) to generate more labeled images to train a classifier\n\nDPT achieves both SOTA performances on image generation and semi-supervised classification tasks on ImageNet and CIFAR-10, demonstrating \"Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels\".\n", "questions": "I do not have questions for this paper.", "limitations": "As stated by the author, filtering of pseudo labels in stage 1 and pseudo images in stage 2 should be investigated to reduce noise.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "(1) The paper is well-written and easy to follow, \n\n(2) The proposed method is simple but effective. And it can leverage any SSL algorithms or generative models in a plug-and-play manner.\n\n(3) The results are very impressive, both in numbers (SOTA in both classification and generation tasks) and samples (Figure 1).", "weaknesses": "(1) Naive combination of SSL and generative models is a straightforward idea; the novelty is limited.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 2c3F for the interest and acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.\n\n## Weakness1: the novelty is limited.\n\nThank you for your feedback. Our method is simple but effective. The novel contributions of our work lie in answering the two key questions and in exploring how diffusion models and semi-supervised classifiers can mutually benefit from each other. We will elaborate more about our novelty in the abstract and introduction parts of our final version. \n\n## Limitation1: filtering of pseudo data should be investigated.\n\nWe appreciate your insightful suggestion. \n\n**Regarding the filtering of pseudo labels in stage 1**, we will add a discussion about it. For instance, we could set pseudo labels, which have lower confidence, to the unconditional class identifier with a higher probability.\n\n**As for the filtering of pseudo images in stage 2**, we've discussed it in the limitation section of our paper. Employing pre-trained models such as CLIP could help us filter out noisy image-label pairs.\n\nWe will elaborate more about the filtering strategy in the appendix section of our paper.\n\n"}, {"review_id": "947JlhL3Qm", "reviewer": "Reviewer_wQJF", "summary": "This work presents an interesting interaction between a generative model and a semi-classifier. The pipeline can boost both the generative sample quality and discriminative performance of the semi-classifier in the few label settings. A three-stage pipeline is designed. First, the semi-classifier is trained on a few label data and the semi-label is acquired for all the unlabeled data. Second, the generative model is trained on both label and semi-label data to achieve comparable sample quality as a full-supervised one. At last, the semi-classifier can further improve its performance from the new samples generated by the learned generative model. DPT with two (i.e., < 0.2%) labels per class is comparable to SOTA supervised baseline (FID 2.52 vs. 2.29).", "questions": "- Is the algorithm stable for different starting samples in the first stage?\n-  what are the key components for the successful interaction between the generative model and semi-classifier?\n- How about the PSNR or other commonly used generation evaluation metrics? Since FID and IS are highly imagenet related.\n- Can it improve the upper bound of generation quality? If combining full supervised label data with this pipeline, will it surpass the 2.29 FID? \n- What about the improvement of unconditional generation?\n- The weaknesses mentioned above should be addressed.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well written and easy to follow.\n- Motivation is clear and the pipeline is simple.\n- Good results. DPT with two (i.e., < 0.2%) labels per class is comparable to SOTA supervised baseline (FID 2.52 vs. 2.29). \n- Interesting interaction between semi-classifier and diffusion model.", "weaknesses": "- 61-63 Specifically, DPT with four labels per class achieves an error rate of 4.53% on 62 CIFAR-10, which even outperforms the performance of SOTA methods on CIFAR-10 63 with 25 labels per class. The Full-flex[1] algorithm has better performance than 4.53% for 4 labels per class on CIFAR-10, which might make this claim incorrect.\n- It's interesting to see if the framework is general to other semi-algorithm and diffusion model pairs.\n- lack of analysis of the key components of the interaction between the generative model and semi-classifier. For example, what is the key factor to make semi-classifier success for generative model in the design? And What is the key factor to make genertive model success for semi-classifier in the design? This should be discussed in the paper.\n\n\n[1] Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer wQJF for the valuable and constructive comments. We respond below to your questions and concerns.\n\n## Weakness1: Claim about DPT with four labels per class may be incorrect. \n\nWe appreciate your pointing out the work [c] and your valuable suggestion. DPT with four labels per class achieves an error rate of 4.53% on CIFAR-10, which is comparable to an error rate of 4.44$\\pm$0.15 % achieved by Full-flex [c]. We will revise our claim in the introduction and appendix of our final version. \n\nMoreover, Full-flex and our work DPT are orthogonal. As DPT is a flexible framework, integrating Full-flex into DPT could possibly further improve performance.\n\n[c] Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data.\n\n##  Weakness2: If the framework is general to other semi-algorithm and diffusion model pairs. \n\nThank you for your valuable question. In principle, DPT can leverage any SSL algorithms or generative models in a plug-and-play manner. In our submission, we have utilized three classical and strong semi-supervised methods including MSN, Semi-ViT, and FreeMatch , as well as two state-of-the-art diffusion models, namely U-ViT (transformer-based) and EDM (U-net-based). Extensive results demonstrate the effectiveness of DPT and suggest that DPT can be generalized to other semi-supervised algorithm and diffusion model pairs. \n\n##  Weakness3 and Q2: lack of analysis of the key components of the interaction.\n\nWe appreciate your feedback. The key component of the interaction between the generative model and semi-classifier is that each can provide high-quality pseudo-data for the other. We explore why classifiers can benefit generative models through class-level visualization and analysis in Appendix H. Besides, we also investigate why generative models can benefit classifiers via class-level visualization and analysis in Appendix I. We will make it clearer in the final version.\n\n## Q1: Is the algorithm stable for different starting samples in the first stage?\n\nThank you for your question. We guess that \"starting samples\" means labeled data in your comment. If we have any misunderstandings, please feel free to post more questions.\n\nThe semi-supervised classifiers (i.e. MSN, FreeMatch) we employed in the first stage are stable (for instance, FreeMatch with four labels per class achieves an error rate of $4.90\\pm0.04$% on CIFAR-10), thus capable of providing high-quality pseudo-labels to the generative models. In return, the generative models can provide high-quality pseudo-images. \n\nBesides, we're running an experiment by randomly sampling labeled data in stage 1 three times on CIFAR-10, and once the results are available, we will update our response. We will add the discussion and experiments in the final version. \n\n## Q3: How about the PSNR or other commonly used generation evaluation metrics? \n\nWe appreciate your insightful suggestion. \n\nTo our best knowledge, calculating PSNR requires comparing two images (e.g. a source image and a super-resolution result) and therefore cannot be adopted to evaluate generative performance directly. \n\nFollowing your suggestions, we **add a new metric $FID_{CLIP}$[a] to evaluate our method without any model pretrained on ImageNet** in Tab. 1 in our response pdf. $FID_{CLIP}$ is similar to FID but replaces the Inception-V3 feature space with CLIP features. DPT with few labels **performs comparably to the fully supervised baseline under the $FID_{CLIP}$ metric**, which remains to show the effectiveness of DPT. We will include the $FID_{CLIP}$ metric in the final version.\n\n[a] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, J. Lehtinen, \"The Role of ImageNet Classes in Fréchet Inception Distance\", ICLR 2023. \n\n##  Q4: Can it improve the upper bound of generation quality?\n\nThanks for your valuable question. When all labels are available, the second stage of DPT becomes equivalent to training a supervised diffusion model with real labels. This is essentially the same as a supervised conditional baseline. Therefore, combining fully supervised labeled data, it will not surpass the baseline (e.g. 2.29 FID of U-ViT).\n\n## Q5: What about the improvement of unconditional generation?\n\nThank you for your valuable question. DPT primarily targets class-conditional generation given a few labels, and it approximates the conditional score function rather than the unconditional one. For unconditional generation, you can sample images using labels sampled from the dataset's label distribution, then ignore those labels. This approach maintains image quality and it is equivalent to unconditional generation.\n\nOnce again, thank you for your constructive feedback. We'll revise our paper according to your suggestions. We kindly request that you consider raising the score accordingly if we addressed your concerns.\n\n"}, {"review_id": "RMI2hMQiDx", "reviewer": "Reviewer_hJXD", "summary": "To advance semi-supervised generative and classification tasks, this paper proposes a simple yet effective training strategy termed dual pseudo training (DPT). The proposed DPT operates in three stages: training a strong semi-supervised learner to predict pseudo labels, training a strong diffusion model using both real and pseudo labels to generate pseudo images, and retraining the semi-supervised learner with a mix of real and pseudo images. Experimental results show the effectiveness of the proposed DPT on semi-supervised generation and classification across various settings.", "questions": "1. Does the \"retraining\" in the 3-rd stage mean fine-tuning or training from scratch?\n\n2. Why do the strong diffusion models with full labels, such as IDDPM, CDM, ADM, and LDM, perform worse than the proposed DPL with few labels? \n\nHow would the proposed DPT perform if full labels are used? It provides the uppper bound.\n\n3. On Line 54, DiT-XL/2-G [7] (FID 2.27) is the SOTA supervised baseline, rather than [5] (FID 3.40). Please carefully check and correct.\n\n5. In Fig. 2 (a), it is no need to indicate the label fraction by the bubble area, as the x-axis has already indicated the label fraction. Different shapes or colors are enough to show the advantage of the proposed DPT.\n\n6. In Table 2, the \"bold\" and \"underline\" look a little messy. Some columns of evaluation metrics have no mark while others have many. A better marking scheme should be figured out to make the table easier to understand.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well-written.\n2. The proposed DPT bridging strong semi-supervised learners and diffusion models by pseudo labels and images is technically sound.\n3. The idea that diffusion models and semi-supervised learners benefit mutually with few labels is interesting and of practical use.\n4. It is impressive that the proposed DPT using 1% labels yields comparable results to the supervised baseline using 100% labels (cf. Table 2).", "weaknesses": "1. Does this paper have any technical novelty? It should be highlighted in the abstract and introduction parts.\n\n2. The proposed DPT improves semi-supervised learning models with generative augmentation via diffusion models. Despite substantial performance improvements, the computing overhead and time budget may be big. It would be better to provide a comparative analysis between the proposed method and existing ones.\n\n3. At the second stage of the proposed DPT, pseudo labels may not always align with ground truth labels, making it hard to control semantics in samples. How does the pseudo label accuracy affect the pseudo image fidelity and semantics? Some illustrative examples are preferred.\n\n4. Would the classifier retrained with a mix of real and pseudo images be used to predict pseudo labels as the first stage? The three stages may form a positive cycle to iteratively improve the pseudo label accuracy and pseudo image semantics.\n\n5. Lines 57-58: The exploration on why classifiers can benefit generative models through class-level visualization and analysis is important for the method validation. It would be better to place it in the main text.\n\n6. In Table 1, the semi-supervised generative model S^3GAN was proposed in 2019. Haven't there been any new methods in the last few years?\n\n7. In tables of image generation results, different methods may differ in the model size, leading to unfair comparison. It would be better to show the number of model parameters in these tables.\n\n8. Many recent related works on semi-supervised learning have not been included, discussed, and compared in this paper, e.g., [a-h].\n\n[a] Zhang et al. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurIPS, 2021.\n\n[b] Tang et al. Stochastic Consensus: Enhancing Semi-Supervised Learning with Consistency of Stochastic Classifiers. ECCV, 2022.\n\n[c] Wang et al. Unsupervised Selective Labeling for More Effective Semi-Supervised Learning. ECCV, 2022.\n\n[d] Tang et al. Towards Discovering the Effectiveness of Moderately Confident Samples for Semi-Supervised Learning. CVPR, 2022.\n\n[e] Lim et al. Class-Attentive Diffusion Network for Semi-Supervised Classification. AAAI, 2021.\n\n[f] Gong, S. et al. (2023). Diffusion Model Based Semi-supervised Learning on Brain Hemorrhage Images for Efficient Midline Shift Quantification. In: Frangi, A., de Bruijne, M., Wassermann, D., Navab, N. (eds) Information Processing in Medical Imaging. IPMI 2023. Lecture Notes in Computer Science, vol 13939. \n\n[g] Alshenoudy, A., Sabrowsky-Hirsch, B., Thumfart, S., Giretzlehner, M., Kobler, E. (2023). Semi-supervised Brain Tumor Segmentation Using Diffusion Models. In: Maglogiannis, I., Iliadis, L., MacIntyre, J., Dominguez, M. (eds) Artificial Intelligence Applications and Innovations. AIAI 2023. IFIP Advances in Information and Communication Technology, vol 675.\n\n[h] Chen et al. Debiased Self-Training for Semi-Supervised Learning. NeurIPS, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer hJXD for the acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.\n\n##  Weakness1: Regarding technical novelty.\n\nThank you for your feedback. Our paper's novelty lies in answering the two key questions and in exploring how diffusion models and semi-supervised classifiers can mutually benefit from each other. We will highlight our novelty in the abstract and introduction parts of our final version. \n\n##  Weakness2: the computing overhead and time budget may be big.\n\nThank you for highlighting this concern. \n\n**Regarding computing overhead and time budget,** we present the computational costs for DPT in Tab. 9 for ImageNet $256\\times256$. The additional time cost over MSN is approximately 201.7%, calculated as $\\frac{Generator ~+ ~DPT ~~extra ~~cost}{Classifier}=\\frac{5813}{2881}=201.7$%. \n\n**In a comparative analysis between DPT and existing methods**, although DPT requires nearly twice the training time compared to the MSN baseline, it's still **more time-efficient** than other methods like Triple-GAN [d], which demands at least 5 times the training time of its classifier. \n\n**We'll further discuss this in Appendix D of our final version.**\n\n[d] Li C, Xu K, Zhu J, et al. Triple generative adversarial networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, 44(12): 9629-9640.\n\n##  Weakness3: Pseudo label accuracy's effect on image fidelity and semantics\n\nThank you for your insightful question. We demonstrate illustrative examples in Appendix F.1. If pseudo labels align with ground truth, DPT generates high-quality images (Fig.5 (a)). If not, DPT may generate samples with incorrect semantics (Fig.5 (d), (g)). Nevertheless, as the number of labels increases, the generation performance of DPT improves, yielding more accurate semantics due to more accurate predictions.\n\n##  Weakness4: Retraining classifier with mixed real and pseudo images for initial pseudo label prediction\n\nWe appreciate your insightful question. In Appendix F.3, we explore using a retrained classifier to predict pseudo labels and train the conditional generative model, forming an additional stage. Tab.13 shows that this indeed improve results. However, we focus on the three-stage strategy in this paper for simplicity and efficiency.\n\n##  Weakness5: Place class-level visualization and analysis in main text.\n\nThanks for your valuable suggestion. We will follow your suggestion and place these details in the main text of our final version.\n\n##  Weakness6: Haven't there been any new semi-supervised generative model in the last few years?\n\nThanks for your valuable comment. In Tab.1, we compare DPT with other semi-supervised methods on ImageNet $\\mathbf{128\\times128}$. To our knowledge, there is no other literature reporting semi-supervised generation results on this dataset in the last few years.\n\n##  Weakness7: It would be better to show the number of model parameters in these tables.\n\nWe appreciate your valuable suggestion. We will add the comparison of the number of generative model parameters in these tables of our final version.\n\n##  Weakness8: Lack of inclusion and comparison with some semi-supervised learning works.\n\nWe appreciate your pointing out these mentioned works and your valuable suggestion, we will add the discussion of these works in the related work in our final version. \n\n##  Q1: Does the \"retraining\" in the 3-rd stage mean fine-tuning or training from scratch?\n\nThanks for your valuable comment. When we conduct experiments on ImageNet, the classifiers we use are MSN and Semi-ViT. When utilizing MSN, we freeze the pre-trained models and replace $\\mathcal{S}$ with $\\mathcal{S} \\cup \\mathcal{S}_2$ to train a classifier from scratch. When utilizing Semi-ViT, we freeze the models, which are pre-trained in a self-supervised manner during the first stage of Semi-ViT, and replace $\\mathcal{S}$ with $\\mathcal{S} \\cup \\mathcal{S}_2$ to train a classifier during the third stage of Semi-ViT. We will clarify this in the method section of our final version.\n\n##  Q2: Why strong diffusion models perform worse than DPT with few labels? How would the proposed DPT perform if full labels are used? \n\nThank you for your question. \n\n**As for why such diffusion models perform worse than DPT with few labels:** Firstly, the supervised baseline of DPT is U-ViT, which outperforms IDDPM, CDM, ADM, and LDM. Secondly, semi-supervised classifier provides U-ViT with high-quality pseudo labels. Therefore, the performance of DPT is comparable to U-ViT and surpasses IDDPM, CDM, ADM, and LDM.\n\n**Regarding how would DPT perform if full labels are used:** When all labels are available, the second stage of DPT becomes equivalent to training a supervised diffusion model with real labels. This is essentially the same as a supervised conditional baseline. Therefore, based on the discussion above, the upper bound of DPT is the supervised baseline (e.g. 2.29 FID of U-ViT).\n\nWe will clarify this in the experiment section of our final version.\n\n##  Q3: Correct SOTA supervised baseline on Line 54 from [5] to DiT-XL/2-G [7].\n\nWe appreciate your feedback. We will correct it in the final version and clarify this in the introduction of our final version to avoid misleading. In our submission, we compared to U-ViT-Huge[5] (FID 2.29),  as shown in the fifth row from the bottom of Tab.2, while U-ViT-Large[5] (FID 3.40), is shown in the ninth row of Tab.2.  \n\n## Q4: no need to indicate the label fraction by the bubble area.\n\nWe appreciate your suggestion regarding our graph design. We will follow your suggestion and change our graph in the final version.\n\n##  Q5: In Table 2, the \"bold\" and \"underline\" look a little messy.\n\nThank you for raising this issue. In the final version of our paper, we will remove the use of “underline” and only introduce “bold” for FID to make the table easier to understand. We appreciate your suggestion.\n\n"}, {"review_id": "jQTsmn7Tgw", "reviewer": "Reviewer_JKn2", "summary": "In this work, the authors propose a novel training strategy, dual pseudo training (DPT), that involves three steps: First, DPT uses state-of-the-art semi-supervised learning methods to predict pseudo-labels on a partially-labeled dataset and diffusion models. Second, DPT trains a conditional generative model with pseudo-labeled data for pseudo-images. Finally, the classifier is re-trained with a dataset which includes both real and pseudo images. DPT surpasses existing strong baseline diffusion models in terms of FID score and semi-supervised learning methods on image classification on CIFAR-10 and ImageNet.", "questions": "Question:\n* The method seems to be sensitive to CFG. If I understand it correctly, a CFG of 0 means no guidance. While methods such as stable diffusion typically use a large CFG for an optimal trade-off between diversity and fidelity (7.5 by default, which is equivalent to 6.5 in the notation of this paper), the method uses CFG less than 1. Why is this large discrepancy?\n\nThe authors are welcome to address the questions in the weakness section.", "limitations": "As mentioned in the weakness section, the discussions of computational efficiency (e.g., in terms of wall clock time) has been omitted. The authors are encouraged to discuss their compute usage and wall clock time as well as the one for their baseline.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The idea of using Semi-Supervised Learning (SSL) to train diffusion models and then improve SSL is novel. This is analogous to distillation methods that sometimes lead to better student performance than the teacher network.\n* The method boosts the performance of both the diffusion model and the discriminative model, becoming the new state-of-the-art method in both fields.\n* The writing is clear. I enjoy reading the paper.\n* The ablations in the appendix are comprehensive and answered several questions that I have.", "weaknesses": "* The baseline diffusion method in this work that leverages full labels is inherently stronger than LDM. Since DPT does not outperform baseline with full supervision, the claim that \"with one or two labels per class, DPT ... surpassing strong diffusion models with full labels, such as IDDPM, CDM, ADM, and LDM\" is misleading to the readers that DPT, with pseudo-labels, leads to better performance than using full labels.\n* Since the authors use a different setting for diffusion method (e.g., with different backbone), the comparison with previous works (e.g., ADM and LDM) is not fair. The authors are encouraged to either use baseline settings from previous methods or migrate their setting to baseline methods to make sure the comparison is fair.\n* The discussions of computational efficiency (e.g., in terms of wall clock time) has been omitted. Diffusion models have been known to be slow in the inference process, especially when applied in the pixel space rather than the latent space. Note that this method is still valuable even if it's not particularly compute efficient, as long as the compute required is reasonable, as many semi-supervised learning methods are also less efficient compared to supervised learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer JKn2 for the interest and acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.\n\n##  Weakness1: claim may mislead to the readers.\n\nWe appreciate your feedback and understand your concern. We will follow your suggestion and try our best to improve the clarity of the writing in the abstract and introduction of the final version to avoid misleading.\n\n\n\n##  Weakness2: The comparison is not fair.\n\nThank you for your valuable comment. We emphasize that the most direct comparable supervised baseline of DPT are U-ViT (on ImageNet) and EDM (on CIFAR-10) with full labels. The choice of these baseline for their performances are nearly the state-of-the-art. Besides, for completeness, we also compare DPT with a large family of prior work. Extensive results already demonstrate the effectiveness of DPT. We will try our best to clarify this thing in the abstract and introduction of the final version.\n\n\n\n##  Weakness3: The discussions of computational efficiency has been omitted.\n\nWe appreciate your valuable feedback. We have presented the computational costs for DPT in Tab. 9 for ImageNet $256\\times256$ and we discuss the computational efficiency ( which will be added to the Appendix D of our final version) as follows:\n\n**Regarding semi-supervised classification:** The additional time cost over MSN is approximately 201.7%, calculated as $\\frac{Generator ~+ ~DPT ~~extra ~~cost}{Classifier}=\\frac{5813}{2881}=201.7$%. Although DPT requires nearly twice the training time compared to the MSN baseline, it's still **more time-efficient** than other methods like Triple-GAN [d], which demands at least five times the training time of its classifier. Furthermore, our use of the DPM-Solver, a fast sampler, ensures that the sampling time is minimal, even in pixel space.\n\n**As for semi-supervised generation:** The extra time (the training time for the classifier) over U-ViT(generator baseline) is approximately 50%, calculated as $\\frac{Classifier}{Generator}=\\frac{2881}{5760}=50$%. \n\n[d] Li C, Xu K, Zhu J, et al. Triple generative adversarial networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, 44(12): 9629-9640.\n\n\n\n##  Q1: Discrepancy in CFG value between stable diffusion and DPT.\n\nThank you for raising this insightful question. We clarify that the CFG value only depends on the generative component. In our experiments on ImageNet $256\\times256$, the supervised baseline U-ViT achieved its best performance with a CFG of 0.4, as measured under the FID metric. We further investigated this by conducting ablation experiments on CFG, and we found that the optimal CFG value for DPT was also 0.4, consistent with U-ViT. Based on the results above, we suggest using the default cfg of 6.5 when intergrating stable diffusion into DPT.\n\n"}]

Author rebuttal:
# Summary of the revision

We sincerely thank the reviewers for their valuable comments, which help to further improve the quality of our work. We have addressed the detailed comments, and summarize the main revision which will be updated in our final version as follows:

## New results 

- We **add the $FID_{CLIP}$ results in our response PDF (Table 1)**.

## Discussion

- We will add the discussion about the $FID_{CLIP}$ metric.
- We will add the discussion about the choices of resolution and number of labels in Experiment.
- We will add more discussion about the filtering strategy in Appendix.
- We will add the discussion about the upper bound of DPT in Experiment.
- We will add the discussion about computational efficiency and computational overhead in Appendix D.
- We will add the discussion about semi-supervised methods mentioned by Reviewer hJXD.

## Writing

- We will add the $FID_{CLIP}$ metric and the number of generative model parameters in tables about image generation results.
- We will fix the claim about unsupervised diffusion models and the performance of DPT on CIFAR-10.
- We will elaborate more about our novelty in Abstract and Introduction.
- We will place class-level visualization and analysis in the main text.
- We will clarify the comparison with previous works (e.g., ADM and LDM).
- We will remove the use of different bubble areas to indicate the label fraction in Fig.2 (a) 
- We will remove the "underline" and only introduce "bold" for FID in Table 2.


We hope you may find the response satisfactory. Please let us know if you have any further feedback.



Meta-review: This paper proposes a method to combine the benefits of generative diffusion models and semi-supervised learning, and show this procedure benefits both tasks. The reviewers provided positive views on the paper on several important aspects. The proposed method is a simple but effective approach that improves both diffusion models and classifiers, and achieves state-of-the-art results on semi-supervised image classification and generation. The paper is well-written with comprehensive experiments. There are also some concerns expressed such as that the comparisons with prior work may be unfair due to different model sizes and settings, and the paper omits discussion of computational efficiency. Overall, this is an interesting paper that may spark more discussion in bringing the two domains together.
Decision: Accept (spotlight)