Semi-supervised Vision Transformers at Scale
Authors: Zhaowei Cai
Avinash Ravichandran
Paolo Favaro
Manchen Wang
Davide Modolo
Rahul Bhotika
Zhuowen Tu
Stefano Soatto
Venue: NeurIPS 2022 Accept (2022)

Abstract: We study semi-supervised learning (SSL) for vision transformers (ViT), an under-explored topic despite the wide adoption of the ViT architectures to different tasks. To tackle this problem, we use a SSL pipeline, consisting of first un/self-supervised pre-training, followed by supervised fine-tuning, and finally semi-supervised fine-tuning. At the semi-supervised fine-tuning stage, we adopt an exponential moving average (EMA)-Teacher framework instead of the popular FixMatch, since the former is more stable and delivers higher accuracy for semi-supervised vision transformers. In addition, we propose a probabilistic pseudo mixup mechanism to interpolate unlabeled samples and their pseudo labels for improved regularization, which is important for training ViTs with weak inductive bias. Our proposed method, dubbed Semi-ViT, achieves comparable or better performance than the CNN counterparts in the semi-supervised classification setting. Semi-ViT also enjoys the scalability benefits of ViTs that can be readily scaled up to large-size models with increasing accuracy. For example, Semi-ViT-Huge achieves an impressive 80\% top-1 accuracy on ImageNet using only 1\% labels, which is comparable with Inception-v4 using 100\% ImageNet labels. The code is available at https://github.com/amazon-science/semi-vit.

Semi-supervised Vision Transformers at Scale
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang,
Davide Modolo, Rahul Bhotika, Zhuowen Tu, Stefano Soatto
AWS AI Labs
(zhaoweic, ra mancoer dmodolo, ztu, soattos] (artoosssaamzza.. @amazon com
Abstract
We study semi-supervised learning (SSL) for vision transformers (ViT), an under- explored topic despite the wide adoption of the ViT architecture to different tasks. tackle this problem, we use a SSL pipeline, consisting of first suprrvees pre-training, followed by supervised fine-tuning, and finally semi-supervised fine- tuning. the semi-supervised fine-tuning stage, we adopt an exponential moving average (EMc)-Tcacher Teacher framework instead of the popular FixMatch, since the former is more stable and delivers higher accuracy for semi semi-supervised vision transformers. In addition, propose probabilistic pseudo mixup mechanism interpolate unlabeled samples and their pseudo labels for improved regularization, which is important for training VITs with weak inductive bias. Our proposed method, dubbed Semi- Vi7, achieves comparable or better performance than the CNN counterparts in the semi-supervised classification setting. Semi-ViT also enjoys the scalability benefits of ViTs that can be readily scaled up to large- size models with increasing accuracy. For example, an impressive 80% top-1 accuracy on ImageNet using only 1 % labels, which is comparable with Inception-v4 using 100% ImageNet labels. The code available at
achieves
com/ amazon sciencee semi-eemiivieeee -vit
Introduction
ImageNet 1% Labels
ImageNet 10% Labels
78
life wottle--
top asouraty
top-1 aocur any
3 &
Saine- Gomu
23
Semi-VIT
88
Semi-ViT
SimCLR Rv2
SimCLRv2
PAWS
83
PAWS
FixMatch
50
numbor of parameters (M)
number paramoters (M)
18 hbbls
10% lobels
100% libes
(a)
(b)
Figure 1: (a) and (b) are the comparisons of Semi- VIT with the state-of-the-art SSL algorithms different model scales, and (c) is the comparison with the state-of-the--ar supervised models.
In the past few years, Vision Transformers (VIT) 18 which adapt the transformer architecture
64 to the visual domain, have achieved remarkable progress in supervised learning
un/self-supervised learning [16] [2 25 and many other computer vision tasks [1]
19
|I
58
(with
architecture modifications). However, ViTs have yet to show the same advantage semi-supers semi-supervised learning (SSL), where only a small subset of the training data is labeled, problem in the middle
between supervised and mn/self-supervii learning. Although several recent methods in SSL have
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the transfer of these methods from Convolutional Neural Networks (CNN) to ViT architectures has yet to show much promise. For example, as discussed in 688, the direct application of Fix [55). one of the most popular SSL methods, to ViT leads to inferior performance (about 10 points worse) than when used with a CNN architecture. The challenge could be potentially caused by the fact that ViTs are known to require more data for training and to have a weaker inductive bias than CNNs [8 However, in this paper we show that semi-supervised ViTs can outperform the CNN counterparts when trained properly, suggesting promising potential to advance SSL beyond CNN architectures.
significantly advanced the field 39
62
55
70
I0C
53
To achieve that, we use the following SSL pipeline: 1) un/self-suppe supervised pre-training on all data (both labeled and unlabeled), followed by 2) supervised fine-tuning only on labeled data, and finally 3) semi- sumi-supe fine-tuning on all data. This pipeline is stable and helps reduce the sensitivity of hyperparameter tuning when training ViTs for SSL. in our experiments. At the final stage of semi-supervised fine-tuning, we adopt the EMA-Teacher framework 622 10.. an improved version of the popular FixMatch (55). Unlike FixMatch, which often fails to converge when training semi- supervised ViT, the EMA-Teacher shows more stable training behaviors and better performance. In addition, we propose probabilistic pseudo mixup for pseudo-labeling based SSL methods, a method that interpolates the unlabeled samples coupled with pseudo labels for enhanced regularization. In the standard mixup [7 the mixup ratio is randomly sampled from a Beta distribution. In contrast, in the probabilistic pseudo mixup the ratio depends on the respective confidence of two mixed-up samples, such that the sample with higher confidence will weigh more in the final interpolated sample. This new data augmentation technique brings negneiliii gains since ViT has weak inductive bias, especially for scenarios where the training is more difficult, egg., without un/self -n/selff-uuerriiee pre-training
data regimes with very few labeled samples (e.g., 1% labels). We call our method Semi- VIT. Notice that Semi- ViT is built on exactly the same design of ViTs (i.e., there are neither additional parameters nor architectural changes).
Semi-ViT achieves promising results on several fronts (Figure Tp. 1) For the first time, we show
that pure ViTs can reach comparable or better accuracy than CNNs on SSI F 2) Semi-ViT can be readily scaled up under the SSL setting. This is illustrated in Figure (a)
(a) and (b) on ViT architectures
at different scales, ranging from ViT-Small to ViT-Huge, and Semi-Vis outperforms the prior art such as SimCLRv2 [T.. 3) Semi-Vit has shown the potential for a substantial reduction of labeling cost. For example, as seen in Figure (c), Semi- with 1 % (10%) ImageNet labels achieves comparable performance of a fully- -uppe-suee Inception-v4 (59) (ConyNeXt-L (45). This implies 100x (10x) reduction in human annotation cost. 4) Semi- achieves the state-of-the-arr SSL results on ImageNet, e.g., 80.0% (84.3%) top-1 accuracy with only 1% (10%) labels. In addition, the substantial boost in performance by Semi-ViT is not isolated on ImageNet: we find an increase of 13%-21% % (7%-10%) top-1 accuracy with 1% (10%) labels over the supervised fine-tuning baselines,
for other datasets including Food-101 iNaturalist 30 and GoogleLandmark (2
2 Semi-supervised Vision Transformers
2.1 Pipeline
Some pipelines for semi-supervised learning exist in the literature. For example: 1) the model is directly trained from scratch using SSL techniques, e.g., FixMatch [55: 2) the model is un/self- supervised pretrained first and finetuned on labeled data later 166 14 22]: 3) the model is self- supervised pretrained first and then finetuned via semi-supervised learning
both labeled and unlabeled data [C]. In this paper, we instead adopt the following pipeline: first, optional self- supervised pre-training
all data without using any labels; next, standard supervised f fine-tuning on available labeled data; and finally, semi cemi-superiised fine-tuning on both labeled and unlabeled data.
This procedure is similar to [5) with the difference that they knowledge distillation final stage. We find that this training pipeline trains semi-supervised vision transformers in a stable manner and achieves promising results, with possibly less hyperparameter tuning.
(29 in their
Although 68 was the first to use transformer architecture for SSL, it does so by combining both CNN and VIT architectures, and requires to use CNN the teacher to produce pseudo labels.

---

CE loss
CE loss
pseudo-label
pseudo-label
logit_s
logit_w
logit ss
logit_ w
θ = shared
student ViT
0'
teacher ViT
VIT
ViT
EMA
A
1 X_W
x_s
X W
X_s
(a) FixMatch
(b) EMA-Teacher
Figure 2: Comparison between Fix.Match (a) and Teacher (b). x//xw is the strongly/weakly augmented view of a sample x, and are the model parameters.
2.2 EMA-Teacher Framework
FixMatch
emerged popular SSL method in the past few years. As discussed in [T it can be interpreted as student-tcacher framework, where the student and teacher models are identical, seen in Figure 2 (a). However, FixMatch has unexpected behaviors, especially when the model incorporates batch normalization (BN) 35 Although ViT uses Layer Normalization (LN) instead of BN normalization, counterparts and often does not converge. This phenomenon was also observed [68] A potential reason this is that the student and the teacher models are identical in FixMatch, which could easily lead to model collapse 22 22 This instability of the identical student-teacher framework has also been observed in other areas, e.g. semi-supervised speech recognition 42
15
still found that the FixMatch with VIT underperforms the CNN
48
28
As suggested
in [1O]. the RMA-Teacher (shown in Figure (b)) is an improved version of the FixMatch, thus we adopt for our Semi-Vin ViT. In the EMA- EMA-Teacher framework, the teacher parameters 0' are updated by the exponential moving average (EMA) from the student parameters 8,
:= m0 (1 m)00
where the momentum decay m is a number close to 1, e.g., 0.9999. The student parameters are updated by standard learning optimization, e.g., SGD AdamW 46. The other components exactly the same as the Fix as seen in Figure 2
This temporal weight averaging can stabilize
the training trajectories 3 36 and avoids the model collapse issue 226 22) Our experiments
also show this EMA-Teacher framework has better results and more stable training behaviors than FixMatch when training SeTi-ViT.
2.3 Semi-supervised Learning Formulation
the MMA-Teacher framework, there are both labeled and unlabeled samples in a minibatch during training. The loss on the labeled samples {(a), vf) No is the standard cross- cross-entropy loss, L = N:
CE 1.E,,E v.). For an unlabeled sample x" E {x(") N. weak and strong augmentation are applied to it, generating ".." and "., respectively. The weak augmented ",, is forwarded through the teacher network, and outputs the probabilities over classes, p = "a f'). Then the pseudo label is produced by ÿ arg max, Pe with its associated confidence o = max Pe- The pseudo label with confidence higher than confidence threshold is then used to supervise the learning of the student on the strong augmented sample x u..
N I oo 2|C(( ,))) N.
L.
(2)
where [:] the indicator function. The overall loss is Li + pha, where μ the trade-off weight. Note that only the pseudo labels with confidence higher than a threshold contribute to the final loss;

---

8:
25
X6
8:
K5
8:
X1
A5
8:
Xg
X5
24
B:
X2
X5
B:
X3
X4
8,
Axg
8,
Ax,
A0
9)
B:
1
2
X3
24
83
26
8
2g
B:
X2
33
36
2g
B:
A5
8
Xg
(a) Pseudo Mixup
(b) Pseudo Mixup+
(c) Probabilistic Pseudo Mixup
Figure 3: Different variations mixup on unlabeled data. The red samples are the ones passing the confidence threshold, but not the blue samples.
the others are instead not used. The philosophy behind this filtering is that the pseudo labels with low confidence are noisier and could hijack the SSI. training.
3
Probabilistic Pseudo Mixup
3.1
Mixup
Mixup
7
performs convex combinations of pairs samples and their labels,
f = Ar; + (1 A)z3
ÿ xys (1 X)yj-
where the mixup ratio X Beta a) E [0, 1). for a E (0, oc). The samples are mixed-up usually in single minibatch during training. Given minibatch B and its shuffled version B, the mixed-up minibatch is B XB + (1 A)B, where À could be either batch-wise element-wise Due the nature of weak inductive bias, VIT more data hungry than CNN, thus effective data augmentation, egg. mixup, is critical for training fully-supervised VIT I8 63
This also applies to Semi-Vif since it inherits the nature of weak inductive bias from VIT. Although it is standard to use mixup in supervised learning, how employ it under pseudo-labeling based SSL framework, e.g., the EMAATeacher is still unclear yet, and we are going to discuss it next.
1 K
3.2 Pseudo Mixup
given an unlabeled sample and its pseudo label (z", ú), only when confidence o not smaller than the confidence threshold т, it will contribute to the loss Lu. as seen in 2 According to their confidence scores, the unlabeled
Under the pseudo-labeling based SSL framework
40 55 5
10
minibatch 8" can be grouped into clean subset Th {xx 0:)lo; 2 -} and a noisy subset Bu 8 1².. One straightforward solution to apply mixup the full unlabeled minibatch 8", with no differentiation between clean and noisy samples, denoted as pseudo mixup, as show in Figure 3 (a). After the pseudo mixup, still only the samples in 8 contribute to the loss, and the samples in 8 are abandoned. In this way, the mixup operation is more than just data augmentation. In fact, sample in T will also contribute to the final loss if it is mixed-up with a sample in 8". As result, it could involve a substantial number of noisy samples into the loss calculation due to the randomness, which, however, is against the philosophy of pseudo-labeling, Since only the clean subset T contributes to the final loss, another choice is to use mixup only on B3 denoted as pseudo mixpp+, as shown in Figure (b). In this way, no sample in the noisy subset 15"h will affect the training.
3.3 Probabilistic Pseudo Mixup
Although the samples in th are noisy, they still carry some useful information for the model to learn. The pseudo mixup above can somehow leverage those information by blending the noisy and clean pseudo samples together. However, the problem is the mixup ratio is randomly generated from a Beta distribution, which does not depend on the confidence of each sample. This ideal. For example, when two samples mixed-up. the sample with higher confidence should have higher mixup ratio,

---

Table 1: Semi- VIT results comparing with line-tuning. The models are self-pretrained by MAE 22
Model
Param
Method
1%
10%
100%
finetune
57.4
73.7
83.7
Base
86M
Semi-ViT
71.0
79.7
finetune
67.1
79.2
86.0
ViT-Large
307M
Semi-Vit ViT
77.3
83.3
finetune
71.5
81.4
86.9
ViT-Huge
632M
Semi-ViT
80.0
84.3
Table 2: The comparison between the FixMatch and the EMA-Teacher. Xmeans the training is failed with accuracy close to 0.
Model
Pretrained
Method
1%
10%
FixMatch
X
ViT-Small
None
A-eecchher
65.6
FixMatch
x
ViT-Base
None
EMA- Teacher
68.
FixMatch
X
74.8
ViT-Base
MAE
65.3
78.1
such that it can weigh more in the final loss. Motivated by this intuition, we propose probabilistic pseudo mixup (Figure 3 (c)), where the mixup ratio À reflects the sample confidence,
X; 0:/ 0j).
(4)
Also, the confidence score at is updated after the mixup operation
0 max oj),
(5)
because the confidence score should align with the majority of the image content. The final clean subset _ = { o''' 2 r} will contribute the final loss. Probabilistic pseudo mixup can enhance regularization, leverage information from all samples, even the noisy ones, and not violate the philosophy of pseudo labeling the same time. It effectively alleviate the issue of weak inductive bias of Bemi-ViT and bring substantial gains, as will be shown our experiments.
Experiments
We evaluate Semi-ViT mainly ImageNet, which consists of 1..28M training and 50K validation images. We sample 10%/1% % labels from the ImageNet training set for the semi-supervised evaluation. We study both scenarios: with and without supervupe pre-training. Without self- pretraeetaiinng. we only evaluate on 10% labels, since learning from scratch on 1% labels is very difficult. When self- celf-pretrained, MAE 2 is mainly used, and we directly use their pretrained models. All learning is optimized with Adam W (46), using cosine learning rate schedule, with weight decay of 0.05. The default momentum decay m of ([ 0.9999. minibatch, N₂ 5N,, and the loss trade-off' 74 as in the implementation of (69
5. The mixup is combination of mixup 75 and Cutmix More details can be found in the appendix.
4.1 Semi-Vii Results
When the model is self-pretrained by MAR
25 we first evaluate the fine-tuning performance of MAE on the labeled data only, as the common practice in celf/un-supervieed learning literature
126 14 22], with results shown in Table [ This already leads to strong semi-supervised basclines, egg., 81.4 top-1 accuracy for ViT-Huge on 10% labels, indicating that MAE strong self-supervised learning technique. However, Semi- VIT has additional significant improvements over the strong baselines for all models, e.g., 8.5- points for 1% labels and 2.9-6.0 points for 10% labels. The fine-tuning results on 100% data are provided as upper-bounde for our Semi-ViT, and their gaps to Semi-ViT are small, e.g., 4.0/2.7/2.6 points for ViT-Base/I Largase//arg//Hugge on 10% labels. An interesting observation is that the larger model is more effective for smaller number of labels, which is consistent with the observations 26.3/18 1..33/8../1..44 points for ViT VTT-Bas//Lrrc/HHuge, which are decreasing. The observation Semi-ViT
LS For example, the fine-tuning gaps between 1% and 100% labels are
5

---

Table 3: The comparison among different mixup variations.
Model
Pretrained
Mixup
1%
10%
EMA-Teacher
65.6
Pscudo Mixup
68.3
ViT-Small
None
Pseudo Mixup+
68.8
ProbPseudo Mixup
70.9
AMA-TTaahhr
68.9
Pscudo Mixup
71.6
ViT-Base
None
Pseudo Mixup+
72.1
ProbPseudo Mixup
73.5
EMA-Teacher
65.3
78.1
Pseudo Mixup
69.5
78.3
ViT-Basc
MAE
Pseudo Mixup+
78.7
ProbPscudo Mixup
71.0
79.7
Table 4: The ablation on the confidence threshold.
Method (ViT-Base)
label
r =
0.3
7
0.4
7
0.5
7
0.6
7 0.7
0.8
0.9
J.MA-Teacher
1%
63.1
64.4
64.6
65.1
65.3
65.1
64.4
63.4
EMA-Tcacher
10%
75.4
76.7
77.2
77.7
77.9
78.1
78.2
77.9
Semi VIT
1%
70.8
71.3
71.3
71.0
70.4
Semi- VIT
10%
79.4
79.5
79.7
79.7
79.6
79.4
79.0
77.2
results is similar, e.g., 12.7/8.7/6.9 points to their upper-bounds on 1 1% labels. These results have shown that vision transformers can also perform very well in semi-supervised learning, as well as supervised learning and un/self-supen snuself-superrvieed learning.
4.2 Ablation Studies
We ablate different factors of Semi-ViT in this section.
FixMatch v.s. EMA-Teacher is compared in Table These experiments do not use the pseudo mixup techniques of Set When the model is not pelf-pretrained, the training of the FixMatch unstable and often failed. When the model is self- self-pretrained, the FixMatch training becomes stable, and starts achieve reasonable results, egg. 74.8 for ViT-Base on 10% labels, which already better than the prior art on ResNet-50, e.g., 73.9 of MPL. 53 and 74.0 of EMAN [TO] But it is only 1.1 points higher than the fine-tuning baseline Table I indicating the FixMatch is not an effective SSL framework for VIT. But the EMe-Teacher achieves much better results, 3.3 points of improvement over FixMatch when self peff-prtrrinee. Even without self self-pretraining, the EMA-Teacher still achieve satisfactory performance, while FixMatch fails.
Different mixup variations on unlabeled data are compared in Table 3 Note that the standard mixup with the implementation of 669 is used the laheled data usual. The EMA-Teacher does not use any mixup mechanism on the unlabeled data, serving as baselines here. When pseudo mixup of Figure (a) is applied the unlabeled data, the performance usually has some substantial gains over the EMA-Teacher baselines, especially for the scenarios where the training more difficult, e.g., without self-pretraining or 1% labels. This shows the importance of using mixup on the unlabeled data for an improved regularization. However, as discussed in Section pseudo mixup could involve many noisy samples into training. On the other hand, pseudo mixup+ of Figure 3 (b) can increase the performance of pseudo mixup constantly, by about 0.5 points, showing that removing those noisy samples does help. In addition, probabilistic pseudo mixup of Figure (c) can further improve the performance of pseudo mixup+ by 1-2 points in all cases. These results imply that those noisy samples do carry some useful information for SSL training, but their weights should be suppressed especially when their confidence scores are low. This data augmentation technique also effectively alleviates the training difficulty of supevvuper vision transformers with weak inductive bias.
Probabilistic Pseudo Mixup
Effect Confidence Threshold ablate the effect of the confidence threshold in Table We find that Semi-Vii is quite robust to the low confidence thresholds. One possible reason is that

---

Table 5: The ablation on the momentum decay of exponential moving average.
Method (ViT-Base)
label
m
0
m
0.9
m
0.99
m
0.999
m 0..999
m
0.99999
IMMA-Teache
1%
x
23.5
49.3
63.1
65.3
59.7
10%
74.8
75.3
76.4
77.2
78.1
77.9
Semi- ViT
69.3
69.7
71.1
71.6
71.0
63.1
Semi-Vil VIT
10%
79.5
79.5
79.6
79.8
79.7
79.0
Table The ablation on supervised fine-tuning.
Method (VTT-Bar ViT-Base)
label
epochs=0
epochs=10
epochs=50
epochs=100
epochs=200
Supervised-Vi ViT
1%
24.7
53.6
57.4
56.9
Supervised-Vi VIT
10%
66.3
72.9
73.7
73.2
E.AA-Teacher
1%
62.7
62.5
60.9
65.3
66.9
E.MA-Teacher
10%
76.5
74.2
77.7
78.1
78.2
Semi- VIT
1%
69.7
69.8
70.4
71.0
70.9
Semi-ViT
10%
79.3
79.4
79.6
79.7
79.6
Semi-ViT uses probabilistic pseudo mixup. When T is low, the low-confidence samples will not hijack the training since their contributions depend on their confidence scores. These imply that the hyperparameter T can possibly be removed (r = 0) Semi-ViT, But the EMA-Teacher has a drop of 2-3 points when the confidence threshold is removed. The final choices of for different Semi- CiT models are shown Table 13 in the appendix.
Effeet of Momentum Decay Table 5 shows the effect of momentum decay m the EMA teacher. Note that when m = O, the frameworks is reduced to the FixMatch. We can find Semi- ViT is robust to m. For 10% labels, Semi-Vit has very minor changes when m decreases 0, but the EMA-Teacher has a drop of 3.3 points. For 1% labels, the training is more challenging, and the choice of m becomes more important. In this case, Semi-ViT has a drop of 2.3 points from m 0.999 to m but the EMA--Teache could fail when m decreases to O. The robustness of Semi-ViT to momentum decay m also attributed to the use of probabilistic pseudo mixup.
Effect of Self-pretraining The self- celf-pretraining of MAE 2 has substantial boost in performance, as seen in Table 3 For ViT-Base, MAE helps to improve by 6.2 and 9.2 points for the Teache with and without probabilistic pseudo mixup, respectively. In addition, helps to train the models in more challenging scenarios, e.g., 1% labels. Without self-pretraining. the training fails to deliver good results 1 1% labels. Notice that, even without pre-training, our Semi-Vit ViT ("ProbPseudo Mixup" in Table 5 also achieves slightly better performance than the CNN counterparts: 70.9 of Semi-Vit ViT--mall v.s. 67.1 of MesMatcch-Reseetti 69.2 (55) of EMAN-ResNet50
T0
when trained
from scratch for 100 epochs.
Effect Supervised Fine-tuning is ablated in Table 6 by varying the number of supervised fine-tuning epochs. The rows of "Supervised-Viie are the numbers of supervised fine-tuning, where the latter remi-supereised finetuning begins. Semi- VIT is still robust to the length of supervised fine-tuning, where the accuracy decrease is 0.4 (1.3) for 10% (1%) labels when supervised fine-tuning removed. However, the performance the -eacher decreases 1.7 (4.2) points. These show sufficient supervised fine-tuning does stabilize the training procedure, especially for less robust framework, e.g., the hurt the performance if it is not sufficient (e.g., epochs=10)
However, notice that supervised fine-tuning could sometimes
www.g..ppccs=1000 for the
Other Self- Self-pretraining Techniques Beyond MAE, we also experiment on other self-pretraining techniques, including MoCo-v3 [6 and DINO 12 in Table By comparing the fine-tumi fine-tuning results, DINO is close to MoCo-v3 for ViT-Base, but much better for ViT-Small, and both of them are better than MAE for ViT-Base suggesting that DINO could be better self- pelf-pretraining technique for smaller scales of ViT models. On top of the strong fine-tume nieettnnng baselines, the semi-supervised fine-tuning. using the EMA- -eacher, still has nontrivial improvements for both DINO and MoCo-v3, e.g., 5.8 (2.1) points on 1% (10%) labels for DINO- MINO-ViT-Baa ViT-Base. In addition, the probabilistic pseudo mixup can further improve over the EMA-Teacher, independent the self self-pretrane pretraining algorithms. And the final Semi- ViT--iii-BB of DINO is 2.1 (0.5) points better than that of MAE on 1 1% (10%) labels.
7

---

Table 7: Semi-Vit results with other self- self-pretraining techniques.
Model
Pretrained
Method
1%
10%
finetune
51.2
69.1
ViT-Small
MoCo-v3 16
EMA-Teacher
61.9
72.3
+ProbPseudo Mixup
64.7
72.9
finetune
58.7
73.9
ViT-Small
DINO 12
RMA-Teacher
66.3
76.3
ProobPeeudo Mixup
68.0
77.1
finetune
66.3
74.5
ViT-Base
MoCo-v3 16
EMA-Teacher
68.9
77.7
+ ProbPPseudo Mixup
72.3
79.2
finetune
65.0
76.0
ViT-Base
DINO
Teacher
70.8
78.1
+ +ProbPscudo Mixup
73.1
80.2
Table 8: The results on ConvNeXt 45
Model
Upper-bound
Method
10%
supervised
61.2
ConvNeXt-T
80.7
EMA-Teacher
70.4
+ +ProbPseudo Mixup
74.1
supervised
64.1
ConvNeXt-S
81.4
EMA-Teacher
71.7
+ +ProbPscudo Mixup
75.1
Other Network Architectures Although in this paper we mainly focus on ViT architectures, the proposed probabilistic pseudo mixup is not limited them. We also try it for CNN architectures, e.g., ResNet (27]. However, we find the direct use of the standard mixup does not improve fully-supervised ResNet performance, so will the probabilistic pseudo mixup for its SSL setting. Instead, evaluate it on the recently proposed ConvNeXt
45 which uses mixup for improved results. Since the goal is fully reproduce the results of 45 all models are trained only for 100 epochs, including the supervised apper-bounde The results in Table & demonstrate that probabilistic pseudo mixup is not limited ViT, but also CNN architectures, e.g., with improvements of 3-4 points, suggesting can be well generalized.
4.3 Comparison with State-of-the-Art
Semi-ViT compared with the state-of-the-as semi-supervised learning algorithms in Table When the model capacity is close, our Semi- ViT has shown much better results than the prior art, e.g. MPL-RN-50
3 v.s. Semi- Semi-ViT-Smaal, CowMix-RN152 [20] v.s. Semi-Vin ViT--ase. S4L-RN50-
4x v.s. Semi- LargL and BinCLRR2+KD-RN15222 RN152-2 x--K [S] v.s. Semi- ViT- The only transformer based SSL method is SemiFormer [68. but requires
CNN as the teacher model and blend convolution and transformer modules together for good performance. However, Semi-Vit VIT is pure ViT based, without any additional parameters and architecture changes, and the Semi-ViT-Sme support that Semi-ViT does advance the state-of-the-art of semi-supervised learning.
model is already better than SemiFormer (77.1 v.s. 75.5). These comparisons
Scalability is an advantage of VIT. and we compare the scalability of Semi- VIT with previous works in Figure I (a) and (b). The comparison has shown that Semi-ViT can achieve better trade-off between model capacity and accuracy and can be scaled up more effectively than the prior art, SimCLRv2 [55]. For example, SimCLRv2 and PAWS [2] scale up the model usually in terms of network depth and width, and they seem to saturate when the model of medium size, e.g., around 300M parameters, but Semi- ViT continues to improve steadily beyond that point.
Semi-Vii ViT also compared with the supervised state-of-the-art in Table TO Our Semi-ViT-Huge is comparable with Inception-v4 [59] but with 100x annotation cost reduction, and comparable with Con 45 (better than Swin-B 44] but with 10x annotation cost reduction. These comparisons imply that Semi- ViT has great potential for labeling cost reduction.

---

Table 9: The comparison with the state-of-the-art SSL models.
Method
Architecture
Param
1%
10%
UDA
ResNet-50
26M
68.8
Fixtatchh
65
ResNet-50
26M
71.5
ResNet-50 (4x)
375M
73.2
CNN
MPL
ResNet-50
26M
73.9
CowMix
20
ResNet-152
60M
73,9
EMAN
10
ResNet-50
26M
63.0
74.0
PAWS 2
ResNet-50
26M
66.5
75.5
SimCLRy2+KD
15
RN152 (3x+SK)
794M
76.6
80.9
DINO 12
ViT-Small
22M
64.5
72.2
Transformer
SemiFrrmer
68
ViT-S+Conv
42M
75.5
Semi-Vis ViT (ours)
ViT-Small
22M
68.0
77.1
Semi-ViT (ours)
ViT-Base
86M
71.0
79.7
Semi-ViT (ours)
ViT-Large
307M
77.3
83.3
Semi-ViT (ours)
ViT-Huge
632M
80.0
84.3
Table 10: The comparison with the state-of-the-art fully supervised models.
Model
Param
Data
top-1
top-5
ResNet-50 p
26M
ImageNet
76.0
93.0
ResNet- 152
60M
ImageNet
77.8
93.8
DenseNet-264
32
34M
ImageNet
77.9
93.9
Inception-v3
24M
ImageNet
78.8
CNN
Inception-v4
59
48M
ImageNet
80.0
95.0
ResNeXt-101
72
84M
ImageNet
80.9
95.6
SENct-154 R1I
146M
ImageNet
81.3
95.5
ConvNeett-L E51
198M
ImageNet
84.3
EfficientNNel-.2 [6]]
480M
ImageNet
85.5
97.5
ViT-Huge [N]
632M
JFT+ImageNet
88.6
Tansformer
DeiT-B [3
86M
ImageNet
81.8
Swin-B 44]
88M
ImageNet
83.3
MAE-VTT-Huge
632M
ImageNet
86.9
Semi-Vis ViT--uge (ours)
632M
1%ImageNet
80.0
93.1
Soni-ViT-Huge (ours)
632M
10% ImageNet
84.3
96.6
4.4 Other Datasets
The generalization of Semi- ViT is evaluated on datasets including Food-101 1 iNaturalist 30 and GoogleLandmark [52]. Since these datasets are beyond ImageNet, assume that the ImageNet dataset is available and the model already supervised pretrained on ImageNet, and then the model finetuned to different target datasets with few labels. The results shown in Table II On these datasets, our Semi- ViT can improve over the fine-tuni fine-tuning baselines by 13-21 (7-10) points on 1 1% (10%) labels. Note that Food- 101, Semi- ViT 1% (10%) labels is close to fine-tuning baseline on 10% (100%) labels, i.e., 82.1 v.s. 84.5 (91.3 v.s. 93.1), indicating that using Semi- VIT can help to annotation costs by about 10 times this dataset.
5 Related Work
The recent works can be roughly and pseudo-labeling based 40 153 10. Consistency-basee methods usually add some noise to the input or the model, and then enforce their feature probability outputs be consistent. For example, to construct two outputs for later consistency regularization, II-model 339 adds noise to the model weights using dropout [57]. Mean-teacher (62] builds a teacher model by EMA updated from the student model, and UDA 70 applies a weak and strong data augmentation to the input. On the other hand, the idea of pseudo-labeling or self-training can traced back to 49), which uses model predictions as hard pseudo labels to guide the learning on unlabeled data. This idea becomes popular in SSL recently 30
Semi-supervised learning has long history of research 78 13
clustered into two groups, consistency-based 33
55
551
5K
10
and some theoretical explanations are available (76 24 In the offline
pseudo labeling
40
71,
the model used to generate pscudo labels is usually frozen updated once

---

Table 11: The Semi-ViT- results on other datasets.
Dataset
# train/test
# class
Method
1%
10%
100%
Finctune
60.9
84.5
93.1
Food-101
75.7K/25.2K
101
Semi- ViT
82.1
91.3
Finetune
19.6
57.3
81.2
iNaturalist
265K/3K
1010
Semi-Vir
32.3
67.7
Finetune
45.3
74.0
91.5
GoogleLandmark 92
200K/15.6K
256
Semi-ViT
61.0
81.0
in a while during training, egg,, at the end of every training epoch, but for online pseudo-labeling the teacher model is updated continuously along with the student. Beyond classification, useudo-labeling has also achieved promising progress in more challenging tasks, e.g., object detection
[55] [1O
[56 33
67
Our Semi- VIT also falls into the category of online pseudo-labeling
Mixup 7 an effective data augmentation technique, which interpolates the input samples and their labels linearly and performs vicinal risk minimization. It has been successfully used in image classification and some other domains, e-g. generative adversarial networks 47
sentence classifica-
tion 2 etc. Other variants have also been developed, e.g., Manifold Mixup 65 that mixes up in the feature space or CutMix 74 which cuts a patch from one image and pastes it into another one. Mixup has also been successfully adopted in self-supervised learning
77 41 and semi-supervised learning 61. Although 7 66 O also use mixup for SSL, they have differences with our probabilistic pseudo mixup: 1) they are consistency-bedt -based SSL framework, but ours is aseudo-labeling based; 2) their mixup ratio is random sampled, but ours depends on the pseudo label confidence; 3) they have only shown successes small CNN architectures and small datasets, e.g., CIFAR [R] and SVHN (5T. but our successes are built on various scales of transformer architectures and large-scale datasets, e.g., ImageNet (54), INaturalist 30, Googlel Gonglel.andmark [52] etc.
7 66
6 Conclusion
In this paper, we propose Simi-ViT for vision transformers based semi-supervised learning. This is the first time that pure vision transformers can achieve promising results semi-supervised learning and even surpass the previous best CNN based counterparts by a large margin. In addition, Semi-ViT inherits the scalable benefits from VIT, and the larger model leads smaller gap to the fully supervised upper-bounds. This has shown to be promising direction for semi-supervised learning. And the advantages of ViT can be well generalized other datasets, suggesting potentially broader impacts. We hope these promising results could encourage more efforts in semi- eemi-supervised vision transformers.
Limitations Our paper only considers the standard semi-supervised classification settings where the full dataset, e.g., ImageNet, is downsampled smaller scales, and not the advanced settings where full ImageNet is used as labeled and additional data, e.g., ImageNet-2IK, is used as unlabeled. And we have only evaluated our approach on the classification task. is unclear whether the same conclusion holds in the case of more advanced classification setting and more challenging tasks, e.g. detection or segmentation.
Potential Negative Social Impacts Semi-ViT has shown that strong models can be obtained with only few labels, e.g., 1%. This increases the good AI models accessibility anyone, which could potentially lead to their inappropriate use.
References
[1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: video vision transformer. In ICCV, pages 6816-6826, IEEE, 2021.
[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael G. Rabbat. Semi-supervised learning of visual features by naraparamt predicting view assignments with support samples. ICCV, pages 8423-8432. IEEF, 2021.
[3] Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consistent explanations of unlabeled data: Why you should average. In ICLR. OpenReview.net, 2019.
10

---

[4] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450 2016.
[5] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 1rXiv:2106.0822 2021.
[6] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR. OpenReview.net, 2020.
[7] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, pages 5050-5060, 2019.
[8] Lucas Beyer, Xiaohua Zhai, Avital Oliver, and Alexander Kolesnikov. S4L: self-supervised semi- supervied learning. In ICCV, pages 1476-1485, IEEE, 2019.
[9] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. ECCV, volume 8694 of Lecture Notes Computer Science, pages 446 461. Springer, 2014.
[10] Zhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Zhuowen Tu, and Stefano Soatto. Exponential moving average normalization for self-supervised and semi-supervised learning. In CVPR, pages 194-203, 2021.
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, volume 12346 Lecture Notes in Computer Science, pages 213-229. Springer, 2020.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supe vision transformers. In ICCV, pages 9630-9640. IEEE, 2021.
[13] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi- superviid learning (chapelle, al., eds.; 2006). IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi. and Geoffrey E. Hinton. A simple framework for contrastive learning visual representations. ICML, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMIR, 2020.
[15] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self- supervised models are strong superviee Iearners. In NeurlPS, 2020.
[16] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pages 9620-9629. IEEE, 2021.
[17] Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Qune Le. Randaugment: Practical automated data augmentation with reduced search space. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, NeurlPS, 2020.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[19] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, pages 6804-6815. TEEF, 2021.
[20] Geoff French, Avital Oliver, and Tim Salimans. Milking cowmask for semi-supervised image classification. In VISIGRAPP, pages 75 24. SCITEPRESS, 2022.
[21] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1776.02677, 2017.
[22] Jean- Bant-Battien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos. and Michal Valko. Bootstrap your own latent- A new approach self- supervised learning. In NeurlPS, 2020.
[23] Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In AAAI, pages 4044-4051, AAAI Press, 2020.
[24] Haiyun He, Hanshu Yan, and Vincent YF Tan. information-thcoreticc characterization of the generalization error for iterative semi-supervised learning. Journal of Machine Learning Research, 23:1-52, 2022.
[25] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
11

---

[26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning, In CVPR, pages 9726-9735. Computer Vision Foundation IEEE, 2020.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. IEEE Computer Society, 2016.
[28] Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori. Momentum pseudo- for semi-supervised speech recognition. In Interspeeck, pages 726-730. ISCA, 2021.
[29] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015.
[30] Grant Van Hor, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2017 dataset. CoRR, abs/ htt//1777.066e 006622,64 2017.
[31] Hu, Li Shen, and Gang Sun. Squeeze-and excitttiittioon networks. In CVPR. pages 7132-7141. Computer
Vision Foundation LEEE Computer Society, 2018.
[32] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 2261-2269. IEEE Computer Society, 2017.
[33] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sobe, and Max Welling, editors, ECCV, volume 9908 of Lecture Notes in Computer Science, pages 646-661. Springer, 2016.
[34] H. J. Scudder III. Probability of error of some adaptive cnttern-recognition machines. IEEE Trans. Inf. Theory, 11(3):33--371 1965.
[35] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 448-456. JoLR.org. 2015.
[36] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads wider optima and better generalization. In UAI, pages 876-885, AUAT Press, 2018
[37] Yannis Kalantidis, Mert Bülent Sariyildiz, Noć Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive leammin. In NeurlPS, 2020.
[38] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[39] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
[40] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges representation learning, ICML, volume 3, 2013.
[41] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-l Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain- agnostic strategy for contrastive representation learning. In ICI.R. OpenReview.net, 2021.
[42] Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert. slimipl: monguage-model-fiee iterative lscudo-labeling. In Interspeech, pages 741 ISCA, 2021.
[43] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Wen Kuo, Kan Chen, Peizhao Zhang. Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In ICLR. OpenReview.net, 2021.
[44] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992-10002. IEAR, 2021.
[45] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint NXN::22011.0545 2022.
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR. OpenReview.net, 2019.
[47] Thomas Lucas, Corentin Tallec, Yann Ollivier, and Jakoh Verheek. Mixed hatches and symmetric discriminators for GAN training. In ICMI,, volume 80 of Proceedings of Machine Learning Research, pages 2850-2859, PMLR, 2018
[48] Vimal Manohar, Tatiana Likhomanenko, Qiantong Xu, Wei-Ning Hsu, Ronan Collobert, Yatharth Saraf, Geoffrey Zweig, and Abdelrahman Mohamed. Kaizen: Continuously improving teacher using exponential moving average for semi-supervised speech recognition. In ASRU, pages 518-525. IEEE, 2021.
[49] Geoffrey McLachlan. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365-36 369,
1975.
12

---

[50] Takeru Miyalo, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 1198911979-1993, 2019.
[51] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[52] Hyeonwoo Noh. Andre A Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In ICCV, pages 3476-3485. TEFE Computer Society, 2017.
[53] Hieu Pham. Zihang Dai, Qizhe Xie, and Quoc V. Le. Meta pseudo labels. In CVPR, pages 11557-11568. Computer Vision Foundation/ / IEEE, 2021.
[54] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):21-2555) 2015.
[55] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-I.iang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
[56] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen- Yu Lee, and Tomas Pfister. A simple semi-supervised learning framework for object detection. arXiv preprint 0rXiv22055.04755 2020.
[57]
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15(1):1929-1958, 2014.
[58] Robin Strudel. Ricardo Gareia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pages 7242-7252. IEFE, 2021.
[59] Christian Szegedy, Sergey loffe. Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In AAAI, pages 4278-4284 AAAI Press, 2017.
[60] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shiens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826. IEEE Computer Society, 2016.
[61]
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICMI., volume 97 of Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 2019.
[62] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurlPS. pages 1195-1204, 2017.
[63] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, volume 139
of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 2021.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pages 5998-6000 2017.
[65] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas. David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6438 34886 PMLR, 2019.
[66] Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In IJCAI, pages 3635-3641. ijeai.org, 2019.
[67] Pei Wang. Zhaowei Cai, Hao Yang, Gurumurthy Swaminathan, Nuno Vasconcelos, Bernt Schiele, and Stefano Soatto. Omni-DETR: Omni-supervised object detection with transformers. In CVPR, 2022.
[68] Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Semi-supervised vision transformers. arXiv preprint arXiv:2111.11066 2021.
[69] Ross
Wightman.
Pytorch image models.
httpa://gittub..cmm// http:///wwwwww..../a.......m
bytorch- iimage-mddell 2019.
[70] Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. In NeurlPS, 2020.
[71] Qizhe Xie, Minh-Thaun Luong, Eduard H. Hovy, and Quoe V. Le. Self-training with noisy student improves imagenet classification. In CVPR, pages 10684-10695. Computer Vision Foundation IEEE, 2020.
[72] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. In CVPR, pages 5987-5995. IEEE Computer Society, 2017.
[73] Weijian Xu, Yifan Xu, Tyler A. Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV, pages 9961-99999999 9970. IEEE, 2021.
13

---

[74] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun. Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, pages 6022-6031. IEEE, 2019.
[75] Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR. OpenReview.net, 2018.
[76] Shuai Zhang. Meng Wang, Sijia Liu, Pin-Yu Chen, and Jinjun Xiong. How does unlabeled data improve generalization in self-training? a one-hidden-laye theoretical analysis. arXiv preprint arXiv:2201.0853 2022.
[77] Zhun Zhong, Liang Zheng. Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, pages 13300--3308. AAAT Press, 2020.
[78] Xiaojin Jerry Zhu. Semi-sun cupi-vupe learning literature survey. Technical report, University of Wisconsin- Madison Department of Computer Sciences, 2005.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the abstract and introduction sections.
(b) Did you describe the limitations of your work? [Yes] See the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the conclusion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [Yes] See the supplemental material. The code will be released upon acceptance.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See the supplemental material. We have tried our best to provide all experimental details. And the code will be released upon acceptance for reproduction purpose.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the supplemental material.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See the references.
(b) Did you mention the license of the assets? [No] Those are common assets.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No] Those are common assets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "vRNddoCQ22W", "reviewer": "Reviewer_Nfe1", "summary": "This paper proposed Semi-ViT, a semi-supervised learning approach for vision transformers. The proposed method consists of three stages: first un/self-supervised pre-training, followed by supervised fine-tuning, and finally semi-supervised fine-tuning. At the semi-supervised fine-tuning.\n\nAt the semi-supervised fine-tuning stage, Semi-ViT adopts two techniques: an exponential moving average (EMA)-Teacher framework and a probabilistic pseudo mixup mechanism, to improve the performance.\n\nSemi-ViT, achieves comparable or better performance than the CNN counterparts in the semi-supervised classification setting.\n\nThe authors also show promising scaling up experiments, such as: Semi-ViT-Huge achieves an impressive 80% top-1 accuracy on ImageNet using only 1% labels, which is comparable with Inception-v4 using 100% ImageNet labels.\n", "questions": "As in \"Strengths And Weaknesses\"", "limitations": "yes. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe proposed method is clearly written. It can be understood easily.\n\nWeakness:\nLack of novelty. The proposed three-stage training, EMA teacher, and the probabilistic Pseudo Mixup are all well-known techniques. (the specific techniques in probabilistic Pseudo Mixup is new, but Pesudo Mixup is very natural). All the experimental results are as expected. I did not learn much new here.\nFor comparisons in Figure 1 (a,b), I'm not sure whether the baseline methods (SimCLRv2, PAWS, EMAN) are also trained in a three-stage manner (e.g., the third semi-supervised stage for SimCLRv2 can be just standard semi-supervised learning with EMA teacher). If not, the merge of three techniques together in Semi-ViT makes this comparison unfair to other methods.\nThe results with 100% data for Semi-ViT in Table 1 should be reported. No matter it's better, equal or worse than the baseline, it is valuable point to make fair comparison with the baselines (the MAE paper only reports the 100% data results). Similarly, 100% data results should be reported in Table 8. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sKpQmAYiYK1", "reviewer": "Reviewer_mtWa", "summary": "Recently pseudo-labeling demonstrated powerful results in many domain, including object detection, speech and image recognition, NLP and others. Current paper continues series of works on pseudo-labeling in context of ViT architecture and understanding different aspects of successful pipeline for ViT models with respect to scaling and reducing supervised data. First, authors proposed probabilistic mixup which allows to use filtered pseudo-labeled data to augment non-filtered pseudo-labeled data: weights of mixup are not sampled from beta distribution but pseudo-label score defines them. This scheme is shown with many experiments and ablations to be very effective and give consistent significant improvement. Second, authors confirm that FixMatch is unstable scheme of training in both cases having or not the self-supervised pretraining in the regime of low supervision (1% or 10% of ImageNet is used as labeled data). Third, authors demonstrated that self-supervised pretraining is complementary to pseudo-labeling and combination together improves results by a lot especially in 1% labeled data setting (this result was shown in several domains too, e.g. speech recognition). Finally authors show great scalability of pseudo-labeling for ViT models (with self-supervised pretraining, supervised finetuning and then EMA pseudo-labeling finetuning) and reach impressive results with only 1% labeled data of ImageNet compared to ImageNet supervised baselines.", "questions": "The paper is very well written with strong results, clear explanation of experiments and settings. I have very little suggestion on improving the paper:\n- Could author do sorting of references, so that their numbers appear in the order they first mentioned in the paper? In this case much simpler to find works from the introduction section.\n- References are great, thanks for this! Only would be great to have extra references to other domains, like speech recognition and NLP, where pseudo-labeling is actively developed too, including EMA and stabilization of training for online pseudo-labeling (model is trained with continuously updated teacher). Let me know if you need particular pointers as I am familiar with these works. There are also 2 recent theoretical papers on explaining why pseudo-labeling works (both teacher-student and online versions) [1,2].\n- Extra observations on divergence and instability of FixMatch are very important to have for future pseudo-labeling development. I like details provided in Sec 2.2 which gives proper combined references to prior works and observations. I would like to see here also references to several speech recognition works and have even stronger overview on the instability training. Yes, it is a bit different variant of FixMatch people use in speech, but the same observation on usage this type of teacher holds - people observe divergence and huge instability especially (if teacher and student share weights) with very little supervision (see [3] paper Fig.1) and they propose either history cache [3] or EMA [4,5] to stabilize the training.\n- line 119 - use set difference not subtraction\n- Maybe I missed but I didn't notice reference to ImageNet\n- Authors did comprehensive experiments showing what is the contribution of self-supervised pretraining, probabilistic mixup in different scenarios, different models, different pretraining. Only one thing remains questions for me: how does the filtering really affect the training? Could it be that with probabilistic mixup we even can avoid it and reduce number of hyperparameters (e.g. have ablation in Table 3 where filtering is not used)? (Just in context, in speech recognition absence of filtering works very well, and filtering techniques so far show only marginal gain for CTC models, where there is no problem for filtering long/short sequences as in seq2seq type of loss). I also didn't find the final filtering threshold in the Appendix, only the set which was grid searched.\n- [Extra question]. I wonder how number of epochs of supervised finetuning influences the overall EMA pseudo-labeling convergence? Are there different scenarios for 1% and 10% labeled data? (See e.g. such study in [3, 4, 5] for speech).\n\nOverall, thanks authors for the deep investigation!\n\n[1] Zhang, S., Wang, M., Liu, S., Chen, P.Y. and Xiong, J., 2022. How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis. ICLR 2022.\n\n[2] He, H., Yan, H. and Tan, V.Y., 2021. Information-theoretic generalization bounds for iterative semi-supervised learning. arXiv preprint arXiv:2110.00926.\n\n[3] Likhomanenko, T., Xu, Q., Kahn, J., Synnaeve, G. and Collobert, R., 2020. slimipl: Language-model-free iterative pseudo-labeling. Interspeech 2021. \n\n[4] Manohar, V., Likhomanenko, T., Xu, Q., Hsu, W.N., Collobert, R., Saraf, Y., Zweig, G. and Mohamed, A., 2021, December. Kaizen: Continuously improving teacher using exponential moving average for semi-supervised speech recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 518-525). IEEE.\n\n[5] Higuchi, Y., Moritz, N., Roux, J.L. and Hori, T., 2021. Momentum pseudo-labeling for semi-supervised speech recognition. Interspeech 2021.\n", "limitations": "Limitations are listed in the conclusion section.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Very well, clearly written paper with all necessary details and deep explanations\n- Comprehensive experimental study of pseudo-labeling for ViT and proper ablations showing consistent results across the board\n- New idea of probabilistic mixup which gives consistent experimental improvement across the board for different scenarios and pipelines\n- Ablations on FixMatch confirming training instability for low supervision setting\n- Ablations showing complementary property of pseudo-labeling and self-supervised pretraining\n- Impressive results with 1% labeled data only\n- Demonstration of scaling property for pseudo-labeling for ViT architecture\n\n**Weaknesses**\n- [not important] Absence of some recent literature on theoretical justification of pseudo-labeling and similar EMA study and stability in other domains (see Questions section on more details)\n- Absence of investigation at what extent filtering is important in the pipeline (regarding that mixup with filtered data helps) - this could be another baseline for probabilistic mixup justification\n- [maybe future work?] Absence of study how many epochs of supervised training / supervised finetuning is needed before starting EMA pseudo-labeling process.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vt1HdGSp6u1", "reviewer": "Reviewer_Uvx3", "summary": "This paper proposes a semi-supervised framework for vision transformers. In which, author introduces two techniques to improve the robustness and performance of ViT in semi-supervised learning. They are 1. EMA-teach network update which is the moving average of the student network. 2. Probabilistic Pseudo Mixup which is a novel mix-up method under pseudo-labelling based SSL framework. ", "questions": "1.\tUnder a more general semi-supervised learning set-up, how does the proposed method work when labelled data and unlabelled data are from different datasets?", "limitations": "1.\tThe large-scaled self-supervised pre-training (MAE) may generate more carbon emission. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper is well written, and the core idea is easy to understand. The proposed method and formulate is clean, straightforward, and easy to re-implement. \n\n2.\tThe proposed method effectively improves the semi-supervised training for ViT. Compared to the baseline, both EMA-teacher updating, and Probabilistic Pseudo Mixup achieve significant improvement.\n\n3.\tThe Probabilistic Pseudo Mixup is novel, which provide a new direction to employ mix-up in ViT under pseudo-labelling based SSL framework.\n\n4.\tThe experimental result is remarkable. Compared to fully supervised finetuning after MAE, this paper is only 2% lower with only 10% imageNet data. In addition, the proposed method works well under various self-supervised pretraining pipelines.\n\nWeaknesses:\n\n1.\tIt will be good to show more ablation study over some hyper-parameters, such as the momentum decay and confidence score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J8xafR-ISSs", "reviewer": "Reviewer_zFZY", "summary": "This work proposes a three steps training framework for pure ViTs, including un/self-supervised pre-training, followed by supervised fine-tuning, and finally semi-supervised fine-tuning. EMA and a probabilistic pseudo mixup mechanism are used and results are competitive.", "questions": "Except that ViTs are sometimes data hungry, what are the main differences between ViTs and CNNs that need the community to pay spatial attention to achieve good SSL results? Does the proposed method also benefit CNNs models?", "limitations": "The necessity to design spatial methods for SSL on ViTs  needs to be clarified.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is well written and easy to follow. \n+ First to use pure ViT for SSL.\n\n- The proposed training pipeline are not new compared with former works, such as [14].\n- The improvements are based on existing works (i.e. EMA-Teacher) that are easy to come up with in the semi-supervised domain. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept