Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed Semi-ViT, a semi-supervised learning approach for vision transformers. The proposed method consists of three stages: first un/self-supervised pre-training, followed by supervised fine-tuning, and finally semi-supervised fine-tuning. At the semi-supervised fine-tuning.

At the semi-supervised fine-tuning stage, Semi-ViT adopts two techniques: an exponential moving average (EMA)-Teacher framework and a probabilistic pseudo mixup mechanism, to improve the performance.

Semi-ViT, achieves comparable or better performance than the CNN counterparts in the semi-supervised classification setting.

The authors also show promising scaling up experiments, such as: Semi-ViT-Huge achieves an impressive 80% top-1 accuracy on ImageNet using only 1% labels, which is comparable with Inception-v4 using 100% ImageNet labels.

strengths_and_weaknesses: Strength:
The proposed method is clearly written. It can be understood easily.

Weakness:
Lack of novelty. The proposed three-stage training, EMA teacher, and the probabilistic Pseudo Mixup are all well-known techniques. (the specific techniques in probabilistic Pseudo Mixup is new, but Pesudo Mixup is very natural). All the experimental results are as expected. I did not learn much new here.
For comparisons in Figure 1 (a,b), I'm not sure whether the baseline methods (SimCLRv2, PAWS, EMAN) are also trained in a three-stage manner (e.g., the third semi-supervised stage for SimCLRv2 can be just standard semi-supervised learning with EMA teacher). If not, the merge of three techniques together in Semi-ViT makes this comparison unfair to other methods.
The results with 100% data for Semi-ViT in Table 1 should be reported. No matter it's better, equal or worse than the baseline, it is valuable point to make fair comparison with the baselines (the MAE paper only reports the 100% data results). Similarly, 100% data results should be reported in Table 8. 
questions: As in "Strengths And Weaknesses"
limitations: yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Recently pseudo-labeling demonstrated powerful results in many domain, including object detection, speech and image recognition, NLP and others. Current paper continues series of works on pseudo-labeling in context of ViT architecture and understanding different aspects of successful pipeline for ViT models with respect to scaling and reducing supervised data. First, authors proposed probabilistic mixup which allows to use filtered pseudo-labeled data to augment non-filtered pseudo-labeled data: weights of mixup are not sampled from beta distribution but pseudo-label score defines them. This scheme is shown with many experiments and ablations to be very effective and give consistent significant improvement. Second, authors confirm that FixMatch is unstable scheme of training in both cases having or not the self-supervised pretraining in the regime of low supervision (1% or 10% of ImageNet is used as labeled data). Third, authors demonstrated that self-supervised pretraining is complementary to pseudo-labeling and combination together improves results by a lot especially in 1% labeled data setting (this result was shown in several domains too, e.g. speech recognition). Finally authors show great scalability of pseudo-labeling for ViT models (with self-supervised pretraining, supervised finetuning and then EMA pseudo-labeling finetuning) and reach impressive results with only 1% labeled data of ImageNet compared to ImageNet supervised baselines.
strengths_and_weaknesses: **Strengths**
- Very well, clearly written paper with all necessary details and deep explanations
- Comprehensive experimental study of pseudo-labeling for ViT and proper ablations showing consistent results across the board
- New idea of probabilistic mixup which gives consistent experimental improvement across the board for different scenarios and pipelines
- Ablations on FixMatch confirming training instability for low supervision setting
- Ablations showing complementary property of pseudo-labeling and self-supervised pretraining
- Impressive results with 1% labeled data only
- Demonstration of scaling property for pseudo-labeling for ViT architecture

**Weaknesses**
- [not important] Absence of some recent literature on theoretical justification of pseudo-labeling and similar EMA study and stability in other domains (see Questions section on more details)
- Absence of investigation at what extent filtering is important in the pipeline (regarding that mixup with filtered data helps) - this could be another baseline for probabilistic mixup justification
- [maybe future work?] Absence of study how many epochs of supervised training / supervised finetuning is needed before starting EMA pseudo-labeling process.


questions: The paper is very well written with strong results, clear explanation of experiments and settings. I have very little suggestion on improving the paper:
- Could author do sorting of references, so that their numbers appear in the order they first mentioned in the paper? In this case much simpler to find works from the introduction section.
- References are great, thanks for this! Only would be great to have extra references to other domains, like speech recognition and NLP, where pseudo-labeling is actively developed too, including EMA and stabilization of training for online pseudo-labeling (model is trained with continuously updated teacher). Let me know if you need particular pointers as I am familiar with these works. There are also 2 recent theoretical papers on explaining why pseudo-labeling works (both teacher-student and online versions) [1,2].
- Extra observations on divergence and instability of FixMatch are very important to have for future pseudo-labeling development. I like details provided in Sec 2.2 which gives proper combined references to prior works and observations. I would like to see here also references to several speech recognition works and have even stronger overview on the instability training. Yes, it is a bit different variant of FixMatch people use in speech, but the same observation on usage this type of teacher holds - people observe divergence and huge instability especially (if teacher and student share weights) with very little supervision (see [3] paper Fig.1) and they propose either history cache [3] or EMA [4,5] to stabilize the training.
- line 119 - use set difference not subtraction
- Maybe I missed but I didn't notice reference to ImageNet
- Authors did comprehensive experiments showing what is the contribution of self-supervised pretraining, probabilistic mixup in different scenarios, different models, different pretraining. Only one thing remains questions for me: how does the filtering really affect the training? Could it be that with probabilistic mixup we even can avoid it and reduce number of hyperparameters (e.g. have ablation in Table 3 where filtering is not used)? (Just in context, in speech recognition absence of filtering works very well, and filtering techniques so far show only marginal gain for CTC models, where there is no problem for filtering long/short sequences as in seq2seq type of loss). I also didn't find the final filtering threshold in the Appendix, only the set which was grid searched.
- [Extra question]. I wonder how number of epochs of supervised finetuning influences the overall EMA pseudo-labeling convergence? Are there different scenarios for 1% and 10% labeled data? (See e.g. such study in [3, 4, 5] for speech).

Overall, thanks authors for the deep investigation!

[1] Zhang, S., Wang, M., Liu, S., Chen, P.Y. and Xiong, J., 2022. How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis. ICLR 2022.

[2] He, H., Yan, H. and Tan, V.Y., 2021. Information-theoretic generalization bounds for iterative semi-supervised learning. arXiv preprint arXiv:2110.00926.

[3] Likhomanenko, T., Xu, Q., Kahn, J., Synnaeve, G. and Collobert, R., 2020. slimipl: Language-model-free iterative pseudo-labeling. Interspeech 2021. 

[4] Manohar, V., Likhomanenko, T., Xu, Q., Hsu, W.N., Collobert, R., Saraf, Y., Zweig, G. and Mohamed, A., 2021, December. Kaizen: Continuously improving teacher using exponential moving average for semi-supervised speech recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 518-525). IEEE.

[5] Higuchi, Y., Moritz, N., Roux, J.L. and Hori, T., 2021. Momentum pseudo-labeling for semi-supervised speech recognition. Interspeech 2021.

limitations: Limitations are listed in the conclusion section.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes a three steps training framework for pure ViTs, including un/self-supervised pre-training, followed by supervised fine-tuning, and finally semi-supervised fine-tuning. EMA and a probabilistic pseudo mixup mechanism are used and results are competitive.
strengths_and_weaknesses: + The paper is well written and easy to follow. 
+ First to use pure ViT for SSL.

- The proposed training pipeline are not new compared with former works, such as [14].
- The improvements are based on existing works (i.e. EMA-Teacher) that are easy to come up with in the semi-supervised domain. 
questions: Except that ViTs are sometimes data hungry, what are the main differences between ViTs and CNNs that need the community to pay spatial attention to achieve good SSL results? Does the proposed method also benefit CNNs models?
limitations: The necessity to design spatial methods for SSL on ViTs  needs to be clarified.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a semi-supervised framework for vision transformers. In which, author introduces two techniques to improve the robustness and performance of ViT in semi-supervised learning. They are 1. EMA-teach network update which is the moving average of the student network. 2. Probabilistic Pseudo Mixup which is a novel mix-up method under pseudo-labelling based SSL framework. 
strengths_and_weaknesses: Strengths:

1.	This paper is well written, and the core idea is easy to understand. The proposed method and formulate is clean, straightforward, and easy to re-implement. 

2.	The proposed method effectively improves the semi-supervised training for ViT. Compared to the baseline, both EMA-teacher updating, and Probabilistic Pseudo Mixup achieve significant improvement.

3.	The Probabilistic Pseudo Mixup is novel, which provide a new direction to employ mix-up in ViT under pseudo-labelling based SSL framework.

4.	The experimental result is remarkable. Compared to fully supervised finetuning after MAE, this paper is only 2% lower with only 10% imageNet data. In addition, the proposed method works well under various self-supervised pretraining pipelines.

Weaknesses:

1.	It will be good to show more ablation study over some hyper-parameters, such as the momentum decay and confidence score.

questions: 1.	Under a more general semi-supervised learning set-up, how does the proposed method work when labelled data and unlabelled data are from different datasets?
limitations: 1.	The large-scaled self-supervised pre-training (MAE) may generate more carbon emission.
review_title: null

Meta-review: metareview: This paper explores Semi-ViT, a semi-supervised learning approach for vision transformers. Semi-VIT build-on three stages pipeline such as SimCLRv2. The authors introduce a probabilistic mixup for the semi-supervised finetuning stage which gives consistent experimental improvements.  Semi-ViT shows strong empirical results as it achieves 80% top-1 accuracy on ImageNet using only 1% labels, which is comparable with Inception-v4 using 100% ImageNet labels,


Demonstrating that ViT+semi-supervised training enables to reach 80% top-1 accuracy on 1% ImageNet is novel and of potential interest to the SSL community.  I therefore recommend acceptance. However, I would encourage the authors to clarify that the three-stages pipeline is not a contribution of the paper and focus the novelty on the probabilistic mixup and the experimental study.
confidence: Certain
Decision: Accept