Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This is an empirical paper focusing on exploring the task of self-supervised learning on lightweight vision transformers (ViTs). In this paper, the authors observed several discoveries, such as: the tiny model fails to benefit from large-scale pre-training data and shows inferior performance on data-insufficient downstream tasks. They further proposed a distillation strategy during pretraining to improve the representation ability of compact ViTs. Experiments are conducted on ImageNet pre-training and multiple downstream tasks and datasets.
strength_and_weaknesses: ### Strengths:

   - It is interesting to explore suitable methods for lightweight vision transformers or other efficient models in the self-supervised learning manner. 

   - This paper provided extensive experiments on ImageNet pre-training and multiple downstream datasets and tasks, such as classification, object detection, and segmentation.


### Weaknesses:

   - Though this is an empirical paper, the novelty and originality in it are fairly limited, as well as the significance and contribution which are also not strong. The observations that tiny models fail to benefit from large-scale pre-training data, and rely more on the downstream dataset scale are a little bit straightforward and not surprising. The use of knowledge distillation for self-supervised learning on lightweight models also has been proposed for a long time, e.g., on low-bit efficient models [1] and mobile-level models [2].

[1] Shen, Z., Liu, Z., Qin, J., Huang, L., Cheng, K. T., & Savvides, M. (2021). S2-bnn: Bridging the gap between self-supervised real and 1-bit neural networks via guided distribution calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2165-2174).

[2] Fang, Zhiyuan, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. "SEED: Self-supervised Distillation For Visual Representation." In International Conference on Learning Representations. 2021.

   - Some statements in this paper are not well supported, such as “lower layers of the pre-trained models matter more than higher ones if sufficient downstream data is provided, while higher layers matter in data-insufficient downstream tasks.” I think a better and fair comparison design is crucial and also necessary for this argument. The current experiments for this part are not rigorous to prove it.

   - The writing and organization of this paper can also be improved. For instance, it’s not clear to me why Table 1 is located in the early part of the paper. I did not get much information from it and do not know what the insight of this table is.

   - Overall, this paper seems a little bit incremental without providing new conclusions or discoveries over previous literature.


clarity,_quality,_novelty_and_reproducibility: The clarity of this paper is qualified. Novelty and originality are somewhat limited. Since this paper did not introduce any new and concrete approach, reproducibility is not applicable.
summary_of_the_review: Overall, this is an empirical paper with some trivial observations which are not well supported by the experiments and some are even not new. Also, the organization and writing can be improved significantly in this paper. Thus, I tend to reject it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to analyze how the lightweight ViTs (e.g., ViT-tiny and DeiT-tiny) perform when using self-supervised pretraining methods. By conducting a variety of experiments, the authors show that: 1) MIM-based methods, like MAE, helps more than contrastive learning based methods, like MOCOv3; 2) Low-level layers matter more than high-level layers when sufficient data for finetuning is available; 3) KD methods help improve the representative ability of high-level layers.

Previous works mostly focus on large-scale models, which often contain more than 80M parameters. In contrast, these paper aims to reveal how lightweight ViTs behave on both ImageNet and some downstream tasks, which makes this paper much more different than previous works on self-supervised methods.
strength_and_weaknesses: Strength
- Analyzing the performance of lightweight ViTs in self-supervised learning manners is important in that in most cases it is not appropriate to use large-scale models but only tiny-sized ones. This paper clearly shows how lightweight ViTs behave under different settings and evaluates the pretrained models on downstream tasks.
- Authors show that low-level layers are more important than high-level layers after pretraining. This is an important signal for the development of SSL methods in future.
- Distilling the knowledge from a large model to lightweight ones is an interesting topic. This paper provides an effective way to do so and analyze how to do distillation helps more for the lifting the model performance.

Weaknesses
- Only two SSL methods are selected for presentation. One is MOCOv3 and the other one is MAE. It would be better if two methods are selected for explanation for each type of SSL method. The conclusion would be more convincing.
- From Table 3, we see that the pre-training benefits little from large-scale data.  With only 10% of the ImageNet 1k data are provided, the classification performance after finetuning is already good. I suppose there are some explanations on the reasons but obviously there is not.
- The analysis is interesting. I am looking forward to taking some message about how to design better network architectures or how to develop more advanced self-supervised learning methods.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and easy to understand. This reveals some secrets behind SSL on visual recognition using lightweight ViT models. The novelty is ok but not significant enough for publication.
summary_of_the_review: The intention of this paper is interesting. There are no relevant papers reporting similar conclusions. Though the novelty of this paper is not significant enough, regarding the thorough experiments that have been done, I give a score of 5 at this moment. I would like to lift the rating if the authors can more clearly explain the significance of the contributions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an analysis of two self-supervised training methods for lightweight vision transformers. The methods are MAE (a masking method) and MoCoV3 (a contrastive method). It shows that although MAE appears stronger when training (w/o labels) and evaluating on ImageNet, the roles reverse on downstream tasks. It analyzes the differences of these two networks and compares them a supervised method (DeitT), by looking at layer-by-layer similarity metrics, as well as analysis of the locality vs. globality of the attention maps. Finally it proposes to use distillation from a large-scale MAE model to improve the tiny MAE. It does this with significant improvements to downstream tasks.
strength_and_weaknesses: Strengths:
- This is a pretty in depth analysis of the difference between a MAE- and a MoCo-based self-supervised ViT.
- The distillation of the MAE-Tiny from MAE-Base leads to significant improvements over the MAE-Tiny without teacher guidance. The final results compared to supervised pretraining is strong and a significant achievement.

Weaknesses:
- The paper mostly presents analysis and new applications of old techniques. It is pretty light on novelties though.
- Not all analysis is tied back to experimental results, which means it's not clear if we can draw actionable conclusions from it. If the analyses had led to more actionable changes that resulted in improvements, the case would have been stronger.
clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is fairly clear, although there are times I wish ablations had been kept separate. For instance, "we adopt the layer-wise attention-based distillation" and then two paragraphs later "For simplicity, we only apply distillation on the attention maps of the last layer". This causes some confusion and it's easier for the reader if you present one main method, and leave ablations to separate sections.

Quality
The analysis and results are sound and of good quality. The experimental evaluation is varied and relevant.

Novelty
As for novelties, this paper mostly presents analysis and the final distilled MAE uses standard methods. The novelty is in applying these to more lightweight ViTs.

Reproducibility
As for reproducibility, it is unclear if source code is forthcoming. 

Suggestions:
- Adding a training data column to tables 4 and 6  would help to make it clear that DeiT used labels and the other methods did not. Having to go through the text to realize this takes time.
- Why not include DeiT in table 6 as well? The distilled MAE actually outperforms the supervised method on several benchmarks. This is significant and should be highlighted.
summary_of_the_review: The papers presents a lot of analysis around lightweight ViTs, and shows the efficacy of student/teacher distillation for mask-based self-supervised pre-training. The paper presents few novelties in terms of technique and method, but the quality and novelty of the analysis and strength of final distilled results are solid contributions. The topic is of broad interest, so many could benefit from this work. Balancing the lack of novelties with the potential benefits it is close to borderline for me. However, I do lean in favor of acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the task of self-supervised learning on lightweight vision transformers (ViTs).  What makes the paper different from existing works (often contain more than 80M parameters. ) is that it aims to reveal how lightweight ViTs behave on both ImageNet and some downstream tasks. The summary of strengths and weaknesses as I summarize from the reviewers' feedback and the paper are:


Strengths
---------------
 - Reviewer PEwP and Reviewer UM5C:  "It is interesting to explore suitable methods for lightweight vision transformers or other efficient models in the self-supervised learning manner."

- Reviewer UM5C: "Authors show that low-level layers are more important than high-level layers after pretraining. This is an important signal for the development of SSL methods in future." 

-  Reviewer UM5C and Reviewer nNkX:  "This paper provides an effective way to do so and analyze how to do distillation helps more for the lifting the model performance."

- Reviewer PEwP and Reviewer nNkX:  "This paper provided extensive experiments on ImageNet pre-training and multiple downstream datasets and tasks, such as classification, -object detection, and segmentation." "The distillation of the MAE-Tiny from MAE-Base leads to significant improvements over the MAE-Tiny without teacher guidance. "


Weaknesses
---------------
- Reviewer PEwP:  "Though this is an empirical paper, the novelty and originality in it are fairly limited, as well as the significance and contribution which are also not strong. "
- Reviewer UM5C: "Only two SSL methods are selected for presentation. One is MOCOv3 and the other one is MAE. It would be better if two methods were selected for explanation for each type of SSL method. The conclusion would be more convincing."
- Reviewer UM5C: From Table 3, we see that the pre-training benefits little from large-scale data. 
- Reviewer nNkX:  "It is pretty light on novelties. "
- Reviewer nNkX:  Not all analysis is tied back to experimental results, which means it's not clear if we can draw actionable conclusions from it. If the analyses had led to more actionable changes that resulted in improvements, the case would have been stronger"


justification_for_why_not_higher_score: This paper explores the task of self-supervised learning on lightweight vision transformers (ViTs) that fits well with our venue.  However, after discussion and despite a response from the authors that addressed some of the concerns raised during the reviews, all reviewers agreed that the level of contribution was too limited in terms of novelty.  Reviewer nNkX's score was a bit more favorable but decided not to champion the paper.

The AC hence suggests that the paper is not yet ready for publication with its current version and encourages the authors to revise the paper in light of the reviewers' comments and submit it to a future venue.
Decision: Reject