Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents comprehensive analyses of self-supervised vision transformers, which provide some new insights about the differences between CL and MIM. For instance, ViTs pre-trained with CL focus on global patterns compared with MIM, collapse into homogeneity while MIM shows more diversity, and reduces the high-frequency signals of the representations but MIM amplifies them. These analyses are helpful for understanding the self-supervised ViTs and show instructions for designing new SSL methods.
strength_and_weaknesses: **Strengths**:

The paper is well-written and easy to follow. The analyses are comprehensive and thorough.

**Weaknesses**:
1. Some expression and figures need to be revised (see below).
2. The proposed method that jointly pre-training with CL and MIM is somewhat not practical due to large training cost, though it is not the key contributions of this paper.

**Questions**:
1. In Figure1, the authors claim that *CL outperforms MIM in small model regimes*, which is inconsistent with the analyses in [a] with MAE. Some explanation should be presented.
2. Since most of the analyses are based on ViT-B, are the observations generalized to larger ViTs (*e.g.*, ViT-L or ViT-H)?
3. In Figure 3, the behaviors of MoCo are rather odd, which are also inconsistent with the observations in [b]. Are the differences from the different settings or the variance of the pre-trained models? My main concern is the reproducibility about the results. Maybe the code and weights to reproduce the results are helpful.
4. The claim in L86 that *the representations of CL contain shape information, so it can help ...* is not well-supported by the analyses on the attention distance in this paragraph.
5. I'm not sure whether L118-127 should present in Sec.2 rather than Sec.3, which focuses on the analyses of layer representations. 
6. In Figure 10, it is not clear to me the meanings of "1e-1" and "1e-4" on top of the figure. And it seems that the changes for CL are larger than MIM, which appears to contradict the observations in Figure5(b) with CKA similarities.
7. The linear evaluation in L247-267 only reveals the linear separability of the representations, which could not support the claim that "Later layers of CL and early layers of MIM are important" well. Is the claim generalized well under the fine-tuning setting?

[a] Wang, S., Gao, J., Li, Z., Sun, J., & Hu, W. (2022). A Closer Look at Self-supervised Lightweight Vision Transformers. ArXiv, abs/2205.14443.

[b] Xie, Z., Geng, Z., Hu, J., Zhang, Z., Hu, H., & Cao, Y. (2022). Revealing the Dark Secrets of Masked Image Modeling. ArXiv, abs/2205.13543.
clarity,_quality,_novelty_and_reproducibility: **Quality & Clarity**: the paper is in general well written. The figures are neat with detailed captions.

**Novelty**: the analyses and findings are largely original and interesting, though the analysis tools used in the paper are borrowed from other works.

**Reproducibility**: Though the code for pre-training and fine-tuning the models are provided in supplementary material, the code to reproduce the figures (or analysis results) is absent, which is the key of this work.
summary_of_the_review: The paper focuses on the self-supervised pre-trained ViTs, and presents several interesting findings, which may help to understand the different behaviors of pre-training with CL and MIM. It may guide future SSL method design and advance research in this field.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper compares two self-supervised classes of methods for vision transformers, namely contrastive learning and masked reconstruction.
Through several quantitative analyses of pre-trained models, the authors expose a series of findings on the learning properties of these methods.
The main findings for contrastive learning are: it captures global low-frequency patterns, its intermediate representations are rather homogeneous, and self-attention is its key component.
Conversely, for masked reconstruction, the main findings are: it focuses on local high-frequency patterns, its intermediate representations maintain diversity, and the MLPs are its key component.

strength_and_weaknesses: Strengths:

- The paper tackles interesting questions in self-supervised learning and provides answers that are thoroughly supported by empirical evidence. With a possible risk of confirmation bias, the findings in this paper validate the anecdotal evidence that has been observed in the community for a while.

- From the reader's perspective, the paper is well-written and well-organized. The authors have done a great job of outlining the research questions and their findings in the introduction.

- The research presented in this work is quite extensive. I appreciate that the authors did not stop at the first finding and instead insisted on probing the models with different tools to highlight their differences from various perspectives.

Weaknesses (or rather suggestions for improvement):

- Complement the discussion by mentioning the findings of "High fidelity visualization of what your self-supervised representation knows about" (Bordes 2022). This work, concurrent with the authors' work, offers qualitative evidence that broadly agrees with the findings of this paper.

- L59-61 "early layers are usually known to capture low-level features, e.g., local patterns, high-frequency signals, and texture information, and later layers capture global patterns, low-frequency signals, and shape information". In the context of convolutional networks, due to their limited receptive field and pyramidal architecture, this statement is considered common knowledge and is supported by a large body of studies. However, this paper focuses on transformers, where this behavior is still under investigation. I suggest providing references to support this statement and possibly clarifying its scope.

- Most of the discussion points compare MoCo vs. SimMIM. However, there are parts that would be more convincing if more methods were added to the discussion. An example of this, L257-267 discuss the difference between SimMIM and MAE in terms of decoder depth, why not include MAE in figure 11?

Minor points:

- Improve figure 1 by adding a grid, separating the plots more, and tweaking the font size. Actually, most figures would benefit from a grid.

- L67: "As shown in Figure 1". Figure 1 deserves a bit of introduction, e.g. what models and methods are being compared and on what, before discussing its meaning. I see that the caption contains all the required information, but it is weird to read the first line of section 2 without knowing what the figure is about.

- Improve figure 10 by connecting all MLP markers with a line, and connecting all self-attention markers with another line, rather than relying on white/gray background to distinguish them. Alternatively, if you want to use a single line for each method, change at least the markers of MLP and self-attn layers.
clarity,_quality,_novelty_and_reproducibility: Quality: high. The core idea is relevant and the execution is well done through a series of well-discussed experiments.

Clarity: high. The paper is well-written and well-organized. The various sections are connected in a logical way.

Originality: low. No new idea is introduced, apart from combining contrastive learning and masked image modeling which can be found in other concurrent works. No new "tool" for probing the learned representations is introduced either. However, new tools and ideas are not always required and the paper shines for the quality of the execution and the discussion.

Reproducibility: high. Although the code is not yet available, it should be possible to fully reproduce the experiments in the paper based on the referenced methods.
summary_of_the_review: This paper offers a quantitative study of various properties of contrastive learning and masked reconstruction in self-supervised vision transformers. Such a study is valuable as it validates what has been observed empirically in the community and provides a tentative analysis of the reasons behind these observations. The paper itself is well-written and clearly outlined. For these reasons, I recommend the paper for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents comparative studies on various facets of two widely used self-supervised learning methods - contrastive learning and masked image modeling. The studies show opposing properties of the two methods: image information (image-level vs. token-level), features representations (low- and high-frequency) and different lead role components (later- and early-layers). The study also demonstrated that their misture outperforms individuals of them.
strength_and_weaknesses: Strengths of the this paper - 

The entire paper, in its current form, is easy to follow. The examination of certain components in each method is clearly understandable. The evidence presented in the paper seems reasonable.

Weaknesses of the paper - 

The major weakness is the lack of novelty and comprehensiveness. The comparisons show the examination of two methods, given a large number number  of similar approaches reported in the community. 
Another weakness of the paper is that the mixture of the two methods is to simply use the weighted combination of the two methods. In spite of its simplicity, this combination form lacks clear motivation and clear discussion on the merging form, e.g. why this form is the best way, and how to merge them in the level of components?

Actions to be taken:

More similar techniques must be taken into account. The comparison aspects should be considered, e.g. components and layers of the systems. The integration of the systems should be properly reasoned.
clarity,_quality,_novelty_and_reproducibility: Originality of this paper: Weak and missing. This paper is only about the evaluation of two standard methods.

Quality of the research: Limited contributions towards the topic. The evidence to justify the argument may be further improved - more metrics and comprehensive evaluation to be undertaken. 

Clarity of the paper: In general, the entire paper is easy to follow. The major concern is that the combination of the two methods is not well explained.
summary_of_the_review: This paper presents the comparisons of two standard methods with several evaluation metrics. The strength of the paper is the easiness to understand the presented work. The major weaknesses of the paper include the missing depth and width of the comparisons. This paper and its presentation in the current form will not be recommended for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper mainly investigates the principle differences between contrastive learning (CL) and masked image modeling (MIM). That’s the CL is shape-biased, which learns low-frequencies, and the MIM is texture-biased, which exploits high-frequencies. The Conclusion is interesting and the analysis is thorough.
strength_and_weaknesses: Strength:  
Well-written, clear analysis and easy to understand.  
The conclusions drawn in the paper may be instructive to the self-supervised learning community.  

Weaknesses:  
(1) The main concern lies in the quantitative experiments. In fact, the paper lacks experiments on downstream tasks, like image classification, object detection or segmentation. The authors should provide corresponding downstream experiments to support the effectiveness of the conclusion other than visualization of principle analysis.  
(2) The principle analysis occupies a large amount of space in the paper, on the contrary, the method derived from the conclusions is rarely involved. Actually, the corresponding method (Section 5) is also the core of the paper.  

clarity,_quality,_novelty_and_reproducibility: Well-written, interesting conclusions, but lack experiments.
summary_of_the_review: I am inclined to reject the current version of the paper since the experiment and the corresponding method are not well explored.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper compares two broad families of SSL pre-training: Masked-Image Modelling versus Contrastive Learning. The paper proposes quantitative and qualitative evaluations shedding light on what these two learning paradigms allow. The tackled topic is timely and answers questions in which the research community focusing on this topic is currently interested. 
The results are extensive, and the evidence presented is clear and reasonable. However, the reviewers pointed out some technical issues in the experimental design but tentatively answered them in the rebuttal. As the reviewers have suggested many improvements, the paper should be improved by considering them to get the paper in better shape further. I am inclined to accept this paper because of its timely message, despite some of the missing results that reviewers suggested. If the paper had to be accepted, I would like the authors to take the job of completing the draft with suggested experiments seriously and update the paper accordingly. 
justification_for_why_not_higher_score: As an empirical analysis paper, not a method one, the bar in terms of the quality of the conducted experiments is extremely high. Some of the points raised by the reviewers show that the experimental protocol could further be improved. 
justification_for_why_not_lower_score: The topic discussed in the paper is really interesting and will be very useful to the community. The experiments presented in the paper, despite the concerns raised in the reviews, remain strong and are sufficient to support the paper's claims.
Decision: Accept: poster