Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is a study on what is the different when learning with random crops and contrastive learning (CL) in a supervised learning (SL) or self-supervised learning (SSL) way, versus random patch masking via self-supervised learning (SSL) as in the Masked image modeling (MIM) variants (MAE/BeiT/CAE). The authors claim that the former is learning a "part-to-whole" task while the latter a "part-to-part" task; both can lead to strong part-aware representations, while it seems that SL excels more on object-level tasks, while SSL methods like CAE or MoCo-v3 are able to learn part-aware representations.
strength_and_weaknesses: ### Strengths

S1: the paper is studying part-level recognition, a very interesging and important task

S2: The paper features an extensive empirical study on many object-level recognition and part-level recognition tasks. 


### Weaknesses

W1: It is unclear to me what the part-to-whole vs  part-to-part analysis offers, especially with respect to part-aware representations: Sections 3.1 and 3.2 seem to suggest the same thing: Both CL and MIM, "[are] potentially capable of learning part-aware representations.". Are there any insights or interesting experiments that are derived from that analysis and distinction? Cause what i see in Sec 4 is an empirical analysis on which losses can learn part-aware representation, that can stand without section 3, really.

W2: It is unclear to me if the "part-to-whole" effect is only a function of the input (random crops) and the contrastive aspect of the loss; I think that what really matters is what is the contrastive loss appled on: If the loss is on aggregated features from the whole crop, I see this making sense. But what is a contrastive loss is applied at the token level?  How would for example DenseCL (Wang, Xinlong, et al. "Dense contrastive learning for self-supervised visual pre-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.) fits in this framework? 

W3: The authors are focusing on random crops, which is only one of the augmentations used for CL - yet there are other augmentations at play. A more fair comparison would use an identical augmentation setup apart from either adding random crops or masking. Are all the other augmentations (color jittering, etc) shared across compared methods?

W4: it is unclear to me, eg from the abstract what is the focus of the study, as I see two  axes that are convoluted: MIM vs CL and SL versus SSL methods. The abstract highlights the first axis, so does Sec 3, but then Figure 2 shows that both CAE (MIM) and MoCov3 (Contrastive) can give part-aware representations, while DeiT (SL) cannot. I think that leaving SL out of the study would make it clearer.

W5: the insights from the empirical study are not analysed enough and can in places seems exargerated:, eg "This implies that in general the self-supervised models are not strong at object-level understanding," this is a very generic statement to make from the results in Table 2. 

Q1: Are you retraining all the models under some otherwise identical setup, or using the publicly available models out of the box? Overall, three are many differences between the models compared beyond the loss (from batchsizes, to augmentations, to optimizers to other ViT-related tricks used in different papers). 

Q2: Although figure 1 and 3 is mostly figurative it seems to me that it exaggerates the effects of the random crops used in practice. Apart from methods that use local crops as in multi-crop, the scale parameter for the random crops is usually not as large as the one used in this figure. What is the crop scale used in all methods, ie what is the percentage of the max width/height that each readnom crop uses?

Q3: Where are the patches seen in Fig 2 and 4 from? Imagenet?

N1: I wonder how things would change for SL if a more recent variant of the DeiT series was used, e.g. Deit III from 
Touvron, Hugo, Matthieu Cord, and Hervé Jégou. "Deit iii: Revenge of the vit." arXiv preprint arXiv:2204.07118 (2022).


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, but the usefulness oft the analysis of Section 3 is to me unclear. The experimental evaluation is exhaustive and in theory reproducible. There seems to be no technical novelty, the paper is a study on SSL/SL and MIM /CL, starting from pretrained models from other papers.

summary_of_the_review: This paper is at its core an empirical study on part-aware representations; the usefulness of the analysis in section 3 is unclear to me, looking forward for some clarifications from the authors on this. There is no technical novelty, while the insights from the study are also not strong, basically verifying the superiority of iBoT over other SSL methods tested on both object and part tasks. I am looking forward to the author's responses.

--- Post rebuttal update:

I want to thank the authors for constructive discussion. I think iff the authors make the edit they promise to the text  and clarify their contributions, I think I can raise my score to borderline accept. Although limited in novelty, the study in this paper can be valuable to the community, if the contributions are not over-claimed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an analysis of existing self-supervised learning methods with vision transformers. Extensive experiments are performed on the off-the-shelf pretrained model with the linear and attentive probes to verify the speculation that contrastive learning is a part-to-whole task and masked image modeling is a part-to-part task. 
strength_and_weaknesses: Pros:

The motivation of this paper is clear, the paper is well-written which makes it easy to follow. The experiment setting makes sense to me.

Cons:

The part-to-part and the part-to-whole difference between masked image modeling and contrastive learning is already mentioned in the CAE paper, which makes the finding and conclusion less interesting and surprising.
Some experiment results do not align with the claim well, for example, in the part-retrieval and part-segmentation experiments, BEiT and MAE which are MIM models do not perform better than contrastive learning or supervised learning.
Based on that, no more instructive conclusion about how to better use the property is given (combining two of them like iBOT shouldn't count as a contribution in this paper), which makes the novelty part weak.

clarity,_quality,_novelty_and_reproducibility: See above section
summary_of_the_review: Overall I think this paper goes one step further than CAE and performs an extensive comparison on various benchmarks, which is appreciated. However, some of the experiment results do not align with the authors' claim and the conclusion is somewhat unsurprising and not enlightening. Thus my rating is 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to understand the two mainstream self-supervised methods. It speculates that masked image modeling is a part-to-part process and contrastive learning is a part-to-whole process. Both empirical results and qualitative analysis are provided to support the claims.
strength_and_weaknesses: Strengths:
1. This paper makes an innovative claim on the current self-supervised learning methods, its conclusion leads people to a more insightful understanding of self-supervised learning.
2. Authors conduct a comprehensive empirical study on both object-level and part-level tasks to support their claims.
3. Many qualitative analyses are made, which give a clear and intuitive explanation.

Weaknesses:
1. For the part segmentation part, the authors' explanation, 'possibly due to pretraining quality in representation encoding, BEiT and MAE perform inferior', is not convincing. I do not think it is safe to draw the conclusion based on only one MIM method CAE. Therefore, whether MIM method can be simply described as a part-to-part process is questionable from the experimental results. 
2. I do not see a clear reason why iBOT performs the best from the authors' explanation. It's unclear why combining part-to-part and part-to-whole processes could generate a better model. I'm afraid that the differences among the models/objectives would matter more than the differences in the training paradigm (contrastive/MIM). This could make the claims in this paper less useful for guidance on future SSL models.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and clearly presented. There are some minor mistakes, in Equations (2) and (3), $P_v$ and $P_m$ are not defined. It sheds some new insights into the current self-supervised learning, which is beneficial to the community. It has a good quality in general, and the experiment details demonstrate its reproducibility. There are some claims in the paper that are not well supported from my point of view, which fails to explain some phenomena in a convincing way.
summary_of_the_review: This paper speculates that contrastive learning is a part-to-whole process and MIM is a part-to-part process and self-supervised learning could learn part-aware representations. It provides an innovative perspective to understand self-supervised learning, supported by comprehensive empirical results and qualitative analyses. The paper is well written but some claims are not well supported. I'm also wondering to what extent the conclusions in this paper can help the future model since the different objectives matter more than the training paradigm (contrastive/MIM). I think this paper is a bit above the borderline.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an interesting study, trying to provide understanding of the differences between MIM and CL self-supervised learning. The draft is not a "method" paper but an "analysis" one, and was rated as such by the committee. Reviewers have praised the importance of the tackled problem, as well as the introduction of a benchmark and experimental protocol for testing part-level understanding. 
However, we have also noted that some of contributions are a bit over-claimed and that the analysis presented in the paper could largely be improved. As an analysis paper, the hypothesis should be very well formulated and properly tested. The paper is unclear in many places and some of the conclusions are only supported by qualitative results, which could be interpreted either way. A lot of constructive and actionable feedback was provided in the discussion phase; I believe that if the authors would selected the studied methods better, added domains, re-written and strengthened the manuscript, this could be a really interesting study. In its current form, I recommend the paper for rejection, and encourage the authors to improve the manuscript and submit to another venue.
justification_for_why_not_higher_score: As an analysis paper, the bar in terms of experimental execution is very high. The paper, in its current form does not meet that bar. A lot of constructive feedback was given in the reviews, and the paper can definitely be strengthened accordingly.
Decision: Reject