Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an SSL framework that is capable of learning different concepts in an image while the previous studies can only capture the dominant concepts. To this end, they proposed MC-SSL based on the components from SiT and DINO, i.e., Group Mask Model Learning (GMML) and Patch Concept Learning (PCL) based on knowledge distillation. In their evaluations, they demonstrated that MC-SSL can perform well with small datasets in multi-class and multi-label downstream tasks. Overall, this is an interesting work with some promising results. 
strength_and_weaknesses: Strength:
- This work demonstrated its ability to differentiate concepts learned from images on small datasets while the majority of related works can only identify the dominant concepts from data.
- This framework avoids the common practice of using a large batch size for training in contrastive learning-based SSL frameworks, while still achieving relatively good performance.
- The authors conduct extensive experiments on multi-class and multi-label downstream tasks to demonstrate the effectiveness of the proposed framework in two settings.
- The results of visualized self-learned concepts are quite impressive and promising.

Weaknesses: 
- The authors claim that their methods work well with small datasets. I am curious as to why not directly use smaller transformer models for these small datasets. How do the authors argue that these improvements indeed are from the proposed MC-SSL rather than related baseline approaches that are not well-trained?
- The notation of shared weights for prediction heads in Figure 2 is somewhat unclear. How exactly does each component share these weights?
- For the ablation studies, I am concerned about the 10% subset of ImageNet. The construction details of this dataset are missing. For example, a biased sampling strategy could cause the model to favor certain tasks or approaches downstream. This requires some clarification.
- How are the different approaches in Table 1 evaluated? Similarly, what trained models are used for Figure 1's visualization? This requires clarification. In addition, I wonder if we feed more data and train longer for the baselines, are they able to capture different concepts? Alternatively, the authors may want to show some learning curves to illustrate this.
- The presentation of this paper should be improved. For instance, the authors may want to highlight their most significant contributions or discuss the most significant differences between their works and those of SiT or DINO. Other minor issues: "our thesis is..." should be replaced with "our hypothesis is...". These notations should be consistent with "Dino → DINO".

clarity,_quality,_novelty_and_reproducibility: 
This paper proposed an interesting SSL framework and presented some encouraging results regarding the learning of various concepts via their framework. However, there are still missing details regarding how the different methods are being evaluated, as well as some empirical evidence and discussion that their framework is truly capable of capturing concepts from images in various settings. In addition, the presentation and structures should be improved so that the readers can easily follow.
summary_of_the_review: Please check the strength and weaknesses for detailed comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the MC-SSL method for self-supervised learning aimed at learning multiple concepts in images. It applies mainly two algorithmic techniques: Group Mask Model Learning (GMML) and learning of pseudo-concepts for data tokens using a momentum encoder framework. The paper is clear and interesting. The approach is novel and the results are good. 
strength_and_weaknesses: Clear and nice paper. I think the presentation was sometimes slightly too high level. I'd like to see just a few more equations around the approach, and some clearer definitions. 

Some other feedback:
* Figure 1 needs to be explained more, how many clusters were used etc, difference between second and third row.
* "they tend to encourage modelling of one dominant class per image using holistic representation and/or disregard the learning of contextual representations" where is the support of this claim? 
clarity,_quality,_novelty_and_reproducibility: Clear, high quality paper. It's novel and useful for the community. Understanding and improving on the differences in what SSL models learn is important as SSL is becoming more and more ubiquitous. I see no code in supplementary material, which is a big drawback in such an empirical work. 
summary_of_the_review: Strong paper with a novel technique (the aspect of pseudo-concepts on patch level) that's interesting and with great results. Given that this is an empirical paper, I believe code should be attached for reproducibility.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper claims to achieve multi-concept learning without dependency on human annotation by applying two patch-level loss simultaneously, including the patch reconstruction loss from SiT and the self-distillation loss from DINO. Extensive experiments and visualizations demonstrate the effectiveness of the proposed method MC-SSL.
strength_and_weaknesses: - Strength:
  - This paper organizes well with clear motivation and detailed experimental results.
  - The paper also explores the usage of popular self-supervised methods on data-scarce situations.

- Weakness:
  - Novelty is limited, especially compared with iBOT [1], which is published in ICLR 2022.
    - According to Table 5, the MC-SSL framework is exactly the same with iBOT combined with an extra auto-encoding reconstruction loss proposed by SiT.
    - According to the footnote in Page 6, the authors claim that the problem of iBOT is the usage of a global DINO head - "iBot focuses on learning a dominant concept via the classification token with DINO loss and hence, models the dominant class, which we think is a limitation of existing SSL methods.", which, however, is also used in MC-SSL according to Table 5. It seems that the multi-concept learning ability mainly comes from the usage of the reconstruction loss of SiT.
    - As iBOT is such an important baseline, it is not included in any of the experiments in Sec. 4.
    - The idea of combining reconstruction loss with patch-level contrastive learning has also been explored by some recent works [2,3].
  - Another important property of MC-SSL is the ability to learn multi-concept without annotations:
    - Different from [4, 5] which explicitly mine inter-image variance and invariance with K-means, MC-SSL only adopts two patch-level loss, which models inter-image variance and invariance in an implicit way (i.e., the intuition in Sec. 3.2). I wonder how an implicit way can achieve a better results than the explicit solutions in [4, 5]? Could you give more explanations about this problem?
    - It would be more interesting to report some numeric results to support the so-called multi-concept learning ability, like performance of unsupervised semantic segmentation.



[1] Zhou J, Wei C, Wang H, et al. ibot: Image bert pre-training with online tokenizer[J]. arXiv preprint arXiv:2111.07832, 2021.

[2] Dong X, Bao J, Zhang T, et al. Bootstrapped Masked Autoencoders for Vision BERT Pretraining[J]. arXiv preprint arXiv:2207.07116, 2022.

[3] Tao C, Zhu X, Huang G, et al. Siamese Image Modeling for Self-Supervised Vision Representation Learning[J]. arXiv preprint arXiv:2206.01204, 2022.

[4] Chen K, Hong L, Xu H, et al. Multisiam: Self-supervised multi-instance siamese representation learning for autonomous driving[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 7546-7554.

[5] Hénaff O J, Koppula S, Shelhamer E, et al. Object discovery and representation networks[J]. arXiv preprint arXiv:2203.08777, 2022.


clarity,_quality,_novelty_and_reproducibility: There is self-contradiction in this paper draft. Check the weakness part for more details.
summary_of_the_review: This is overall an interesting paper. However, the discussions can not be fully supported by the experiments. Furthermore, there exits self-contradiction and remaining unclear explanations in this paper draft version. I would like to see the author responses for more discussions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an approach for self-supervised training of Transformers based on the idea of masking and predicting. Two objectives have been utilized during the training process. One is patch reconstruction and the other one is named "patch concept classification" to learn different concepts/classes for data tokens. The results of the paper on downstream tasks show some improvements compared to some of the existing approaches.
strength_and_weaknesses: Strength:
	- A set of experiments have been conducted to demonstrate the effectiveness of the approach compared to some baselines.

Weaknesses:

1-  My main concern about this submission is the lack of novelty and proper comparison to SOTA. Recently, there are a bunch of well-known works existing in the literature with the same idea of self-supervised training of Transformers with masking and prediction. While I can see some discussions for BEiT and MAE, I can see only a footnote about iBOT as a very similar approach published in ICLR 2022 and almost no comparison with iBOT and BEiT in the experiments. These two approaches are very similar to the proposed one and there are only differences in defining tokenizer. I am not fully convinced by the note that the authors mentioned in the footnote about iBOT that it focuses on learning a dominant concept. I would like to see detailed discussions, experiments and proper comparisons to show the differences between the proposed and existing approaches.

2-  Comparisons on the object detection and semantic segmentation downstream tasks are not provided while the paper claims that the proposed approach would learn all semantic concepts in an image. This claim can be evaluated better on dense prediction tasks. Most of the existing approaches with the same idea such as MAE, iBOT, and BEiT have been evaluated on dense prediction tasks as downstream tasks.

3- I need more clarification on the "Patch Concept Learning" objective while we are feeding the image and its corrupted version to student and teacher networks. To me, it makes more sense if we learn the concept by feeding two different augmented views to the student and teacher and not the same view. This task to me seems like a trivial task at least for those patches which are not masked and are exactly the same for two networks. I would like the authors to clarify this in more detail.

clarity,_quality,_novelty_and_reproducibility: To me, the method is not novel enough and there are not a sufficient set of experiments to show the differences with the existing approaches.
 The implementation details are provided for the approach and it follows the same strategy as the existing approaches such as DINO, MAE, etc. I think that the method would be reproducible. 
summary_of_the_review: While there have been efforts to develop an idea for self-supervised pre-training of Transformers and evaluate it, however, there are some missing experiments that are very important to find out the benefit of this approach compared to the existing approaches in the literature. In my opinion, the paper needs major revision to fill those gaps I mentioned in the weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: *Summary*: This paper proposes to use a mixture of recontruction and self-distillation losses to learn an image representation. The authors argue that general self-distillation losses use a global feature and thus focus only on single dominant objects. In contrast, the authors propose - (1) patch-level self-distillation loss; (2) l1 reconstruction loss. The method is evaluated on image classification benchmarks - multiclass and multilabel settings. In the comments, the authors also provide segmentation results by training a linear classifier.

*Strengths*: (1) The problem setting studied in this paper is quite important. Global level losses indeed focus on single objects, which limits their applications. (2) Extensive empirical evaluation on multiple different datasets and settings. (3) Ablation experiments bring out the importance of image corruption, epochs, output dimension, and global feature learning. 

*Weaknesses*: (1) The central claim of the paper is that the proposed loss function enables multi-concept learning. The reviewers (rsxz, 85mW) , and the AC feel that this claim is under-evaluated. Datasets used in Table 1,2,3 (and majority of 5) are about single object recognition. The only results relevant to the claim are in Table 4. This table does not compare against other methods that use a patch-based loss (MAE, iBOT, data2vec, SimMIM, BeIT). (2) As pointed out by reviewer 85mW, it is unclear what enables MC-SSL's multiconcept learning - is it the reconstruction loss; the patch self-distillation loss; both?. Table 5 lacks this analysis. The relevant analysis would disable the reconstruction loss from MC-SSL, and the resulting method would be similar to iBOT. (3) The model settings used vary according to the dataset  and Table 5 shows that the global head actually hurts multi-label performance on VOC. This implies that the proposed solution needs to be catered heavily to the downstream task, a property not noted in the work.
justification_for_why_not_higher_score: This paper presents several issues - (1) lack of proper empirical comparisons; (2) ablation analysis; (3) correctly evaluating the central claim in the paper. Reviewers are aligned in not accepting the paper.
Decision: Reject