Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces self-supervised representation learning, which combines two well-known existing methods (i.e., contrast learning and mask autoencoder learning) in a way that appropriately obtains their respective advantages. Contrastive learning with Simese networks is faster in optimization compared to mask auto-encoder training, but is relatively poor at extracting spatially local information. On the other hand, masked encoder learning has performance advantages by making the learned representation have the ability to extract spatially local information, but the optimization is very slow. Since the main structural difference between the two existing methods is the preparation of the input, the proposed method uses visible patches and mask patches as two separate sets of patches required for contrastive learning. This method also proposed a novel mask region separation strategy that makes learning more difficult to obtain stronger representations. This strategy uses importance sampling using weights defined similarly to the attention weights for each value in ViT's self-attention module while input is the additional global token instead of local patches. Experiments have demonstrated that the learned representation through the proposed method (called ccSSL) was effective in terms of classification ability in linear probing task and finetuning task, and transferring ability in multiple downstream tasks.
strength_and_weaknesses: * Strength
1. The advantages and disadvantages of the two existing representation learning methods are well described and appropriately utilized in the method design.
2. The mask region separation strategy is novel and appears to be as effective as in tab 5. (The ablation experiments shown in tab 5 are well designed to support the use of the strategy.)
3. The learned representation through the proposed method was effective in all the evaluation tasks which are widely used in verifying the effectiveness of the representation learning.


* Weaknesses
1. Some informative ablation studies may be necessary. E.g., the effect of various masking rate.
2. Detailed information on how the [CLS] token is constructed is lacked or insufficient.
3. It is necessary to experimentally confirm the advantages and disadvantages of the two existing representation learning methods and whether the proposed method has acquired each of these advantages well.
clarity,_quality,_novelty_and_reproducibility: - Quality and novelty are sufficient to meet the ICLR standards.
- It is difficult to pinpoint exactly why the method performs well because there are not a few ablation studies needed for the clarity of the methodology.
- As the explanation of  constructing [CLS] token, it is not straightforward to reproduce this method with this manuscript only.
summary_of_the_review: As this paper introduces a new and effective expression learning method by appropriately utilizing the advantages of the two well-known methods, I am on the positive side and I think it will attract the attention of many researchers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a contrasting-aided contextual masked image modeling framework, termed ccMIM. The main motivation behind this paper is to use CLS tokens to mask semantic patches so that the reconstruction target could be more difficult. To make the CLS token contains more semantic context, ccMIM introduces a global alignment loss (InfoNCE or Self-Distillation) on the CLS token. Lots of experiments are conducted to show the superiority of ccMIM.

strength_and_weaknesses: ### Strengths

1. The motivation makes sense. The paper writing is easy to understand.
2. Results on image classification / object detection / semantic segmentation all show improvements.

### Weaknesses

1. As shown in Fig.1, the importance score of each patch is computed with the CLS token before feeding into ViT-Encoder. This means the CLS token has no interaction with image patches. So is the importance ranking generated from the CLS token really reliable? More descriptions and visualizations should be added for clear understanding.
2. Bringing contrastive learning to MIM is already well-studied in many works. The more important component in this paper is the importance-based sampling strategy. However, sampling semantic-less patches for harder self-supervised learning objectives is suitable for not only MIM but also contrastive learning methods. It's important to conduct such a sampling strategy to contrastive learning methods (e.g., BYOL [Grill et al., 2020] or SimCLR [Chen et al., 2020b]) and see if consistent performance gains could be achieved.
3. ccMIM feeds both visible patches and masked patches into ViT encoder. The extra computation costs are not mentioned in the paper.
4. Improvements compared to previous methods are a bit marginal.
5. Experiments on larger architectures (e.g., ViT-L) could bring more authority.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is easy to follow despite some details needing clarification (see Weaknesses).
- Quality: The paper addresses a practical problem in an intuitive way, but the experimental results seem insufficient to fully validate the method.
- Novelty: Brings contrastive learning to MIM is already well-studied in many works. The importance-based sampling strategy is new to the self-supervised learning community. But it should be better to discuss such a strategy with contrastive learning.
- Reproducibility: Hyper-parameters and implementation details are all included in the paper.
summary_of_the_review: This paper presents ccMIM. The core component behind ccMIM is to sample semantic-less patches for harder self-supervised learning objectives. Despite better results being achieved, the core contribution of ccMIM is not fully studied (see weakness).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 1, Adopt importance sampling to select the masked patches with richer semantic information for reconstruction, instead of random sampling as done in previous MIM works.
2, Propose a new contrastive loss that aligns the tokens of the vision transformer extracted from the selected masked patches and the remaining ones.
3, The proposed contextual MIM and contrastive learning are synergetically performed in a loop with fast convergence and strong performance on downstream tasks without ad-hoc augmentations
strength_and_weaknesses: Strength
1, Novel framework for synergizing MIM and contrastive learning in a close-loop.
2, Improvement over MAE: Better than MAE with shorter training epochs.
Weaknesses
1, Do you need to extract the mask set feature? Will that slow down the training?
2, Why the s means the importance of the patches? Will the global token have global information at the beginning of the training?
3, Can you compare the real training time? Your algorithm seems to be much more complicated than the original MAE. (thus More time over one epoch).
4, Which token do you use for the downstreaming classifcation task?


clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, Novelty, and Reproducibility are OK.
summary_of_the_review: The method provides a better version of the original MAE with better performance and faster convergence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper introduces self-supervised representation learning, which combines two well-known existing methods (i.e., contrast learning and mask autoencoder learning) in a way that appropriately obtains their respective advantages. The main motivation behind this paper is to use CLS tokens to mask semantic patches so that the reconstruction target could be more difficult. To make the CLS token contains more semantic context, ccMIM introduces a global alignment loss (InfoNCE or Self-Distillation) on the CLS token.

**Strengths:** The motivation makes sense. The paper writing is easy to understand. Results on image classification / object detection / semantic segmentation all show improvements.

**Weaknesses:** The following major concerns were identified by the reviewers:

1. missing some ablation studies (e.g., effect of mask rate) [pWuU]
2. missing information about how [CLS] token is constructed [pWuU]
3. need to confirm in experiments whether the advantages/disadvantages of the 2 representation learning methods is present in the proposed combination model. [pWuU]
4. CLS token has no interaction with image patches. Is it reliable? [ZDzp]
5. the core part is the sampling strategy. but there are missing experiments using the same patch sampling strategy for other CL methods, to show the effectiveness of the sampling strategy. [ZDzp]
6. what is the extra computation cost? [ZDzp, tKmv]
7. marginal improvement compared to previous methods. [ZDzp]
8. missing experiments on larger architectures (ViT-l) [ZDzp]
9. Why is s the importance of the patch? [tKmv]
10. Will the global token have global information at beginning of training? [tKmv]
11. Which token is used for downstream classification? [tKmv]


justification_for_why_not_higher_score: The methodology is somewhat incremental, since it combines two previous works, MIM and contrastive learning. 
justification_for_why_not_lower_score: Although the novelty is somewhat incremental, combining contrastive learning and MIM is done in a clever way, to achieve synergy between the two methods, and the results are good.  MIM and Contrastive learning are two important branches of self-supervised learning, and their combination is interesting.
Decision: Accept: poster