Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies an interesting layer-grafting technique to promote the combination of CL and MIM for representation learning. Specially, the authors find that directly combining such two paradigms shows the negative influence due to the gradient conflicts, and discovers that the sequential cascade in a certain order can address this problem and consistently improve the performance of representation learning. 
strength_and_weaknesses: The strength of this submission can be summarized as follows

1)	The authors systematically studied the combination of CL and MIM and analyzes the conflict in the gradient direction that incurs the incompatibility of the two paradigms.

2)	Different from the straightforward combination, the authors proposed to reconcile the conflicts of two paradigms via layer grafting, namely, a cascade way. Specifically, the authors show the order the layer grafting make the different effect, where MIM contributes more in the lower layer and CL contributes more in the higher lower. Then, by means of their advantages of CL and MIM, the authors introduce the layer-grafted pretraining method to enhance representation learning.

3)	A range of experiments have demonstrated that the layer-grafted pretraining significantly improves the performance of either CL or MIM, and achieves the new state-of-the-art compared with the current strong baselines including MoCoV3 and SIM. More extensive analysis of LR search or ablation study as well as the finetuning, and segmentation further confirms the designing intuition and advantages of layer-grafted techniques.

4)	The submission is well written with clear logic, and the topic of this work is well motivated based on sufficient evidence on the direct combination. 

Minor weakness

1)	LR search directly reflects the grafting degree among layers, which can be studied further in other different datasets to confirm whether it consumes much empirical experience.

2)	The results of the straightforward combination of MIM and CL can be added to tables when compared with the state-of-the-art baselines. This will strengthen the advantages of the proposed layer-grafting techniques.

3)	Layer-grafting pretraining for more general scenarios can be discussed to motivate more explorations in the future.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written with high quality, and novelty.

The whole algorithm is reproducible.
summary_of_the_review: The paper provides a new road for representation learning. The observed phenomenon is interesting and important. Then, a novel method is proposed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a simple method to combine masked image modeling (MIM) and contrastive learning (CL) into a joint image pretraining framework. Specifically, MIM and CL are applied to the intermediate block and the end block respectively during training. Two steps are conducted: MIM is applied first and then its learning rate is decreased and CL is added into training. 

strength_and_weaknesses: Strengths:
1. The proposed method is simple and easy to follow. The writing is clear.
2. Experimental results show improvement over models either trained by MIM or CL separately on the few shot learning setting.

Weaknesses:
1. While the simplicity of the method is desirable, the novelty of the proposed method is relatively limited. It is well known that combining CL and MIM cannot bring significant improvement, as found in [a,b]. In the paper, the authors only compare performance with [a] without a deep analysis of their differences. Reference [b] is missing. The authors are suggested to compare their method in more detail with existing methods to clarify their contributions, e.g. why their strategy is better than other methods.

[a] C. Tao, et al. Siamese image modeling for self-supervised vision representation learning. arXiv preprint arXiv:2206.01204, 2022.

[b] Z. Huang, et al. Contrastive masked autoencoders are stronger vision learners. arXiv preprint arXiv:2207.13532, 2022.

2. The motivation for solving conflict gradients of MIM/CL may be problematic and need further verification. There are several factors ignored in current experiments which make the conclusion less convincing.
- The gradient discrepancy is not severe according to Fig.1: most numbers are still positive, which may imply the gradients may not be the root cause.
- Do those statistical results remain the same across the whole tranining process? Or it is only like this at specific training point (e.g. 100 epoch as in Fig. 1). In other words, MIM and CL may not always have gradient conflict, so the effect of gradient issue may be mistakenly enlarged.

3. The performance is not competitive when compared with existing methods. The finetune result on ImageNet is the same as original MAE, and worse than [a,b] and others like iBot, showing the limits of applying this method in real use. Compared with finetune results, few shot results are less concerned since the paper mainly claims a better pipeline/training strategy over initial MAE and CL.
clarity,_quality,_novelty_and_reproducibility: See my comments above.
summary_of_the_review: See my comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new approach to combine both Contrastive learning (CL) and Mask Image Modeling (MIM) to take advantage of the best of both worlds. An analysis shows that a simple combination of the losses does not work well because there are some conflicts between the losses. To solve this problem, the paper introduces a two-step training approach strategy where the model is first trained with a MIM loss, and then fine-tuned with a CL loss. The proposed approach is evaluated on ImageNet in multiple scenarios: linear, 1% few-shot, 10% few-shot, and fine-tuning. The proposed approach improves performances in these scenarios. 
strength_and_weaknesses: **Strengths**
- The paper proposes a new approach to combine both Contrastive learning (CL) and Mask Image Modeling (MIM) to take advantage of the best of both worlds. 
- Combining CL and MIM losses does not seem straightforward. An analysis in the paper shows naive joint optimization of CL and MIM losses does not work well because there are some conflicts between the losses. Figure 1 shows the cosine similarity between MIM and CL gradients for each layer. For each layer, there are negative values that indicate the MIM and CL are optimized in opposite directions. The paper also gives a reason to justify these conflicts: the MIM loss is designed to reconstruct the input so it should not be invariant to data augmentations, whereas the CL loss is designed to ensure that the model remains invariant to some data augmentations.
- The paper introduces a simple two-step training approach strategy to reduce the conflicts between MIM and CL losses. First, the model is trained with a MIM loss. Second, the model is fine-tuned with a CL loss. The proposed strategy is not specific to a Vision Transformer architecture and can be used with multiple Vision Transformer architectures. An empirical analysis shows that the order is important so the model should be trained with the MIM loss and it is not a good idea to start with the CL loss.
- The proposed approach is evaluated on ImageNet in multiple scenarios: linear, 1% few-shot, 10% few-shot, and fine-tuning. The proposed approach improves performances in these scenarios for both ViT-B/16 and ViT-L/16 architectures.


**Weaknesses**
- The paper does not introduce new string technical contributions. The proposed approach relies on existing techniques, but the combination and context seem novel.
- The proposed approach seems specific to Vision Transformer architectures and may not adapt to other model architectures like ConvNets. *[post-rebuttal] After the author answer, it does not seem to be a weakness because it can be used with ConvNets.*
- I think adding studying the transfer learning performances will improve the quality of the paper. It is quite standard to study how the learned representations transfer to some downstream tasks. I think analysing the transfer learning performances will be more interesting than the visualization for feature space (Figure 3). *[post-rebuttal] The authors added new transfer learning experiments to answer this concern. Table C shows the performances of an UperNet with different pre-training methods.*


Minor: there is an error at the end of the intro: “performing MIM->CL will considerably damage the performance.” It should be “CL->MIM”

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and understand. It clearly identifies a problem and proposes a solution. The solution is validated empirically in multiple scenarios. I think some experiments may be difficult to reproduce because some implementation details are missing. For example, there are a lot of details about the optimizer but not a lot about the data preprocessing/data augmentation. I encourage the authors to release their code or to add the implementation details in an appendix.  

summary_of_the_review: Overall, I like the idea of combining both Contrastive learning (CL) and Mask Image Modeling (MIM) to take advantage of the best of both worlds. A problem to combine CL and MIM losses is well identified and a solution is proposed, but the proposed solution relies on existing techniques.  

------------------------------------------

[initial post rebuttal]

After reading all the reviews, the rebuttal and the updated paper, I decrease my score to 5. Overall, I think the idea of combining MIM and CL is interesting and well motivated in the paper. However, as point out in multiple reviews, the performance gains on some benchmarks do not seem significant. In the fine-tuning setting, which is popular setting, the proposed approach is 0.8pt worst than C-MAE (Table A). The performance gain seems significant only in the few shots and linear settings. I think it is important to see a more consistent and significant gains on most of the benchmarks to validate the proposed approach. 

------------------------------------------

I updated my review and rating to ignore the recent works and to take into account the linear and few-shot settings are the main targets of the paper. I would suggest to update the paper to reflect it. For example, the abstract is not clear about this point.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to learn better visual representation by leveraging both contrastive learning (CL) and masked image modeling (MIM). The authors claim that simultaneously performing CL and MIM pertaining will lead to worse results. To mitigate this issue, the paper proposes Layer Grafted Pre-training, which finetunes a pretrained MIM model (e.g., MAE) using contrastive learning (e.g., MoCo-v3 and VICReg). The pretrained models are evaluated on ImageNet with linear and fine-tuning protocols.
strength_and_weaknesses: Strength:
- Combining contrastive learning and masked image modeling is a popular problem and has attracted much attention. This paper provides a very simple solution to benefit from contrastive learning and masked image modeling at the same time.
- The paper shows high quality in writing. The presentation of both methods and experiments is intuitive. It is easy to follow and comprehend.

Weakness:
- The paper presents limited novelty and contribution. The method simply finetunes a strong MIM pretrained model using contrastive learning, and then transfer it to downstream classification task. It can be viewed as an intermediate fine-tuning, e.g.finetune a MIM model using ImageNet22k and transfer to ImageNet1k,  which is used in BEiT and adopted by some follow-up papers. The intermediate fine-tuning has been demonstrated to be very effective and can bring 1.5% Top-1 accuracy. 
- The experiments are insufficient to demonstrate the effectiveness. The method only improves 0.3% top-1 accuracy when fine-tuning on ImageNet. In addition, fine-tuning experiments are only conducted using ViT-B/16 backbone. The results of ViT-L/16 should be reported.
- The experiments in Table 4 show that the method doesn't work when using VICReg. The fine-tuning result is the same as the original MAE, i.e., 83.6%. 
- What about conducting multi-task learning but performing masked image modeling at lower layers? This is an important baseline according to your motivation.
- There are some papers (e.g., A, B, C, etc) that also combine contrastive learning and masked image modeling, including a few listed in related work. More comparison and discussion can be added.
- The paper can also be viewed as using self-supervised pretraining (CL) improves self-supervised pertaining (MIM), which is similar to [D]. Discussion can be added.

[A] Siamese image modeling for self-supervised vision representation learning.

[B] MimCo: Masked Image Modeling Pre-training with Contrastive Teacher

[C] Masked Image Modeling with Denoising Contrast

[D] Self-Supervised Pretraining Improves Self-Supervised Pretraining
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity, limited novelty, and borderline quality.
summary_of_the_review: Overall, I think the paper is well-written and motivated. But it has limited technical contribution and provides few new insights and knowledge. The method is also not that effective. I tend to give a reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes Layer Grafted Pre-training, a simple two-step approach for bridging two recent self-supervised learning methods, which are Masked Image Modeling (MIM) and Contrastive Learning. The key observation is that both objectives have conflict aspects, and separating them into lower and higher layers can enhance the stability of joint optimization of MIM and CL. Specifically, in a cascade fashion, the proposed method pre-trains early layers under MIM loss and then trains the other layers under CL loss. The extensive experiments showed the effectiveness of the proposed method on image classification tasks such as few-shot classification and linear evaluation.
strength_and_weaknesses: Strengths
- The writing is clear and easy to understand.
- This manuscript tackles a recently popular and interesting problem that unifies two large paradigms (i.e., Contrastive Learning (CL) and Masked Image Modeling (MIM)) that divide recent self-supervised learning.
- The proposed method outperforms the state-of-the-art self-supervised learning baselines on the image-classification tasks.

Weaknesses
- Overall, the authors validate the proposed method on CL-centric evaluation metrics (e.g., linear evaluation, few-shot classification). One of the main differences between CL and MIM is that "MIM helps the model to learn rich local structures within the image" (as mentioned in the manuscript), which can be beneficial to dense prediction-type downstream tasks such as object detection and semantic segmentation. For example, MAE [1] has shown superiorities of MIM in COCO object detection and segmentation, and on ADE20K semantic segmentation, while it shows poor performance on linear probing compared to fine-tuning on image classification tasks. However, the proposed method only highlights the benefits of image classification tasks, which are close to the benefits of CL. Moreover, improvements compared to CL are much smaller than MIM. In short, my question is, " Is the proposed method a unified method of CL and MIM or a slightly better variant of CL?" If so, does the proposed method inherit the benefits of MIM? (e.g., dense prediction tasks)
- The key observation of the manuscript is that CL and MIM loss objectives are in conflict. However, I am questionable that the proposed MTL combination is the right way to conclude it. Specifically, the authors enforce the model to discriminate against all the augmentations under MIM loss, while invariant against the same augmentations under CL loss. I think combining MIM without augmentation (or minimal) and CL with augmentation is a more straightforward approach with less conflict. Furthermore, in the experiments section, the authors also adopt MAE models, which are pre-trained without augmentations. I would like to ask the authors to provide more explanations and valuable insights about this.
- In the preliminary section 2.1, it is unclear which features are used for contrastive learning; which are they between outputs of the encoding function $f$ or the decoding function $d$? Does the decoding function $d$ be discarded after training MIM?

[1] He et al., "Masked Autoencoders Are Scalable Vision Learners," CVPR 2022
clarity,_quality,_novelty_and_reproducibility: Clarity 
- Clear, only minor flaws

Quality
- Technically strong results

Novelty
- Minor variations to existing techniques. But its empirical finding is valuable to be shared

Reproducibility
- Detailed configurations are provided in the manuscript.
summary_of_the_review: Overall, I have several concerns, as I mentioned above weaknesses section.
For now, I weakly recommend accepting this manuscript.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies an interesting layer-grafting technique to promote the combination of CL and MIM for representation learning. Specially, the authors find that directly combining such two paradigms shows the negative influence due to the gradient conflicts, and discovers that the sequential cascade in a certain order can address this problem and consistently improve the performance of representation learning. A range of experiments have demonstrated that the layer-grafted pretraining significantly improves the performance of either CL or MIM, and achieves the new state-of-the-art compared with the current strong baselines including MoCoV3 and SIM. More extensive analysis of LR search or ablation study as well as the finetuning, and segmentation further confirms the designing intuition and advantages of layer-grafted techniques. For example, this paper has shown its strengths on some benchmarks, including linear evaluation, few-shot classification, and ADE20K segmentation.

The clarity and novelty are obviously above the bar of ICLR. While the reviewers had some concerns on the limitation of performance, the authors did a particularly good job in their rebuttal. Note that there are two points to be merged into the updated version: 1) if the few-shot setting is main focus, this paper needs to present more analysis on why the method is specifically useful for the few-shot and the differences in features required between few-shot and fine-tuning; 2) if the linear and few-shot settings are the main targets of the paper, the authors are strongly suggested to update the paper to reflect it. For example, the abstract and title are not clear about this point, and should be updated. Overall, after a zoom meeting, all of us have made a consensus to accept this paper for publication! Please include the additional experimental results and further explanation in the next version.
justification_for_why_not_higher_score: According to my expertise and reviewing process, this paper should belong to an Accept with poster.
justification_for_why_not_lower_score: According to my expertise and reviewing process, this paper should belong to an Accept with poster.
Decision: Accept: poster