SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation

Authors: Yike Yuan, Huanzhang Dou, Fengjun Guo, Xi Li
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised learning, Masked image modeling

Abstract:
This paper represents a neat yet effective framework, named SemanticMIM, to integrate the advantages of masked image modeling (MIM) and contrastive learning (CL) for general visual representation. We conduct a thorough comparative analysis between CL and MIM, revealing that their complementary advantages fundamentally stem from two distinct phases, i.e., compression and reconstruction. Specifically, SemanticMIM leverages a proxy architecture that customizes interaction between image and mask tokens, bridging these two phases to achieve general visual representation with the property of abundant semantic and positional awareness. Through extensive qualitative and quantitative evaluations, we demonstrate that SemanticMIM effectively amalgamates the benefits of CL and MIM, leading to significant enhancement of performance and feature linear separability. SemanticMIM also offers notable interpretability through attention response visualization.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729343020444

thread_type:
official_review

<thread_object>
creator: reviewer_weyn

<content>
summary:
This paper proposes SemanticMIM, a framework that integrates the strengths of masked image modeling (MIM) and contrastive learning (CL) to achieve general visual representation learning. Specifically, SemanticMIM leverages a proxy architecture to compress information from image tokens into proxy tokens and then reconstructs masked tokens, thereby combining the semantic awareness of CL with the spatial sensitivity of MIM. As the approach only modifies the encoder architecture, it can be seamlessly integrated into any MIM framework. Extensive experiments further validate the effectiveness of the model and offer interpretability through attention visualization.

strengths:
1. The analysis of the fundamental principles underlying CL and MIM is insightful and thought-provoking.

2. The proposed SemanticMIM is well-motivated, which leverages the strengths of both CL and MIM.

3. SemanticMIM improves the performance of two MIM methods when applied using the training scheme introduced in this paper.

weaknesses:
1. Some important details about the training process are missing. For instance, are the [PROXY] tokens supervised with specific targets, similar to the supervision used in CL? Are the [MASK] tokens trained using the same loss function employed in typical MIM frameworks? Clarifying these aspects would enhance the reproducibility and understanding of the proposed approach.

2. The classification of SSL methods into only CL and MIM may not be entirely accurate. Earlier works also introduced a variety of pretext tasks, such as image colorization and rotation prediction, which play an important role in the development of SSL methods. A more comprehensive overview of these earlier approaches can be found in the survey “Self-supervised visual feature learning with deep neural networks: A survey”.

3. The reproduced performance of BEiT and Maskfeat in this paper appears much lower than the original results reported in their respective papers. Could you clarify why the official training schemes were not followed? Additionally, if the official training schemes were applied, how much improvement would SemanticMIM bring to these MIM methods?

4. The citation format in some sections of the text requires correction. For example, “Self-supervised learning (SSL) algorithms Liu et al. (2021); Balestriero et al. (2023) have emerged as …” should be revised to “Self-supervised learning (SSL) algorithms (Liu et al. 2021; Balestriero et al. 2023) have emerged as …” to conform to general academic writing practices.

questions:
From Fig. 14, it appears that different layer depths yield different types of attention maps. In Fig. 1, are all the displayed attention maps derived from the same layer depth? Providing this clarification will help readers better understand the visual interpretability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730145062943

thread_type:
official_review

<thread_object>
creator: reviewer_efcx

<content>
summary:
This paper proposes a self-supervised learning method to pretrain vision transformers named SemanticMIM. SemanticMIM tries to take the advantage from both masked image modeling and contrastive learning by leveraging a proxy architecture that customizes interaction between image and mask tokens, bridging these two phases to archieve general visual representation with the property of abundant semantic and positional awareness. I think the novelty of this paper is good. But my presentation is poor and the experiments are so insufficient.

strengths:
1. The novelty seems good, a very interesting method to combine the MIM and CL together;
2. Many analysis experiments enhance the quality of this paper;

weaknesses:
1.I do not think attention response (figure 1&7&8) can represent the the quality of the learned features. For example, the attention responses of models using the CLIP model as a supervisory signal all appear poor, but their actual performance is often much better.
2.It seems that the authors did not use \citep{} correctly while writing and instead used \cite{}. This resulted in an inconsistent citation format throughout the entire paper.
3.I suggest the authors to provide a comparison on training cost, because the proposed method seems have larger computational cost.
4.Experimental results are insufficient: longer epochs and larger models (L-scale) need to be validated, which is crucial for self-supervised learning. Downstream task results are also needed, such as semantic segmentation and object detection. (This is the biggest concern!)

questions:
Do you think the downstream task performance is good enough to evaluate a self-supervised model? If so, the current MIM methods have achieved very good results and I think the proposed method has NO potential to reach such high results. If not, what metrics is needed do you think to evaluate a pretrained model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730364441193

thread_type:
official_review

<thread_object>
creator: reviewer_h81s

<content>
summary:
The paper introduces SemanticMIM, a framework that combines Masked Image Modeling (MIM) and Contrastive Learning (CL) for enhanced visual representation.

Key Contributions:

Theoretical Insights: It analyzes the complementary strengths of MIM and CL, emphasizing their different approaches to semantic modeling—CL focuses on global semantics, while MIM emphasizes local details.

Framework Design: SemanticMIM integrates CL's benefits within the MIM framework using a proxy architecture that combines compression and reconstruction processes into a single learning framework.

Experimental Results: The framework demonstrates significant performance improvements in distinguishing specific object semantics and identifying relevant features, outperforming both MIM and CL in various tasks.

Conclusion: SemanticMIM effectively captures global and spatial information, leading to notable advancements in visual representation for downstream applications.

strengths:
The strengths of this paper lie in its innovative integration of Masked Image Modeling (MIM) and Contrastive Learning (CL) within a unified framework, effectively leveraging the advantages of both approaches. Additionally, SemanticMIM excels at capturing both global semantics and local features, enhancing the ability to distinguish specific object semantics. Finally, in addition to quantitative experiments, the paper also includes extensive qualitative analyses and visualizations to demonstrate the effectiveness of the framework.

weaknesses:
The paper's limitations include insufficient quantitative experiments, particularly a lack of tests with models of varying sizes across different settings. Additionally, while the paper provides segmentation results for downstream tasks, the promising attention effects raise expectations for object detection performance as well. It would be beneficial for the authors to include more experimental results in the rebuttal stage.

questions:
The shortcomings in the experimental section raise concerns about whether this training method can scale to larger models and be applied to a broader range of downstream tasks. If the authors can provide additional experimental results, I would be willing to reconsider and potentially increase the score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730744398403

thread_type:
official_review

<thread_object>
creator: reviewer_zqbb

<content>
summary:
This paper first compares the pros and cons of two related but different lines of work: masked image modeling (MIM) and contrastive learning (CL). They then propose SemanticMIM which brings the pros of CL, i.e. suppression, and global representation, etc, into MIM. Experiments show improvement in several tasks.

strengths:
1. The analysis of MIM and CL can be beneficial to the community.
2. Introducing [PROXY] tokens between [CLS] and [MASK] as the solution is simple.
3. Visualization in the experiments highlights the claims of the paper.

weaknesses:
1. The presentation is not good. The figures do not help explain the methods.

questions:
1. The layout of Fig. 1 can be changed to fit its goal of comparing MIM and CL as the following.
Image A BeiT-A MoCov-A Ours-A
Image B BeiT-B MoCov-B Ours-B
The current layout is confusing at first glance since A and B are "up vs. down" on the left subfigure but "left vs. right" on the right subfigure.

2. Fig. 1 needs a clearer caption to explain it. The caption does not explain what the reader should be observing here. In what way is "ours" better than the baselines? The paper only explains it later in L49, L246, and L264. Moreover, in L49, how does a reader realize the following claim "MIM focuses on the reconstruction of partially corrupted images, serving as a pretext task that facilitates the model’s ability to infer local patterns from contextual information rather than grasping global semantics" from the figure? The color yellow for the boxes is too similar to the chosen colormap.

3. Fig. 2 can be also made clearer. What is the purpose of the target generators here? What is being trained? The [CLS] tokens and [MASK] tokens should be indicated in this figure. The token color of Fig.2 and Fig.3 should match.

4. In Fig 3, why does the [MASK] token in MIM go straight to [TARGET]? As in Fig. 2, same as CL, MIM also outputs some tokens (blue). The major difference of [MASK] token having positional embedding should be indicated in Fig.2 or 3.

5. In L262, there is a missing space between SimMIM and the citation.

6. In Fig. 4, it does not seem like compression for SemanticsMIM since the number of [MASK] matches the number of [IMG].

7. In L344, semanticMIM -> SemanticMIM.

8. The baselines seem outdated (2021~2022). Can SemanticMIM be compared to [1] or [2]?

9. In the experiment section, why use the term  "[CLS] token with i.e. [PROXY] token" but not just "[PROXY] token"? It is confusing to read.

10. I think the caption in Fig.5 and Fig. 6 should be y-axis (singular).

11. In Sec. 4.4, as a comparison, how many [IMAGE] tokens are there?

12. In Tab. 1, Why compare to only Linear for PascalVOC and only FT for ADE20K? Is using the CLS and Patch tokens as auxiliary inputs for the classifier necessary in the ImageNet experiments? Why is there no "using CLS or Patch tokens" for PascalVOC and ADE20K?

[1] MIMIC: Masked Image Modeling with Image Correspondences. CVPRW 2024.
[2] Learning Vision from Models Rivals Learning Vision from Data. CVPR 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732365510620

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
We appreciate all reviewers for their careful reading and insightful feedback and we are pleased to see that all reviewers found our proposed analysis and method to be novel, insightful, and beneficial to the community. The main concerns raised by the reviewers focused on presentation and experiments. We have addressed these questions in detail, revised the manuscript accordingly, and conducted additional experiments. Here is a brief summary:

1. We have added experiments on scaling up the model and serving as backbone for object detection (see response to Reviewer H81s and Efcx).
2. We have redrawn all the figures to enhance the quality of the presentation.
    - Fig.1: We adjusted the layout of the images, used red solid boxes to annotate query image patches. We rewrite caption with descriptions of the model properties emerged for better understanding. We also pointed out the layers corresponding to the attention responses for clarity.
    - Fig.2: We modify the token color, matched with Fig.3 and Fig.4 to make it less confusing and mark the query tokens.
    - Fig.3: Token colors are matched and position ids are added to each kind of tokens.
    - Fig.4: We match the token colors, add target tokens and adjust the token numbers to emphasize compression.
3. We have corrected the citation format errors and some typos in the manuscript.
4. We clarified the concept of CL and MIM in our manuscript to make them more rigorous. 
5. In the experimental section, we included more implementation details of the modification of SemanticMIM compared to general MIM framework.

Thanks again for all reviewer's comments and acknowledgment of our work. We sincerely look forward to the opportunity to engage in further discussions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a visual pre-training algorithm that unifies masked image modeling and contrastive learning. It first formulates these two types of pre-training methods using one framework, and then proposes a new pipeline that uses compressed image tokens to guide the model to reconstruct the masked tokens. Although with a smaller number of [PROXY] tokens, the overall framework is **not** essentially different from the vanilla MIM -- note that MIM, with a large masking ratio, can force the model to work on highly-compressed image tokens. Also, the experiments are not strong enough to validate the effectiveness; in particular, experiments on larger backbones and downstream tasks are not complete.

The overall rating of this paper falls on the borderline (5/5/6/6). The reviewers are mostly concerned about the writing (some typos, missing technical details, etc.) and experiments (small models, short training schedules, weaker baselines, etc.). Given this many points to resolve, the AC cannot suggest acceptance this time and suggests the authors to continue improving this work.