Morphing Tokens Draw Strong Masked Image Models

Authors: Taekyung Kim, Byeongho Heo, Dongyoon Han
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised learning

Abstract:
Masked image modeling (MIM) has emerged as a promising approach for pre-training Vision Transformers (ViTs). MIMs predict masked tokens token-wise to recover target signals that are tokenized from images or generated by pre-trained models like vision-language models. While using tokenizers or pre-trained models is viable, they often offer spatially inconsistent supervision even for neighboring tokens, hindering models from learning discriminative representations. Our pilot study identifies spatial inconsistency in supervisory signals and suggests that addressing it can improve representation learning. Building upon this insight, we introduce Dynamic Token Morphing (DTM), a novel method that dynamically aggregates tokens while preserving context to generate contextualized targets, thereby likely reducing spatial inconsistency. DTM is compatible with various SSL frameworks; we showcase significantly improved MIM results, barely introducing extra training costs. Our method facilitates MIM training by using more spatially consistent targets, resulting in improved training trends as evidenced by lower losses. Experiments on ImageNet-1K and ADE20K demonstrate DTM's superiority, which surpasses complex state-of-the-art MIM methods. Furthermore, the evaluation of transfer learning on downstream tasks like iNaturalist, along with extensive empirical studies, supports DTM's effectiveness.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730398313666

thread_type:
official_review

<thread_object>
creator: reviewer_l29y

<content>
summary:
This paper proposes Dynamic Token Morphing (DTM) to mitigate the identified spatial inconsistencies discriminative problem. This method can be used fot both masked image modeling methods and contrastive methods and accelerating the training. The achieved results look good and I think the novelty is also good. But I concern several weaknesses as I noted below. I may change the rate after reviewing other reviewers' review.

strengths:
1.A interesting insight on masked image modeling pre-training methods and good results are achieved across all scale models including ViT-S/B/L.
2.The proposed method can be used for both MIM and contrastive learning mothods

weaknesses:
1.Why not compare your method with hierarchical ViTs, such as ConvMAE [1], HiViT [2], iTPN [3], GreenMIM [4]. Is your method still effective on these methods? I suggest conducting an experiment on maybe one Small scale model.
2.The authors argue that the current methods meets the trouble of spatial inconsistencies, but existing ones actually have achieved very good results, such as EVA [5], EVA02 [6], Fast-iTPN [7]. How the authors expain that? If you think the proposed method can still improve their performance, I think conducting experiments is necessary.
3.Missing COCO experiments

[1]ConvMAE: Masked Convolution Meets Masked Autoencoders, [NeurIPS 2022]
[2]HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer, [ICLR2023]
[3]Integrally Pretrained Transformer Pyramid Networks, [CVPR2023]
[4]Green Hierarchical Vision Transformer for Masked Image Modeling, [NeurIPS 2022]
[5]EVA: Exploring the Limits of Masked Visual Representation Learning at Scale [CVPR2023]
[6]EVA-02: A Visual Representation for Neon Genesis
[7]Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration [TPAMI2024]

questions:
See the second point of Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730897734411

thread_type:
official_review

<thread_object>
creator: reviewer_gb6h

<content>
summary:
This paper investigates the issue of spatially inconsistent target representations during pre-training. The authors conducted quantitative and qualitative experimental observations of this phenomenon to discuss the impact of such spatial inconsistencies, particularly on the performance of downstream classification tasks. To address this issue, based on the MIM baseline, they propose an advanced token aggregation method termed Dynamic Token Morphing (DTM), which integrates tokens with contextual associations. The paper reports the classification results on ImageNet-1k and the segmentation performance on ADE20k, demonstrating consistent improvements with DTM across the SSL framework and the SLIP model.

strengths:
1.This paper provided an extensive discussion and demonstration of the spatial inconsistency in pre-trained representations, illustrating the significance of this issue in the domain of self-supervised learning.
2. DTMs re-characterize visual features, mitigating the inflexibility associated with patch-based representations.
3. This paper integrated DTM with multiple baseline methods, such as MAE, BEiT v2, BYOL, etc., demonstrating the effectiveness and convenience of the proposed approach.

weaknesses:
1. Although DTM presents an intuitive feasibility in addressing the spatial inconsistency of visual representations, the explanation in the methods section is not very intuitive or detailed, which may lead to comprehension errors. For instance, the process of how to perform k iterations based on bipartite matching to obtain the final token morphing matrix M, and how to derive the aggregated results from M, is not clear. Perhaps an algorithm flowchart would be beneficial.
2. Some details in the writing of the article could be further optimized. For example, on line 264, whether generating vi with xiM should not include masked tokens, as there is an inconsistency with Figure 4 (the input of the target encoder does not include masked tokens). Additionally, in Figure 4, the lines connecting the Morphing Matrix to Vi and Ui are different; does this indicate different operations?
3. The authors emphasize in the abstract that DTM brings about faster training, but when comparing with other methods, especially the baseline method, the best results are achieved using the same number of epochs. Does this imply that the overall training time was reduced?

questions:
In addition to weakness, there are also some questions:
1. In line 239, "we do not employ a superpixel-based clustering for our method, which is inefficient as well." Is the decision to avoid superpixel-based clustering solely due to its low performance? Could the reasons be analyzed further?
2. Please explain why the absence of a dynamic mechanism results in lower performance than the baseline in Table 7.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731052714080

thread_type:
official_review

<thread_object>
creator: reviewer_xemo

<content>
summary:
The paper proposes a novel masked image modeling method. The paper identified the issue of the existing pre-training models: the spatial consistency in target representations despite proximity and contextual similarity. Then the author proposed a context-aware token aggregation method called Dynamic Token Morphing (DTM). The experiments show that the proposed DTM method is general enough to help improve various SSL frameworks and achieve SOTA ImageNet-1k performance. In addition, the computation complexity add-on of the proposed method is not large.

strengths:
- The paper achieves SOTA results on ImageNet-1k. 
- The proposed method is applicable to various SSL frameworks such as MAE, BEiT and BYOL.
- The paper is very well written and has extensive experiments and visualizations.
- The "Pilot Study" section gives very good narrative to the paper and provides intuitive motivations.

weaknesses:
The paper is very well-written and achieves SOTA results, I don't have weakness to add.

questions:
- For the Token encoding section of "4.1 Preliminary" section, consider adding more introduction/references to the concept introduced, for example "online encoder" and "target encoder.
- It will be easier for readers to follow if there is an additional figure about how to obtain the morphing matrix M.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732225411399

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the **Reviewers xEMo, gb6h, L29Y** for their thorough and constructive reviews with valuable and insightful advice. We appreciate the positive comments from all reviews: 

**Reviewer xEMo** - 1) our **state-of-the-art (SOTA) results** on ImageNet-1K, 2) **wide applicability** to masked image modeling (MIM) and self-supervised learning (SSL) methods, 3) **high clarity** of the manuscript, 4) **extensive experiments/visualizations**, and 5) the **insightful motivation** and **very good narrative to the paper provided in the pilot study**;

**Reviewer gb6h** - 1) **extensive discussions/demonstrations** of spatial inconsistency in our work that highlight our method’s significance, 2) **effective solution** addressing a novel problem, and 3) **broad applicability** to both MIM and SSL;

**Reviewer L29Y** - 1) **interesting insights** on MIM pre-training, 2) **strong results** achieved across all model scales including ViT-S/B/L, and 3) **wide applicability** of our method to MIM and SSL frameworks.

We have made every effort to address all the comments by carefully revising the paper. Throughout the revised paper, we highlighted the newly added or edited materials **in blue**. Please refer to the following summarization of the revised materials as follows:
* Complemented the explanation of the introduced concepts and revised some explanations with references in “4,1 Preliminary” (**Q1** by **Reviewer xEMo**)
* Added an illustrative figure of the morphing matrix M in Fig. 5 (**Q2** by **Reviewer xEMo**)
* Provided additional clarifications in Section 4.2 (**Q1** by **Reviewer gb6h**)
* Added an algorithm flowchart (in Section B of Appendix) to detail the flow of the Token Morphing Function (**Q1** by **Reviewer gb6h**, **Q2** by **Reviewer xEMo**)
* More refined writings in Section 4.1 and Fig. 4 (**Q2** by **Reviewer gb6h**)
* Added visualization of the spatial inconsistency of EVA-CLIP-g/14 in Fig. A in Appendix with a quantitative comparison in its caption (**Q2** by **Reviewer L29Y**)

Moreover, alongside our revisions, we tried our best to resolve the reviewer’s comments:
* Provided a detailed clarification of the “faster training” property (**Q3** by **Reviewer gb6h**)
* Provided an explanation for not using superpixel-based clustering (**Q4** by **Reviewer gb6h**)
* Clarified the reason for the lower performance without the dynamic mechanism in Table 7 (**Q5** by **Reviewer gb6h**)
* Showcased the effectiveness of DTM on hierarchical ViTs (**Q1** by **Reviewer L29Y**)
* Added new COCO detection results to show the effectiveness of DTM (**Q3** by **Reviewer L29Y**)

Detailed responses to other comments are left in individual comments. Due to the page limit, we send "Applicability on other targets" and "Applicability on SSL frameworks" to Section C in Appendix.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an algorithm for visual pre-training. It was based on masked image modeling, and the method was motivated by the so-called spatial inconsistency between image tokens. To alleviate it, the proposed method involves merging (morphing) visual tokens into larger units according to inter-token similarity, and the morphed tokens are used as basic units in the masked image modeling procedure.

Overall, the method is interesting and shows good practice. All three reviewers are positive on this submission, making it a case of clear acceptance. However, the AC still has a few concerns about the submission.

* First, the paper does not clearly connect the proposed method with spatial inconsistency; I was not convinced why grouping similar visual tokens can solve the inconsistency issue, although qualitative experiments (e.g. the example in Figure 1) seem to suggest that such trained models enjoy better spatial consistency.
* Upon this, I am a bit doubtful about why the method works -- does it work by alleviating the inconsistency issue, or by finding a more proper difficulty for the MIM task (note that grouping visual tokens can decrease the difficulty and make some "unpredictable" units predictable)? The paper lacks sufficient analysis and discussion on this point.
* The writing of this paper needs major improvement; the current form makes it difficult to understand many things. Adding some figures in the rebuttal makes things better, but the AC still had a difficult time reading the paper. Reviewers also pointed out this issue.
* In addition, I do not think the experiments are sufficient to fully validate the effectiveness -- there are only limited ImageNet tests with some transfer tests on ADE20K and FGVC sets, and the ablations are minimal. By the way, I agree with Reviewer **gb6h** that convergence (shown in Figure 6) cannot be solid evidence.

The AC discussed this case with the SAC in length. With three positive reviews (6/6/8), a final decision of acceptance was made, but the above concerns do exist. The AC strongly suggests the authors to take them into consideration and try to improve the final paper.