MORPHING TOKENS DRAW STRONG MASKED IMAGE MODELS

## Abstract

Abstract Masked image modeling (MIM) has emerged as a promising approach for training Vision Transformers (ViTs). The essence of MIM lies in the token-wise prediction of masked tokens, which aims to predict targets tokenized from images or generated by pre-trained models like vision-language models. While using tokenizers or pre-trained models are plausible MIM targets, they often offer spatially inconsistent targets even for neighboring tokens, complicating models to learn unified and discriminative representations. Our pilot study identifies spatial inconsistencies and suggests that resolving them can accelerate representation learning. Building upon this insight, we introduce a novel self-supervision signal called Dynamic Token Morphing (DTM), which dynamically aggregates contextually related tokens to yield contextualized targets, thereby mitigating spatial inconsistency. DTM is compatible with various SSL frameworks; we showcase improved MIM results by employing DTM, barely introducing extra training costs. Our method facilitates training by using consistent targets, resulting in 1) faster training and 2) reduced losses. Experiments on ImageNet-1K and ADE20K demonstrate the superiority of our method compared with state-of-the-art, complex MIM methods. Furthermore, the comparative evaluation of the iNaturalists and fine-grained visual classification datasets further validates the transferability of our method on various downstream tasks.
Code is available at https://github.com/naver-ai/dtm .

## 1 Introduction

Since the success of Vision Transformers (ViTs) (Dosovitskiy et al., ), numerous training strategies have been developed for ViTs, including self-supervised learning (SSL) methods (Chen et al., ; He et al., ; Grill et al., ; Caron et al., ).
Recent advances in masked image modeling (MIM) (Zhou et al., ; He et al., ; Peng et al., ; Baevski et al., ; Heo et al., ; Kim et al., ) have solidified its position as a primary SSL approach for ViT.
The crux of the MIM methods is leveraging token-wise optimization objectives by predicting masked tokens to match given targets. MIM methods have explored various approaches to assign effective target tokens, employing various pre-trained models including vision-language models (Bao et al., ; Peng et al., ), utilizing momentum encoders (Baevski et al., ; Zhou et al., ) or directly exploiting patchified images (He et al., ; Xie et al., ).

While tokenizers or pre-trained models have proven effective as MIM targets (Bao et al., ; Peng et al., ; Li et al., ; Wei et al., ), we argue they often generate spatially noisy token representations of class information (*e.g*., inconsistent class labels), which may incur negative impacts when utilized as pre-training targets; for example, a pre-trained vision-language model exhibits spatially inconsistent prediction results for tokens, as shown in Fig. 1. To further explore the behavior of token representations from pre-trained models, we analyze the effects of spatially inconsistent representations from the perspective of the models’ capability. Our pilot exploration shows reduced discriminability in zero-shot classification and continuous similarity metrics without token aggregation. Another pilot distillation experiment reveals that targets by context-unaware token aggregation disrupt pre-training.
As a result, spatially inconsistent targets challenge learning one-to-one token maps, leading to suboptimal representation learning.

Bearing this in mind, we introduce a novel token contextualization method called Dynamic Token Morphing (DTM), where token morphing links contextually similar tokens and aggregates them to produce coherent and comprehensive representations.
We conjecture that training can be accelerated through the guidance of composite representations of morphed tokens derived from the aggregation of contextually related tokens. Specifically, we encode the token-wise target representations and derive matching relations among tokens using DTM. The merging process is applied to
both online and target tokens
regarding the matching relation and aligns each morphed token with the corresponding morphed target token. The range of morphing can vary from a single token to all tokens, covering from token-wise to image-level representation learning. Among various options, we opt for bipartite matching for morphing, achieving both efficiency and efficacy.

Through extensive experiments, we verify our method’s general applicability and scalability. DTM could improve fine-tuning accuracies on ImageNet (Russakovsky et al., ) classification and ADE20K (Zhou et al., ) segmentation when equipped with various SSL frameworks for pre-training on ImageNet, demonstrating its versatility as a fundamental component.
Comparison with previous SSL methods verifies its superiority.
Our method showcases further transferability on the iNaturalist (Van Horn et al., ) and fine-grained visual classification datasets (Horn et al., ; Krizhevsky, ; Khosla et al., ).

Figure: (a) Input image
Refer to caption: x1.png

## 2 Related Work

Masked image modeling. Inspired by the promising performance of masked language modeling (MLM), BEiT (Bao et al., ) successfully extends MLM into the computer vision domain, using an external offline tokenizer from Dall-E (Ramesh et al., ). iBOT (Zhou et al., ) jointly trains the target encoder and the online tokenizer to remove the dependency on the external tokenizer.
Data2vec (Baevski et al., ) incorporates a momentum encoder to perform feature-level masked prediction tasks, leveraging representations from the multiple layers of neural networks.
MAE (He et al., ) and SimMIM (Xie et al., ) demonstrate efficient masked image modeling by directly reconstructing masked input pixels without any tokenizer.
On the other hand, several attempts have been made to exploit the pre-trained model as a tokenizer.
BEiT v2 (Peng et al., ) pre-trains a codebook for CLIP (Radford et al., ) to discretize a semantic space.
MVP (Wei et al., ) exploits a tokenizer pre-trained with multimodal data to enhance the semantics for MIM. A line of studies (Wei et al., ; Ren et al., ) using CLIP as a teacher to generate target representations have also been highlighted. Our method aims to utilize a teacher model more effectively, including CLIP, rather than just using it as a raw pre-trained model.

Token aggregation methods.
Token aggregation can conceptually be categorized as a token clustering method and usually aims for efficiency. Hard clustering methods like K-Means (Lloyd, ), K-Medoids (kme, ), and Density-Peak Clustering with K-Nearest Neighbors (DPC-KNN) (Jiang et al., ) enforce each data to belong to a single cluster exclusively.
Bipartite matching (Karp et al., ) also aggregates data in a hard clustering manner, which optimizes pairs elements from two disjoint sets given objective function.
Meanwhile, soft clustering is defined to let data belong to multiple clusters.
LIT (Pan et al., ) employs deformable token merging layers to aggregate tokens between stages. Furthermore, some token pruning methods (Rao et al., ; Xu et al., ; Tang et al., ; Liang et al., ) can be categorized into token aggregation methods; however, they intensely focused on compressing tokens to aim for a cost-efficient vision transformer.
While the above token aggregation methods have mainly been employed to boost efficiency, our approach diverges significantly. We take the concept of token aggregation to address spatially noisy target tokens in token-level supervision, thereby enhancing the efficacy of MIM in terms of precision.

## 3 Pilot Study

In this section, we study the behavior of token representations generated by pre-trained models and their negative impacts on pre-training. Our study can be summarized as follows:

- •
Revealing spatial inconsistency and its impact (§3.1).
We begin by visualizing token representations from a pre-trained vision-language model to define spatial inconsistency: as shown in Fig. 1, the inconsistent class predictions across different patches suggest spatially noisy token representations.
We further investigate impact of spatial inconsistency, as relying on the noisy token representations may underutilize the pre-trained model’s capability.
The quantitative analyses in Table 1 show that enhancing spatial coherence in the token representations results in improved performance, which highlights the importance of addressing spatial inconsistency.
- •
Spatial inconsistency in representation learning (§3.2). Our study shifts to a practical focus on pre-training for representation learning under supervision (*i.e*., target). We first reveal that the lack of spatial coherence in token representations significantly hinders learning and using such noisy representations as targets can weaken the supervision model. We then assess token aggregation methods as the supervision to confirm the noticeable efficacy of a context-based method for handling spatial inconsistency.

**Table 1: Addressing spatial inconsistency boosts accuracy. We compare the ImageNet accuracy of zero-shot/linear probing via average pooled token-wise logit. Post-hoc morphed patch representations enhance accuracies, indicating that addressing spatial inconsistency improves precision. Linear probing is trained for 25 epochs, and we use the fixed half of the whole tokens for aggregation.**
| Token<br>Aggregation | Zero-shot image<br>classification ($\%$) | Linear probing ($\%$) | Averaged patch-wise cosine<br>similarity with [CLS] |
| --- | --- | --- | --- |
|  | 26.5 | 73.2 | 0.53 |
| ✓ | 30.8 (+4.3) | 77.6 (+3.2) | 0.56 (+0.3) |

### 3.1 Spatial Inconsistency

Given the input image in Fig. 1(a), we visualize the token-wise zero-shot classification results without and with token aggregation in Fig. 1(b) and 1(c), respectively.
Correct and incorrect tokens are marked in green and red, respectively, with a gradient to darker shades of red, indicating a more significant deviation from the true class.
Despite the proximity and contextual similarity among tokens, wrong tokens in the green box (bottom right) in Fig. 1(b) exhibit spatially inconsistent prediction results (French Bulldog, Gorilla, Bull Mastiff, Hotdog, and Newfoundland Dog) while tokens in Fig. 1(c) show correct or relatively consistent predictions (French Bulldog).
Moreover, token aggregation for predictions yields improved accuracy, with 113 correct tokens out of 196 and a gap of 31 tokens compared to the counterpart.
This highlights the spatial inconsistency among tokens from a pre-trained model, which could potentially disrupt representation learning when used as supervision.

Impact of spatial inconsistency.
To quantitatively assess the impact of spatial inconsistency at a dataset scale, we compute the token-wise ensemble prediction using a global pool, with and without token aggregations. Fundamentally, we predict class scores for each token and average these scores across all tokens within the given image. When predicting with token aggregation, we group semantically relevant tokens and average their representations group-wise prior to token-wise ensemble prediction. We employ CLIP-B/16 (Radford et al., ) for our study, so we aggregate 98 tokens for token aggregation, which occupy half of the entire tokens.

Table 1 below reports the results of zero-shot image classification, linear probing, and averaged cosine similarity with the [CLS] token on ImageNet-1K. Note that we do not perform any extra training here. Our results show that using aggregated tokens consistently exceeds the performances of those without token aggregation across all metrics. This suggests that aggregating contextually related tokens can address inconsistencies and lead to performance gains.

Note that the final metric we use is a continuous metric that computes patch-wise similarity. Specifically, it measures the cosine similarity between the [CLS] token and each patch, which is also a patch-wise metric but continuous(^1^11 We believe a continuous metric to assess spatial inconsistency could track network responses continuously compared with the discrete classification metrics.).
As observed in Table 1, the averaged similarities are computed to 0.56 vs. 0.53 for each case. This new metric, directly linked to the [CLS] token, suggests a more direct relationship between improved accuracy and reduced spatial inconsistency. The trend aligns with the other discrete metric, depicting patch-wise classification of 30.8$\%$ and 26.5$\%$ with and without token aggregation, respectively.
The quantitative assessments, using both continuous and discontinuous metrics, demonstrate that token representations exhibit spatial inconsistency. Taking this further, we believe more effectively addressing inconsistency would yield a greater impact.

### 3.2 Spatial Inconsistency in Representation Learning

Figure: Figure 2: Representation learning with various supervisions. We illustrate our study’s base representation learning framework along with different self-supervision signal function $f$. We evaluate four variants of the self-distillation target: (1) token-wise supervision (baseline), (2) downsampled supervision (3) supervision after bipartite matching layer-wise, (4) superpixel supervision, and (5) supervision by token morphing.
Refer to caption: x4.png

**Table 2: Self-distillation signals should be aware of context. We evaluate the ImageNet-1K linear probing accuracy of self-distillation supervisions (Fig. 2). The baseline demonstrates efficacy as token-wise supervision directly benefits the tokens; only more refined signals could surpass it (lower rows). Upper rows showcase simpler targets, utilizing fundamental methods that only achieve efficiency without accuracy gains.**
| Distillation signals | Linear prob ($\%$) | Speed↓ (ms/img) |
| --- | --- | --- |
| (1) Token-wise (baseline) | 70.9 | 0.629 |
| (2) Downsampling | 68.8 (-2.1) | 0.647 |
| (3) Layer-wise Bipartite Matching (Bolya et al., ) | 70.2 (-0.7) | 0.633 |
| (4) Superpixel clustering (Achanta et al., ) | 72.6 (+1.7) | 0.740 |
| (5) Token morphing via Bipartite Matching | 72.2 (+1.3) | 0.653 |

We now study token aggregations for handling spatial inconsistency in representation learning.
This analysis aims to highlight the spatial inconsistency in supervision signals (*i.e*., self-distillation or pre-training targets) that potentially diminish the quality of representations.
Our comparison ranges distillations with the signals of 1) token-wise, 2) downsampling, 3) layer-wise bipartite matching during encoding, 4) superpixel, and 5) token morphing via bipartite matching. Note that token morphing will be introduced in §4.1, but we show its effectiveness in advance.
We pre-train ViT-B/16 for 50 epochs and perform linear probing for 50 epochs on ImageNet-1K for each configuration.

Table 2 exhibits distillation with context-aware token aggregation methods such as superpixel clustering and bipartite matching-based token morphing
outperforms that of the token-wise distillation approach.
We first observe token representations indeed exhibit spatial inconsistency,
as we claimed. Furthermore, token aggregation methods for contextually related tokens could manage this inconsistency.
Since the downsampling approach aggregates tokens without context awareness, and the layer-wise bipartite matching approach risks harming intermediate representations during encoding, both approaches exhibit diminished returns.
The findings highlight the significance of employing context-aware token aggregations that preserve token count while addressing spatial inconsistency.

Despite superpixel clustering’s promising results, our pilot study finds that our method learning with 1) superpixel clustering and 2) context-aware bipartite matching yields accuracies of 87.1% and 87.9%, respectively, over 100 epochs trained on ImageNet-100. Therefore, we do not employ a superpixel-based clustering for our method, which is inefficient as well.

## 4 Method

We observed that 1) supervision signal by a pre-trained model often produces noisy and spatially inconsistent token-level targets for learning, closely linked with performance degradation;
2) naive token aggregation methods could partially handle spatial inconsistency but are insufficient as a supervisory signal;
3) a well-designed method is favorable for considering context and reducing noise more effectively. Motivated by the observations, we introduce an advanced token aggregation method called Dynamic Token Morphing (DTM) for masked image modeling and self-supervised learning. DTM contextually aggregates tokens to derive random numbers of morphed tokens aiming to encapsulate diversified semantic information. The core idea of DTM is illustrated in Fig. 3, where the DTM module is straightforwardly added to the baseline (*e.g*., MIM). DTM encourages conventional token-level MIMs by aligning morphed tokens from online and target encoders by reducing spatial inconsistency concerning context.

Figure: Figure 3: Token Morphing offers diverse and highly contextualized signals. Dynamic Token Morphing (DTM) aligns the representations of tokens based on a dynamic aggregation of contextually related ones. Note that DTM processes morphed tokens to create more diverse and generalized targets. Blue and Green tokens denote the representations of the image patches processed by online and target models, respectively. Gray tokens represent masked tokens.
Refer to caption: x5.png

### 4.1 Preliminary

Token encoding.
Given an image $x$, we patchify the image into N patches $\{x_{i}\}_{i=1}^{N}$.
We select positions $\mathcal{M}\subset\{1,2,...,N\}$ of masked patches in a block-wise manner with a masking ratio $r\in(0,1)$ so that $|\mathcal{M}|=\lfloor rN\rfloor$.
We mask the image patches by replacing the image patches of the position in $\mathcal{M}$ to a learnable mask token $e_{[mask]}$.
To this end, the input of the encoders becomes $\{x_{i}^{\mathcal{M}}\}_{i=1}^{N}$, where $x_{i}^{\mathcal{M}}=e_{[mask]}$ for $i\in\mathcal{M}$ and $x_{i}^{\mathcal{M}}=x_{i}$ for $i\notin\mathcal{M}$.
The masked patches $\{x_{i}^{\mathcal{M}}\}_{i=1}^{N}$ are concatenated with a learnable CLS token $e_{[CLS]}$ and fed into the online encoder $f_{\theta}$ with a subsequent linear head $h_{\theta}$ and the target encoder $f_{\xi}$, and become encoded online tokens $\{\textbf{u}_{i}\}_{i=1}^{N}$ and $\{\textbf{v}_{i}\}_{i=1}^{N}$ where $\textbf{u}_{i}=h_{\theta}(f_{\theta}(x_{i}^{\mathcal{M}}))$ and $\textbf{v}_{i}=f_{\xi}(x_{i}^{\mathcal{M}})$, respectively.

Token morphing. Differing from token aggregation methods, which prioritize efficiency via token reduction, token morphing specifically targets the spatial inconsistency in token representations by taking context into account. It connects contextually relevant tokens to smooth them without reducing the number of tokens (*i.e*., maintaining the token count).
We define the process with the token morphing function $\phi_{R}(\cdot)$ based on the morphing schedule $R$, a sequence of token numbers to morph.
Here, $\phi_{R}$ is a general notation for a function
$\phi_{R}:\mathbb{R}^{N\times d}\rightarrow\{0,1\}^{\bar{n}\times N}$
that calculates similarity using a matching algorithm (*e.g*., bipartite matching or K-Means clustering) and returns a token morphing matrix $M=[M_{ij}]\in\{0,1\}^{\bar{n}\times N}$, where $\bar{n}$ means the number of token groups after morphing and $d$ denotes the feature dimension.
Eventually, we can morph tokens following Eq. 3.

Figure: Figure 4: Overview of Masked Image Modeling via Dynamic Token Morphing (DTM). For a token morphing schedule of DTM, we aggregate the dynamic range of tokens using morphing matrix $M$ derived from target tokens $\{\textbf{v}_{i}\}_{i=1}^{N}$. Specifically, we randomly sample a number of remaining tokens $\bar{n}$ and an iteration number $k$ to dynamically schedule token morphing (i.e., $\{r_{p}\}_{p=1}^{k}$), forming $\bar{n}$ morphed tokens $\{\hat{\textbf{u}}_{i}\}_{i=1}^{\bar{n}}$ and $\{\hat{\textbf{v}}_{i}\}_{i=1}^{\bar{n}}$. Then, we align representations of the corresponding online and target morphed tokens.
Refer to caption: x6.png

### 4.2 Overall Dynamic Token Morphing Pipeline

We present a more advanced token morphing method, Dynamic Token Morphing (DTM), here.
DTM’s dynamic nature stems from its simultaneous exploration of multiple and diverse token morphing cases.
This design is based on the insight that morphing numerous tokens enhances the denoising effect while morphing fewer tokens retains detailed token representations. Furthermore, to achieve diversified morphed tokens, we divide the morphing process into multiple processes to create multiple morphed tokens. DTM encompasses three key components: 1) dynamic scheduler for token counts, 2) token morphing via scheduler, and 3) aligning morphed tokens.
The overall framework of DTM is described in Fig. 4.

Dynamic scheduler.
DTM generates multiple morphed tokens to ensure diversity and an extensive range of token variations. To this end, we first sample the final number of morphed tokens $\bar{n}\sim\mathcal{U}(\bar{N},N)$ to remain after token morphing and the iteration number $k\sim\mathcal{U}(1,K)$ from uniform distributions, where $\bar{N}$ represents the minimum number of morphed tokens and $K$ denotes the maximum number of iteration for sampling.
Then, we define a token count scheduler $R=\{r_{p}\}_{p=1}^{k}$, a sequence of token numbers $r_{p}$ that determines the number of tokens to morph for each iteration dynamically.
Rather than sampling a sequence of random numbers $r_{p}$ that satisfies $\sum_{p=1}^{k}r_{p}=N-\bar{n}$, we simply divide $N-\bar{n}$ by $k$ for constant counts:

$$ $\displaystyle r_{p}=\begin{cases}\big{\lfloor}(N-\bar{n})/k\big{\rfloor},& \text{if }p<k\\ N-\bar{n}-(k-1)\big{\lfloor}(N-\bar{n})/k\big{\rfloor},&\text{if}p=k.\end{cases}$ (1) $$

Token morphing via dynamic scheduler.
The token morphing function $\phi_{R}$ is dynamically changed based on randomly sampled $\bar{n}$ and $k$.
While our DTM is universal to any token matching methods, we employ bipartite matching (Karp et al., ) to search for contextually similar tokens efficiently.
Specifically, we split tokens into two groups, with each token in the first group matched to its closest cosine similarity counterpart in the second group.
Eventually, we obtain the token morphing matrix $M$, a contextual relation among tokens, from the target token representations $\{\textbf{v}_{i}\}^{N}_{i=1}$ as follows:

$$ $\displaystyle M=\phi_{R}\big{(}\{\textbf{v}_{i}\}^{N}_{i=1}\big{)},$ (2) $$

where $M_{ij}=1$ indicates that the $j^{\text{th}}$ token $\textbf{v}_{j}$ will be aggregated to the $i^{\text{th}}$ token $\textbf{v}_{i}$.
In addition, we let every token be assigned to a specific cluster, even in cases where it forms a single token cluster, and each token should retain its exclusive association with a single cluster (*i.e*., $\sum_{i}\sum_{j}M_{ij}=N$ and $\sum_{i}M_{ij}=1$).
For the repetition $p{=}1,\ldots,k$,
$r_{p}$ token pairs are successively matched.
Based on the token morphing matrix $M$ obtained by $\phi_{R}$, the representations for online and target morphed tokens are derived by:

$$ $\displaystyle\hat{\textbf{u}}_{i}=\sum_{j}M_{ij}\textbf{u}_{j},\quad\hat{ \textbf{v}}_{i}=\sum_{j}M_{ij}\textbf{v}_{j},$ (3) $$

where $\{\hat{\textbf{u}}_{i}\}_{i=1}^{\bar{n}}$ and $\{\hat{\textbf{v}}_{i}\}_{i=1}^{\bar{n}}$ denote for the representations of online and target morphed tokens, respectively.
Note that the morphed tokens are representative tokens for each token group, with their representations being smoothed specific to their respective groups.

Aligning morphed tokens.
We formulate the objective function by accumulating DTM losses, which aligns the representations of the corresponding online and target morphed tokens derived by DTM.
The DTM loss with sampled $\bar{N}$ and $k$ is formulated as follows:

$$ $\displaystyle\mathcal{L}_{\text{DTM}}(\bar{n},k)=\sum_{i=1}^{\bar{n}}w_{i}d( \hat{\textbf{u}}_{i},\hat{\textbf{v}}_{i}),$ (4) $$

where $d(\cdot)$ is a distance function and
$w_{i}=\sum_{j}M_{ij}$ is a number of tokens aggregated for the $i^{\text{th}}$ online and target morphed tokens.
Here, we utilize $w_{i}$ to consider all tokens aggregated for the morphed tokens.
The DTM loss can be extended to token-wise or image-level losses when $\bar{n}=N$ or $\bar{n}=1$, respectively. We adopt Cosine distance for the distance function in eq. 4 for all the DTM losses.
To further enhance the dynamic nature of our method, we apply multiple DTM losses, each derived from its corresponding morphing case.
The final objective function is the summation of all DTM losses, which is defined as:

$$ $\min_{\theta}\ \sum_{l=1}^{L}\mathcal{L}_{\text{DTM}}(\bar{n}_{l},k_{l})\quad \text{s.t.}\ \bar{n}_{l}\sim\mathcal{U}(\bar{N}_{l},N)\ \text{and}\ k_{l}\sim \mathcal{U}(1,K_{l}),$ (5) $$

where $L$ denotes the total number of simultaneously employed DTM losses.

**Table 3: ImageNet-1K performance comparisons. All models were pre-trained/fine-tuned on ImageNet-1K. We evaluate the improvements in fine-tuning accuracies of competing methods using different supervisions and ours for ViT-{S/16, B/16, L/16} with a resolution of $224\times 224$. ADE20K semantic segmentation results (Seg) using ViT-B/16 are compared as well. All our models are pre-trained for 300 epochs.**
| Method |  | Pre-training epochs | Supervision | ViT-S | ViT-B | ViT-L | Seg |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ViT-S | ViT-B | ViT-L |  |  |  |  |  |  |  |
| Supervised models |  |  |  |  |  |  |  |  |  |
| DeiT (Touvron et al., ) | ICML 2021 | - | - | - | Label | - | 81.8 | - | - |
| DeiT-III (Touvron et al., ) | ECCV 2022 | - | - | - | Label | - | 83.8 | 84.2 | 49.3 |
| Cosub (Touvron et al., ) | CVPR 2023 | - | - | - | Label | 81.5 | 84.2 | 85.3 | 49.3 |
| MaskSub (Heo et al., ) | arXiv 2023 | - | - | - | Label | 81.7 | 84.2 | 85.3 | 50.2 |
| Self-supervised models |  |  |  |  |  |  |  |  |  |
| MoCo v3 (Chen et al., ) | ICCV 2021 | 300 | 300 | 300 | Pixel | 81.7 | 83.2 | 84.1 | 47.3 |
| BEiT (Bao et al., ) | ICLR 2021 | 300 | 800 | 800 | DALL-E | 81.7 | 83.2 | 85.2 | 47.1 |
| DINO (Caron et al., ) | ICCV 2021 | 3200 | 1600 | - | Feature | 82.0 | 83.6 | - | 46.8 |
| SplistMask (El-Nouby et al., ) | arXiv 2021 | 300 | 300 | - | Pixel+Feat | 81.5 | 83.6 | - | 46.8 |
| iBOT (Zhou et al., ) | ICLR 2022 | 3200 | 1600 | 1000 | Feature | 82.0 | 84.0 | 84.8 | 50.0 |
| MAE (He et al., ) | CVPR 2022 | - | 1600 | 1600 | Pixel | - | 83.7 | 85.6 | 48.1 |
| SimMIM (Xie et al., ) | CVPR 2022 | - | 800 | - | Pixel | - | 83.8 | - | - |
| MaskFeat (Wei et al., ) | CVPR 2022 | - | 1600 | 1600 | Feature | - | 84.0 | 85.7 | - |
| ExtreMa (Wu et al., ) | arXiv 2022 | - | 300 | - | Feature | 81.8 | 83.7 | - | 47.9 |
| CAE (Chen et al., ) | arXiv 2022 | 300 | 1600 | 1600 | Pixel+Feat | 82.0 | 83.9 | 86.3 | 50.2 |
| CMAE (Huang et al., ) | arXiv 2022 | - | 1600 | 1600 | Pixel+Feat | - | 84.4 | - | 50.1 |
| FD-CLIP (Wei et al., ) | arXiv 2022 | - | 300 | - | CLIP B/16 | - | 84.9 | - | 52.8 |
| BEiT v2 (Peng et al., ) | arXiv 2022 | - | 300 | 300 | CLIP B/16 | - | 85.0 | 86.6 | 52.7 |
| data2vec (Baevski et al., ) | ICML 2022 | - | 800 | 1600 | Feature | - | 84.2 | 86.6 | - |
| mc-BEiT (Li et al., ) | ECCV 2022 | - | 800 | 800 | VQGAN | - | 84.1 | 85.6 | 47.0 |
| MVP (Wei et al., ) | ECCV 2022 | - | 300 | 300 | CLIP B/16 | - | 84.4 | 86.3 | 52.4 |
| SdAE (Chen et al., ) | ECCV 2022 | - | 300 | - | Pixel | - | 84.1 | - | 48.6 |
| MSN (Assran et al., ) | ECCV 2022 | - | 600 | - | Feature | - | 83.4 | - | - |
| BootMAE (Dong et al., ) | ECCV 2022 | - | 800 | 800 | Pixel+Feat | - | 84.2 | 85.9 | 49.1 |
| CAN (Mishra et al., ) | arXiv 2022 | - | 1600 | 800 | Pixel | - | 83.6 | 84.7 | - |
| SemMAE (Li et al., ) | NIPS 2022 | - | 800 | - | Pixel | - | 83.3 | - | 46.3 |
| ConMIM (Yi et al., ) | ICLR 2023 | 300 | 800 | 1600 | Dictionary | 82.0 | 83.7 | 85.5 | 46.0 |
| RC-MAE (Yi et al., ) | ICLR 2023 | 1600 | 1600 | 1600 | Pixel | 82.0 | 83.6 | 86.1 | - |
| MixedAE (Chen et al., ) | CVPR 2023 | - | 1600 | - | Pixel | - | 83.9 | - | 49.8 |
| SIM (Tao et al., ) | CVPR 2023 | - | 1600 | - | Feature | - | 83.8 | - | - |
| HPM (Wang et al., ) | CVPR 2023 | - | 800 | 800 | Pixel | - | 84.2 | 85.8 | 48.5 |
| MIRL (Huang et al., ) | NeurIPS 2023 | - | 300 | 300 | Pixel | - | 84.1 | 85.4 |  |
| DeepMIM (Ren et al., ) | arXiv 2023 | - | 300 | - | CLIP B/16 | - | 84.8 | - | - |
| dBOT (Liu et al., ) | ICLR 2024 | - | 1600 | 1600 | Feature | - | 84.5 | 86.6 | 49.5 |
| LUT (Kim et al., ) | ECCV 2024 | 400 | 1600 | 1600 | Pixel | 82.0 | 84.2 | 86.0 | 49.5 |
| CrossMAE (Fu et al., ) | arXiv 2024 | 800 | 800 | 800 | Pixel | 79.3 | 83.7 | 85.4 | - |
| DTM (ours) | - | 300 | 300 | 300 | CLIP B/16 | 83.2 | 85.4 | 86.7 | 53.1 |

## 5 Experiment

In this section, we compare the ImageNet-1K classification and ADE20K segmentation performances. We explore the strengths of DTM including applicability on SSL frameworks and other pre-trained target models (e.g., SLIP),
and training efficiency. We analyze the importance of dynamics and faster convergence of DTM. In appendix, we provide additional visualized examples of spatial inconsistencies. We also investigate the transferability of DTM and the applicability of superpixel algorithms. Then, we conduct ablation studies on various configurations of DTM.
Implementation details including pre-training and fine-tuning details are listed in Section E of the Appendix.

**Table 4: Applicability of DTM on SLIP. We apply DTM to SLIP (Mu et al., ), a more improved language-image pre-trained model to demonstrate our method’s applicability beyond CLIP. We employ ViT-{S/16, B/16, L/16} with a resolution of $224\times 224$. All models are pre-trained for 300 epochs on ImageNet-1K.**
| Target models | Method | ViT-S | ViT-B | ViT-L |
| --- | --- | --- | --- | --- |
| SLIP | Baseline | 81.8 | 84.0 | 85.7 |
| DTM | 82.1 (+0.3) | 84.5 (+0.5) | 86.1 (+0.4) |  |

**Table 5: Applicability of our DTM on various SSL frameworks. We train DTM with other SSL methods to show its broader applicability. We adopt MAE (He et al., ) with the CLIP teacher (Radford et al., ), BEiT v2 (Peng et al., ), and BYOL (Grill et al., ) to cover general SSL frameworks. We employ the ViT-B/16 architecture with a resolution of $224\times 224$. All the models are pre-trained for 100 epochs.**
| Framework | Method | FT ($\%$) |
| --- | --- | --- |
|  | MAE + CLIP | 82.6 |
| Feature MIM | MAE + CLIP + DTM | 83.2 (+0.6) |
| (CLIP teacher) | BEiT v2 | 84.2 |
|  | BEiT v2 + DTM | 84.4 (+0.2) |
| Image-level SSL | BYOL | 81.7 |
| BYOL + DTM | 82.1 (+0.4) |  |

**Table 6: Efficiency of context-aware token aggregations. We report fine-tuning accuracies and throughputs for each configuration, which are pre-trained with ViT-B/16. We compare DTM with Bipartite matching, DTM with K-Means clustering, and layer-wise K-Means clustering. For the layer-wise K-Means clustering, we use constant numbers of clusters and iterations. DTMs both outperform the baseline with large margins. Moreover, DTM with K-means clustering surpasses layer-wise K-means clustering, demonstrating the superiority of DTM.**
| Case | Throughput (image/s) | FT ($\%$) |
| --- | --- | --- |
| Baseline | 1458 | 84.3 |
| Layer-wise K-Means clustering | 1265 | 85.1 |
| DTM (K-Means clustering) | 489 | 85.4 |
| DTM (Bipartite matching) | 1446 | 85.4 |

### 5.1 ImageNet-1K Classification

We compare the fine-tuning accuracy of our method with previous state-of-the-art self-supervised methods on ImageNet-1K (Russakovsky et al., ) datasets.
The comparisons include supervised learning and SSL methods with various supervision signals.
When the target model is CLIP, we only compare models pre-trained with CLIP B/16 for 300 epochs for a fair comparison.
Table 3 reports the fine-tuning accuracies of ViT-S/B/L backbones.
Our baseline simply employ negative cosine loss with a vanilla CLIP model as the target model.
We observe that our MIM pre-trained by DTM achieves 83.2%, 85.4%, and 86.7% top-1 accuracies with ViT-S/16, ViT-B/16, and ViT-L/16, respectively, which outperforms state-of-the-art performances across the scales.
Specifically, our method surpasses MVP (Wei et al., ), DeepMIM (Ren et al., ), and BEiT v2 (Peng et al., ) by 1.0$\%$p, 0.6$\%$p, and 0.4$\%$p on ViT-B/16, respectively.
Moreover, our method outperforms other methods that leverage diverse supervision, demonstrating our method’s superiority among self-supervised learning methods.

Additionally, we extend DTM’s pre-training to 800 epochs, which improves fine-tuning accuracy of 85.5% on IN-1K.
This result highlights 1) DTM merits longer pre-trainings and 2) DTM trained for 800 epochs surpasses others trained for 1600 epochs such as BeiT v2 (Peng et al., ). We believe even longer pre-training of DTM would lead to further improvements.

### 5.2 Effectiveness of Our Method

On other targets. To confirm our method’s applicability, we apply DTM with the SLIP (Mu et al., ) to verify its effectiveness across different target models.
We compare fine-tuning accuracies of the baseline and our method pre-trained by target patches from SLIP.
Specifically, we generate target tokens using SLIP along with its projector.
We pre-train ViT-B/16 with a resolution of $224\times 224$ for 300 epochs and fine-tune for 100 epochs on ImageNet-1K (Russakovsky et al., ).
As shown in Table 4, our DTM successfully improves the fine-tuning accuracy of the baselines pre-trained with SLIP by 0.5$\%$p, which reveals its general applicability beyond CLIP (Radford et al., ).

Applicability on SSL frameworks.
We apply DTM on BEiT v2, MAE with the CLIP teacher (Radford et al., ), and BYOL (Grill et al., ) to verify its applicability to various SSL frameworks.
As shown in Table 5, our DTM successfully improves the fine-tuning accuracy of MAE + CLIP, BEiT v2, and BYOL by 0.5$\%$p, 0.2$\%$p, and 0.4$\%$p, which reveals its general applicability to various frameworks beyond our baseline.

Efficiency.
Table 6 shows that bipartite matching is efficient and effective, significantly boosting the accuracy (+1.1%p) with only a 1% speed loss. While K-Means (Lloyd, ) also exhibits considerable improvements, it significantly degrades the training speed. Layer-wise K-Means shows accelerate the K-Means method by aggregating tokens within layers at the cost of degraded representations, leading to lower accuracy.

Transferability.
We verify the improved transferability of our pre-trained model.
We compare fine-tuning accuracies of the baseline and our proposed model on iNaturalist datasets (Van Horn et al., ), which are highly imbalanced with different numbers of images per class, and Fine-Grained Visual Classification (FGVC) datasets.
Table A and Table B in the Appendix show our DTM loss significantly improves the baseline with large margins, which reveals enhanced transferability.

### 5.3 ADE20K Semantic Segmentation

We further evaluate semantic segmentation performance on ADE20K (Zhou et al., ) to verify the transferability of our pre-trained model. We follow the training and evaluation protocol (He et al., ); the models are fine-tuned for 160K iterations using UperNet (Xiao et al., ) with a batch size of 16 and a resolution of 512$\times$512. We initialize UperNet with our pre-trained ViT-B/16. Detailed hyperparameters for semantic segmentation fine-tuning can be found in Appendix.
The first right column in Table 3 shows the mIoU performance comparison.
Our method also outperforms the previous state-of-the-art results with a margin of 0.3$\%$p, validating its superiority over other SSL methods.
This result signifies that our method effectively enhances discriminability for dense prediction tasks.

### 5.4 Analysis

Ablation study on the importance of dynamics.
Table 7 presents an ablation study on the effectiveness of the dynamic mechanism on DTM.
We employ ViT-B/16 (Dosovitskiy et al., ) with a resolution of $224\times 224$.
The models are pre-trained for 300 epochs and fine-tuned for 100 epochs on ImageNet-1K (Russakovsky et al., ).
For token morphing without the dynamic mechanism, half of the total 196 tokens are aggregated for each image.
As shown in Table 7, token morphing with dynamic mechanism significantly improves the baseline while performance degrades without the dynamic mechanism, demonstrating the importance of dynamic mechanism in DTM.

Faster convergence of DTM.
We analyze the behaviors of models pre-trained by DTM and token-wise objectives in Fig. 5.
All the approaches employ CLIP representations for the target.
As shown in Fig. 5(a), the fine-tuning accuracies of DTM surpass the accuracy of the token-wise supervision approaches.
Fig. 5(b) shows that the model pre-trained by DTM exhibits a lower fine-tuning loss than the baseline model, verifying better convergence.
Finally, Fig. 5(c) verifies the consistent effect of DTM across various pre-training epochs.

Figure: (a) Fine-tuning accuracy
Refer to caption: extracted/5917117/materials/analysis/comp_acc_v2.png

**Table 7: Ablation study on dynamic mechanism. We investigate the efficacy of the dynamic mechanism in DTM, revealing its significant contribution to our method.**
| Method | $\mathcal{L}_{\text{Token_Morphing}}$ | Dynamic | FT ($\%$) |
| --- | --- | --- | --- |
| Baseline | - | - | 84.3 |
| DTM | ✓ | - | 84.0 (-0.3) |
| DTM | ✓ | ✓ | 85.4 (+1.1) |

## 6 Conclusion

We have introduced a novel masked image modeling method based on token morphing to address the negative impacts of spatially inconsistent target representations during pre-training.
We have first analyzed the existence and impacts of spatial inconsistency in target representations.
Specifically, we qualitatively observed spatial inconsistency among tokens from pre-trained models despite proximity and contextual similarity. We then investigated the impacts on downstream tasks, including zero-shot/linear classification. We have further validated that context-aware token aggregation methods enhance the pre-training capability of the target tokens, while arbitrary aggregation, like downsampling, tends to disrupt them.
Based on the observations, we have proposed Dynamic Token Morphing (DTM), which dynamically aggregates contextually associated tokens with a random number of matching tokens.
Subsequently, our method aligns the representations of morphed tokens with the representations of their corresponding morphed targets. Our extensive experiments verified its general applicability, scalability, and superiority. We further validated its transferability, and subsequent analyses supported our method. Our study showcased that a token aggregation method could boost representation learning performance, with the potential for broader future applications and research directions.

Limitation.
Despite the potential of our method, we have verified its applicability only up to ViT-L/16. Resource limitations prevented us from performing more experiments on larger-scale models such as ViT-G.

## Appendix A More Examples on Spatial Inconsistency

We further investigate the spatial inconsistency of patch representations generated by a pre-trained model across various samples.
We visualize the token-wise zero-shot classification results without
and with token aggregation using CLIP (Radford et al., ).
Specifically, correct and incorrect tokens are marked
in green and red, respectively, with a gradient to darker shades of red, indicating a more significant
deviation from the true class. .
As shown in Fig. A, patch representations without token aggregation reveal spatially inconsistent patch-wise predictions compared to the prediction results with token aggregation, which reveals the spatially inconsistent representations among patches.
In addition, predictions with token aggregation exhibit significantly more correctly predicted patches than predictions without token aggregation.

## Appendix B Investigation on Transfer learning

iNaturalist datasets.
We verify the improved transferability of our pre-trained model.
We compare fine-tuning accuracies of the baseline and our proposed model on iNaturalist 2018, iNaturalist 2019, and mini iNaturalist 2021 (Van Horn et al., ), which are highly imbalanced with different numbers of images per class.
All the models are ViT-B/16 with a resolution of 224$\times$224.
Following the protocol (Kornblith et al., ), we perform grid searches on learning rates and weight decay and report the maximum accuracy and the mean and standard deviation of the accuracies.
Table A shows our DTM loss significantly improves the baseline with large margins, which reveals enhanced transferability.

Fine-Grained Visual Classification (FGVC) datasets.
We further validate the transferability of our method.
Following the evaluation protocol as above, we conduct comparisons on FGVC datasets.
Specifically, we evaluate fine-tuning accuracies on Birds, CUB-200, CIFAR-10, CIFAR-100, and Dogs through grid searches with different learning rates and weight decays.
As shown in Table B, our method outperforms the baseline overall across the datasets, which shows superior transferability and tuning robustness.

## Appendix C Compatibility with Superpixel Algorithms

We perform additional experiments to use the concept of superpixels (Achanta et al., ; Chang et al., ) in ours, both directly to tokens (layer-wise superpixel) and can be combined with DTM. The layer-wise superpixel method uses constant numbers of superpixels and iterations.
Table C shows superpixel-based methods enjoy notable gains but do not exceed the bipartite matching one.
While the layer-wise superpixel method also utilizes superpixels, the layer-wise token aggregation across the encoder layers bipartite matching approach risks harming intermediate representations during encoding

### C.1 Ablation Study on the Number of Schedules

Our method can further enhance the diversity of target morphed tokens by employing multiple schedules within a single iteration.
Thus, we study the effects of learning diverse morphed tokens derived from multiple morphing schedules simultaneously.
We compare the fine-tuning accuracy of models pre-trained by our DTM, varying the number of schedules.
We pre-train ViT-B/16 with a resolution of $224\times 224$ for 100 epochs and fine-tune for 100 epochs on ImageNet-1K (Russakovsky et al., ) and ImageNet-100 (Russakovsky et al., ).
Table D reveals that exploring diverse target morphed tokens improves the representation capability of pre-trained models, leading to increased fine-tuning accuracies of at least 0.1$\%$p and 0.2$\%$p on ImageNet-1K and ImageNet-100 compared to the model pre-trained by the single DTM approach, respectively.
However, the goal of experiencing diverse morphed tokens at once appears to be attained with double scheduling, resulting in no additional gains in performance through further exploration.
Given that utilizing two schedules yields the best and is most efficient among all other multiple scheduling options, we adopt the double morphing scheduling approach.

**Table A: Transfer learning results on iNaturalists. We further present the end-to-end fine-tuning accuracies on the iNaturalist 2018, iNaturalist 2019, and mini iNaturalist 2021 datasets (Van Horn et al., ). We report the best results along with the mean ± std of the set of accuracies obtained from grid searches for each method.**
| Method | iNat 2018 | iNat 2019 | iNat 2021-mini |
| --- | --- | --- | --- |
| Baseline | 75.0 (74.6±0.6) | 81.1 (79.8±1.0) | 75.8 (75.2±0.6) |
| DTM (Ours) | 78.5 (77.4±0.6) | 81.9 (81.2±0.6) | 78.4 (77.5±0.6) |

**Table B: Transfer learning results on Fine-Grained Visual Classification (FGVC) datasets. We present the end-to-end fine-tuning accuracies on multiple datasets, reporting the best results along with the mean ± std of the accuracies from grid searches. Our Dynamic Token Morphing (DTM) outperforms the baseline at the best accuracies overall.**
| Method | Birds | CUB-200 | CIFAR-10 | CIFAR-100 | Dogs | Average |
| --- | --- | --- | --- | --- | --- | --- |
| Baseline | 87.3 (86.5±0.6) | 87.1 (86.8±0.6) | 99.2 (99.1±0.0) | 92.0 (91.9±0.3) | 86.9 (86.8±0.1) | 90.5 |
| DTM (Ours) | 88.8 (88.2±0.4) | 88.8 (88.1±0.4) | 99.3 (99.2±0.0) | 92.3 (92.1±0.2) | 87.9 (87.8±0.2) | 91.4 (+0.9) |

**Table C: Applicability of DTM on the superpixel algorithm. All the studies report fine-tuning accuracies for each configuration pre-trained using ViT-B/16. All models are pre-trained for 100 epochs on ImageNet-100. Here, the Layer-wise superpixel method denotes a token reduction approach that generates superpixel tokens across the layers. While the superpixel algorithm is applicable to DTM, Bipartite matching exhibits the best performance, demonstrating the superiority of our design choice. We mark the default settings for the study in gray.**
| Method | Fine-tuning ($\%$) |
| --- | --- |
| Baseline | 79.5 |
| Layer-wise superpixel (Achanta et al., ) | 86.7 |
| DTM (Superpixel clustering) (Chang et al., ) | 87.1 |
| DTM (Bipartite matching) | 87.9 |

**Table D: Ablation study on the number of DTM schedules. We study the effect of exploring diverse morphed tokens through multiple scheduling. We report fine-tuning accuracies for each configuration, which are pre-trained with ViT-B/16. All the backbones are pre-trained for 100 epochs on ImageNet-1K Russakovsky et al. (). Simultaneous exploration using multiple morphing schedules further enhances the performance. We mark the default settings for the study in gray.**
| Method | Case | IN-1K Fine-tuning $(\%)$ | IN-100 Fine-tuning $(\%)$ |
| --- | --- | --- | --- |
| Baseline |  | 83.5 | 79.5 |
| DTM | 1 | 84.8 | 87.6 |
| 2 | 84.9 | 87.9 |  |
| 3 | 84.8 | 87.8 |  |
|  | 4 | 84.8 | 87.8 |

**Table E: Ablation study on various configurations. We investigate the effectiveness of our design choices for DTM: morphing a random number of tokens, gradual morphing by multiple morphing steps, and target normalization. We report fine-tuning accuracies for each configuration. We adopt ViT-B/16 with a resolution of $224\times 224$. All the models are pre-trained for 100 epochs on ImageNet-100 Russakovsky et al. (). While other configurations improve the baseline well, our design yields the best accuracy. We mark the default settings for the study in gray.**
| Method | Configurations | Fine-tuning ($\%$) |
| --- | --- | --- |
| Baseline |  | 79.5 |
| DTM (ours) | Fixed number of morphing tokens | 87.4 |
| Single step morphing | 87.7 |  |
| + Target normalization He et al. () | 87.7 |  |
| Default | 87.9 |  |

**Table F: Ablation studies on a single DTM schedule. We perform ablation studies on loss functions, ranges of token morphing steps, and target normalization. All the studies report fine-tuning accuracies for each configuration pre-trained using ViT-B/16. All models are pre-trained for 100 epochs. We mark the default settings for the study in gray.**
| Case | Fine-tuning ($\%$) |
| --- | --- |
| $\ell_{1}$ | 84.5 |
| $\ell_{2}$ | 84.6 |
| Smoothed $\ell_{1}$ | 84.4 |
| Cosine distance (Cos) | 84.8 |

## Appendix D Additional ablation studies

### D.1 Ablation Study on Randomness of DTM.

Randomness in the number of morphing tokens.
Our DTM randomizes the number of morphing tokens since fixing the number of morphing tokens does not adequately generate diverse morphed tokens.
To verify the effectiveness of the randomness, we compare the fine-tuning accuracies of models pre-trained using DTM with random and fixed numbers of morphing tokens.
We pre-train and fine-tune the models for 100 epochs on ImageNet-100 (Russakovsky et al., ).
As reported in the 2nd and 5th rows of Table E, exploring diverse morphed tokens improves the fine-tuning accuracy by 0.5$\%$p. This result demonstrates the impact of varying numbers of morphing tokens.

Randomness in gradual token morphing.
As morphed tokens can vary by the number of morphing iterations, we apply randomness in token morphing iterations.
We compare fine-tuning accuracies of the pre-trained models with random and fixed iteration numbers for morphing to validate the effect of randomness.
As shown in the 3rd and 5th rows of Table E, randomly selecting the iteration number for morphing enhances the performance, confirming its effectiveness.

### D.2 Ablation Study on target normalization.

Target normalization is proven to have a significant impact on MAE (He et al., ). Thus, we verify the impact of target normalization on our DTM.
We employ ViT-B/16 with a resolution of $224\times 224$ for comparison.
As shown in the 4th and 5th rows of Table E, target normalization does not yield a positive effect on our DTM, resulting in a decrease in fine-tuning accuracy from 87.9$\%$ to 87.7$\%$.

### D.3 Ablation study

We conduct ablation studies on loss functions, the range of token morphing steps.
We also study the impacts of the number of morphing schedules.
We pre-train ViT-B/16 with a resolution of $224\times 224$ for 100 epochs and fine-tune for 100 epochs on ImageNet-1K (Russakovsky et al., ).

Loss function.
We compare various options for the loss function in our method.
We compared $\ell_{1}$, $\ell_{2}$, smoothed $\ell_{1}$, and cosine distance.
As shown in Table LABEL:tab:ablation_loss_ftn, the model pre-trained using cosine distance outperforms the models with other distance functions.

Range of token morphing iterations.
We compare fine-tuning accuracies of the pre-trained models while varying the ranges that randomly samples the number of morphing iterations in Table LABEL:tab:ablation_step.
While our DTM works for all the sampling ranges, K=14 works best.

## Appendix E Implementation details

Pre-training on ImageNet-1K.
The pre-training recipe for DTM mainly follows the recipe of BEiT v2 (Peng et al., ).
Table G reports the implementation details for pre-training.
We train our framework with ViT-S/16, ViT-B/16, and ViT-L/16 for 300 epochs using AdamW with momentum (0.9, 0.98) and a batch size of 1024.
We use a learning rate of $1.5\times 10^{-4}$ with cosine decay and warmup 10 epochs.
We employ the CLIP base models (Radford et al., ) with its visual projector as a target model across all scales of ViT. Block-wise masking is used with a ratio of 0.4 following (Bao et al., ; Peng et al., ).
Cosine distance is used as a distance metric for the objective according to an ablation study in Appendix.
We adopt the hyperparameters for ViT-B/16 and ViT-L/16 pre-training from BEiT v2.
Specifically, we use layer scales of 0.1 and $1\times 10^{-5}$ for ViT-B/16 and ViT-L/16, respectively.
We employ both relative positional embeddings and shared relative positional embeddings.
The maximum gradient value is constrained to 3.0.
We apply color jittering followed by random resizing and cropping for data augmentation.
The hyperparameters for ViT-S/16 replicate the settings of ViT-B/16.
For the hyperparameters of DTM, we used 2 for $L$, 1 for $\bar{N}_{1}$, 147 for $\bar{N}_{2}$, and 14 for all $K_{l}$.
We also pre-train various SSL frameworks through our DTM with the same fundamental setups.

Fine-tuning on ImageNet-1K.
We fine-tune our pre-trained models on ImageNet-1K (Russakovsky et al., ) by default following the standard protocol (He et al., ; Peng et al., ).
Specifically, pre-trained ViT-S/-B/-L are fine-tuned for 300, 100, and 50 epochs, respectively. Optimization is performed with AdamW using a weight decay of 0.05.
We use a layer-wise learning rate decay of 0.6 for ViT-S and ViT-B and 0.8 for ViT-L.
Learning rate is set to $5\times 10^{-4}$ with a linear warmup for 10 epochs for ViT-S and ViT-B and 5 epochs for ViT-L.
We adopt commonly used values for RandAugment, Mixup, Cutmix, and Label Smoothing.
On the other hand, we employ relative positional embeddings.
Stochastic depth is applied with values of 0.1, 0.1, and 0.2 for ViT-S/16, ViT-B/16, and ViT-L/16, respectively.
The overall recipe is detailed in Table H.

**Table G: Hyperparameters for pre-training on ImageNet-1K.**
| Hyperparameters | ViT-S/16 | ViT-B/16 | ViT-L/16 |
| --- | --- | --- | --- |
| Layers | 12 | 12 | 24 |
| Hidden size | 384 | 768 | 1024 |
| FFN inner hidden size | 1536 | 3072 | 4096 |
| Attention heads | 6 | 12 | 16 |
| Layer scale | 0.1 | 0.1 | 1e-5 |
| Patch size | $16\times 16$ |  |  |
| Relative positional embeddings | ✓ |  |  |
| Shared relative positional embeddings | ✓ |  |  |
| Training epochs | 300 |  |  |
| Batch size | 1024 |  |  |
| Adam $\beta$ | (0.9, 0.98) |  |  |
| Base learning rate | 1.5e-4 |  |  |
| Learning rate schedule | Cosine |  |  |
| Warmup epochs | 10 |  |  |
| Gradient clipping | 3.0 |  |  |
| Dropout | ✗ |  |  |
| Drop path | 0 |  |  |
| Weight decay | 0.05 |  |  |
| Data Augment | RandomResizeAndCrop |  |  |
| Input resolution | $224\times 224$ |  |  |
| Color jitter | 0.4 |  |  |

**Table H: Hyperparameters for fine-tuning on ImageNet-1K.**
| Hyperparameters | ViT-S/16 | ViT-B/16 | ViT-L/16 |
| --- | --- | --- | --- |
| Fine-tuning epochs | 300 | 100 | 50 |
| Warmup epochs | 10 | 10 | 5 |
| Layer-wise learning rate decay | 0.6 | 0.6 | 0.8 |
| Batch size | 1024 |  |  |
| Adam $\epsilon$ | 1e-8 |  |  |
| Adam $\beta$ | (0.9, 0.999) |  |  |
| Base learning rate | 5e-4 |  |  |
| Learning rate schedule | Cosine |  |  |
| Repeated Aug | ✗ |  |  |
| Weight decay | 0.05 |  |  |
| Label smoothing $\varepsilon$ | 0.1 |  |  |
| Stoch. depth | 0.1 | 0.1 | 0.2 |
| Dropout | ✗ |  |  |
| Gradient clipping | ✗ |  |  |
| Erasing prob. | 0.25 |  |  |
| Input resolution | $224\times 224$ |  |  |
| Rand Augment | 9/0.5 |  |  |
| Mixup prob. | 0.8 |  |  |
| Cutmix prob. | 1.0 |  |  |
| Relative positional embeddings | ✓ |  |  |
| Shared relative positional embeddings | ✗ |  |  |

**Table I: Hyperparameters for fine-tuning on ADE20K.**
| Hyperparameters | ViT-B/16 |
| --- | --- |
| Input resolution | $512\times 512$ |
| Peak learning rate | 5e-5 |
| Fine-tuning steps | 160K |
| Batch size | 16 |
| Adam $\epsilon$ | 1e-8 |
| Adam $\beta$ | (0.9, 0.999) |
| Layer-wise learning rate decay | 0.75 |
| Minimal learning rate | 0 |
| Learning rate schedule | Linear |
| Warmup steps | 1500 |
| Dropout | ✗ |
| Stoch. depth | 0.1 |
| Weight decay | 0.05 |
| Relative positional embeddings | ✓ |
| Shared relative positional embeddings | ✗ |

Fine-tuning on ADE20K.
Table I summarizes the fine-tuning recipe of ViT/16 for the semantic segmentation task on ADE20K (Zhou et al., ).
We employ AdamW with momentum (0.9, 0.999) and warm-up for 1500 iterations.
The learning rate is linearly scheduled with a value of $5\times 10^{-5}$
We apply layer-wise learning rate decay of 0.75, stochastic depth of 0.1, and weight decay of 0.05.
The model is fine-tuned using 8 V100-32GB GPUs.

Transfer learning.
We follow the fine-tuning recipes for DTM to conduct transfer learning to iNaturalist datasets, including iNaturalist 2018, iNaturalist 2019, and mini iNaturalist 2021 Van Horn et al. () and FGVC datasets, including Birds, CUB-200, CIFAR-10, CIFAR-100, and Dogs. However, we additionally perform grid searches of learning rates and weight decay. Specifically, we fine-tune the models with learning rates of $2.5\times 10^{-5}$, $5\times 10^{-5}$, and $1\times 10^{-4}$ and weight decays of 0.05 and 0.1.

Applicability on various SSL frameworks.
When pre-training and fine-tuning models with MAE (He et al., ), BEiT v2 (Peng et al., ), and BYOL (Grill et al., ), we follow their vanilla training recipes. We pre-train and fine-tune ViT-B/16 for 100 epochs. However, we use CLIP (Radford et al., ) target features instead of patchified images to pre-train MAE.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - For the Token encoding section of "4.1 Preliminary" section, consider adding more introduction/references to the concept introduced, for example "online encoder" and "target encoder.
- It will be easier for readers to follow if there is an additional figure about how to obtain the morphing matrix M.
soundness: 4
strengths: - The paper achieves SOTA results on ImageNet-1k. 
- The proposed method is applicable to various SSL frameworks such as MAE, BEiT and BYOL.
- The paper is very well written and has extensive experiments and visualizations.
- The "Pilot Study" section gives very good narrative to the paper and provides intuitive motivations.
summary: The paper proposes a novel masked image modeling method. The paper identified the issue of the existing pre-training models: the spatial consistency in target representations despite proximity and contextual similarity. Then the author proposed a context-aware token aggregation method called Dynamic Token Morphing (DTM). The experiments show that the proposed DTM method is general enough to help improve various SSL frameworks and achieve SOTA ImageNet-1k performance. In addition, the computation complexity add-on of the proposed method is not large.
weaknesses: The paper is very well-written and achieves SOTA results, I don't have weakness to add.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: In addition to weakness, there are also some questions:
1. In line 239, "we do not employ a superpixel-based clustering for our method, which is inefficient as well." Is the decision to avoid superpixel-based clustering solely due to its low performance? Could the reasons be analyzed further?
2. Please explain why the absence of a dynamic mechanism results in lower performance than the baseline in Table 7.
soundness: 2
strengths: 1.This paper provided an extensive discussion and demonstration of the spatial inconsistency in pre-trained representations, illustrating the significance of this issue in the domain of self-supervised learning.
2. DTMs re-characterize visual features, mitigating the inflexibility associated with patch-based representations.
3. This paper integrated DTM with multiple baseline methods, such as MAE, BEiT v2, BYOL, etc., demonstrating the effectiveness and convenience of the proposed approach.
summary: This paper investigates the issue of spatially inconsistent target representations during pre-training. The authors conducted quantitative and qualitative experimental observations of this phenomenon to discuss the impact of such spatial inconsistencies, particularly on the performance of downstream classification tasks. To address this issue, based on the MIM baseline, they propose an advanced token aggregation method termed Dynamic Token Morphing (DTM), which integrates tokens with contextual associations. The paper reports the classification results on ImageNet-1k and the segmentation performance on ADE20k, demonstrating consistent improvements with DTM across the SSL framework and the SLIP model.
weaknesses: 1. Although DTM presents an intuitive feasibility in addressing the spatial inconsistency of visual representations, the explanation in the methods section is not very intuitive or detailed, which may lead to comprehension errors. For instance, the process of how to perform k iterations based on bipartite matching to obtain the final token morphing matrix M, and how to derive the aggregated results from M, is not clear. Perhaps an algorithm flowchart would be beneficial.
2. Some details in the writing of the article could be further optimized. For example, on line 264, whether generating vi with xiM should not include masked tokens, as there is an inconsistency with Figure 4 (the input of the target encoder does not include masked tokens). Additionally, in Figure 4, the lines connecting the Morphing Matrix to Vi and Ui are different; does this indicate different operations?
3. The authors emphasize in the abstract that DTM brings about faster training, but when comparing with other methods, especially the baseline method, the best results are achieved using the same number of epochs. Does this imply that the overall training time was reduced?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the second point of Weaknesses.
soundness: 3
strengths: 1.A interesting insight on masked image modeling pre-training methods and good results are achieved across all scale models including ViT-S/B/L.
2.The proposed method can be used for both MIM and contrastive learning mothods
summary: This paper proposes Dynamic Token Morphing (DTM) to mitigate the identified spatial inconsistencies discriminative problem. This method can be used fot both masked image modeling methods and contrastive methods and accelerating the training. The achieved results look good and I think the novelty is also good. But I concern several weaknesses as I noted below. I may change the rate after reviewing other reviewers' review.
weaknesses: 1.Why not compare your method with hierarchical ViTs, such as ConvMAE [1], HiViT [2], iTPN [3], GreenMIM [4]. Is your method still effective on these methods? I suggest conducting an experiment on maybe one Small scale model.
2.The authors argue that the current methods meets the trouble of spatial inconsistencies, but existing ones actually have achieved very good results, such as EVA [5], EVA02 [6], Fast-iTPN [7]. How the authors expain that? If you think the proposed method can still improve their performance, I think conducting experiments is necessary.
3.Missing COCO experiments

[1]ConvMAE: Masked Convolution Meets Masked Autoencoders, [NeurIPS 2022]
[2]HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer, [ICLR2023]
[3]Integrally Pretrained Transformer Pyramid Networks, [CVPR2023]
[4]Green Hierarchical Vision Transformer for Masked Image Modeling, [NeurIPS 2022]
[5]EVA: Exploring the Limits of Masked Visual Representation Learning at Scale [CVPR2023]
[6]EVA-02: A Visual Representation for Neon Genesis
[7]Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration [TPAMI2024]

Meta-review: This paper presents an algorithm for visual pre-training. It was based on masked image modeling, and the method was motivated by the so-called spatial inconsistency between image tokens. To alleviate it, the proposed method involves merging (morphing) visual tokens into larger units according to inter-token similarity, and the morphed tokens are used as basic units in the masked image modeling procedure.

Overall, the method is interesting and shows good practice. All three reviewers are positive on this submission, making it a case of clear acceptance. However, the AC still has a few concerns about the submission.

* First, the paper does not clearly connect the proposed method with spatial inconsistency; I was not convinced why grouping similar visual tokens can solve the inconsistency issue, although qualitative experiments (e.g. the example in Figure 1) seem to suggest that such trained models enjoy better spatial consistency.
* Upon this, I am a bit doubtful about why the method works -- does it work by alleviating the inconsistency issue, or by finding a more proper difficulty for the MIM task (note that grouping visual tokens can decrease the difficulty and make some "unpredictable" units predictable)? The paper lacks sufficient analysis and discussion on this point.
* The writing of this paper needs major improvement; the current form makes it difficult to understand many things. Adding some figures in the rebuttal makes things better, but the AC still had a difficult time reading the paper. Reviewers also pointed out this issue.
* In addition, I do not think the experiments are sufficient to fully validate the effectiveness -- there are only limited ImageNet tests with some transfer tests on ADE20K and FGVC sets, and the ablations are minimal. By the way, I agree with Reviewer **gb6h** that convergence (shown in Figure 6) cannot be solid evidence.

The AC discussed this case with the SAC in length. With three positive reviews (6/6/8), a final decision of acceptance was made, but the above concerns do exist. The AC strongly suggests the authors to take them into consideration and try to improve the final paper.

Decision: Accept (Poster)