Morphing Tokens Draw Strong Masked Image Models
Authors: Taekyung Kim
Byeongho Heo
Dongyoon Han
Venue: ICLR 2025 Poster (2025)
Keywords: Self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We identify spatial inconsistencies of Masked Image Modeling targets tokenized from pre-trained models and resolve the issue through our proposed Dynamic Token Morphing

Abstract: Masked image modeling (MIM) has emerged as a promising approach for pre-training Vision Transformers (ViTs). MIMs predict masked tokens token-wise to recover target signals that are tokenized from images or generated by pre-trained models like vision-language models. While using tokenizers or pre-trained models is viable, they often offer spatially inconsistent supervision even for neighboring tokens, hindering models from learning discriminative representations. Our pilot study identifies spatial inconsistency in supervisory signals and suggests that addressing it can improve representation learning. Building upon this insight, we introduce Dynamic Token Morphing (DTM), a novel method that dynamically aggregates tokens while preserving context to generate contextualized targets, thereby likely reducing spatial inconsistency. DTM is compatible with various SSL frameworks; we showcase significantly improved MIM results, barely introducing extra training costs. Our method facilitates MIM training by using more spatially consistent targets, resulting in improved training trends as evidenced by lower losses. Experiments on ImageNet-1K and ADE20K demonstrate DTM's superiority, which surpasses complex state-of-the-art MIM methods. Furthermore, the evaluation of transfer learning on downstream tasks like iNaturalist, along with extensive empirical studies, supports DTM's effectiveness.

Published as a conference paper at ICLR 2025
MORPHING TOKENS DRAW STRONG MASKED IMAGE MODELS
Taekyung Kim", Byyongho Heo, Dongyoon Han" NAVER Al Lab (taekyung.Ke k,, bh. bh.heo, dongyoon Sangyoon.han)enanee haaeeerrrrcaan
ABSTRACT
Masked image modeling (MIM) has emerged a promising approach for pre- training Vision Transformers (ViTs), MIMs predict masked tokens token-wise to recover target signals that are tokenized from images or generated by pre-trained models like vision-language models. While using tokenizers or pre-trained models is viable, they often offer spatially inconsistent supervision even for neighboring tokens, hindering models from learning discriminative representations. Our pilot study identifies spatial inconsistency supervisory signals and suggests that ad- dressing it can improve representation learning. Building upon this insight, we introduce Dynamic Token Morphing (DTM), a novel method that dynamically aggregates tokens while preserving context generate contextualized targets, thereby likely reducing spatial inconsistency. DTT is compatible with various SSL frameworks; we showease significantly improved MIM results, barely intro- ducing extra training costs. Our method facilitates MIM training by using more spatially consistent targets, resulting in improved training trends evidenced by lower losses. Experiments on ImageNet- 1K and ADE20K demonstrate DTM's Ms su- periority, which surpasses complex state-of- oftt--of-the-ar MIM methods. Furthermore, the evaluation of transfer learning on downstream tasks like iNaturalist, along with extensive empirical studies, supports DTM's effectiveness. Code available at https::///// tut com nwveravverreaaaa ai/dmm.
INTRODUCTION
Since the success of Vision Transformers (ViTs) (Dosovitskiy et al., 2021), numerous training strate- gies developed for ViTs, including self-supervised learning (SSL) methods (Chen al., 2020; He et al., 2019; Grill et al., 2020; Caron et al., 2021). Recent advances in masked image modeling (MIM) (Zhou et al., 2022; He et al., 2022: Peng al., 2022: Baevski et al., 2022; Heo al., 2025; primary SSL approach for VIT. The crux of the MIM
Kim et al., 2024) solidified its position methods leveraging token-wise optimization objectives by predicting masked tokens to match given targets. MIM methods explored various approaches to assign effective target tokens, em- ploying supervision from various pre-trained models, including vision-language models (Bao al., 2022; Peng et al., 2022), utilizing momentum encoders (Baevski et al., 2022; Zhou et al., 2022), or directly exploiting patchified images (He et al., 2022; Xie et al., 2022).
While tokenizers pre-trained models have proven effective supervisory signals for MIM tar- gets (Bao et al., 2022: Peng et al., 2022; Li et al., 2022b; Wei et al., 2022c), we argue they often generate spatially noisy token representations with respect to class information (i.e., inconsistent class labels across tokens), which may impede training when utilized as pre-training targets. For example, a pre-trained vision-language model exhibits spatially inconsistent per-token prediction results, as shown in Fig. 1. To explore the spatial inconsistency in token representations from pre- trained models, analyze its effects on the models' capability. Our pilot exploration shows low accuracy metrics such as zero-shot classification without token aggregation (i.e., denoising). Our subsequent pilot study on representation learning shows that using supervisory signals alone sig- nals after token aggregation without preserving context both disrupt pre-training. Since token-wiis objectives are standard, spatially inconsistent targets likely challenge learning one-to-ome mapping.
"Equal contribution

---

Published as a conference paper at ICLR 2025
(a) Inpur image
(b) W/o aggregation (zero-shot: 26.5%
(c) Aggregation (2cro-shot: 30.8%)
Figure 1: What is spatial consistency among visual tokens? We schematically visualize loken-wise zero- shot classificution results to illustrate the spatially inconsistent token predictions. With the input image (a), the following results (b) and (c) display the predicted classes for each token within four example bounding boxes without/with token aggregations, respectively. We depict the differences between the predicted and ground trund-truth classes by varying the lightness of red, whereas the green represents the correct prediction. Each result yields 113 corrected tokens with aggregation and 82 without aggregation out of a total of 196 tokens, respectively; aggregation gives fewer spatially inconsistent representations. The zero-shot accuracies (reported in Table 1) support spatial consistency's 's connection to the model's ability.
Bearing this in mind, we introduce a novel token contextualization method called Dynamic To- ken Morphing (DTM), where token morphing aggregates related tokens while preserving context to produce coherent representations for the supervisory signal. We conjecture that training can be accelerated through the guidance of composite representations of morphed tokens derived from the preserved context even after token aggregations. Specifically, we encode the token-wise target rep- resentations and derive matching relations among tokens using DTM. The token merging process is applied to both online and target tokens considering their matching relation; it aligns each mor- phed token with the corresponding morphed target token while preserving the number of original tokens. The range of morphing can vary from single token to all tokens, covering from token- wise to image-level representation learning. Among various options, we opt for bipartite matching for morphing, achieving both efficiency and efficacy.
Through extensive experiments, we verify our method's general applicability and scalability. DTM could improve fine-tuning accuracies on ImageNet (Russakovsky et al., 2015) and ADE20K (Zhou et al., 2017), which achieve state-of-the-an results. The effectiveness of our method is supported by accelerated fine-tuning trends after DTM pre-training, which highlights how spatially consistent tar- gets are crucial. Our method shows further transferability on the iNaturalist (Van Horn et al., 2018) and fine- -ra-graindd visual classification datasets (Van Horn et al., 2015; Krizhevsky, 2009; Khosla et al., 201 We further demonstrate DTM's broad applicability with other supervisory signals and SSL frameworks and provide a deeper understanding of our design insights through ablation studies.
2 RELATED WORK
Masked image modeling. Inspired by the promising performance of masked language modeling (MLM), BETT (Bao et al., 2022) successfully extends MLM into the computer vision domain, using an external offline tokenizer from Dall-E (Ramesh et al., 2021). iBOT (Zhou et al., 2022) jointly trains the target encoder and the online tokenizer to remove the dependency on the external tok- enizer. Data2vec (Baevski al., 2022) incorporates momentum encoder to perform feature-level masked prediction tasks, leveraging representations from the multiple layers of neural networks. MAE (He al., 2022) and SimMIM (Xie et al., 2022) demonstrate efficient masked image model- ing by directly reconstructing masked input pixels without any tokenizer. On the other hand, several attempts have been made to exploit the pre-trained model as tokenizer. BEiT v2 (Peng et al., 2022) pre-trains codebook for CLIP (Radford et al., 2021) to discretize semantic space. MVP (Wei et al., 2022b) exploits a tokenizer pre-trained with multimodal data to enhance the semantics for MIM. A line of studies (Wei et al., 2022e; Ren al., 2023) using CLIP target representations have also been highlighted. Our method aims to utilize a teacher model more effectively, including CLIP, rather than just using it raw pre- pre-trained model.
teacher to generate
Token aggregation methods. Token aggregation can conceptually be categorized as a token cluster- ing method and usually aims for efficiency. Hard clustering methods like K-Means (Lloyd, 1982), K-
2

---

Published as a conference paper at ICLR 2025
Medoids (kme, 1990), and Density-Peal Clustering with K-Nearest Neighbors (DPC-KNN) (Jiang et al., 2019) enforce each data to belong to a single cluster exclusively. Bipartite matching (Karp et al., 1990) also aggregates data in a hard clustering manner, which optimizes pairs from two disjoint sets given objective function. Meanwhile, soft clustering is defined to let data belong to multiple clusters. LIT (Pan al., 2022) employs deformable token merging layers to aggregate tokens be- tween stages. Furthermore, some token pruning methods (Rao et al., 2021; Xu et al., 2022; Tang al., 2022; Liang al., 2022) can be categorized into token aggregation methods; however, they intensely focused on compressing tokens to aim for a cost-efficient vision transformer. While the above token aggregation methods have mainly been employed to boost efficiency. our approach diverges significantly. We take the concept of token aggregation to address spatially noisy target tokens in token-level supervision, thereby enhancing the efficacy of MIM in terms of precision.
PILOT STUDY
This section studies how supervisory signals from the target encoder (e.g., pre-trained model) impact representation learning. In MIM or token-based SSL frameworks, these signals are typically token representations generated by the pre-trained model, which can deliver token-wise supervision. Our study is structured into two parts:
Defining spatial inconsistency and revealing its impact ($3.1). We begin by visualizing pre- dicted token representations from a pre-trained vision-language model. Fig. 1 reveals inconsistent class predictions across tokens including neighbors, which we define as spatial inconsistency. We investigate the impact of spatial inconsistency, as the token-wise noisy predictions may ob- scure the pre-trained model's capability. The quantitative analyses Table show that reducing the inconsistency in token representations improves classification performance without training, which highlights spatial consistency's 's significance.
Handling spatial inconsistency in representation learning ($3.2). Our study extends the first study by exploring a pre-training scenario using a spatially inconsistent target, where a vision- language model provides token-wise supervision. We speculate that the lack of spatial coherence in the token-wise supervision hinders training, as noisy targets may weaken the supervisory sig- nal. We then assess token aggregation methods to address the inconsistency and confirm the effectiveness of context-preserving approaches.
3.1 SPATIAL INCONSISTENCY
We define spatial inconsistency through a practical example (see Fig. 1). Given the input image in Fig. la, we visualize the token-wise zero-shot classification results without/with token aggregation in Fig. 1b and Ic, respectively. Correct and incorrect tokens are marked in green and red, respec- tively, with gradient to darker shades of red, indicating more significant deviation from the true class. Despite the proximity and contextual similarity among tokens, wrong tokens in the green box (bottom right) in Fig. lb exhibit spatially inconsistent prediction results (French Bulldog, Gorilla, Bull Mastiff, Hotdog, and Newfoundland Dog) while tokens in Fig. Ic show correct or relatively consistent predictions (French Bulldog). Moreover, token aggregation for predictions improves curacy yielding 113 correct tokens out of 196 - 31 correct tokens more than the counterpart. This highlights the spatial inconsistency among tokens from pre-trained model, which blurs its perfor- mance and could potentially disrupt representation learning when used as supervisory signal.
Impact of spatial inconsistency. To quantitatively assess spatial inconsistency's impact in a trained model, we compute ensembled token-wise predictions using global pooling, with/without token ag- gregations. Fundamentally, we predict class scores for each token and average these scores across
Table 1: Addressing spatial inconsistency boosts accuracy. We compare the ImageNet accuracy of zero- shol/lneea probing via average pooled token-wise logit. Post-hoc morphed patch representations enhance ac- curacies, indicating that addressing spatial inconsistency improves precision. Linear probing trained for 25 epochs, and we use the fixed half of the whole tokens for aggregation.
Token Aggregation
Zero-shot image classification (%)
Averaged patch-wise cosine similarity with [CLS)
Linear probing (%)
26.5
73.2
0.53
30.8 (+4.3)
77.6 (+3.2)
0.56 (+0.3)
3

---

Published as a conference paper at ICLR 2025
Target encoder
Supervisory Signal
Lasst
F
Online encoder fo
Figure 2: Representation learning with various supervisions. We illustrate our study's base representation learning framework along with different supervisory signal functions f. We evaluate four variants of the distil- lation target: 1) token-wise supervision (baseline); 2) downsampled supervision; 3) supervision after bipartite matching layer-wise; superpixel supervision: supervision by token morphing.
Table 2: Refined supervisory signals improve representation learning. We evaluate the ImageNet-1K linear probing accuracy of supervision types (Fig. 2). Token-wise denotes the traditional baseline, which distills token-wise supervision; only refined signals preserving context surpass (upper rows). Lower rows showcase simpler targets, using basic methods that lose context by decreasing tokens, achieving efficiency only. argue that the improved accuracy presumably reveals the presence of spatial inconsistency.
Supervisory signals
Linear prob (%)
Speed (ms/img)
(1) Token-wise (baseline)
70.9
0.629
(2) Superpixel clustering (Achanta al., 2012)
72.6 (+1.7)
0.740
(3) Token morphing (our method)
72.2 (+1.3)
0.653
(4) Downsampling
68.8 (-2.1)
0.647
(5) Layer-wise Bipartite Matching (Bolya et al., 2022)
70.2 (-0.7)
0.633
all tokens within the given image. When predicting with token aggregation, we group semantically relevant tokens and average their representations group-wise prior to ensembling token-wise predic- tions. We employ CLIP-B/16 (Radford al., 2021) for our study and set to aggregate 98 tokens for token aggregation, half of the entire tokens.
Table reports the results of zero-shot image classification, linear probing, and averaged cosine similarity with the [CLS] ] token on ImageNet-IK. Note that we do not perform any extra training on the model here. Our results show that using aggregated tokens consistently exceeds the performances of those without token aggregation across all metrics. This suggests aggregating tokens can address inconsistencies and lead to performance gains. The final metric we use is continuous metric that computes patch-wise similarity; it measures the cosine similarity between the [CLS] token and each patch, which is also a patch-wise wise metric but continuous!.
As observed in Table I, the averaged similarities are computed to 0.56 vs. 0.53 for each case. This new metric, directly linked to the [CLS] token, suggests a more direct relationship between im- proved accuracy and reduced spatial inconsistency. The trend aligns with the other discrete metric, depicting patch-wise classification of 30.8% and 26.5% with and without token aggregation, respec- tively. The quantitative assessments, using both continuous and discontinuous metrics, demonstrate that token representations exhibit spatial inconsistency. Taking this further, we believe more effec- tively addressing inconsistency would yield a more significant impact.
3.2 SPATIAL INCONSISTENCY IN REPRESENTATION LEARNING
We extend the study to highlight how the spatial inconsistency in supervisory signal would affect pre- training quality. Specifically, our study is conducted in practical representation learning scenario using a -LIP-distillation method (Peng et al., 2022) with a target encoder for token-wipg wise distillation during training. We argue that poor representation may suggest spatial inconsistency in the given supervisory signal. We further argue that context-preserving preserving token aggregation (e.g., smoothing tokens but preserving their amount) reduces inconsistency.
We quantitatively compare supervisory signals in oaken-wise distillation: 1) bascline token-wiss wise signal: 2) superpixel clustering (Chang et al., 2023); 3) token morphing?: 4) downsampling: 5) layer-wise bipartite matching (Bolya et al., 2022). We pre-train ViT-B/16 for 50 epochs and linear- probe trained for 50 epochs ImageNet-1KG 1K; CLIP-B/16 is employed for the target encoder.
We believe a continuous metric to assess spatial inconsistency could track network responses continuously compared with the discrete classification metrics. Token morphing will he introduced in 1. We show its effectiveness here in advance.
4

---

Published as a conference paper at ICLR 2025
Ouline Tokens
1 Morphing Cases
Morphed Masked
LUTN
Dynamic
Token
Marphed Target Tokens
Morphing
LoTH
Morphing
Morphed Masked Token
LOTM
Target Tokens
Morphed Target Tokens of DTM
Figure 3: Token morphing offers diverse contextualized signals. Dynamic Token Morphing (DTM) aligns token representations by dynamically aggregating contextually related tokens create more diverse and diver- sified targets. Blue and Green tokens denote the representations of the image patches processed by online and target models, respectively. Gray tokens denote masked tokens.
Table 2 exhibits distillation with refined methods (i.e., superpixel clustering and token morphing) improves accuracy; this presumably confirms the presence of spatial inconsistency in the baseline token-wise supervision. However, the naive downsampling method damages intermediate repre- sentations without considering context, leading to diminished returns. While context-aware aggre- gation, such the layer-wise bipartite matching, could reduce the inconsistency, it merges tokens from early layers possibly losing distinctive information without fully restoring them to their origi- nal amounts. Finally, our study provides insights for designing an improved MIM in our upcoming method, where the MIM target utilizes vision-language model.
4 METHOD
We observed that 1) supervisory signals from pre-trained models often produce noisy and spatially inconsistent token-wise targets for learning, closely linked with performance degradation; 2) naive token aggregation methods could partially handle spatial inconsistency but are insufficient as a supervisory signal; 3) well-designed method is favorable for considering context and reducing noise more effectively. Motivated by the observations, we introduce an advanced token aggregation method called Dynamic Token Morphing (DTM) for masked image modeling and self-supervised learning. DTM contextually aggregates tokens to derive random numbers morphed tokens aim- ing encapsulate diversified semantic information. The core idea of DTM illustrated in Fig, 3, where the DTM module is straightforwardly added to a MIM baseline. DTM encourages conven- tional token-wer wise MIM by aligning morphed tokens from online and target encoders by reducing spatial inconsistency while preserving context.
4.1 PRELIMINARY
Token encoding. Given an image x, we patchify the image into N patches {a; We select positions M C 21, N) of masked patches in block-wise manner (Bao et al., 2022; Peng et al., 2022) with masking ratio
(0, 1) so that |M| = [. We mask the image patches by replacing the image patches of the position in M learnable mask token C[mask]. Specifically, the patches become (x,M). Ji=1, where Z.M. mjmkk)] for M and x;M I; for M. The masked patches (arM) = are concatenated with learnable [CLS] token and fed into the online encoder fe with subsequent linear head he while the original patches { are fed into the target encoder fç. and become encoded online tokens {u:} and encoded target tokens {v:) respectively. where u, he (x: and representations for self-supervision while the online encoder learns to encode representations from the given images (Grill et al., 2020; Zhou al., 2022; Baevski et al., 2022).
fz.).. Here, the target encoder generates target
5

---

Published as a conference paper at ICLR 2025
A A-M(1AA)
Schedulling
Virgh
Marpphing Marriss B
(R, Jevs
Online
encodor
(M.)
Dynamic Token Marphing
Figure 4: Overview of Masked Image Modeling via Dynamic Token Morphing (DTM). For token morph- ing schedulc of DTM, we aggregate the dynamic range tokens using morphing matrix M derived from target tokens [v:] Specifically, we randomly sample number remaining tokens and an iteration number k to dynamically schedule token morphing (i.e., { (rpip=1). forming morphed tokens {ú, )2 and {v, M Then, align representations of the corresponding online and target morphed tokens.
Token morphing. Unlike the traditional token aggregation methods, which prioritize efficiency via token reduction, token morphing preserves context to address the spatial inconsistency in token rep- resentations. It connects contextually relevant tokens to smooth them without reducing the number of tokens. We define the process with the token morphing function @x(-)) based the morphing schedule R, a sequence of token numbers to morph. Here, ΦR is a general notation for a func- tion Φn vxd - {0, 1 J that calculates similarity using matching algorithm (e.g., bipartite matching" or K-means clustering) and returns a token morphing matrix M = [M,y] E {0, 1) 1 where ñ means the number of token groups after morphing, and d denotes the feature dimension. Each token is ultimately assigned a smoothed representative (i.e., prototype) token (see eq. (3)).
4.2 MASKED IMAGE MODELING USING DYNAMIC TOKEN MORPHING
Here, we present a more advanced token morphing method, Dynamic Token Morphing (DTM), which designed dynamically morph tokens for MIM training. DTM's dynamic nature stems from avoiding fixed token morphing and instead simultaneously deriving multiple cases. This design based on the insight that morphing numerous tokens enhances the denoising effect while morphing fewer tokens retains detailed token representations. Furthermore, to achieve diversified morphed tokens, the morphing process repeated to generate multiple morphed tokens. DTM processes three key elements: 1) dynamic scheduler for token counts; 2) token morphing via scheduler; 3)
aligning morphed tokens. The overall framework of DTM is described in Fig.
Dynamic scheduler. DTM generates multiple morphed tokens to ensure diversity and an extensive range of token variations, as illustrated in Fig. 3. To this end, we first sample the final number of morphed tokens ń ~ U(N, N) to remain after token morphing and the iteration number A ~ U1, K) from uniform distributions, where represents the minimum number morphed tokens and K denotes the maximum number of iteration for sampling. Then, define a token count scheduler = { p=1. sequence of token numbers Tp E I that dynamically determines the number of tokens to morph for each iteration. Rather than sampling sequence of random numbers by for constant counts:
rp that satisfies ΣØ= Tp N-
n. we simply divide
(N - kk/k N - n 1) R)/k],
ifp
Tp
(1)
pf
Token morphing via dynamic scheduler. Our token morphing function ΦR progressively morphs tokens; When the morphing target is to reduce N tokens to ń, we design the morphing function
'Despite superpixel clustering's potential in $3.2, we tested our method employing 1) superpixel clustering and 2) context-aware bipartite matching yields accuracies of 87. $7.1%% 1% and 87.9%, respectively (see Table E) on ImageNet-100 1000. This prevents from using superpixel clustering in method.

---

Published as a conference paper at ICLR 2025
Imput Tokens
Murphing Matrix
Morphed Tokens
Ai
Als
Niger Ney
Myr
Ma
Nermalos
Par
Figure 5: Illustrative description of morphing matrix M A the illustration of morphing matrix M. green and white entries denote Mij = 1 and My = O, respectively, where the (i. entry indicates whether the j-th token representations V, is aggregated into the i-th morphed token representations v. Multiplying the morphing matrix M by the token representations {v:}g-1 with subsequent normalization via the number of the aggregated tokens Σ, Mi yields morphed token representations (v: )= as formulated in eq. (3). If we distribute the morphed tokens to their aggregated tokens and arrange the tokens, then we can achieve image representations with smoothed representations. Note that morphing matrix is generated for each morphing case, as shown in Fig. 3.
to conduct k-step iterative morphing. The goal of p-th iteration is to reduce "p N tokens using morphing, where ry is given to ΦR according the dynamic scheduler R. {rp]pe p= Note that the
final number of tokens is ñ (i.e., N - Ep=1"p "p ft).
We eventually obtain the token morphing matrix M, contextual relation among tokens, from the target token representations {v, L.W as follows:
M = on on ((v:)
(2)
indicates that the 5th token v, will be aggregated to the hth morphed token vi. as depicted in Fig. 5. To illustrate the detailed process of ΦR, suppose we have completed (p 1) number of iterations, resulting in partially morphed target token representations { N, where The goal of the p-th iteration is to morph the rp-most similar tokens. Thus, we apply the bipartite matching (Karp al., 1990) on {v. Np thereby derive the p-th intermediate morphing matrix M [M[]] E {0, 11. 1,,,,--XX,,, where each entry indicates whether a token is morphed or isolated. During the bipartite matching step, we split tokens into two groups, with each token the first group matched its closest cosine similarity counterpart in the second group. We repeat the process for & iterations and gather all morphing matrices with normalization MP = MP/ Σj MD, to build the morphing matrix M = ! MP, where M f0, 1 In addition, we let every token be assigned to specific cluster, even in cases where it forms single token cluster itself, and each token should retain its exclusive association with a single cluster (i.e., E, M,j and Σx Mij = 1). Note that we generate the (p + 1)-th partially morphed target token representations {v 1 No+1 by vf = E, MGyy" for the (p + 1)-th Ji=1
where My
Np NN - - Tq.
Ji=1 obtain Tp number of token pairs and
iteration. The overall process of the morphing function on that outputs the token morphing matrix M is described in Algorithm in $B through a simplified pseudo-code.
Finally, the morphed representations for both online {ū, and target tokens {ie are de- rived based on the token morphing matrix M obtained by ΦR. This involves multiplying the morphing matrix M with the online [ui, u2...., uN] E R xd and target token representations [V1, V2...., vN] E R Xxd followed by normalization with the number of aggregate tokens:
ú E, M,
V, E,
(3)
M,
Note that the morphed tokens are representative tokens for each token group, with their representa- tions being smoothed specific to their respective groups.
Aligning morphed tokens. We formulate the objective function by accumulating DTM losses, which aligns the representations of the corresponding online and target morphed tokens derived by DTM. The DTM loss with sampled ñ and k is formulated as follows:
LDIM( k) = Σ wid(ú; Vz),
(4)
7

---

Published as a conference paper at ICLR 2025
Table 3: ImageNet-IK performance comparisons. All models pre- pee-trrined/inn-tuueee ImageNet- IK. evaluate the improvements fine-tumi fine-tuning accuracies competing methods using different supervisions and ours for ViT-,S/16, B/16, L/16] with a resolution of 224 x 224. ADE20K semantic segmentation results (Seg) using ViT-B/16 are compared as well. All our models are pre-trained for 300 epochs.
Pre-training epochs
Method
Supervision
VIT-S VIT-B
VIT-L
Seg
VIT-S VIT-B VIT-
Supervised models
DefT (Touvron al., 2021)
ICML 2021
Label
81.8
DeiT-IIl (Touvron et al., 2022)
BCCV 2022
Label
83.8
84,2
49.3
Cosub (Touvron et al., 2023)
CVPR 2023
Label
81.5
84.2
85.3
49.3
MaskSub (Heo al., 2025)
CVPR 2025
81.7
84.2
85.3
50.2
Self-sopervised models
MoCo v3 (Chen al., 2021)
ICCV 2021
300
Pixel
81.7
83.2
84.1
47.3
DINO (Caron
2021)
ICCV 2021
3200
1600
Feature
82.0
83.6
46.8
SplistMask (El-Nouby et 2021)
arXiv 2021
300
300
Pixel-Feat.
81.5
83.6
45.7
BEIT (Bao ct al., 2022)
ICLR 2022
300
800
800
DALL-E
81.7
83.2
85.2
47.1
iBOT (Zhou ct al., 2022)
ICLR 2022
3200
1600
1000
Feature
82.0
84.0
84.8
50.0
MAR (He al, 2022)
CVPR 2022
1600
1600
Pixel
83.7
85.6
48.1
SimMIM (Xie et al., 2022)
CVPR 2022
800
Pixel
83.8
MaskTeat (Wei et ol., 2022a)
CVPR 2022
1600
1600
Feature
84.0
85.7
FD-CLIP (Wei ct 2022c)
arXiv 2022
300
CLIP B/16
84.9
52.8
BETT v2 (Peng al., 2022)
arXiv 2022
300
300
CLIP B/16
85.0
86.6
52.7
CAN (Mishra
2022)
arXiv 2022
1600
800
Pixel
83.6
84.7
data2vec (Baevski et 2022)
ICMI, 2022
800
1600
Feature
84.2
86.6
mc-BBT (Li al., 20226)
BCCV 2022
800
800
VOGAN
84.1
85.6
47.0
MVP (Wei al., 2022h)
ECCV 2022
300
300
CLIP B/16
84.4
86.3
52.4
SdAE (Chen al,, 2022)
ECCV 2022
Pixel
48.6
MSN (Assman ul., 2022)
BCCV 2022
600
Feature
83.4
BootMAE (Dong et al., 2022) 2022)
BCCV 2022
800
800
Fextl-Feat.
84.2
85.9
49.1
SemMAE (Li et al., 2022a)
NeurIPS 2022
800
Pixel
83.3
46.3
DecpMIM (Ren al., 2023)
arXiv 2023
300
CLIP B/16
84.8
AdPE (Wang al., 2023b)
arXiv 2023
1600
Pixel
84.4
86.3
$1.5
ExtreMa (Wu et al., 2023)
TMI 2023
300
Feature
81.8
83.7
47.9
CAE (Chen el al., 2023b)
IJC 2023
300
1600
1600
Fixel-Feat.
82.0
83,9
86.3
50.2
CMAE (Huang et al., 2023b)
TPAMI 2023
1600
1600
Fexel-Feet.
84.4
50.1
ConMIM (Yi et al., 2023)
ICLR 2023
800
1600
Dictionary
82.0
83.7
85.5
46.0
RC-MAE (Yi al., 2023)
ICLR 2023
1600
1600
1600
Pixel
82.0
83.6
86.1
MixedAE (Chen et al., 2023a)
CVPR 2023
1600
Pixel
83,9
49.8
SIM (Tao et al., 2023)
CVPR 2023
1600
Feature
R3.8
HPM (Wang el al,, 2023a)
CVPR 2023
800
800
Pixel
84.2
85.8
48.5
MIRL (Huang et al., 2023a)
NeurIPS 2023
300
Pixel
84.1
85,4
CrossMAE et al. 2024)
arXiv 2024
800
800
Pixel
79.3
83.7
85.4
dBOT (Liu al., 2024)
ICLR 2024
1600
1600
Feature
84.5
86.6
49.5
LUT (Kim ct al., 2024)
ECCV 2024
400
1600
1600
Pixel
82.0
84.2
86.0
49.5
MI-MAF (Huang et al., 2025)
arXiv 2025
400
Pixel
84.1
49.3
DTM (ours)
300
300
300
CLIP B/16
83.2
85.4
86.7
53.1
where d(-) is distance function and w; = Σ3 Mij is a number of tokens aggregated for the 7th online and target morphed tokens. Here, we utilize w; to consider all tokens aggregated for the morphed tokens. The DTM loss can be extended to token-wise
image-level losses when ñ N or ñ = 1, respectively. We adopt Cosine distance for the distance function in eq. (4) for all the DTM losses. To further enhance the dynamic nature of our method, we apply multiple DTM losses, each derived from its corresponding morphing case. The final objective function is the summation of all DTM losses, which defined as:
min Σ Lou s.t. nu ~ ((N, and ką U(1, K)),
(5)
where L denotes the total number of simultaneously employed DTM losses.
EXPERIMENT
This section first reports the ImageNet- classification and ADE20K segmentation performances. We explore DTM's strengths in terms of its efficiency and applicability. Finally, we study the impact of DTM's dynamic nature and pre-training capability. Note that additional results on DTM pre- trained models' transfer learning ($C), further DTM's applicability ($D), more empirical studies ($E), and the implementation details (SF) in Appendix.
8

---

Published as a conference paper at ICLR 2025
Table 4: Efficiency of context-aware token aggregations. We report fine-tuning accuracies and throughputs for each configuration, which are pre-trate pre-trained with VIT-B/16. We compare DIM with Bipartite matching. DTM with K-means clustering, and layer-wise K-means clustering. For the layer-wise K-means clustering. we use constant numbers of clusters and iterations. DTMs hoth outperform the bascline with large margins. Moreover, DTM with K-means clustering surpasses layer-wise K-means clustering, demonstrating the superiority of DTM.
Case
Throughput (image/s)
FT (%)
Baseline
1458
84.3
Layer-wise K-means clustering
1265
85.1
DTM (K-means clustering)
489
85,4
DTM (Bipartite matching)
1446
85.4
5.1 IMAGENET-IK CLASSIFICATION
We compare the fine-tuning accuracy of our method with previous state-of-the-art self-supervised methods on ImageNet- IK (Russakovsky al., 2015). The comparisons include supervised learn- ing and SSL methods with various supervisory signals. When the target model is CLIP, we only compare models pre-trained with CLIP B/16 for 300 epochs for fair comparison. Table 3 ports the fine-tuning accuracies of ViT-S/B/L backbones. Our baseline simply employs negative cosine loss with a vanilla CLIP model as the target model. We observe that our MIM pre-trained by DTM achieves 83.2%, 85.4%, and 86.7% top-1 accuracies with ViT-S/16, ViT-B/16, and ViT- L/16, respectively, which outperforms sfte-offthe-ee our method surpasses MVP (Wei et al., 2022b), DeepMIM (Ren et al., 2023), and BEÎT v2 (Peng et al., 2022) by 1.0%p. 0.6%p, and 0.4%p on ViT-B/16, respectively. Moreover, our method out- performs other methods that leverage diverse supervision, demonstrating our method's superiority among self superviedd learning methods.
performances across the scales. Specifically,
Additionally, extend DTM's pre- pre-training to 800 epochs, which improves fine-tuning accuracy of 85.5% % on ImageNet- IK. This result highlights 1) DTM merits longer pre-trainings and 2) DTM trained for 800 epochs surpasses others trained for 1600 epochs such BeiT v2 (Peng al., 2022). We believe even longer pre-training of DTM would lead to further improvements.
5.2 ADE20K SEMANTIC SEGMENTATION
We further evaluate semantic segmentation on ADE20K (Zhou et al., 2017) to verify the transfer- ability of our pre-trained model. We follow the training and evaluation protocol (He et al., 2022); model is fine-tuned for 160K iterations using UperNet (Xiao al., 2018) with a batch size of 16 and resolution of 512. We initialize UperNet with our pre-trained ViT-B/16. Detailed hyper- parameters for semantic segmentation fine-tuning can be found in Appendix. The first right column in Table 3 shows the mloU performance comparison. Our method also outperforms the previous state-of-the-as results with a margin of 0.3%p. validating its superiority over other SSL methods. This result signifies that our method effectively enhances discriminability for dense prediction tasks.
5.3 EFFECTIVENESS OF OUR METHOD
Efficiency. Table 4 shows that bipartite matching is efficient and effective, significantly boosting the accuracy (+1.1%p) with only a 1% speed loss. While K-means (Lloyd, 1982) also exhibits con- siderable improvements, it sigmificantly degrades the training speed. Layer-wise K-means shows accelerate the K-means method by aggregating tokens within layers at the cost of degraded repre- sentations, leading to lower accuracy.
Transferability. We verify the improved transferability of our pre-trained model. We compare fine- tuning accuracies of the baseline and our proposed model on iNaturalist datasets (Van Horn et al., 2018), which are highly imbalanced with different numbers of images per class, and Fine-Graing Visual Classification (FGVC) datasets. Table C and Table D in the Appendix show our DTM loss significantly improves the bascline with large margins, which reveals enhanced transferability.
5.4 EMPIRICAL ANALYSIS
Impact of the dynamic nature of DTM. Table presents an ablation study on the effectiveness of the dynamic nature of DTM. We employ ViT-B/16 (Dosovitskiy et al., 2021) with resolution of

---

Published as a conference paper at ICLR 2025
Top-1 Accuracy on Imagelet-11
Beselne DIM
Top- Accuraccy on ImageNet-
DTH
Trainino
Beseline
DM
Fine-tunigg Epochs
Fine-tuning Epochs
Pre-training Epochs
(a) Fine-tuning accuracy
(b) Train loss
(c) Acc vs. pre-train epochs
Figure 6: Visualizations with DTM. We plot top-1 accuracies and (b) training losses during fine-tuning on ImageNet-IK for models pre-trained by Dynamic Token Morphing (DTM) versus its baseline. (c) confirms impact of pre-training epochs for DTM over the baseline. We train the ViT-B/16 architectures with a resolution of 224 x 224. In both (a) and (b), DTM consistently exhibits a substantial gap compared to the baseline during the entire fine-tuning phase. indicating that DTM offers stronger supervision that facilitates training. DTM consistently improves the baseline regardless
pre-traiming epochs, as shown in (c).
Table 5: Empirical study on DTM's dynamic nature. investigate efficacy of dynamic scheduling DTM, which reveals its significant contrihution to our method.
Method
LTkeen Meephing
Dynamic
FT Acc. (%)
Baseline
84.3
DTM
84.0 (-0.3)
DTM
85.4 (+1.1)
224 224. The models are pre-trained for 300 epochs and fine-tuned for 100 epochs on ImageNet- IK (Russakovsky et al., 2015). For token morphing without dynamic scheduling, half of the total 196 tokens are aggregated for each image. Table exhibits that token morphing with the dynamic scheduler significantly improves the baseline while its absence incurs performance degradation, which highlights the importance of dynamic nature DTM.
Improved training frends with DTM pre-training. We analyze the effectiveness of DTM pre- training and the baseline pre-training method using the MIM's token-wise objective in Fig. the methods employ CLIP representations for the MIM targets. As shown in Fig. 6a, the fine- tuning accuracies DTM surpass the baseline and starts from significantly higher initial point. Furthermore, Fig. 6b shows that the model pre-trained by DTM exhibits a lower fine-tuning loss than the baseline model suggesting presumably better loss convergence. Finally, Fig. 6c reveals the consistent advantages of DTM pre-training across various pre-training epochs.
CONCLUSION
We have introduced a novel masked image modeling method based on the proposed token morphing address spatially inconsistent target representations during pre-training. We have first revealed the existence and impacts of spatial inconsistency target representations. Specifically, we have quali- tatively observed spatial inconsistency among tokens from pre-trained models despite proximity and contextual similarity. We have then investigated a representation learning scenario through accuracy metrics like zero-shot linear classification. Our study has validated that pereext-preserving token aggregation methods enhance the pre-training capability of the target, while arbitrary aggregation, like downsampling, disrupts it. Based on the observations, phing (DTM), which dynamically aggregates contextually associated tokens with randomness and iteratively generates diverse sets of morphed tokens. MIM training performed by aligning the representations of morphed tokens from the online and target encoders. Our extensive experiments have verified its scalability and performance superiority. We have further validated its applicability and effectiveness through empirical studies. We believe that our insights token aggregation sidering context preservation could boost supervisory signals representation learning and beyond, with the potential for broader future applications and research directions.
have proposed Dynamic Token Mor-
Limitation. Despite the potential of DTM, we have verified its applicability only up to VIT-L/16. Resource limitations restricted larger-scale experiments such with VIT-G.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Partitioning Around Medoids (Program PAM), chapter 2, pp. 68-125. John Wiley & Sons, Ltd, 1990. ISBN 9780470316801 doi: http://b...rr//11100002299944540
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süisttrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI),34111) (411))222241) 2012.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.07141, 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning speech, vision and language. Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), ICML, volume 162 of Proceedings of Machine Learning Research, pp. 1298-1981111 1312. PMLR, 17- 23 Jul 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations (ICLR), 2022.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0946) 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, October 2021.
Shuning Chang. Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from a token sparsification view. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6195-6205, 2023.
Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit- Yan Yeung. Mixed autoencoder for self-supervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22742-22751, June 2023a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. International Journal of Computer Vision (IJCV), 132(():208-22) 2233 2023b.
Xinlei Chen, Saining Xie, and Kaiming He. empirical study of training self-supervised vision transformers. arXiv preprint S10T:::20040005577 2021.
Yabo Chen, Yuchen Liu, Dongsheng Jiang. Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In European Conference on Computer Vision (ECCV), pp. 108-124. Springer, 2022.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. arXiv preprint arXiv:2207.071 0100070.11110 16, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2 /2. 10740, 2021.
11

---

Published as a conference paper at ICLR 2025
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024.
Letian Fu, Long Lian, Renhao Wang. Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A. Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. arXiv preprint arXiv:2401. 14391, 2024.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach supervisedd learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271 1-2244,, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722. 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Byeongho Heo, Taekyung Kim, Sangdoo Yun, and Dongyoon Han. Masking meets supervision: A strong learning alliance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Guoxi Huang, Hongtao Fu, and Adrian G. Bors, Masked image residual learning for scaling deeper vision transformers. In Advances in Neural Information Processing Systems, volume 36, pp- 57570-57582. Curran Associates, Inc., 2023a.
Tao Huang, Yanxiang Ma. Shan You, and Chang Xu. Learning mask invariant mutual information for masked image modeling. arXiv preprint arXhv:2502.19718 2025.
Zhicheng Huang. Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023b.
Jianhua Jiang, Yujun Chen, Xianqiu Meng, Limin Wang, and Keqin Li. A novel density peaks clustering algorithm based on k nearest neighbors for improving assignment process. Physica A: Statistical Mechanics and its Applications, 523:702-713, 2019.
Richard M Karp. Umesh V Vazirani, and Vijay V Vazirani. An optimal algorithm for on-line bi- partite matching. In Proceedings of the twenty-second annual ACM symposium Theory of computing, pp. 352-358, 1990.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In Fürst Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Puttern Recognition (CVPRW), Colorado Springs, CO, June 2011.
Taekyung Kim, Sanghyuk Chun, Byeongho Heo, and Dongyoon Han. Learning with unmasked tokens drives stronger vision learners. European Conference on Computer Vision (ECCV), 2024.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Alex Krizhevsky, Learning multiple layers of features from tiny images. In Tech Report, 2009.
Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. In Advances in Neural Information Processing Systems, volume 35, pp. 14290-14303 Curran Associates, Inc., 2022a.
Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, and Ling- Yu Duan. me-beit: Multi-choice discretization for image bert pre-training In European Conference on Computer Vision (ECCV), 2022b.
12

---

Published as a conference paper at ICLR 2025
Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang. and Pengtao Xie. EVit: Ex- pediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), 2022.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target represen- tations for masked autoencoders. In The Twelfth International Conference Learning Represen- tations (ICL.R, 2024.
S. Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2): 129-137, 1982. doi:
Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Saraa, Aaron Maschinot, and Dilip Krishnan. A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint arXiv:2210.168700 2022.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, arXiv preprint arXiv:2112.12750 2021.
Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less more: Pay less attention in vision transformers. Proceedings of the AAAI Conference Artificial Intelligence, 36(2): 2035-2043, Jun. 2022.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BETT v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning (ICML), pp. 8748-8763, PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp.
PMLR, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Informa- fion Processing Systems (NeurIPS), 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classi- fiers generalize ImageNet? In Proceedings of the 36th International Conference Machine Learning (ICML), pp. 5389-5400. PMLR, 2019.
Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep super- vision for masked image modeling. arXiv preprint arXiv:2303.08817 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJV),, 115(3):111-522 2015.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15388 2023.
Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 12165-12172 12174, June 2022.
Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang. Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for supervuud vision representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2132-2141, June 2023.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In Proceedings of the 38th International Conference Machine Learning (ICML), pp. 10347- 10357. PMIR, 2021.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European Confer- ence on Computer Vision (ECCV), pp. 516-533, Springer, 2022.
Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, and Hervé Jégou. Co-training 21 submodels for visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11701-11710, June 2023.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10375-10385, June 2023a.
Xiao Wang, Ying Wang, Ziwei Xuan, and Guo-Jun Qi. Adpe: Adversarial positional embeddings for pretraining vision transformers via maet. arXiv preprint arXiv:2303.07598 2023b.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14668-14678, June 2022a.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: dultimodality-guide visual pre-training. In European Conference on Computer Vision (ECCV), pp. 337-353. Springer, 2022b.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.
2022c
Zhirong Wu, Zihang Lai, Xiao Sun, and Stephen Lin. Extreme masking for learning instance and distributed visual representations. Transactions on Machine Learning Research, 2023, ISSN 2835-8856.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV). Springer, 2018
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9653-9663, June 2022.
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: flow-fast token evolution for dynamic vision trans- former. Proceedings of the AAAI Conference Artificial Intelligence, 36(3):2964-2972 Jun. 2022.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. In The Eleventh International Conference Learning Representations (ICLR), 2023.
14

---

Published as a conference paper at ICLR 2025
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In International Conference on Learning Repre- sentations (ICLR), 2022.
15

---

Published as a conference paper at ICLR 2025
Appendix
This appendix includes additional experimental analyses of our proposed method:
A Additional examples of spatial inconsistency among patches from pre-trained models
SB
An algorithm for Token Morphing Function
SC: Transferability of DTM on iNaturalist and Fine-Grained Visual Classification (FGVC) datasets
SD Applicability DTM on other targets and SSL frameworks
SE: Ablation studies on compatibility with superpixel algorithms (Achanta al., 2012) into DTM, the number of morphing schedules, effects of randomness in the number of morphing tokens, randomness in gradual token morphing, and target normalization
$F: Implementation details for both pre-training and fine-tuning on ImageNet-IK (Russakovsky et al., 2015) and fine-tuning on ADE20K (Zhou et al., 2017)
A MORE EXAMPLES ON SPATIAL INCONSISTENCY
EVA-CLIP. We extend analysis explore the spatial inconsistency of visual token predictions produced by other supervisory models. We employ a strong and larger-scale model: EVA-01-CLIP. g/14 (Sun al., 2023), which the teacher model for EVA-02 (Fang al., 2024). Following the analysis in Fig. 1, we visualize token-wise zero-shot classification results with and without token ag- gregation. Consistent with our earlier approach, we aggregate 128 tokens, corresponding the total tokens. Fig. A demonstrates that token-wise wise zero-shot predictions without token aggrega- tion exhibit spatially inconsistent token-wer liken-wise predictions compared those with token aggregation, similar to the behavior observed in the CLIP case. Zero-shot accuracies on ImageNetV2 (Recht et al., 2019) computed by the token-wise ensemble prediction via global pooling. follow a similar trend, where aggregation enhances zero-shot performance (53.1% % 51.2%) reflecting the capa- bility supervisory signals. This suggests that even a stronger pre-trained model can benefit from token aggregation when used as supervisory signal.
half of
16.00
7--
(a) Inpot image
Wio Aggrogation (zcro-shot: 51.2%%
(c) Aggrogation (zcro-shot: 53.19)
Figure A: Visualization of spatial inconsistency using EVA-CLIP. We present token-wirg prediction results of a sample for zero-shot image classification using EVA-01-CLIPA spatial inconsistent predictions among patches. For the given input images, we visualize the predicted classes for each patch in the four bounding boxes through token-wise zero-shot classification with and without token aggregation, respectively. Precisely, we depict the differences between predicted classes and ground- truth class by varying the lightness of red colors. We observe that the prediction results of the morphed tokens are more likely to align with the input patches, leading to reduction in the spatial inconsistency incurred by token-wise predictions.
(Sun et al., 2023) to visualize the
We further investigate the spatial inconsistency of patch representations generated by a pre- trained model across various samples. We visualize the token-wise zero-shot classification results
CLIP.
16

---

Published as a conference paper at ICLR 2025
(a) Input image
(b) w/o Aggrogation
(0) Aggrogation
Figure B: Visualization of spatial inconsistency using CLIP. We present patch-wise prediction results of sample for zero-shot image classification visualize the spatial inconsistent predictions among patches. For the given input images (a), (b), and (c), we visualize the predicted classes for each patch in the bounding boxes through patch-wise zero-shot classification with and without token aggregation, respectively. Precisely, we depict the differences between the predicted classes and the ground-truth class by varying the lightness of red colors. We observe that the prediction results of the morphed tokens more likely to aliga with the input patches, leading to reduction the spatial inconsistency incurred by patch-wise predictions.
without and with token aggregation using CLIP (Radford et al., 2021). Specifically, correct and incorrect tokens marked in green and red, respectively, with gradient to darker shades of red, indicating a more significant deviation from the true class. As shown in Fig. B, patch representa- tions without token aggregation reveal spatially inconsistent token-wise predictions compared the prediction results with token aggregation, which reveals the spatially inconsistent representations among patches. In addition. predictions with token aggregation exhibit significantly more correctly predicted patches than predictions without token aggregation.
B ALGORITHM FOR TOKEN MORPHING FUNCTION
In this section, describe the generation process of the morphing matrix M for the token morphing function dn. This function utilizes from the target representations {u the dynamic scheduler R, and the iteration number à in Algorithm 1.
17

---

Published as a conference paper at ICLR 2025
Algorithm 1: Token Morphing Function (øn)
1: input: token representation [vo iteration k. scheduler R = {rp)
2: define N
define vi f- Vi for EE [1,
4: for porpe E {l,...,k: k:)
k-iterative morphing
5: M - BIPARTTEMAACHING(*
6:
Mº f Mayl MyylE Ey'= My for all i
# Normalize # Morph matched tokens
v+l EE= Mjjy for [1, rpl
7:
8:
n n n Tp
9:
refurn M = M
10:
function BIPARTITEMATCHINGGY"
Standard bépurtite matching algorithm
11: (S,, t- random 2 n..
Split for Bipartite matching
12: sim 4- [Sim(y], vy) for (i.3) E S x S
Measure similarity
13:
sort(sim, orderw descending") [ry]]
# Threshold for top-ry. similarity
14:
Mtj 1; MP M MM.. Sim(y? v22 in S
return M
16:
end function
Table A: Applicability DTM on SLIP. We apply DTM to SLIP (Mu et al., 2021), more improved language- image pre-trained model to demonstrate our method's applicability beyond CLIP. We employ ViT-(S/16, B/16, L/16] with resolution of 224 x 224. All models are pre-trained for 300 epochs on ImageNet-1K.
Target models
Method
VIT-S
VIT-B
ViT-L
Baseline
81.8
84.0
85.7
SLIP
DTM
82.1 (40.3)
84.5 (+0.5)
86.1 (40.4)
C ASSESSING TRANSFER LEARNING
iNaturalist datasets. We verify the improved transferability of our pre-trained model. We com- pare fine-tuning accuracies of the baseline and proposed model on iNaturalist 2018, iNaturalist 2019, and mini iNaturalist 2021 (Van Horn et al., 2018), which are highly imbalanced with different numbers of images per class. All the models are V with resolution of x24x221 Following the protocol (Korbblth et al., 2019), we perform grid searches on learning rates and weight decay and report the maximum accuracy and the mean and standard deviation of the accuracies. Table C shows our DTM loss significantly improves the baseline by large margins, demonstrating enhanced transferability.
Fine-Grained Visual Classification (FGVC) datasets. We further validate the transferability of our method. Following the evaluation protocol as above, we conduct comparisons on FGVC datasets. Specifically, we evaluate fine-tuning accuracies on Birds, CUB-200, CIFAR-10, CIFAR- 100, and Dogs through grid searches with different learning rates and weight decays. As shown in Table D, our method outperforms the baseline overall across the datasets, which shows superior transferabilitty and tuning robustness.
D MORE INVESTIGATION ON EFFECTIVENESS OF OUR METHOD
On other targets. To confirm our method's applicability, we apply DTM with the SLIP (Mu al., 2021) to verify its effectiveness across different target models. We compare fine-tuning accuracies of the baseline and our method pre-trained by target patches from SLIP. Specifically, we generate target tokens using SLIP along with its projector. We pre-train ViT-B/16 with resolution of 224 224 for 300 epochs and fine-tune for 100 epochs on ImageNet- IK (Russakovsky et al., 2015). As shown in Table A, DTM successfully improves the fine-tuning accuracy of the baselines pre-trained with SLIP by 0.5%p. which reveals its general applicability beyond CLIP (Radford et al., 2021).
Applicability to SSL frameworks. We apply DTM BEiT v2. MAE with the CLIP teacher (Rad- ford et al., 2021), and BYOL (Grill et al., 2020) to verify its applicability to various SSL frame- works. As shown in Table B, our DTM successfully improves the fine-tuning accuracy of MAE
18

---

Published as a conference paper at ICLR 2025
Table B: Applicability of our DTM various SSL frameworks. We train DTM with other SSL methods show broader applicability.
adopt MAE (He et al., 2022) with CLIP teacher (Radford et al., 2021), BEiT v2 (Peng et al., 2022), and BYOL (Grill et al., 2020) to cover general SSL frameworks. We employ the ViT-B/16 architecture with a resolution of 224 x 224. All the models are pre-trained for 100 epochs.
Framework
Method
FT (%)
MAE CLIP MAE CLIP+ DTM
82.6
Feature MIM
83.2 (+0.6)
(CLIP teacher)
BEiT v2
84.2
BEiT v2 + DTM
84.4 (+0.2)
BYOL
81.7
Image-level SSL
BYOL + DTM
82.1 (+0.4)
Table C: Transfer learning results INaturalists. We further present the end-to-end fine-tuning accuracies on the iNaturalist 2018, iNaturalist 2019, and mini iNaturalist 2021 datasets (Van Horn al., 2018). We report the best results along with the mean std the set of accuracies obtained from grid searches for each method.
Method
iNat 2018
iNat 2019
iNat 2021-mini
Baseline
75.0 (74.6::06)
81.1 (798110)
75.8 (75.2:10.6)
DTM (Ours)
78.5 (77.4+0.6)
81.9 (81.210.6)
78.4 (77.5:10.6)
Table D: Transfer learning results on Fine-Grained Visual Classification (FGVC) datasets. We present the end-to-end fine-tuning accuracies multiple datasets, reporting the best results along with the mean 1 std of the accuracies from grid searches. Our Dynamic Token Morphing (DTM) outperforms the baseline the best accuracics overall.
Method
Birds
CUB-200
CIFAR-10
CIFAR-100
Dogs
Average
Baseline
87.3 (86.5=0.6)
87.1 (86.8+0.6)
99.2
+0.0)
92.0 (91.9
86.9 (86.820.1)
90.5
DTM (Ours)
88.8 (88.2=0.4)
88.8 (8.4)
99.3 (09 20.0)
92.3 (92.1
87.9 (87.8+0.2)
91.4 (+0.9)
CLIP, BEiT v2, and BYOL by 0,5%p, 0.2%p, and 0.4%p, respectively, which reveals its general applicability to various frameworks beyond our baseline.
E ADDITIONAL EMPIRICAL STUDIES
E. COMPATIBILITY WITH SUPERPIXEL ALGORITHMS
We perform additional experiments to use the concept of superpixels (Achanta et al., 2012; Chang al., 2023) in method, both directly tokens (layer-wise superpixel) and can be combined with DTM. The layer superpixel method uses constant numbers of superpixels and iterations. Table E shows superpixel-based methods enjoy notable gains but do not exceed the bipartite match- ing one. While the layer-wise superpixel method also utilizes superpixels, the layer-wise token aggregation across the encoder layers the bipartite matching approach risks harming intermediate representations during encoding
E.2 ABLATION STUDIES ON RANDOMNESS IN DTM
The number of dynamic schedules. Our method can further enhance the diversity of target mor- phed tokens by employing multiple schedules within a single iteration. Thus, we study the effects of learning diverse morphed tokens derived from multiple morphing schedules simultancously. We compare the fine-tuning accuracy of models pre-trained by our DTM, varying the number of sched- ules. We pre-train ViT-B/16 with a resolution of 224 x 224 for 100 epochs and fine-tune for 100 epochs on ImageNet-IK (Russakovsky et al., 2015) and ImageNet- 100 (Russakovsky et al., 2015). Table F reveals that exploring diverse target morphed tokens improves the representation capabil- ity of pre-trained models, leading to increased fine-tuning accuracies of at least 0. 1p and 0.2%p on ImageNet-1K 1K and ImageNet-100 compared to the model pre pre-trained by the single DTM ap- proach, respectively. However, the goal of experiencing diverse morphed tokens at once appears to be attained with double scheduling, resulting in additional gains in performance through further
19

---

Published as a conference paper at ICLR 2025
Table E: Applicability of DTM the superpixel algorithm. All the studies report fine-tunie tine-tuiigg accuracies for each configuration pre-trained using ViT-B/16. models are pre-trained for 100 epochs ImageNet- 100. Here, the Layer-wise superpixel method denotes token reduction approach that generates superpixel tokens across layers. While the superpixel algorithm is applicable to DTM, bipartite matching exhibits the hest performance, demonstrating the superiority of our design choice. The default settings for the study are marked gray.
Method
Fine-tuning (%)
Baseline
79.5
Layer-wise superpixel (Achanta et al., 2012)
86.7
DTM (Superpixel clustering) (Chang et al., 2023)
87.1
DTM (Bipartite matching)
87.9
Table F: Ablation study the number of DTM schedules. We study the effect of exploring diverse morphed tokens through multiple scheduling. We report fine-tuning accuracies for each configuration, which are pre- trained with ViT-B/16. All the backbones are pre-trained for 100 epochs on ImageNet-IK Russakovsky et al. (2015). Simultaneous exploration using multiple morphing schedules fürther enhances the performance. The default settings for the study marked in gray
Method
Case
IN-IK Fine-tuning (%)
IN-100 Tine-tuning (%)
Baseline
83.5
79.5
84.8
DTM
2
84.9
87.9
3
84.8
87.8
4
84.8
87.8
exploration. Given that utilizing two schedules yields the best and is most efficient among all other multiple scheduling options, we adopt the double morphing scheduling approach.
Randomness in the number morphing tokens. Our DTM randomizes the number of morphing tokens since fixing this number does not adequately generate diverse morphed tokens. To verify the effectiveness of the randomness, we compare the fine-tuning accuracies of models pre-trained using DTM with random or fixed numbers of morphing tokens. We pre-train and fine-tune the models for 100 epochs on ImageNet- 100 (Russakovsky et al., 2015). As reported in the 2nd and 5th rows of Table G, exploring diverse morphed tokens improves the fine-tuning accuracy by 0.5%pp This result demonstrates the impact of varying the number of morphing tokens.
Randomness in gradual token morphing. As morphed tokens vary by the number of morphing iterations, we apply randomness to token morphing iterations. We compare fine-tumin ane-ttuing accuracies of the models pre-trained with random fixed iteration numbers for morphing to validate the effect of randomness. As shown in the 3rd and 5th rows of Table G, randomly selecting the iteration number for morphing enhances the performance, confirming its effectiveness.
E.3 ON TARGET NORMALIZATION
Target normalization is proven to have a significant impact on MAE (He et al., 2022). Thus, we verify the impact of target normalization on our DTM. We employ ViT-B/16 with a resolution of 224 x 224 for comparison. As shown in the 4th and 5th rows of Table G, target normalization does not yield positive effect on our DTM, resulting in decrease fine-tuning accuracy from 87.9% to 87.7%.
E.4 FURTHER STUDIES
We conduct ablation studies on loss functions and the range of token morphing steps. We also examine the impacts of the number of morphing schedules. We pre-train ViT-B/16 with a resolution of 224 x 224 for 100 epochs and fine-tune for 100 epochs on ImageNet-IK K (Russakovsky et al., 2015).
20

---

Published as a conference paper at ICLR 2025
Table G: Ablation study various configurations. We investigate the effrectivnness of design choices for DTM: morphing a random number tokens, gradual morphing by multiple morphing steps, and target normalization. We report fine-tuning accuracies for each configuration. We adopt ViT-B/16 with resolution of 224 x 224. All the models are pre-trained for 100 epochs on ImageNet- 100 Russakovsky et al. (2015). While other configurations improve the bascline well, our design yields the best accuracy. We mark the default settings for the study gray
Method
Configurations
Fine-tuning (%)
Baseline
79.5
Fixed number morphing tokens
87.4
Single step morphing
87.7
DTM (ours)
Target normalization He et al. (2022)
87.7
Default
87.9
Table H: Ablation studies single DTM schedule. We perform ablation studies on loss functions, ranges of token morphing steps, and target normalization. All the studies report fine-tuning accuracics for cach con- figuration pre-trained using ViT-B/16. All models are pre-trained for 100 epochs. The default settings for the study are marked in gray...
(a) Loss function
(b) Morphing steps
Case
Fine-tuning (%)
Case
Fine-tuning (%)
ε1 f2 Smoothed &
84.5
4(1 7)
84.7
84.6
U(1, 14)
84.8
84.4
(111 28)
84.7
Cosine distance (Cos)
84.8
Loss function. We compare various options for the loss function in our method. We compared é,, lz. smoothed f,, and cosine distance. As shown in Table Ha, the model pre-trained using cosine distance outperforms the models with other distance functions.
Range of token morphing steps. We compare fine-tuning accuracies of the pre-trate pre-trained models while varying the ranges that randomly sample the number of morphing iterations in Table Hb. While our DTM works for all the sampling ranges, K=14 works best.
F IMPLEMENTATION DETAILS
Pre-training on ImageNet-IK. The pre-training recipe for DTM mainly follows the recipe of BEiT v2 (Peng et al., 2022). Table I reports the implementation details for pre-training. We train our framework with ViT-S/16, ViT-B/16, and ViT-L/16 for 300 epochs using AdamW with momentum (0.9, 0.98) and batch size of 1024. We use a learning rate of 1.5 x 10 with cosine decay and warmup 10 epochs. We employ the CLIP base models (Radford et al., 2021) with its visual projector target model across all scales of VIT. Block-wise masking is used with ratio of 0.4 following (Bao et al., 2022; Peng et al., 2022). Cosine distance used as a distance metric for the objective according to ablation study in Appendix. We adopt the hyperparameters for ViT-B/16 and ViT-L/16 pre-training from BEiT v2. Specifically, we use layer scales of 0.1 and 1 x 10-5 for ViT-B/16 and ViT-L/16, respectively. We employ both relative positional embeddings and shared relative positional embeddings. The maximum gradient value is constrained to 3.0. We apply color jittering followed by random resizing and cropping for data augmentation. The hyperparameterss for VIT-S/16 replicate the settings of ViT-B/16. We also pre-train various SSI. frameworks through our DTM with the same fundamental setups.
Fine-tuning on ImageNet-1K. We fine-tune our pre-trained models on ImageNet-1 1K (Russakovsky et al., 2015) by default following the standard protocol (He et al., 2022; Peng et al., 2022). Specifi- cally, pre-trained ViT-S/-B/-L are fine-tuned for 300, 100, and 50 epochs, respectively. Optimization is performed with Adam W using a weight decay of 0.05. We use a layer-wise learning rate decay of 0.6 for VIT-S and ViT-B and 0.8 for ViT-L. Learning rate is set to 5 10-4 with linear warmup for 10 epochs for VIT-S and ViT-B and 5 epochs for ViT-L. We adopt commonly used values for Ran- dAugment, Mixup, Cutmix, and Label Smoothing. On the other hand, employ relative positional
21

---

Published as a conference paper at ICLR 2025
embeddings. Stochastic depth is applied with values of 0.1, 0.1, and 0.2 for ViT-S/16, ViT-B/16, and ViT-L/16, respectively. The overall recipe is detailed in Table J.
Fine-tuning ADE20K. Table K summarizes the fine-tuning recipe of VIT/16 for the semantic segmentation task on ADE20K (Zhou et al., 2017). We employ AdamW with momentum (0.9, 0.999) and warm-up for 1500 iterations. The learning rate is linearly scheduled with value of x 10-6 We apply layer-wise learning rate decay of 0.75, stochastic depth of 0.1, and weight decay of 0.05. The model is fine-tuned using V V100-32GB GPUs.
Transfer learning. We follow the fine-tuning recipes for DTM to conduct transfer learning to iNat- uralist datasets, including iNaturalist 2018, iNaturalist 2019, and mini iNaturalist 2021 (Van Horn et al., 2018) and FGVC datasets, including Birds, CUB-200, CIFAR-IO, CIFAR-I00, and Dogs. However, we additionally perform grid searches of learning rates and weight decay. Specifically, fine-tune the models with learning rates of 2.5 x 10-5, 5 x 10-5, and x 10 and weight decays of 0.05 and 0.1.
Applicability on various SSL frameworks. When pre-training and fine-tuning models with MAE (He et al., 2022), BEiT v2 (Peng et al., 2022), and BYOL (Grill et al., 2020), we follow their vanilla training recipes. We pre-train and fine-tune V for 100 epochs. However, we use CLIP (Radford et al., 2021) target features instead of patchified images to pre-train MAE.
Table I: Hyperparameters for pre-training on ImageNet-IK
Hyperparameters
VIT-S/16
VIT-B/16
VIT-L/16
Layers
12
12
24
Hidden size
FFN inner hidden size
1536
3072
4096
Attention heads
12
16
Layer scale
0.1
1e:5
Relative positional embeddings
Shared relative positional embeddings
Training epochs
300
Batch size
1024
Adam
(0.9, 0.98)
Base leurning
1.5e-4 Cosine
Leamingg rate schedule
Warmup epochs
Gradient clipping
3.0
Dropout
Drop puth
Weight decay
0.05
Data Augment
RandonResizeAndCrrp
Input resolution
224 x 224
Color jitter
22

---

Published as a conference paper at ICLR 2025
Table J: Hyperparameters for fine-tuning on ImageNet-IK.
Hyperparameters
ViT-S/16
ViT-B/16
ViT-L/16
tinentunngg epochs
300
100
50
Warmup epochs
Layer-wia biser-wise learning rate decay
0.6
0.6
0.8
Batch size
1024
Adam
1c-8
Adam |3
(0.9, 0.999)
Base learning rate
Sc-4
Learming rate schedule
Cosine
Repeated Aug
X 0.05
Weight decay
Label smoothing
0.1
Stoch. depth
0.1
0.1
0.2
Dropout
Gradient clipping
Erasing prob.
0.25
Input resolution
224 224
Rand Augment
9/0.5
Mixup proh.
Cutmix prob.
1.0
Relative positional embeddings
Shared relative positional embeddings
X
23

---

Published as a conference paper at ICLR 2025
Table K: Hyperparameters for fine-tuning on ADE20K.
Hyperparameters
VIT-B/16
Input resolution
512 512
Peak learning rate
5e-5
Fine-tuning steps
160K
Batch size
16
Adam é
lc-8
Adam /
(0.9, 0.999)
Layer-wise learning rate decay
0.75
Minimal learning rate
Learning rate schedule
Linear
Warmup steps
1500
Dropout
Stoch. depth
0.1
Weight decay
0.05
Relative positional embeddings
Shared relative positional embeddings
24


Reviews:
[{"review_id": "8GjhviBqXz", "reviewer": "Reviewer_xEMo", "summary": "The paper proposes a novel masked image modeling method. The paper identified the issue of the existing pre-training models: the spatial consistency in target representations despite proximity and contextual similarity. Then the author proposed a context-aware token aggregation method called Dynamic Token Morphing (DTM). The experiments show that the proposed DTM method is general enough to help improve various SSL frameworks and achieve SOTA ImageNet-1k performance. In addition, the computation complexity add-on of the proposed method is not large.", "questions": "- For the Token encoding section of \"4.1 Preliminary\" section, consider adding more introduction/references to the concept introduced, for example \"online encoder\" and \"target encoder.\n- It will be easier for readers to follow if there is an additional figure about how to obtain the morphing matrix M.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper achieves SOTA results on ImageNet-1k. \n- The proposed method is applicable to various SSL frameworks such as MAE, BEiT and BYOL.\n- The paper is very well written and has extensive experiments and visualizations.\n- The \"Pilot Study\" section gives very good narrative to the paper and provides intuitive motivations.", "weaknesses": "The paper is very well-written and achieves SOTA results, I don't have weakness to add.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OX2Mtjnvvr", "reviewer": "Reviewer_gb6h", "summary": "This paper investigates the issue of spatially inconsistent target representations during pre-training. The authors conducted quantitative and qualitative experimental observations of this phenomenon to discuss the impact of such spatial inconsistencies, particularly on the performance of downstream classification tasks. To address this issue, based on the MIM baseline, they propose an advanced token aggregation method termed Dynamic Token Morphing (DTM), which integrates tokens with contextual associations. The paper reports the classification results on ImageNet-1k and the segmentation performance on ADE20k, demonstrating consistent improvements with DTM across the SSL framework and the SLIP model.", "questions": "In addition to weakness, there are also some questions:\n1. In line 239, \"we do not employ a superpixel-based clustering for our method, which is inefficient as well.\" Is the decision to avoid superpixel-based clustering solely due to its low performance? Could the reasons be analyzed further?\n2. Please explain why the absence of a dynamic mechanism results in lower performance than the baseline in Table 7.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.This paper provided an extensive discussion and demonstration of the spatial inconsistency in pre-trained representations, illustrating the significance of this issue in the domain of self-supervised learning.\n2. DTMs re-characterize visual features, mitigating the inflexibility associated with patch-based representations.\n3. This paper integrated DTM with multiple baseline methods, such as MAE, BEiT v2, BYOL, etc., demonstrating the effectiveness and convenience of the proposed approach.", "weaknesses": "1. Although DTM presents an intuitive feasibility in addressing the spatial inconsistency of visual representations, the explanation in the methods section is not very intuitive or detailed, which may lead to comprehension errors. For instance, the process of how to perform k iterations based on bipartite matching to obtain the final token morphing matrix M, and how to derive the aggregated results from M, is not clear. Perhaps an algorithm flowchart would be beneficial.\n2. Some details in the writing of the article could be further optimized. For example, on line 264, whether generating vi with xiM should not include masked tokens, as there is an inconsistency with Figure 4 (the input of the target encoder does not include masked tokens). Additionally, in Figure 4, the lines connecting the Morphing Matrix to Vi and Ui are different; does this indicate different operations?\n3. The authors emphasize in the abstract that DTM brings about faster training, but when comparing with other methods, especially the baseline method, the best results are achieved using the same number of epochs. Does this imply that the overall training time was reduced?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6wEnUPv1CU", "reviewer": "Reviewer_L29Y", "summary": "This paper proposes Dynamic Token Morphing (DTM) to mitigate the identified spatial inconsistencies discriminative problem. This method can be used fot both masked image modeling methods and contrastive methods and accelerating the training. The achieved results look good and I think the novelty is also good. But I concern several weaknesses as I noted below. I may change the rate after reviewing other reviewers' review.", "questions": "See the second point of Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.A interesting insight on masked image modeling pre-training methods and good results are achieved across all scale models including ViT-S/B/L.\n2.The proposed method can be used for both MIM and contrastive learning mothods", "weaknesses": "1.Why not compare your method with hierarchical ViTs, such as ConvMAE [1], HiViT [2], iTPN [3], GreenMIM [4]. Is your method still effective on these methods? I suggest conducting an experiment on maybe one Small scale model.\n2.The authors argue that the current methods meets the trouble of spatial inconsistencies, but existing ones actually have achieved very good results, such as EVA [5], EVA02 [6], Fast-iTPN [7]. How the authors expain that? If you think the proposed method can still improve their performance, I think conducting experiments is necessary.\n3.Missing COCO experiments\n\n[1]ConvMAE: Masked Convolution Meets Masked Autoencoders, [NeurIPS 2022]\n[2]HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer, [ICLR2023]\n[3]Integrally Pretrained Transformer Pyramid Networks, [CVPR2023]\n[4]Green Hierarchical Vision Transformer for Masked Image Modeling, [NeurIPS 2022]\n[5]EVA: Exploring the Limits of Masked Visual Representation Learning at Scale [CVPR2023]\n[6]EVA-02: A Visual Representation for Neon Genesis\n[7]Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration [TPAMI2024]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)