MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Masked Image Modeling Representations
Authors: Benedikt Alkin
Lukas Miklautz
Sepp Hochreiter
Johannes Brandstetter
Venue: ICLR 2025 Poster (2025)
Keywords: self-supervised learning
masked image modeling
instance discrimination
computer vision
contrastive learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce MIM-Refiner, a refinement process of MIM(masked image modeling)-models using an ensemble of instance discrimination heads attached at intermediate layers to leverage the best pre-trained MIM representations.

Abstract: We introduce MIM (Masked Image Modeling)-Refiner, a contrastive learning boost for pre-trained MIM models. MIM-Refiner is motivated by the insight that strong representations within MIM models generally reside in intermediate layers. Accordingly, MIM-Refiner leverages multiple instance discrimination (ID) heads that are connected to different intermediate layers. In each head, a nearest neighbor ID objective constructs clusters that capture semantic information which improves performance on downstream tasks, including off-the-shelf and fine-tuning settings.

The refinement process is short and simple -  yet highly effective. Within a few epochs, we refine the features of MIM models from subpar to state-of-the-art, off-the-shelf features. Refining a ViT-H, pre-trained with data2vec 2.0 on ImageNet-1K, sets a new state-of-the-art in linear probing (84.7\%) and low-shot classification among models that are pre-trained on ImageNet-1K. MIM-Refiner efficiently combines the advantages of MIM and ID objectives, enabling scaling ID objectives to billion parameter models using relatively little compute. MIM-Refiner compares favorably against previous state-of-the-art SSL models on various benchmarks such as low-shot classification, long-tailed classification and semantic segmentation.

Published as a conference paper at ICLR 2025
MIM-REFINER: A CONTRASTIVE LEARNING BOOST FROM INTERMEDIATE TRAINED MASKED IMAGE
MODELING REPRESENTATIONS
Benedikt 1.2 Lukas Miklautz
Sepp Hochreiter
Johannes Brandstetter
EELLI Unit Linz, Institute for Machine Learning, JKU Linz, Austria
Emmi AI GmbH. Linz, Austria "NXAI GmbH, Linz, Austria Faaculty of Computer Science, University of Vienna, Vienna, Austria "alkin@ml, jku.at
ABSTRACT
We introduce MIM (Masked Image Modeling)-Refiner, contrastive learning boost for pre-trained MIM models. MIM-Refiner is motivated by the insight that strong representations within MIM models generally reside in intermediate layers. Accordingly, MIM-Refine leverages multiple instance discrimination (ID) heads that are connected to different intermediate layers. In each head, nearest neighbor ID objective constructs clusters that capture semantic information which improves performance downstream tasks, including off-the-shelf and fine-tuning settings.
The refinement process is short and simple - yet highly effective. Within few epochs, we refine the features of MIM models from subpar to state-of-the-art, off- the-shelf features. Refining ViT-H, pre-trained with data2vee 2.0 on ImageNet- K, sets new state-of-ther thet in linear probing (84.7%) and low-shot classification among models that are pre-trained on ImageNet- IK. combines the advantages of MIM and ID objectives, enabling scaling ID objectives to billion parameter models using relatively little compute. MIM-Refiner compares favorably against previous state-of-the-art SSL models on various benchmarks such as low-shot classification, long-tailed classification and semantic segmentation. Project page: https:/ hhmps://mm-iae //mm/mmmmm/mm jku. qithub. 10/MIM Re finer
efficiently
MIM-Refiner VIT-H/14
INTRODUCTION
84
Self-supervised learning (SSL) has attracted consid- erable attention, owing to its data efficiency and gen- eralization ability (Liu et al.) 2021). SSL leverages pre-training tasks and creates intricate input represen- tations without the need for explicit supervision via expensive annotations. In computer vision, Masked Image Modeling (MIM) (Chen et al.) [2200b] |innent et al. 2010: Pathak et al. 2016) has emerged as one of the prevalent SSL pre-training paradigms, enabling an
Accuracy [9]
MUGS WIT-L/26
82
v3
VIT-N16
MAF-CT
VIT-LI7-BN.
1BOT
VITIL/6
80
BYOL
2essNet-200x2
2020/06
2022.04
2022/03
202 2024/02
efficient pretraining of large models on unlabeled data
Figure 1: Linear probing state-of-the-art on ImageNet- 1K over the last four years.
by reconstructing masked parts of the input images.
The success of MIM is driven by methods like Masked Autoencoder (MAE) (He et al.) 2022). data2vec (Baevski al., 2022 2023), and others (Bao al. 2022) Xie et al. 2022). For example, MAE opens the door for sparse pre-training of Vision Transformers (ViTs) (Dosovitskiy et al.) 202I) by masking large parts of the image and not processing the masked areas. The computational efficiency, coupled with the data efficiency of generative reconstruction task (Xie et al. 2023
El-Nouby et al. 2021; Singh et al. 2023) fosters the scaling to larger architectures on datasets of limited size. However, MIM models tend to spread their attention across the whole image (Walmer ct al. 2023). When adapting to downstream tasks, a sufficient amount of labels required to rewire the attention to focus important regions in the image. In the absence thereof, MIM models perform

---

Published as a conference paper at ICLR 2025
WUrLE Sight
Celech LCI PrEe
Bunt97
ImageNet-1K Linear Probing D2V2-Refined
MIM-Refiner
84.7%
MAE-Refined
83
V VABSE Pheeiee
20%
MAE-CT
Accuracy
I-JERA
MIM
10
81
+5.3%
IBOT
D2V2 DINO
MAE
Mo v3
L/16
H/14
77
L/7
1000
2000
3000
4000
Pre-t Pre-training GPU Hours
mio-ome
Pride
Figure 2: Left: Downstream evaluations of pre-trained SSL models. MIM-Refinee effectively combines their respective advantages of MIM and ID without suffering from their respective disad- vantages. Right: Representation quality of SSL methods evaluated via linear probing. MIM-Refiner advances the state-of-the-art on ImageNet- K pre-training in low- and high-compute regimes.
poorly. In contrast, for example, Instance Discrimination (ID) (He et al. 2020: Chen et al. 2020c
methods implicitly focus on objects and form semantic clusters in the latent space (Caron et al. 1021 which eases adaption to downstream tasks the absence of vast amounts of labels. In summary, the most important desiderata for efficient SSL pre-training methods in computer vision are rich representations of the input - ideally in the form of semantic clusters in the latent space alongside efficiency in both compute and data, and, most notably, favorable scaling to larger architectures.
In this work, we show that MIM models have different types of blocks: those that mainly improve the pre-training objective and others that responsible for abstraction within the MIM encoder. The origin of this behavior can be traced back to the fact that MIM architectures usually comprise a large ViT encoder together with a very light- weight decoder. For larger models, the light-weight decoder reaches a point, where it cannot further improve the pre-training objective on its own and passes part of the reconstruction task back to the last encoder blocks. Consequently, the feature quality for downstream tasks of the later blocks degrades, and, somewhat unusual, the representation quality peaks in the middle blocks of the encoder.
Based on these insights, we introduce MIM-Refiner, a simple - yet highly effective sequential refinement approach tailored to MIM models. MIM-Refiner applies an ensemble of ID heads that enforce semantic clusters via an ID objective. Most importantly, those ID heads are attached to intermediate blocks the encoder including those that exhibit peak representation quality, instead of only single ID head attached to the last block.
Experimentally, we show that within few epochs, MIM-Refiner refines the features of MIM model to (i) incorporate the beneficial properties of ID objectives (ii) preserves the advantages of the MIM model (iii) exploits the synergies of both methods to improve upon each individual pre- training objective, advancing the state-of-the-an ahr across various benchmarks, see Figure 2 Extensive evaluations show the potential of MIM- Refiner for training sarge-scale vision foundation models.
Our contributions can be summarized as follows:
1. We show via a detailed analysis that MIM models have different types of blocks: those that mainly improve the pre-training objective and others that are responsible for abstraction.
2. We introduce MIM-Refiner, a sequential approach to refine the representation of a pre- trained MIM model to form semantic clusters via an ID objective. Motivated by the findings in (1), MIM-Refiner is designed to exploit the intermediate representations via an ensemble of ID heads attached to multiple encoder blocks.
3. We experimentally show the effectiveness and generality of MIM-Refinee by refining multitude of MIM models of various scales, which achieve new state-of-the-ee results in a broad range of downstream tasks.
2

---

Published as a conference paper at ICLR 2025
2
BLOCK TYPES IN MASKED IMAGE MODELING
MAE
600
Decoder
70
MAE L/66
VIT-8/16
MAE H/14
VIT-L166
#Parameters (M)
500
VIT-H/14
K-NN Accuracy [%6]
50
400
300
96.0%
99.0%
30
92.1%
98.1%
100
10
76.7%
97.1%
0
MAE
data2vec 2.0
VT Blocks
(a) Encoder-decoder Parameter Count
(b) ImageNet- KK k--N Accuracy Block
80
0.8
60
Relative Improvement
0.6
Accuracy (86)
50
40
0.2
30
0.0
IBOT
MAE MMf-Refined
Reconstruction on Loss
-0.4
K-NN Accuracy
10%
1%
S-shot Labels per
2-shet
1sshot
(c) ImageNet-IK Low-shot Accuracy.
(d) Relative Improvements per Block.
Our analysis reveals different block types in MIM models. (a) MIM models are asymmetri- cally designed where the encoder has most of the parameters, indicated by the percentages in the bars. (b) The representation quality of MAE encoders (measured by A-NN accuracy) peaks in the middle blocks before degrading when the later blocks take over parts of the decoder's 's task, Various MIM models and also semantic segmentation task follow this pattern (see Appendix B.4) (c) The decline in representation quality later blocks primarily contributes to the degradation in downstream performance. ID methods (represented by iBOT) and our MAE-Refined do not suffer from this issue. (d) Correlation of the relative improvement reconstruction loss and k-NN accuracy per block. The relative improvement is the difference between subsequent blocks divided by the maximum difference over all blocks. Figure (b) and Figure in the appendix show the raw k-NN accuracies and reconstruction losses from which the relative improvement obtained. Middle blocks form abstract representations (large improvements in the k--N accuracy, almost no improvement in reconstruction loss), later blocks take over parts of the reconstruction task (decrease the k-NN accuracy, large improvement in the reconstruction loss). Additional details can be found in Appendix D.2
Figure 3:
We start with an analysis, which - to the best of our knowledge - the first that clearly reveals the different block types in MIM models. Motivated by these findings, we introduce targeted refinement process of blocks that harm downstream performance Section
Different blocks due to asymmetric encoder-decoder design. MIM models, such as MAE (He et al. 2022) and data2vec 2.0 (Baevski al. 2023) enable efficient pre-training of large models. In terms of architecture, the encoder and decoder are intentionally designed asymmetrically. The encoder, the one hand. is large ViT that discards 75% of the input patches through masking to drastically reduce training costs. The decoder, on the other hand, operates on the full sequence length by concatenating mask tokens to the encoded visible patches and, thus, is typically very
3

---

Published as a conference paper at ICLR 2025
lightweight compensate for the increased number of tokens (Figure Bay. As models increase in size, the decoder eventually reaches a point where it cannot further improve the pre-training objective on its own. Consequently, it begins to delegate a portion of the reconstruction task back to the last encoder blocks. This transfer adversely affects the feature quality for downstream tasks associated with those blocks (Figure (b)), especially when only few labels are available (Figure 3c). We observe this phenomenon by correlating the relative improvement in the reconstruction loss vs the k-NN accuracy (Figure Roughly speaking, the blocks of the encoder operate in three different regimes:
1. In early VTT blocks, general purpose features are learned, which improve the reconstruction loss and the k-NN accuracy simultaneously.
2. In middle ViT blocks, abstractions are formed. The reconstruction loss improves only slightly, while the k-NN accuracy improves drastically.
3. In late ViT blocks, features are prepared for the reconstruction task. The reconstruction loss improves at faster rate, while the k-NN accuracy decreases.
What to learn from these findings? Naively using the last encoder block features uses features suited for reconstruction, not for downstream tasks. If lots of labels are available, this can be compensated by fine-tuning the last encoder blocks on the downstream task. However, if not enough labels are available, or the last encoder blocks are not fine-tuned, downstream performance suffers.
One would think that simply using a larger decoder solves these problems. However, there multiple problems with this solution. (i) The decoder commonly operates on the full sequence length, making it costly to increase its size. (ii) Scaling the decoder can decrease performance as shown, for example, in MAE (He et al., 2022) (Table 1). (iii) Models that can use a deeper decoder (such as CrossMAE (Fu et al. | 20244) also show degrading representation quality in later blocks (see Appendix Figure 7d). Instead of changing established MIM pre-training procedures, we ask the question: can we leverage and further improve the strong intermediate representations of MIM models?
3 METHOD
We propose MIM-Refiner, a novel approach aimed at improving downstream performance by refining the later blocks of a pre-trained MIM model. MIM-Refiner leverages the abstract intermediate representations with an ensemble of Instance Discrimination (ID) heads, which are attached to multiple blocks towards the end of the encoder, as visualized on the left side of Figure 4 The resulting experimental improvements in various downstream tasks are discussed in Section
Inspired by NN based contrastive learning (Dwibedi et al., 202 Azabou et al. 202I). we propose Nearest Neighbor Alignment (NNA). NN contrastive objectives introduce an inter-sample correlation by retrieving NNs samples batch and subsequently applying an objective between the samples and their NNs. In practice, the NN retrieval is typically done by tracking samples from previous iterations in a fisst-inffirs--o in the queue are from a previous model state which creates tradeoff between the benefit of the NN-swap and the worse signal from the out-of-sync samples. We argue that the NN-swap does not offer a benefit for negative samples, since they are already different images, and instead, degrades the signal from the contrastive objective. We therefore propose to use the NN only for the alignment of positive samples, as visualized on the right side of Figure
queue (Dwibedi et al., [202 He et al., 2020). Therefore, the samples
4 Omitting the NWas for the negatives stabilizes training for large models and slightly improves performance (see Table I.. NNA is variant of NNCLR and we visualize their difference Figure I3
Formally, given a batch of features 2 = {21...., =xx and a queue Q the NNA objective is: exp(NN 24, -0)2/7)
NNA - -log- C
exp(NN(2z, Q) 21/7)+ + Σ exp(SG(zz) 2//7 #5]
NN(2, ② = argmax QEQ
(2)
where 2; is the anchor, NN(2, Q) is the positive, 2, are the negatives, is the temperature, and SG is the stop-gradient operation. | + j] denotes the Iverson bracket that evaluates to if / j and to 0 i j. All vectors are assumed to be normalized length 1.
4

---

Published as a conference paper at ICLR 2025
MIM-Refiner
Block
Block 1
Copy
Block
Block
2
Copy
Block
Block
Block 3
Copy
Block
Block 4
Block 4
Copy
Block
ID Head
Sample in Batch
NN in Queue
Block
Block
Copy
Block
Head
Attracting Force
Repelling Force
ID Head
Decoder
Stop Gradient
Comparison of different pre-training schemes. ID uses single ID head, whereas MIM models use light-weight decoder to train an encoder. MIM-Refiner attaches multiple ID heads to the later third of the blocks of pre-trat are-trained MIM encoder. Right: Nearest Neighbor Aligmment (NNA). An anchor sample is attracted by its NN and simultaneously repelled from other samples in the batch. The NN is retrieved from
Figure 4: Left:
queue of samples from previous batches.
4 EXPERIMENTS
We refine series of MIM models, namely MAE (He al. 2022), data2vec 2.0 (Baevski al. 2023
(abbreviated as D2V2), dBOT (Liu al. 2022 and CrossMAE (Fu et al. 2024). These models were pre- trainddnd on ImageNet- 1K (Deng al. 2009). which we also for the refinement. Models refined for 20/30 epochs using peak learning rate of 4e-4 with layer- wise decay 0.65. ID heads attached after each block in the last third. Further implementation details listed in Appendix C For visual clarity, we compare our best model against previous state-of-the-arr models and show the full result tables in the Appendix. We evaluate models in classification (low- and many-shot), feature evaluation, clustering, transfer learning and semantic segmentation.
4.1 ABLATION STUDY
Table I: Ablation study by refining data2vec 2.0 Baevski et al.
2023 models.
Default settings
(a) Head Count L/16
(b) Head Count H/14
(c) Head Schedule L/16
Heads
k-NN
#Heads
k-NN
Schedule
k-NN
80.2
80.7
Constant
81.0
2
80.2
81.1
Uniform Decay
80.9
4
80.9
4
81.7
Staggered Decay
81.0
8
81.0
82.1
Staggered Step
80.9
12
81.0
12
82.3
One Hot
80.7
(d) NN-swap L/16
NN-swap H/14
(f) Augmentations L/16
Swap Neg.
k-NN
Swap Neg.
k-NN
Color/blur
k-NN
1-shot
X
81.0
82.3
81.0
61.7
80.9
80.7
X
80.5
63.4
(a) Adding multiple heads at intermediate features improves k-NN accuracy by 0.8%. The best settings include the blocks with the best representation quality (see Figure Adding additional heads before the best blocks does not improve performance while increasing training costs. (b) The benefit doubles to 1.6% % for VIT-H since deeper models degrade more (see Figure 3b). Additionally,
5

---

Published as a conference paper at ICLR 2025
investigate where to attach the ID heads in Appendix [8.2 where we find that simply attaching ID head to the last (or 12 for ViT-H) blocks to perform best across model scales.
(c) Scheduling the loss weight of each intermediate head is not necessary, i.c. simply summing all losses is sufficient to achieve good performances. "Uniform Decay" decays the loss weight of all intermediate heads during training. "Staggered Decay" starts by decaying the first head and gradually starts to decay more and more heads as training progresses. "Staggered Step" disables the first head after some time followed by gradually disabling more and more heads. "One Hot" trains the encoder with only one intermediate head at time where training starts with the earliest intermediate head and gradually iterates through the heads. Details to the loss schedules are in Appendix D.14
(d) Using the NN swap only for aligning the positive with the anchor gives small but consistent improvement on smaller models. (e) The improved signal quality is cruicial to avoid training instabilities in larger models. These instabilities manifest in sudden representation quality drop mid-training leading to much worse final model.
(f) Relying only on the data-driven augmentation of the NN-swap by omitting color/blur augmenta- tions, is beneficial for certain downstream tasks such as extreme low-shot classification (Lehner et al. 2024
We show more results without color/blur augmentations in Appendix B.5
4.2
LOW-SHOT AND FEATURE EVALUATIONS
We evaluate the ability of our models to perform low-shot classification Table Additionally, linear probing and k-NN accuracy reported which computed from the features of the frozen encoder. These metrics are typically correlated to low-shot performance as indicates that the representation is already linear separable which cases drawing decision boundaries given only few labels. For linear probing, we use the protocol of DINOv2 (Oquah et al. 2023) which includes using features of the last four blocks in its hyperparameter grid. Therefore, linear probing evaluates the features at the end of the encoder, while k-NN accuracy evaluates only the features of the last block.
MIM-Refiner drastically improves upon MIM models and other SSL models. In the 1-shot tings D2V2-Refined-H sets a new state-of-the-art of 64.2%, outperforming the 63.6% of MAWS- which is pre-trained on private dataset with 2000x the size of ImageNet- IK.
6.5B Singh 2023
Table 2: Low-shot and feature evaluations of recent SSL models ImageNet- IK MIM-Refiner significantly improves upon MIM models and previous state-of-ther of-the--ttt SSL models. Appendix Table [0] extends this comparison to more methods and to models that were trained more data (such as DINOv2) where MIM-Refiner outperforms DINOv2-g in some benchmarks.
Low-shot Evaluation
Feature Eval
VIT
Method
1-shot
2-shot
5-shot
1%
10%
Probe
k--N
MAE
14.3
34.9
56,9
67.7
79.3
77.5
60.6
D2V2
24.1
58.8
72.1
75.1
81.5
78.2
51.8
iBOT
48.5
58.2
66.5
73.3
79.0
81.1
78.0
L/16
Mugs
52.9
62.3
69.4
76.2
80.3
82.1
80.4
MAF-Refined
57.8
66.3
72.0
76.1
81.2
82.8
81.5
D2V2-Refined
61.7
69.6
73.9
78.1
82.1
83.5
81.0
MAE
7.2
14.1
40.2
72.8
81.2
78.2
58.1
D2V2
21.6
60.8
74.2
77.6
83.3
80.4
48,0
H/14
MAE-CT
49.4
59.6
67.4
74.4
81.7
82.3
79.1
MAE-Refined
59.5
68.5
73.8
77.4
82.1
83.7
82.3
D2V2-Refined
64.2
71.3
75.5
78.1
83.5
84.7
82.3
MAE
17.8
29.1
62.9
73.6
82.0
79.7
67.1
2B/14
MAE-Refined
58.2
68.6
74.8
78.7
82.5
84.5
83.2
4.3
CLUSTER EVALUATIONS
We compare the cluster performance 4IM-Refiner against recent SSL models in Table 3 We apply mini-batch k-means (Sculley 2010) 100 times the validation of ImageNet- 1 and select

---

Published as a conference paper at ICLR 2025
Table 3: k-means cluster performance and class separation on ImageNet of recent SSL models.
MIM-Refiner drastically improves performance of MiM-models and even outperforms DINOv2-g which has 2x more parameters and is trained on on 100x more data.
Cluster Performance
Class Separation
VIT
Method
ACC
NMI
AMI
ARI
SIL (t)
DBS (t)
D2V2
10.5
45.1
19.5
2.5
-9.1
6.4
iBOT
52.2
80.5
67.0
33.4
13.3
3.5
L/16
Mugs
54.3
78.6
65.5
22.4
14.9
3.3
D2V2-Refined
67.4
86.3
76.2
40.5
37.1
2.2
D2V2
9.9
45.8
18.0
2.6
-10.8
6.5
H/14
D2V2-Refined
67.3
87.2
77.9
42.2
34.5
2.3
H/16
MAE-CT
58.0
81.8
69.3
36.8
g/14
DINOv2 (LVD-142M)
47.7
76.3
63.6
5.1
30.4
2.8
ACC: 19.3 S1L:
ACC: 70.3 SIL: 32.4
ACC: 64.3 SIL: 10.0
(a) D2V2
(b) D2V2-Refined
(c) Mugs
Figure 5: UMAP (McInnes et al. 2018 plots of ViT-L embeddings using all 53 food related classes of ImageNet. The corresponding k-means cluster accuracy (ACC) and class separation measured in silhouette score (SIL) is shown below each plot. The clustering after refinement (b) is visually more condensed and better separated with corresponding improvements in ACC and SIL than before refinement (a). Mugs (c) does not separate the clusters that well, as shown by the merged clusters in the middle and the lower SIL score. The colors show the 53 ground truth food classes.
the with the lowest k-means loss for comparison (average performance is reported in Appendix T2).e We report commonly used metrics for measuring Cluster Performance w.r.t. the ground truth clustering: Cluster Accuracy (ACC) (Yang
20100, Normalized Mutual Information (NMI) (Kvalseth) 1987), Adjusted Mutual Information (AMI) (Nguyen et al.) 2009), Adjusted Rand Index (ARI) (Hubert & Arabie 1985), where higher values indicate better match the found clustering with the ground truth labels. Further, we measure the Class Separation in the embedded space using the
score (DBS) (Davies Bouldin 1979) and silhouette score (SIL) (Rousseeuw 1987) w.r.f. the ground truth classes. The DBS measures the separation and compactness of classes, where lower values are better and zero is the minimum. The SII ranges from -100 100, where higher values better. Negative SILs relate to mismatches between classes and embedding, where scores close to zero indicate a potential overlap classes.
Table 3 shows that MIM-Refiner greatly improves various clustering metrics. The reached ACC of 67.4% outperforms the current sate-of--the-nn of 61.6% reached by TEMI MSN (Adaloglou et al. 2023). Figure illlustrte this drastic increase in cluster performance and class separation visually. Additionally. we evaluate combining multiple models using the TURTLE (Gadetsky et al. 2024 framework in Appendix Table 13
7

---

Published as a conference paper at ICLR 2025
Table 4: Transferring MIM-Refiner models to other datasets. "VTAB-6" reports the average accuracy over six datasets from the VTAB benchmark (Zhai et al. 2019) and "VTAB-1K" is the average over all 19 datasets the VTAB-IK BK benchmark. ADE20K reports the mean intersection over union (mloU) of semantic segmentation probe. MIM-Refiner learns general features that can easily be transferred to various datasets and tasks. Table confirms this finding on additional models and Appendix B.11 shows individual VTAB performances for VTAB-6 probing and V fine-tuning.
iNat18 Fine-tuning
Linear Probe
VTAB-IK
ViT
Method
1-shot
5-shot
10-shot
iNat18
VTAB-6
ADE20K
Fine-tuning
MAE
7.1
51.6
68.9
42.8
83.0
33.6
73.7
iBOT
15.8
51.5
65.5
56.0
87.6
35.6
70.8
L/16
Mugs
19.5
53.2
66.9
61.5
87.9
34.8
68.0
MAE-Ref.
19.0
58.0
71.7
60.6
88.5
37.3
75.2
MAE
6.5
53.3
71.7
43.0
83.2
35.5
72.7
H/14
MAE-CT
16.5
60.1
74.7
62.8
88.8
37.6
75.7
MAE-Ref.
20.9
62.4
75.4
64.6
89.3
39.4
75.9
MAE
10.0
53.7
72.2
51.0
85.4
37.3
74.1
2B/14
MAE-Ref.
22.5
63.5
76.5
69.6
89.8
40.3
75.6
4.4 TRANSFER LEARNING EVALUATIONS
We investigate generalization of pre-trained MIM-Refiner models to other datasets in Table which shows the benefits of MIM-Refiner models when transferring pre-trained representation. We consider a variety classification downstream tasks and a semantic segmentation task. MAE-Refined consistently improves over MAE and state-of-the- trrt SSL methods. Additionally MIM-Refiner further improves when scaling up to a 2B parameter model.
4.5 FINE-TUNING WITH LARGE AMOUNTS OF LABELS
MIM models typically outperform ID methods when given enough labels to fine-tune the model. As refinement process employs ID objective, investigate whether MIM-Refiner involuntarily degrades performance given an abundance of labels. To this end, we fine-tune MIM models and their
refined version ImageNet-1K (Deng al. 2009), i8 (Horn al. 2018 and ADE20K (Zhou
et al. 2019) using linear classification or UperNet (Xiao et al. 2018 segmentation head. Table shows small but consistent improvement of MIM-Refinee models.
Table 5: Full fine-tuning using 100% of the labels. MIM-Refiner consistently improves performance slightly even though ID methods perform worse than MIM models this benchmark. Appendix Table [8 confirms this pattern on additional MIM models. "Robustness" shows the average perfor- mance of the trained IN-IK classifier robustness datasets (individual results in Table T9). ID and
JEPA (Assran et al. models (shown in gray) are not competitive with MIM(-Refiner models.
2023
VIT-L/16
ViT-H/14
Model
IN-1K
Robustness
iNat18
ADE20K
IN-1K
Robustness
iNat18
D2V2
86.6
60.2
81.0
54.4
86.6
63.2
79.6
D2V2-Ref.
86.7
60.5
81.6
54.4
86.8
64.1
79.8
dBOT
85.8
55.3
81.9
53.1
87.1
63.7
84.1
dBOT-Ref.
85.9
55.3
82.1
53.3
87.1
64.0
84.5
TOT
84.8
47.7
76.9
51.1
Mugs
85.2
46.4
76.9
50.2
I-JPAA
84.9
50.9
75.9
While improvements in fine-tuning with large amounts of labels might marginal when compared to the gains MIM-Refiner on other benchmarks, it is important to consider that these benchmarks extremely competitive and advancements thereon are made in small increments. Gains of one
8

---

Published as a conference paper at ICLR 2025
percentage point larger unrealistic when keeping data and model size constant. Additionally, MIM-Refiner is designed to be compute efficient, requiring only a couple of epochs of training, whereas training from scratch requires multiple hundred epochs. The fact that MIM- MeM--efiner still improves fine-tuning with large amounts of labels slightly shows that our method can efficiently learn strong semantic representations, even improving upon MIM models in their "strong suit" (fine-tuning with large amounts of labels).
4.6 DISCUSSION & FUTURE WORK
We experimentally show that MIM-Refiner efficiently combines the advantages of MIM and ID methods without suffering from their respective disadvantages. When comparing to the current state- of-the-art models in ID (iBOT, Mugs). MIM-Refiner consistently outperforms them across a variety of benchmarks, most of the time by large margins. Additionally, the efficiency of MIM-Refiner allows us to scale up model size beyond the largest ID models, without requiring unreasonable amounts of compute. To put it into perspective, the currently largest ID model trained on ImageNet- 1K are ViT-L models (300M parameters) whereas MIM-Refinee can effortlessly scale up to 2B parameter models. Even DINOv2 Oquab et al. (2023), ) vision foundation model trained on private high-quality dataset with 142M images, trains only IB parameter model. When comparing against the current state-of-the-arr MIM models, MIM Refiner drastically improves performance in few-shot settings while also showing slight improvements in many-shot settings. Overall, we have not found a single setting where refining MIM models would be undesirable.
MIM-Refiner shows strong scaling behavior, however we are only able to show results on ImageNet- 1K due to resource constraints. When scaling up to ImageNet-2IK, we would first need to pre-train large-scale MIM models ourselves, including tuning hyperparameters (learning rate, training duration, ...). This requires a lot of compute, which is within our current compute budget. Note that for ImageNet- K, we simply download the pre-trained checkpoints and only need compute for the refinement process. Additionally, going beyond ImageNet-21IK web-scale datasets is also heavily restricted by the lack of publicly available high-quality datasets for ID training (such as LVD- 142M). However, MIM and ID models have been shown to scale well larger datasets (Singh al. 2023 Oquab et al. 2023). which suggests that also MIM-Refinee would scale well to larger datasets.
5 RELATED WORK
5.1 PRE-TRAINING IN COMPUTER VISION
Following the success of generative pre-training of transformers (Vaswani et al.
2017) in language
modeling (Devlin et al. 2019 Radford et al., 2018), similar directions were explored in computer
vision (Dosovitskiy et al. 2021 Xie et al. 2022] Wei et al. 2022 Chen et al. 2020a). With the
introduction of Vision Transformers (Dooovisskiy et al. 2021 large Masked mmge Modeling (MIM)
models could be efficiently trained (He et al. 2022: Baevski al. 2023 Singh et al. 2023) by
using the ability of transformers to effortlessly process sparse input by dropping masked patch tokens in the input and subsequently reconstructing the masked parts. In order to optimize the MIM pre-training objective, models have to infer the missing regions by efficiently encoding foreground and background alike which leads to a rich representation.
Building on rich features learned by MIM models has been explored in various ways. MAWS (Singh ct al. 2023) first pre-trains an MAE, followed by weakly supervised training on a billion-scale dataset using billion-scale models. Semi ViT (Cai et al. 2022) uses a pre-trained MAE as starting point for semi-supervised learning. Segment Anything (Kirillov et al., (023) use MAE as basis for a segmentation foundation model. MIM-Refiner also builds on the rich features of MIM models and
refines them with ID objective to ease adaption to downstream tasks.
Instance Discrimination (ID) is another branch of self-supervised learning that uses augmentations to create multiple views of the same sample where the task is then to find matching pairs within the views from all samples within a batch (Chen et al. 2020c He et al. 2020: Dosovitskiy et al. 2016, Wu et al. 2018) or align features of one view with the features from another (Grill et al. 2020 Caron a 2021). We use the terminology that views from the same sample are "positive pairs' and views of different samples are "negative pairs". When describing a single view of a sample, it is called the "anchor" to which all other views in batch are either "positives" or "negatives".

---

Published as a conference paper at ICLR 2025
NN-based ID (Dwibedi et al. 2021 Azabou et al. 2021 extends this setting to use NNs in various introduces Nearest
ways to create views or otherwise augment samples during training. Neighbor Alignment, which is a modification of previous NN-based ID methods use the NN only for the alignment part, i.e. pulling the anchor closer to the NN of its positives while pushing the anchor away from its negatives.
5.2 COMBINING MIM AND ID
Adding MIM to ID methods has emerged as powerful pre-training scheme. However, in contrast
to ID models, the MIM objective in the end-to-end training either uses significantly lower masking
ratios with mask tokens getting processed by the encoder (Zhou al. 2021 Oquab et al. 2023), 3) or
requires a target encoder to encode the unmasked image (Huang et al.) 2022b Assran et al. 2022).
Both drastically increase the computational requirements the encoder operates on the full sequence length. Consequently, these models either require copious amounts of compute to train (Oquab) et al. 2023) or limit themselves to relatively small models (Zhou et al. [021] Huang et al.) (022b). Contrary, MIM models have shown success and scalability to large models with comparatively littl compute (He et al., 2022 Baevski al. 2023 (Singh et al. (2023). MIM-Refiner can build these models which allows us to scale to large model sizes without a large amount of compute. Our largest model, HLAE-Refined-22 contains approximately twice the parameters of the currently largest uni-modal contrastive model DINOv2-g (Oquab et al. 2023) and can be trained on two orders of magnitude less data.
Attempts to preserve the efficiency while training MIM and ID objectives end-to-end have been less successful (Lehner et al. 2024 Jiang et al., (023), where both works came to the conclusion that sequential training (MIM + ID) circumvents various problem with end-to-end training. First, a powerful encoder trained solely with a MIM objective. Second, the encoder is trained with ID objective while preserving the rich features in early blocks with a layer- fayer-wise learning rate decay (Clark ct al, 2020) in lower blocks and either constraining changes in early blocks (Jiang et al. 2023
or completely freezing them (Lehner et al. 2024).
MIM-Refiner is also a sequential MIM - ID method. In contrast to our work, previous works start from the representation after the last MIM encoder block and are therefore highly reliant good representation thereof. This can be seen for example on MAE-CT (Lehner et al. 2024) where their
ViT-H/14 model is worse than their ViT-H/16 model, despite using 30% more FLOPS. Additionally,
previous works (Lehner et al. 2024 Jiang et al. (023) omit current go-to techniques such as multi-
crop augmentation (Caron et al.) (0200) which has been shown to improve performance (Caron et al.)
2021 Zhou et al. 2021 2022).
Training models with additional losses from intermediate layers dates back the early deep learning days (Lee et al.) [015] Szegedy et al.) (2015) where these auxiliary losses were used to alleviate optimization issues in lower layers. MIM- relates deep supervision in the sense that we use multiple ID) heads attached at intermediate layers where each head produces a loss for training.
6 CONCLUSION
We introduce MIM-Refiner, a procedure to refine pre-trained MIM models. Motivated by the insights that the representation quality MIM models peaks in the middle of the encoder, we employ ensemble of instance discrimination heads attached multiple blocks towards the end of the encoder, including the blocks where representation quality peaks, to improve upon the best existing representation. We train this ensemble for short duration to improve the representation for downstream tasks such as classification, clustering or semantic segmentation.
Our refined MIM models learn strong features from ImageNet- IK alone that can be readily used for downstream tasks without fine-tuning the model but also improve performance when the model is fine-tuned, particularly in few-shot settings. Our models outperform competitors that were also trained on ImageNet- I and sometimes also ones that were trained on more data or use larger models.
Extensive evaluations show the potential of MIM- for training large-scale vision foundation models that provide general-purposs off-the-shelf features for broad range of downstream tasks.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
The full codebase used for all experiments in this paper together with the exact hyperparameter configurations that were used for each experiment and pre-trained models can be found in github repository: https///ittub..oom/m//k////allia Refinerr
We provide the hyperparameters for training and evaluation pipelines in Appendix Section D together with additional implementation details in Appendix Section
ACKNOWLEDGEMENTS
We thank Johannes Lehner for helpful discussions and suggestions.
We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at IT4Innovations, Czech Republic, MeluXina at LuxProvide, Luxembourg, LUMI at CSC, Finland and Leonardo at CINECA, Italy. We acknowledge access to LEONARDO at CINECA, Italy, via an AURELEO (Austrian Users at LEONARDO supercomputer) project.
The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064) PRIMAL (FFG-873979), S3AI (FFG-872172), EPILEPSIA (FFG- 892171), AIRI FG 9-N (FWF-36284, FWF-36235), ALsGrrenHcatingGrids (FFG- 899943). IN- TEGRATE (FFG-892418), ELISE (H2020-ICT-201993 ID:: 951847), Stars4Waters (HORIZON- CL6-2021 L.6-2021-CLIMANN -CLINATT-LMT--0--0 We thank AKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
ETHICS STATEMENT
Our work proposes training methodology that can readily reuse existing models, which makes use of the expended energy for large-scale pre- pre-training of MIM models instead of training models from scratch, which would need much more energy than refining existing models. Additionally, our models learn strong general-purpose features that be readily used for broad range of downstream tasks without needing to fine-tune or retrain a model. This further reduces the energy consumption and carbon emissions required for training state-of-the-ar models in computer vision tasks.
The strong ceneral-purposs features of our models provide a fundamental advancement to computer vision, inheriting their potential benefits. On the one hand, our models could be used for environmental monitoring of endangered species where little data is available. On the other hand, the strong few-shot performance of our models could lead to overconfidence in rare discase classifiers where sufficient data is indispensable and checking with healthcare professional is necessary to avoid misdiagnoses.
REFERENCES
Nikolas Adaloglou, Felix Michels, Hamza Kalisch, and Markus Kollmann. Exploring the limits of deep image clustering using pretrained models. In BMVC, pp. 297-299. BMVA Press, 2023.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for lfficeefficient learning. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXI, volume 13691 of Lecture Notes in Computer Science, pp. 456-773. Springer, 2022.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael G. Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 15619-15629, IEFE, 2023.

---

Published as a conference paper at ICLR 2025
Mchdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C. Johnson, Kiran Bhaskaran-Nair, Max Dabagia, Keith B. Hengen, William R. Gray Roncal, Michal Valko, and Eva L. Dyer. Mine your view: Self-supervised learning through across-sample prediction. CoRR, abs/2102. 101000001,,. 10106, 2021.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cottrell, and Julian J. McAuley. Rezero is all you need: fast convergence at large depth. In Cassio P. de Campos, Marloes H. Maathuis, and Erik Quaeghebeur (eds.), Proceedings of the Thirty-Sevents Conference Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021, volume 161 of Proceedings of Machine Learning Research, pp. 1352-1361. AUAI Press, 2021.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for supervvue learning in specch, vision and language. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 1298-1318 PMLR, 2022.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 Proceedings of Machine Learning Research, pp. 1416-1429. PMLR, 2023.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In The Tenth International Conference Learning Representations, ICLR 2022, Viriual Event, April 25-29, 2022. OpenReview.net, 2022.
Zhaowei Cai,, Avinash Ravichandran, Paolo Favaro, Manchen Wang. Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurIPS, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and Hsuan- -ieun Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Monireal, QC, Canada, October 10-17 17, 2021, pp. 9630-963444 IEFF, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 2020a.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020c.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. Int. J. Comput. Vis., 132(1):00--223 2024.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9620-9629. IEEF, 2021.
12

---

Published as a conference paper at ICLR 2025
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014. Columbus, OH. USA, June 23-28, 2014, pp. 3606-3613. IEEE Computer Society, 2014.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders discriminators rather than generators. In 8th International Conference Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
David L. Davies and Donald W. Bouldin. A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell., 1(2):224-227, 1979.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248 248-255. IEEE Computer Society, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (I), pp. 4171-4186. Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans. Puttern Anal. Mach. Intell., 38(9):1734-1744 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9568-9577. IEEE, 2021.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jégou, and Edouard Grave. Are lcage-scale datasets necessary for self-supervised pre-training? CORR, abs/21 12.14400, 2021.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transac- tions on pattern analysis and machine intelligence, 28(4):594-611 2006.
Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A. Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. CoRR, 0bo/2200.114311, 2024.
Artyom Gadetsky, Yulun Jiang, and Maria Brbic. Let go of your labels with unsupervised transfer. arXiv preprint arXiv:2406.022. 2024.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee -ienn Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pp- 2980-2988 IEEE Computer Society, 2017.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9726-9735. Computer Vision Foundation/ IEEE, 2020.
13

---

Published as a conference paper at ICLR 2025
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, pp. 19979-15988, 2022.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR (Poster). OpenReview.net, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1oXYv:1606.08415.. 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of distri-iittiittiinn generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8769-8778. Computer Vision Foundation / IEEE Computer Society, 2018.
Po-Ya Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurlPS, 2022a.
Zhicheng Huang. Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autocncoders are stronger vision learners. CoRR, abs/2207. 13532,, 2022b.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218, 1985.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org. 2015.
Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nett 2023.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, pp. 3992-4003. IEEE, 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Tarald O Kvalseth. Entropy and correlation: Some comments. IEEE Transactions on Systems, Man, and Cybernetics, 17(3)517-559, 1987.
Chen-Yu Yu Lee, Saining Xie, Patrick W. Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2015, San Diego, California. USA, May 9-12, 2015, volume 38 JMLR Workshop and Conference Proceedings. J..g. 2015.
14

---

Published as a conference paper at ICLR 2025
Johannes Lehner, Benedikt Alkin, Andreas Fürst, Elisabeth Rumetshofer, Lukas Miklautz, and Sepp Hochreiter. Contrastive tuning: A little help to make masked autoencoders forget. In AAAI Conference on Artificial Intelligence, AAAI 2024, February 20-27, 2024, Vancouver, Canada, AAAI Press, 2024.
pp. 200229965299
Collin Leiber, Lukas Miklautz, Claudia Plant, and Christian Böhm. Benchmarking deep clustering algorithms with clustpy. In ICDM (Workshops), pp.
IEEE, 2023.
Yanghao Li, Hanzi Mao, Ross B. Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV (9), volume 13669 Lecture Notes in Computer Science, pp. 280-296 Springer, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. ECCV volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer. 2014.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engi- neering, 35(1):857-876, 2021.
Xingbin Liu, Jinghao Zhou, Tao Kong. Xianming Lin, and Rongrong Ji. Exploring target representa- tions for masked autoencoders. CoRR, abs/2209.05 009917, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1166l 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Julien Mairal. Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more. CoRR, abs/1912.08 165, 2019.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018
Xuan Vinh Nguyen. Julien Epps, and James Bailey. Information thcoretic measures for clusterings comparison: is a correction for chance necessary? In JCML, volume 382 of ACM International Conference Proceeding Series, pp. 1073-1080. ACM, 2009.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722 729. IEEE, 2008
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran. Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. CoRR, abs/2304.07193, 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 3498-3505. IEEE Computer Society, 2012.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2536-2544. IEEE Computer Society, 2016
15

---

Published as a conference paper at ICLR 2025
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Coumapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in python. J. Mach. Learn. Res., 12:2825-2830 2011.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1015-1018. ACM Press. ISBN 978- 14503334594
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre- mre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021.
Peter Rousseeuw. Silhouettes: graphical aid the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
D. Sculley. Web-scale k-means clustering. In WWW. pp- 1177-1178. ACM, 2010.
Mannat Singh, Laura Gustafson, Aaron Adeock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross B. Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 794- 804. IEEE, 2022.
Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross B. Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of MAE pre-pretraining for billion-scale pretraining. CoRR, 0et/22333.1446. 2023.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9. IEEE Computer Society, 2015.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data- efficient learners for self-supervised video pre-training In NeurlPS, 2022.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 32 42. IEEE, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie. Yoshua Bengio, and Pierre-Antoina Manzagol. Stacked denoising autoencoders: Learning useful representations deep network with local denoising criterion. J. Mach. Learn. Res., 11 11:3371 17771-3340 1-3408, 2010.
Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching matters: Investi- gating the role of supervision in vision transformers. In CVPR, pp. 7486-7496. IEFF, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In ICLR (Poster). OpenReview.net, 2019a.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019b.
16

---

Published as a conference paper at ICLR 2025
Luya Wang, Feng Liang, Yangguang Li, Honggang Zhang, Wanli Ouyang, and Jing Shao. Repre: Improving supervieee vision transformer with reconstructive pre-training. In Luc De Raedt (ed.), Proceedings of the Thirty-Firsh International Joint Conference Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 1437-1443. ijeai.org, 2022.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. CoRR, abs/1804.0320%% 2018.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan L. Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, pp. 14648-14658. IEEE, 2022.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, pp- 1112-1122. Association for Computational Linguistics, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-par non-parametric instance discrimination. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3733-3742. Computer Vision Foundation IEEE Computer Society, 2018.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp. 3485-3492. IEEE Computer Society, 2010.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V. volume 1 11209 of Lecture Notes in Computer Science, pp. 432 4322444. Springer, 2018
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In JEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 9643-9653. IEBE, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Yixuan Wei, Qi Dai, and Han Hu. On data scaling in masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 00365-10374, 2023.
Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. Image clustering using local discriminant models and global integration. IEEE Transactions on Image Processing, 19(10): 2761-2773, 2010.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping WU, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 111.:147000000 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3): 302-321, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan L. Yuille, and Tao Kong. ibot: Image BERT pre-training with online tokenizer. CoRR, abs/21 11.072221. 2021.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi-granular self superviee learning framework. CoRR, abs/2203. 005/2203.14415..1 2022.
17

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
One limitation of MIM-Refiner is that it requires batch normalization (loffe & Szegedy 2015) layers the ID head. Without them, performance decreases significantly. Similar observations have been made in other contrastive learning literature Chen et al. (202Tp. The batch normalization layers significantly decrease scalability distributed hardware setups as each layer requires
synchronization of batch statistics Chen et al. (2020c).
MIM-Refiner addresses a common issue with MIM models: their typically lightweight decoder often delegates part of the reconstruction to the encoder, resulting in subpar representations for the later encoder blocks in downstream tasks. Alternatively, one could simply argue for larger decoder. However, a larger decoder increases computational costs since the decoder typically operates on the full sequence length. Additionally, the direction of a larger decoder was explored to certain extent in the original MAE paper (He et al. 2022), where larger decoders performed worse in fine-tuning and linear probing. Successor models such as CrossMAE perform well with a deeper decoder but still show decreasing representation quality in later layers, shown in Appendix B.4
During early development, we tried various ways to propagate the intermediate representation towards the end. While we found that simple ensemble of contrastive heads attached to later blocks works very well, there might be even better ways to leverage the rich intermediate MIM representations. We explored the following variants in with little success: (i) completely deleting the last few blocks (ii) reinitializing the last few blocks while setting the weights/biases of the last projection in the attention/MLP to 0 which leads to the result of the previous block being propagated to the end via the residual connection (iii) gating the last few blocks via ReZero (Bachlechner et al. (021).
To address the limitation of MIM-Refiner requiring batch normalization layers. we explore another variant that copies the weights of the intermediate block with the highest k-NN accuracy to all subsequent blocks and sets the weights/biases of the last projection in the copied attention/ML.P blocks to 0. This is similar to the above mentioned approach (ii), except that the weights of the copied blocks are not random but copied from an intermediate block. We then train the model with only a single head attached at the last block. This drastically reduces the number of batch normalization layers. 6 shows that such an approach can yield competitive performances on smaller models, but is outperformed by
larger ones.
Table 6: Copying the peak-represeentation block to subsequent blocks while setting the last atten- tion/MLP projection weights/biases to 0 can improve scalability to even larger distributed setups due to requiring less batch normalization layers. This approach ("Copy Blocks") is competitive to ensemble of ID heads ("MIM-Refiner") on smaller models but is worse on larger models.
k-NN
MAE L/16
MAE H/14
Copy Blocks MIM-Refiner 81.5
81.5
81.8
82.3
Another approach to improve scalability to larger distributed systems is to only aggregate batch statisties within a node. This avoids costly inter-node communication and instead only requires intra-node connections which is typically much faster.
As MIM-Refiner is quite computationally efficient even with the batchnorm synchronization limitation, we do not explore these approaches further.
18

---

Published as a conference paper at ICLR 2025
B EXTENDED RESULTS
B.1 EXTENDED REPRESENTATION QUALITY COMPARISON
We compare linear probing, I-shot classification performance and runtime against various image models (including non-public ones) in Figure 6
Imageket IK Lnzzrr Probing D2V2-Raflned
ImagehetXK 1 shot Casafiication
D2V2-Refiied H/2.4 64.2%
m
64.7%
28/24
MAS-Refined
DINOV2 9/14
MSN 1,7
SWAGG
MAE-CT
Accunscy
Nugs 1/B8
IBOT
D2V2 piva
OpenCLIP
FLOPS
G/M
MAE
MoCo v3
1/36
B/8
H//4
U/44
1/7
1000
2000
1000
4000
Pre-training
Pro Dataset
Figure 6: Left: Representation quality of SSL methods evaluated via linear probing. MIM-Refiner advances state-of-the art among models pre-trained on ImageNet-IK K in low- and high-compute regimes. Right: MeMM-Reinnr advances state-of-thh-aan of- in the extreme setting of 1-shot classification despite being trained on orders of magnitude less data. Size of dots corresponds to model FLOPS.
B.2 WHERE TO ATTACH ID HEADS?
Table 7 abbates different choices of where to attach ID heads on a D2V2 pre-trained ViT-L/16 and ViT-H/14. On ViT-L/16, there is almost no difference of where to attach the IDD heads in the last third of blocks. We tried to transfer this insight to ViT-H/14 14 where the default setting of attaching ID heads to all later blocks performs better. We therefore use the default setting of attaching ID heads the last 8 blocks (ViT-L. and ViT-2B) or the last 12 blocks (ViT-H).
Table 7: Spacing heads out more across the later ViT blocks can achieve comparable performances for ViT-L/16 but does not generalize VIT-H/14. The default setting of attaching ID heads to all later blocks generalizes well across model scales.
Block Indices
k-NN
20,24
80.9
Block Indices
k-NN
16,20,24
81.1
22,24,26,28,30,37
82.0
15, 15, 15,1,22
81.1
16,18,20,22222222,2.....22
82.1
18,20,22,24
81.0
20-32
82.3
17-24
81.0
(b) VIT-H/14
(a) VIT-L/16
B.3 FREEZING EARLY BLOCKS
Freezing early blocks as form of regularization to preserve MIM features (similar to related works (Lehner et al. 2024 Jiang et al. (023)) is not necessary, as shown in Table g Note that we still use a layer-wise learning rate decay (Clark et al. 2020).
Table 8:
Freezing early blocks is not necessary and slightly decreases performance. Ablation conducted with D2V2-L/16. We freeze the first 6 layers to refine MAE-2B to save memory/compute.
#Frozen k-NN
0
81.0
80.9
12
80.6
19

---

Published as a conference paper at ICLR 2025
B.4 INTERMEDIATE REPRESENTATION ANALYSIS OF ADDITIONAL MIM METHODS
We visualize the N-NN accuracies of various MIM models in Figure 7 where all of them show the pattern that the representation quality of larger models degrades towards the end of the encoder.
70
MAE L/16
35
MAE L/16
MAE H/14
MAE H/14
KAN Accuracy [%6]
50
mon
20
30
10
10
5
VTT Blocks
VTT Blocks
(a) MAE (He et al.
2022 k-NN classification
(b) MAE He et al.
2022 segmentation probe
D2V2 8/16
CrossMAE 5/16
60
D2V2 H/14
CrossMAE B/16
K-NN Accuracy [16]
60
KANN Accuracy [96
40
40
20
20
VT Blocks
(c) data2vec 2.0 Baevski et al. 2023
K-NN classifica-
tion
(d) CrossMAE et al.
2024 K-NN classification
70
dBOT L/16
70
CAE B/16
#BOT H/14
K-NNN Accurscy [96]
R-NN Accurscy [96]
5.0
50
30
30
10
10
VIT Blocks
VT Blocks
(e) dBOT
Liu et al. 2022 k- k-NN classification
(f) CAE (Chen et al. 2024 k-NN classification
Figure 7: Intermediate representation analysis of various MIM models. As models get bigger, the k-NN accuracy degrades more towards the end of the encoder, especially for L/16 and H/14 models. This pattern is consistent over various MIM models and across tasks. k--N accuracy is calculated on ImageNet-1K (Deng et al. 2009). (b) shows the mloU of linear segmentation probes on ADE20K Zhou al. 2019
20

---

Published as a conference paper at ICLR 2025
B.5 LOW-SHOT CLASSIFICATION WITHOUT COLOR AUGMENTATIONS
MAE-CT (Lehner et al. 2024) showed that omitting color augmentations can lead to performance gains for low-shot classification, especially larger models. We therefore train ViT-H/14 models also without color augmentations. We use the same hyperparameters as with color augmentations (see Table 24) except that we disable color augmentations (i.e. we train with only crop flip augmentations) and half the training duration. confirrs the findings of (Lehner et al. 2024) as omitting color augmentations also improves low-shot performance of MIM-Refiner
Table ImageNet- low-shot and feature evaluations of D2V2-Refined-E H//1 with and without color augmentations. Omitting color augmentations improves ImageNet-IK low-shot performance.
Low-shot Evaluation
Feature Eval
Model
Color/blur
I-shot
2-shot
5-shot
1%
10%
Probe
k-NN
X
64.7
72.0
75.9
79.1
83.5
84.1
82.1
D2V2-Refined
64.2
71.3
75.5
78.1
83.5
84.7
82.3
B.6 EXTENDED IMAGENET-IK 1K LOW-SHOT AND FEATURE EVALUATIONS
Table 10: Extending Table 2 with additional MIM-Refiner models and more SSL models. The last row-group compares the best MIM-Refiner model to models with longer sequence length (MSN- L/7) and models that were pre-trained on more data. Parentheses show pre-training dataset size. MIM-Refiner consistently improves MIM models, outperforms other SSL models with the same pre-training data and even outperforms DINOv2-gi g/1 some settings.
Low-shot Evaluation
Feature Eval
VIT
Method
1-shot
2-shot
5-shot
1%
10%
Probe
k-NN
CrossMAE
16.8
34.0
52.4
63.2
77.7
74.4
53.4
MAE
14.3
34.9
56.9
67.7
79.3
77.5
60.6
dBOT
28.0
46.9
62.4
70.0
80.2
77.8
61.3
D2V2
24.1
58.8
72.1
75.1
81.5
78.2
51.8
CAE
28.1
56.5
68.4
71.4
79.5
80.0
66.9
L/16
iBOT
48.5
58.2
66.5
73.3
79.0
81.1
78.0
Mugs
52.9
62.3
69.4
76.2
80.3
82.1
80.4
CrossMAE-Reffinn
50.3
60.9
68.2
71.7
79.3
81.8
79.9
MAE-Refined
57.8
66.3
72.0
76.1
81.2
82.8
81.5
dBOT-Refined
57.4
66.0
71.7
76.6
81.6
83.3
81.3
D2V2-Refined
61.7
69.6
73.9
78.1
82.1
83.5
81.0
MAE
7.2
14.1
40.2
72.8
81.2
78.2
58.1
dBOT
23.9
45.0
63.0
73.0
82.1
79.0
60.0
D2V2
21.6
60.8
74.2
77.6
83.3
80.4
48.0
I-JEPA
35.1
47.9
59.9
73.3
79.5
79.3
71.6
H/14
MAE-CT
49.4
59.6
67.4
74.4
81.7
82.3
79.1
MAE-Refined
59.5
68.5
73.8
77.4
82.1
83.7
82.3
dBOT-Refinel
59.2
67.6
72.9
77.3
82.5
84.0
82.0
D2V2-Refined
64.2
71.3
75.5
78.1
83.5
84.7
82.3
MAE
17.8
29.1
62.9
73.6
82.0
79.7
67.1
2B/14
MAE-Refine
58.2
68.6
74.8
78.7
82.5
84.5
83.2
L/16
iBOT (14M)
37.4
49.9
61.9
70.9
80.3
82.7
72.9
L/7
MSN (1.3M)
57.1
66.4
72.1
75.1
80.7
H/14
D2V2-Refined (1.3M)
64.2
71.3
75.5
78. 1
83.5
84.7
82.3
g/14
DINOv2 (142M)
60.5
68.3
74.4
79.1
83.8
86.5
83.5
21

---

Published as a conference paper at ICLR 2025
B.7 EXTENDED IMAGENET-IE IK CLUSTER EVALUATIONS
Table 11 extends the main paper results (Table 3 with additional models. Table 12 shows the average clustering results of the 100 mini-batch k-means as described in Section 4.3 We that MIM-Refiner has also better average performance than competitor methods.
Best k-means cluster performance and class separation on ImageNet of recent SSL models. This table extends Table 3 from the main paper with additional comparison and refined models.
Table 11:
Cluster Performance
Class Separation
VIT
Method
ACC
NMI
AMI
ARI
SIL (f)
DBS (4)
MAE
18.5
55.2
29.1
6.9
-5.9
5.0
D2V2
10.5
45.1
19.5
2.5
-9.1
6.4
iBOT
52.2
80.5
67.0
33.4
13.3
3.5
L/16
Mugs
54.3
78.6
65.5
22.4
14.9
3.3
MAF-Refined
61.8
84.0
72.6
40.7
21.4
2.9
D2V2-Refined
67.4
86.3
76.2
40.5
37.1
2.2
MAE
14.3
50.2
24.2
4.3
-7.8
5.2
D2V2
9.9
45.8
18.0
2.6
-10.8
6.5
H/14
MAE-Refined
64.6
85.3
74.6
45.5
21.0
2.9
D2V2-Refined
67.3
87.2
77.9
42.2
34.5
MAE
19.9
54.1
33.1
6.2
-3.6
4.8
2B/14
MAF-Refined
63.0
85.0
74.4
44.0
14.0
3.2
g/14
DINOv2
47.7
76.3
63.6
5.1
30.4
2.8
Table 12: Average k-means cluster performance ImageNet of recent SSL models. MIM-Refiner drastically improves performance of unrefined models and outperforms competitors.
Cluster Performance
VIT
Method
ACC
NMI
AMI
ARI
MAE
17.9
54.5
28.9
6.5
D2V2
10.2
44.5
19.3
2.3
iBOT
50.5
80.0
66.6
31.6
L/16
Mugs
50.7
77.4
64.4
18.1
MAE-Refined
60.6
83.5
71.9
35.0
D2V2-Refined
60.6
83.9
72.9
30.0
MAE
13.8
49.8
24.4
4.2
D2V2
9.7
45.2
18.1
2.5
H/14
MAE-Refined
63.2
84.7
74.0
40.1
D2V2-Refined
60.4
84.5
74.3
28.4
MAE
19.2
53.5
32.9
5.7
2B/14
EAE-Refined
59.4
84.0
73.4
38.0
g/4
DINOv2
46.8
75.6
62.7
3.5
22

---

Published as a conference paper at ICLR 2025
Figure shows additional analysis the cluster structure in each of the blocks for the refined and unrefined MAE and D2V2 models with ViT-H/14 14 on ImageNet-IK, Corresponding to the k-NN that MAE and D2V2 have higher cluster accuracy (ACC) in
accuracy analysis in Figure the intermediate blocks than in the last block. MIM-Refiner turns this behaviour around and causes steep increase of ACC starting from the intermediate block and continuing almost 70% in the last layer. The silhouette score (SIL) confirms this as well. The early blocks allow no separation of the ground truth ImageNet-IK, classes leading negative SIL values, whereas later blocks of the refined models increase separation by 30-50% compared to unrefined counterparts. Interestingly, MAE-Refined is not plateauing in SIL and ACC compared to D2V2-Refinee pointing to potential room for improvement in MAE refinement by using additional ID heads earlier layers. The bottom part Figure measures the pairwise cluster label similarity between subsequent blocks in terms of normalized mutual information (NMI) (K 1987) as (NMT(y: M6+1)) 100, where are the k-means cluster labels at block The low cluster label similarity in the early blocks for models indicates that the found cluster labels focus on different clusterings. The later blocks of HIM--efined models have high alignment with similarities of more than 90%. The higher ACC w.r.t the ground truth ImageNet-IK classes in the later blocks indicates that they focus This is in contrast to the overall lower cluster label similarity in unrefined models, which focus on different cluster structures in each block.
object-centric features.
MAE
D2V2
D2V2
MAE-Refined
D2V2 Refined
DZV2-Refined
Cluster Accuracy (86)
Silhouette Score
10
40
30
-10
20
-20
10
-30
0
40
MAE
MAE-Refined
D2V2
Cluster Label Simitarity (%)
D2V2 Refined
80
70
60
VIT Blocks
Figure 8: Block-witd wise cluster analysis for refined and unrefined MAE and D2V2 (H/14). Upper Left: Cluster accuracy w.r.t. lowest k-means loss per block. Upper Right: Silhouette score w.r.t. ground truth ImageNet- classes. Bottom: Pairwise cluster label similarity between subsequent blocks (details in Section B.7.
23

---

Published as a conference paper at ICLR 2025
B.8 MULTI- CLUSTERING EVALUATIONS
We evaluate unsupervised cluster accuracy of MIM-Refiner in combination with other foundation models using the TURTLE (Gadetsky al. 2024) framework. The results in Table 13 show that MIM-Refiner learns features that are complementary to features learned from foundation models. Combining multiple MIM-Refiner models boosts unsupervised classification accuracy of individual models surpassing the best k-means accuracy of D2V2-Reff-H
(67.3%). TURTLE with the feature spaces of MAE-Ref., dBOT-Ref. and D2V2-Ref improves the state-of-the-art of unsupervised classi- fication accuracy using only ImageNet- K for pre-training to 71.6%. When additionally including foundation models that were pre-trained on web-scale data, MIM-Refiner consistently improves performance. MAE-Refined in combination with DINOv2 (Oquab al. (023). CLIP (Radford et al. 2021) and SWAG (Singh et al. 2022) sets a new state-of-the- of 76.8%.
We conduct this study by implementing MIM- MIM-Refiner models into the official implementation of TURTLE and train with the recommended default settings. We do not tune any hyperparameters.
Table 13: Unsupervised classification evaluation using the TURTLE (Gadetsky et al. 2024) frame- work. MIM-Refiner models synergize well with foundation models such DINOv2 (Oquab et al.. 2023), CLIP (Radford et al., (2021) and SWAG (Singh et al. 2022). MIM-Refiner in combination with TURTLE (Gadetsky al. (0244) sets new state-of-the-art
ImageNet-1K unsupervised classification without additional data (7 1.6%) and with additional data (76.8%). Model sizes are H/14 for MIM-Refinee and SWAG (Singh et 2022). g/14 for DINOv2 (Oquab et al. 2023) and L/14 for
CLIP (Radford et al. 2021).
NetgeNet-11 1K pre-training
Web-scale pre-training
MAE-Ref. dBOT-Ref.
D2V2-Ref.
DINOv2
CLIP
SWAG
ACC
MIM- LLM-Refiner only
X
X
X
61.7
X
X
X
X
62.2
X
X
X
70.4
X
X
X
71.6
MIM-Refiner + DINOv2
X
X
X
68.5
X
X
X
58.3
X
X
74.0
X
X
73.7
MIM Refiner + DINOv2 + CLIP
X
X
72.9
X
69.4
75.0
X
75.0
MIM MIM-Refiner DINOv2 CLIP + SWAG
74.8
/
X
74.9
/
V
76.4
X
/
V
76.8
24

---

Published as a conference paper at ICLR 2025
B.9 COCO OBJECT DETECTION AND INSTANCE SEGMENTATION
We fine-tune MIM models and their refined versions on COCO Lin et al. 2014) using the Mask R-
CNN (He al. 2017) configuration of the ViTDet (Li et 2022 framework from detect ror 2
We train for 100 epochs using a batch size of 64, peak learning rate of le-4 with the AdamW (Kingma Ba 2015 Loshchilov & Hutter 2019) optimizer and multi-step schedule. The results Table 14 further underline that refined models are competitive with unrefined models on dense downstream tasks. Note that MIM models are extremely good this type of downstream task. For example. training pre-trained MAE-H further via weakly weakly-supervised training on web-scale dataset of billion images even degraded performance vs a plain ImageNet- pre-trained MAE by 1.0 A A and 1.3 Apmusk (Singh et al. 2023).
MAE
dBOT
D2V2
Model
Apbox
A Apmask
Apbox
Apmask
Apbox
Apmask
MIM
55.2
49.1
54.7
48.7
56.0
49.5
MIM-Refiner
54.9
48.8
54.6
48.6
55.5
49.3
Table 14: Fine-tuning results for L/16 models on COCO object detection and instance segmentation.
B.10
EXTENDED TRANSFER LEARNING RESULTS
Table 15 extends Table with additional results and comparison to more SSL models.
Table 15: Extending Table with additional MIM-Refiner models and additional SSL models.
MIM-Refiner learns general features that can easily be transferred to various datasets and tasks.
INat18 Fine-tuning
Linear Probe
VIT
Method
1-shot
5-shot
10-shot
iNat18
VTAB-6
ADE20K
CrossMAE
5.4
45.6
63.7
39.8
81.2
30.9
MAE
7.1
51.6
68.9
42.8
83.0
33.6
dBOT
7.1
53.1
71.2
44.6
83.4
34.1
D2V2
5.5
53.1
70.6
39.4
81.5
38.8
CAE
7.4
54.5
71.2
46.4
84.8
36.5
L/16
iBOT
15.8
51.5
65.5
56.0
87.6
35.6
Mugs
19.5
53.2
66.9
61.5
87.9
34.8
CrossMAE- Ref
18.0
52.9
67.8
60.5
88.3
34.3
MAE-Ref
19.0
58.0
71.7
60.6
88.5
37.3
dBOT-Ref
18.3
58.8
73.0
61.7
88.5
38.4
D2f2-Refined
15.2
56.3
71.8
52.0
85.9
41.0
MAE
6.5
53.3
71.7
43.0
83,2
35.5
dВОТ
6.8
53.1
73.2
46.2
84.6
36.1
D2V2
5.9
55.7
73.4
41.7
83.1
42.4
H/14
MAE-CT
16.5
60.1
74.7
62.8
88.8
37.6
MAE-Ref
20.9
62.4
75.4
64.6
89.3
39.4
dBOT-Ref
20.0
60.9
75.8
65.8
89.5
37.6
D2V2-Refined
16.1
$9.2
74.8
$4.4
87.1
43.7
MAE
10.0
53.7
72.2
51.0
85.4
37.3
2B/14
MAE-Ref
22.5
63.5
76.5
69.6
89.8
40.3
L/7
MSN
17.0
38.0
48.1
25

---

Published as a conference paper at ICLR 2025
B.11 VTAB INDIVIDUAL DATASET RESULTS
We show the individual accuracies for each VTAB (Zhai et al. main paper. Table shoss linear probing results on VTAB and Table 17 shows fine-tuning results on V. We use only six datasets for linear probing, as the other datasets are quite different to the ImageNet- IK images seen during training and therefore benefit heavily from fine-tuning which makes them more suited for evaluation via fine-tuning instead of linear probing.
2019) dataset from Table of the
Table 16: Linear probing accuracy six VTAB (Zhai et al. 2019 datasets from the "Natural" category: Caltech101 (Fei-Fei al.] 2006), CIFARTO0 (Krizhevsky 2009). DTD (Cimpoi 2014), Flowers (Nilsback & Zisserman 2008). Pets (Parkhi et al. 2012) and Sun397 (Xiao et al. 2010
VTAB Dataset
VIT
Method
CF100
CT101
DTD
FL102
Pets
Sun397
Average
CrossMAE
81.4
88.8
75.6
84.2
84.1
72.7
81.2
MAE
80.0
91.8
75.6
86.3
89.6
74.7
83.0
dBOT
84.3
91.3
75.7
88.0
86.0
75.1
83.4
D2V2
85.0
89.8
74.0
84.9
80.7
74.6
81.5
CAE
87.0
92.3
76.1
88.4
89.2
75.9
84.8
L/16
iBOT
89.3
91.3
78.1
95.8
93.7
77.1
87.6
Mugs
89.5
90.5
78.0
96.8
95.3
77.2
87.9
Cro-sMAE-Ref
88.7
93.5
79.0
95.7
95.2
78.1
88.3
MAE-Ref
89.1
91.9
79.0
96.2
95.8
78.8
88.5
JBOT-Ref
90.4
91.3
78.6
95.9
95.5
79.2
88.5
D2V2-Ref
88.9
88.8
73.9
92.1
94.7
77.1
85.9
MAE
81.0
90.3
76.9
85.9
89.5
75.3
83.2
лВОТ
85.5
91.7
77.7
88.1
88.2
76.3
84.6
D2V2
87.1
91.4
77.4
85.9
79.9
76.9
83.1
IJJEA
87.1
92.7
72.5
90.4
92.4
74.9
85.0
H/14
MAE-CT
87.7
93.9
80.1
97.0
95.0
79.2
88.8
MAE-Ref
90.1
92.0
80.4
97.5
96.0
79.8
89.3
JBOT-Ref
91.7
92.1
80.6
96.7
96.1
80.1
89.5
D2V2-Ref
90.4
89.0
75.9
93.3
95.6
78.4
87.1
MAE
82.5
92.0
78.2
90.5
91.8
77.1
85.4
2B/14
MAE-Ref
90.8
92.6
81.1
97.7
96.5
80.3
89.8
Table 17: Fine-tuning accuracy of all 19 VTAB-IK (Zai et al.
2019) datasets (averages are reported Table 4. Row-groups correspond to ViT-L/16, ViT-H/14 and ViT-2B/14 models respectively.
Natural
Specialized
Mechod
CF100
C1101
DTD
P1.002
Pets
SVAN
Sun307
Cametyon
EuroSAT
Resisc45
Relioooptyy
Clevr Cours
CloveDist
DMLab
ANORB-AR
2VORB-B
35pm
MAE
50.3
90.1
68.0
90.8
92.77
92.2
35.1
85,5
93.2
$1.7
72.7
93.2
63.9
55.8
84.5
93.9
58.6
$1.9
61.7
Mugs
69.4
89.2
71.4
95.4
92.9
7W.1
47.6
85.2
94.0
86.2
74.7
75.1
53.0
$1.1
77.9
46.8
45.5
26.
32.6
MAF-Ref
62.2
90.6
71.5
97.22
93.7
91.77
49.1
83.3
94.3
36.8
73.4
92.5
64.3
$7.4
84.3
33.4
60L.5
39.0
46.5
73.9
MAE-CT
93.2
72.3
97.3
93.4
90.7
18.3
94.4
36.1
75.0
93.0
64,9
58.3
$3.6
91.9
38.0
43.7
MAE-Ref
62.3
91.3
73.1
97.9
93.9
89.5
49.7
86.8
94.9
87.7
75.1
92.9
63.33
57.5
85.9
93.5
62.2
36.0
49.3
MAE-Ret Ref
61.8
91.0
78.0
97.5
94.5
91.3
49.
94.7
87.0
73.8
92.6
64.7
58.8
85.3
61.2
37.9
40.9
26

---

Published as a conference paper at ICLR 2025
B.12 EXTENDED COMPARISON OF FINE-TUNING ON IMAGENET- 1K AND INAT18
Table 18 extends Table 5 with additional MIM models. MIM-Refiner consistently improves also on fine-tuning with abundance labels. Table 19 shows individual performances on robustness and domain generalization benchmarks.
Table 18:
Full fine-tuning using 100% of the labels. MIM-Refiner consistently improves performance slightly even though ID methods typically perform worse than MIM models on this benchmark. This table extends Table from the main paper with additional MIM models and SSL models.
VIT-L/16
VIT-H/14
Model
ImageNet-Il 1K
iNat18
ImageNet-I1
iNat18
MAE
85.7
80.7
86.7
82.7
MAE-CT
85.4
80.9
86.8
82.9
MAB-Refined
85.6
80.9
86.9
83.3
CrossMAE
84.9
77.7
CrofsMAE-Refined
85.1
78.4
D2V2
86.6
81.0
86.6
79.6
D2V2-Refined
86.7
81.6
86.8
79.8
dBOT
85.8
81.9
87.1
84.1
dBOT-Refined
85,9
82.1
87.1
84.5
IBOT
84.8
76.9
Mugs
85.2
76.9
I-JEPA
84.9
75.9
Table Robustness and domain generalization evaluation on ImageNet-CCorrupiie Henddrycks
& Dietterich 2019) ImageNet-A(dversan
(Hendrycks et al. 2021b), ImageNet-
R(endition) (Hendryeks et al. 2021a) and ImageNet-Sketch (Wang et al. 2019b). For ImageNet-C
report the mean corruption error (Hendrycks & Dietterich 2019). MIM-Refiner consistently
improves robustness, particularly on larger models. Table reports the average accuracy of IN-A, IN-R and IN-Sketch.
VIT
Method
IN-C (L)
IN-A (1)
IN-R (1)
Sketch (1)
Validation (1)
CrossMAE
41.0
51.8
57.1
42.0
84.9
L/16
CrossMAE-Ref..
40.7
52.1
57.0
42.1
85.1
MAE
39.2
56.2
60.0
45.6
85.7
L/16
MAE-Ref.
39.0
57.0
60.1
45.9
85.6
D2V2
34.0
66.1
64.4
50.0
86.6
L/16
D2V2-Ref.
33.5
67.2
64.3
50.1
86.7
dBOT
36.1
60.0
60.5
45.5
85.8
L/16
IBOT-Reef
36.1
60.3
60.4
45.3
85.9
iBOT
37.8
47.6
53.7
41.9
84.5
L/16
Mugs
39.9
47.8
52.0
39.3
84.6
MAE
34.6
67.8
64.1
48.9
86.7
H/14
MAE-Ref.
34.3
68.0
65.1
49.7
86.9
D2V2
30.7
72.9
65.7
51.0
86.6
H/14
Def2-Ref.
30.2
74.1
66.1
52.1
86.8
31.8
71.1
68.1
51.8
87.1
H/14
defT-Ref.
31.8
72.3
68.0
51.8
87.1
H/14
IJEPA
37.4
51.7
57.4
43.7
81.7
MAE
32.6
68.4
65.4
50.0
86.7
2B/14
MAE-Ret Ref.
32.2
68.9
66.2
50.5
86.8
27

---

Published as a conference paper at ICLR 2025
B.13 FINE-TUNING VIT-2B MODELS
Table 20 shows results for fine-tuning VTT-2B models.
Table 20: Fine-tuning VIT-2B models using 100% of the labels. As fine-tuning these models expensive, freeze the first 6 of the 24 blocks to memory and compute.
ViT-2B/14
Model MAE MAE-Refined
ImageNet-Ik IK
iNat18
86.7
82.2
86.9
83.2
B.14 IMPACT OF MULTI-CROP AUGMENTATION
Multi-crop augmentation CCaron of ID methods (Caron et al. 2020 2021 Zhou et al. 2021 2022) and also improves MIM-Refiner significantly, where the k-NN accuracy D2V2-Refined L/16 drops by 2.6% when omitting multi-crop augmentations. When comparing to drops of other models, this a relatively small drop. For example, the performance of DINO B/16 drops by 7.2% and iBOT B/16 drops by 5.6% when omitting multi-crop augmentations (see Table 10 in (Zhou et al. 2021
2020) has been shown greatly improve the performance
B.15 HIGH-DIMENSIONAL h-NN
The linear probing protocol of DINOv2 (Oquab et al.) 2023) includes the possibility to concatenate features of the last blocks as input to the linear probe. We find that this outperforms using only features of the last block in most cases. As the best linear probe uses features from last blocks and the fact that the N--N and linear probing metrics are typically correlated (Oquab et al. 2023), we report the k-NN accuracy using only the features of the last block in Table to use the last 4 blocks for one metric and only the last block for the other. In Table 21 we investigate using the concatenation of features from the last 4 blocks for a k-NN classifier. Most models benefit from using more features, especially MIM models.
Table 21: ImageNet-1KK IK k-NN accuracy at 224x224 resolution of the [CLS] token of the last block the concatenation of the [CLS] tokens the last blocks.
#Blocks
ViT
Method
4
Delta
MAE
60.6
63.3
+2.7
D2V2
51.8
52.9
+1.1
iBOT
78.0
78.9
+0.9
L/16
Mugs
80.4
80.1
-0.3
MAE-Refined
81.5
81.5
0.0
D2V2-Refined
81.0
81.7
+0.7
MAE
58.1
61.4
+3.3
D2V2
48.0
52.2
+4.2
I-JEPA
71.6
72.3
+0.7
H/14
MAE-CT
79.1
78.6
-0.5
82.3
82.5
+0.2
D2V2-
82.3
83.4
+1.1
g/14
DINOv2
83.0
83.9
+0.9
B.16 MIM-REFINER ON SMALLER MODELS
HIM--efinnr builds on pre-trained MIM models which excel at larger scales (ViT-L and upwards). mainly focus large-scale models in our paper. However, MIM-Refiner also improves MIM
28

---

Published as a conference paper at ICLR 2025
models smaller scales (ViT-B). Additionally, combinations of MIM and ID methods have been explored in various works (Huang et al. 2022b] Wang et al. 2022 Yi et al. 2023). However, as these methods introduce significant runtime overhead over MIM models, they mainly focus smaller models (ViT-L. and smaller) where the pre-trained models are also often not published, which makes comprehensive comparison against these models impossible. Nevertheless,
show that MIM- Refiner is complementary to these methods by refining Contrastive MAE (Huang et al.) 2022b) ViT-B/16 model with MIM-Refiner. Table 22 shows that MIM-Refiner also significantly improves representation quality on smaller models.
Table 22: MIM-Refinee also significantly improves VIT-B models. Methods that improve MIM by incorporating ID already during pre-training are orthogonal MIM-Refiner where the refinement process also significantly improves representation quality these models.
VIT-B/16
k-NN
5-shot
2-shot
1-shot
МАE
51.1
43.1
27.1
14.0
MAE-Refined
76.6
64.5
58.6
50.0
CMAE 76.7
43.3
31.2
21.7
CMAE-Refined
78.5
70.1
65.7
57.9
B.17 RUNTIME OVERHEAD OF ID QUECE
We benchmark the overhead of the queue with its topk lookup in Table The queue only adds small amount of overhead as it is only needed for the forward pass (not for the backward pass) and queue operates in the bottleneck dimension (256 for models) of the head. Our setup for the queue closely follows the one introduced in Dwibedi et al. (202T). We estimate the runtime of three configurations where the last one used for training with topl NN-swap and (iii) queue with top20 NN-swap. We train these configurations for short amount of time on single GPU with the maximal possible batchsize and report the average runtime per sample. Note that the overhead is small that random runtime fluctuations that are common in modern GPU setups can slightly distort the results.
models: (i) no queue (ii) queue
Queue size
topk
L/16
H/14
2B/14
0
12.8s
30.1s
76.7s
65K
12.9s
30.2s
76.9s
65K
20
13.0s
30.5s
77.5s
Table 23:
Runtime per sample for different queue configurations. The queue adds only small amount of overhead. MIM-Refiner uses a queue size of 65K with a top20 NN lookup.
C IMPLEMENTATION DETAILS
C.I EVALUATIONS
To avoid slight performance differences due to minor implementation details or version changes and facilitate fair comparison between models, run all evaluations own in accordance with the suggested hyperparameters of the respecitve methods.
C.2 HARDWARE
All models are pre-trained on multiple nodes of 4XA100-64GB GPUs where ViT-L uses 4 nodes (i.e. 16 GPUs), VIT-H nodes of 4xA100 (i.e. 32 GPUs) and VIT-2B uses 16 nodes (i.e. 64 GPUs). For evaluations, we use a mix of 4xA100-644GB nodes, 8xA100 100-40G nodes and various smaller nodes that vary in number of GPUs. We estimate the total number of A100 GPU-hours used for this project to be 40K hours. This estimate includes everything from initial exploration, method development, analysis and evaluations.
29

---

Published as a conference paper at ICLR 2025
C.3 VISION TRANSFORMER
The architecture of our models follows the ones from the respective MIM model that is refined. That is a pre-norm architecture for MAE (He et al., 2022) and a post-norm architecture for data2vec 2.0 (Baevski et al. 2023). We attach the ID heads to the [CLS] tokens and also use the [CLS] token for evaluation.
We download the official checkpoints from the respective MIM works. Note that the official Cross- MAE model is pre-trained for less epochs than all other MIM models. For dBOT we use the models where a teacher of the same size is used (i.e. dBOT-L used MAE-L as teacher and dBOT-H used MAE-H as teacher).
C.4 HEAD ARCHITECTURE
We use a three layer MLP with hidden dimension 2048 as projector and two layer MLP with hidden dimension 4096 as predictor. Each linear projection is followed by a GELU (Hendrycks & Gimpel 2016) and a batchnorm (loffe & Szegedy 2015) layer. For the last linear projection in projector and predictor, no GELU is used.
C.5 REFINEMENT HYPERPARAMETERS
Hyperparameters for the refinement stage are listed Table Following MAE-CT (Lehner et al. 2024), we initialize all ID heads first by training them with frozen encoder to ensure good learning signal from the start of the refinement process. For this initialization, we use the same hyperparameters in Table 24 that and toPI-NN lookup. As we do not use a momentum encoder during training, we instead track an EMA of the encoder and use the EMA then for downstream tasks. As ViT-2B is very expensive to train, we freeze the first 6 blocks (for refinement and also for evaluation). As shown in Table this slightly reduces performance but also reduces memory consumption and runtime.
20 epochs for all models, learning rate of 2
Table 24: MIM-Refiner hyperparameters.
Parameter
Value
Parameter
Value
30 (MAEXBOT LJII)
NNCLR Heads
Epochs
20 (MAE 2B, data2vec 2.0)
Weight Decay
le-5
Batch Size
1024 511, 2B) AdamW
Temperature
0.2 0.3 (II), 0.35 (2B)
Optimizer
topk-NN k
20
Learning Rate
4c-4
NN-swap for Positives
/
Momentum
G, = 0.9, B2 = 0.95
NN-swap for Negatives
x
Learning Schedule
Warmup Decay
Augmentation
Warmup Epochs
Color & Blur Settings
see BYOL
End Learning Rate
1e-6
Global Views
Encoder
Global View Resolution
224
Layer-wise I.R Deray
0.65
Global View Scale
[0.25. 1.0]
Freeze Blocks
(L/H), (2B)
Local Views
Weight Decay
0.05
Local View Resolation
96 (L), 98 (H. 28) [0.05, 0.25
EMA
Scale
D EVALUATION DETAILS
D. GPU HOURS BENCHMARK
For benchmarking GPU hours, we follow the setup from MAE-CT (Lehner et al. 2024): we conduct comparison by implementing all methods in single code-base and conducting short training on a single A 100 40GB PCLe card. These runs are executed mixed precision training mode and with the highest possible batchsize that is power 2. The runtime of these benchmark runs is then extrapolated to the reported number of epochs. Benchmarks are conducted in pytorch 2.1 with CUDA 12.1. FLOPS are measured with the fvcore library?]
3 For the 1-shot classification plot in Figure 2 we do not take into account that some models train on higher resolutions (e.g. DINOv2) for visual clarity.
30

---

Published as a conference paper at ICLR 2025
D.2 MAE INTERMEDIATE REPRESENTATION ANALYSIS
We analyze how well the features of a VIT block are suited for reconstruction by training an MAE with a decoder attached after every ViT block. We use the same parameters as for training from scratch (He et al., 2022) but reduce training duration to 20 epochs, warmup to 5 epochs and the depth of all decoders to 2. The encoder remains fully frozen during training and only the decoders are trained.
For the visualization in Figure 3d we calculate the delta from one block to the next. We do this for both the k-NN accuracy and the reconstruction loss. Additionally, we divide by the maximum delta of each metric to transform both metrics into a similar value range and upper bound Figure 2 shows the reconstruction loss per block and the same plot for MAE L/16, where similar behavior can be observed.
We conduct the same analysis for refined models and show results in Figure 10
MAE B/16
L/16
0.46
MAE H/14
0.8
Train Reconstrution Lees
Relative Improvement
0.44
0.4
0.42
0.2
0.0
0.40
-02
Reconstruction on Lass
k-NN Accuracy
MAE-L/16
Figure 9: Left: Reconstruction loss per block of MAEs. Right Relative improvement of reconstruc- tion loss and k-NN accuracy for MAE L/16.
MAELIE
3.5
MAS Ruf L3G
MAg-Relleet LAG
2.46
IZLA
72
0.5
ThE Inptovement
2.44
0.42
0.8
4.0
Macamituttoo Low
A-N Accur ary
(a) Reconstruction Loss
(b) N-NN Accuracy
(c) Relative Improvement
Figure 10: Feature analysis of refined models. (a) The reconstruction loss increases in later blocks after refinement. (b) Representation quality increases drastically in later encoder blocks. (c) The relative improvements of MAL-Refined-H/1k shows that the refinement improves representation quality at the cost of reconstruction loss, which benefits downstream performance.
D.3 IMAGENET-IK IK LOW-SHOT EVALUATION DETAILS
For the 1, 2 and 5-shot benchmarks we train logistic regression (Caron et al. 2021 Assran et al.
2022) using the [CLS] token after the last encoder block with the eyanure (Mairal 2019) library.
As MIM models benefit from fine-tuning in this setting (Assran et al. (022). MAE and data2vec 2.0 tunet instead. We report the average of three dataset splits from MSN (Assran et al.) 2022)
In the 1 % and 10% low-shot benchmark, all models are fine-tuned with hyperparameters similar Cai et al. 2022) As the
to those used in related works (Lehner al. 2024 Assran al. 2023
31

---

Published as a conference paper at ICLR 2025
parameters vary between 1 1%/10% and also between model sizes, refer to the codebase for the exact protocols.
For a fair comparison, conduct the low-shot evaluations of DINOv2 at 224 resolution. We study the impact of the higher resolution where we observe minimal gains at the original resolution (5 (518x518). Note that DINOv2 first trains at 224x224 followed by short training at 518x518 resolution.
Table 25: ImageNet- IK low-shot evaluation DINOv2 g/14 4 on higher resolutions.
resolution
#patches
FLOPS [G]
1-shot
2-shot
5-shot
224x224
256
291
60.5
68.3
74.4
518x518
1369
1553
61.1
68.8
74.8
D.4 IMAGENET-IE 1K E--N CLASSIFICATION DETAILS
For k-NN classification, we follow the protocol of DINO (Wu et al. 2018 Caron et al. (02T). We train a soft k-NN classifier weighted by cosine similarity with k = 10. For MIM models, higher values for A are beneficial, so we tune this parameter for MAE and data2vec 2.0.
D.5 IMAGENET- K LINEAR PROBING EVALUATION DETAILS
For linear probing, we use the protocol from DINOv2 Oquab et al. 2023) for publicly released models and the values from the original papers otherwise. We train for 50 epochs using SGD with momentum 0.9. As data augmentation we use RandomResizedCrop and Horizone HorizontalFlip The DINOv2 protocol sweeps over the following hyperparameters by training multiple linear classifiers once:
13 learning rates ranging from 0.0001 to 0.5
Use the last block output or concatenate the output of the last 4 blocks
Use the [CLS) token or the concatenation of [CLS] and [AVG] token
As the linear probes trained on the concatenation of the last 4 blocks have more features and more parameters discriminate between classes, they tend be the best within the sweeped parameters. Note that we evaluate the representation of the last block in isolation via k-NN classification. We investigate k-NN classification with features from the last 4 blocks in Appendix B.15
D.6 IMAGENET- K CLUSTER EVALUATION DETAILS
For each considered model in the clustering experiments Section 4.3 we used the CLS token embeddings of the ImageNet validation and preprocessed the embeddings using L2 normal- ization. For conducting mini-batch k-means and calculating the cluster related metrics we used the scikit-lees -lean package (Pedregosa al. 201T). except for calculating the cluster accuracy where we used the implementation in ClustPy (Leiber et al. 2023). The UMAP plots in Figure [S] where generated by applying UMAP on top of the L2 normalized CLS token embeddings of the 53 food related classes of ImageNet for each model. We use the default UMAP parameters of umap-lea leaa (Mcinnes et al. 2018) for all plots (0_neighbors=1))
D.7 INAT18 18 TRANSFER LEARNING EVALUATION DETAILS
We report the accuracy on the validation set averaged over three seeds.
For 1-shot classification iNat18, 2023). We also attempted to fine-tune MIM models where some models fail to exceed random performance and therefore also linear probing.
the linear probing protocol from DINOv2 (Oquab et al.
For 5-shot and 10-shot classification on iNat18, we fine-tume all models. The hyperparameters for fine-tuning (Table 26) inspired by MAWS (Singh al. 2023)
32

---

Published as a conference paper at ICLR 2025
Table 26: Hyperparameters for fine-tuning on iNat18 low-shot classification.
Parameter
Value
Parameter
Value
Epochs
30
Label smoothing
0.1
Batch size
Data Augmentation
Oplimizer
AdamW
Sesize
256
Learning mite
le-3
interpo l
hicubic
Layer-wise decay
0,75
RandomRe
224
Weight decay
0.05
scale
[0.08, 1.0] bicubic
Momentum
0.999
interpolation
Learning schedule
lincar warmup
cosine decay
Randomilori zont alrlip bormalize
0.5
Warmup epochs
ImageNet statistics
D.8 TRANSFER LEARNING LINEAR PROBING
For transfering the pre-trained features to iNat18, six VTAB datasets and ADE20K (Table 4 we use the DINOv2 (Oquab et al., 2023) linear probing protocol as described in Appendix D.5 For ADE20K and iNat18, we reduce the hyperparameter grid to fit into 40GB GPU memory.
D.9 ADE20K SEMANTIC SEGMENTATION LINEAR PROBE
Large models (such as VIT-H or ViT-2B) are expensive to train on ADE20K. Therefore, we opt for simple light evaluation protocol compare our models on a segmentation task:
We keep resolution 224x224
We freeze the encoder
We train linear classifier similar to DINOv2 (Oquab et al. 2023) that predicts a class for each patch. The resulting low-resolution prediction is then upsampled to 224x224 resolution.
For evaluation, use the original resolution image and slide 224x224 window over the image with a stride of 170 pixels and average the logits per pixel.
As intermediate representations are commonly used for semantic segmentation, we use features from the last block, the 5th last block, the 9th last block and the 13th last block. Compared to simply using the last blocks, this improves performance for all compared models.
D.10 FINE-TUNING ON VTAB-1K
For fine-tuning models VTAB-1K 1K provide the hyperparameters in Table 27 We search for the best learning rate for each dataset by fine-tuning the model 25 times (5 learning rates with seeds each) on the 800 training samples and evaluating them on the 200 validation samples. With the best learning rate, we then train each model 5 times on concatenation of training and validation split, evaluate on the test split and report the average accuracy.
Table 27: Hyperparameters for fine-tuning on VTAB-IK.
Parameter
Value 64
Epochs
Parameter
Valuc linear warmup - cosine decay
Learning mite schedule Warmup epechs
Batch size Socds
Data Augmentation
Optimizer
Adamw
Learning rate
1e.3, 7.5e 4, 5.0c.4, 2.5e-4, 1.0e
Interrol lat on size Normal
Rasize
bicubic 224x224 ImageNet. IK statistics
Layer-wise decay
Weight decay
0.05
Momentum
81 0.9, 8g 0.999
33

---

Published as a conference paper at ICLR 2025
D.11 FINE-TUNING WITH 100% LABELS
For fine-tuning with 100% of the labels (Table 5 we use the hyperparameters provided in MAE (He) al. 2022) for both 1 and ImageNet- IK (see Table(28) As D2V2 models are unstable with the default learning rate of the MAE fine-tuning protocol, we use the highest stable learning rate out of 5e-4, 2.5c-4 and le-4.
For ViT-2B/14 (Table 20), we freeze the first 6 of the 24 blocks to reduce computational costs. Additionally, as the 2B models are sometimes unstable with learning rate of le-3, we reduce it to 7.5e-4 or 5e-4 using the largest stable learning rate.
To fine-tune I-JEPA (Assran al., 2023), we adjust hyperparameters match their tin-ttunig setting of a ViT-H/16 We reduce peak stochastic depth from 0.3 to 0.25. To fine-tune on iNat18, we found that a learning rate le-3 performs better than the le-4 used in the original work.
Table 28: Hyperparameters for fine-tuning on ImageNet- IK and 8 many-shot classification.
Value
Parameter
Value
Epochs
Train Data Augmentation
Batch size
1024
RandomRes s zedCrop
224 [0.08, 1.0] bicubic y
Stochastic depth
acale
interpolation
Peak rule
0.2 (L/2B)) 0.3 (H)
sandomborizzottaane
Decay
Optimizer
AdamW 1e.3
RandAug
magnitude
Learning tale
magn:itude_sttd
0.5
Layer-wise decay
Freeze Blocks
Momentum
Weight decay
91
(L/If) (2B)
0.9, 82 0.999
0,05
Cutmix a Test Augmentation
Normalise
Mixup
ImageNer statistics
0.8
1.0
learning nite schedule
Warmup epochs
linear warmup
cosine decay
Rea 1 se interpolation
256
bicubic
End Learning Rate
6
Cent arcrap Normalize
224
smoothing
ImageNer statistics
D.12 ADE20K SEMANTIC SEGMENTATION FINE-TUNING
We fine-tune ViT-L models using an UperNet (iiao et al. 2018) segmentation head to predict segmentation mask. We follow common practices and fine-ture tune on 512x512 resolution, where we interpolate the absolute positional embedding from 224x224 to 512x512, add relative position bias to the attention layers (initialized to 0) (He et al. 2022) and introduce layerscale (Touvron al., 2021) (initialized to 1). A common augmentation pipeline is used that consists of random rescaling, random horizontal flipping, color jitter and padding if necessary. We train for 160K updates using a batchsize of 16, learning rate of 2c-5, weight decay 0.05, linear warmup for 1.5K update steps followed by cosine decay, stochastic depth rate 0.2, dropout 1, layer-wise learning rate decay 0.95, We evaluate after 160K update steps using sliding window of 341 pixels and report mloU over the validation set.
D.13 k-NN CLASSIFICATION AND SEMANTIC SEGMENTATION PROBE PER BLOCK
For the per-block analysis in Figure 3b and Figure we follow the respective settings of k-NN classi- fication (Appendix Section D.4) and semantic segmentation linear probing (Appendix Section D.9p. The only change is that for semantic segmentation linear probing per- block, we fix the learning rate to and only use the patch tokens of the respective block as input to the linear probe.
34

---

Published as a conference paper at ICLR 2025
D.14 LOss SCHEDULE VISUALIZATIOS
We visualize the schedules used for scheduling the loss weight of ID heads attached at intermediate blocks in the ablation study (Table in Figure 11
Decny
Staggered Decay
6.8-
0.0
Loo Waggt
5.6
0.6
0.2
022
0.0
16
12
Eooch
Saaggeed Shep
One Hot
1.0
8 ock
23
Blok
5.8
Savi::
Irea Welge
Slock
5.8
1.4
0.2
0.0
0.0
Figure 11: Loss weight schedules for the ablation in Table [ For visual clarity, small offsets added when values overlap (for "One Hot" and "Staggered Step").
35

---

Published as a conference paper at ICLR 2025
E REPRESENTATION DEGRADATION IN OTHER MODALITIES
We investigate feature degradation of masked pre-trained models from different domains in Figure 12 which show similar trends, suggesting that our methodology could be extended other modalities.
VideaMAE
RoBERTa
VideoMMAE 1/16
Large
VideoMAE B/16
30
Arcuacy
K-N Accurscy (96
25
20
Blocks
VIT
(a) VideoMAE >> ImageNet-IK
(b) RoBERTa MNLI
ESC-50 Enviromennial Sound Classifcation
AudioMAE B/16
55
AudioMAE 8/16
50
KAN Acourssy 180)
Accurey
45
20
40
Blocks
(c) AudioMAE + SpeechCommands V2
(d) AudioMAE + ESC-50
Figure 12: Feature degradation investigation on other modalities. (a) We evaluate VideoMAE (Tong
et al., 2022) on ImageNet-1K K classification by treating each image as single frame video. (b) Entailment classification using RoBERTa (Liu et al. 2019) on the MNLI (Williams et al.
2018 task
of the GLUE (Wang | (2119a) benchmark. (c) AudioMAE (Huang et al.) [202aa) is evaluated on
SpeechCommands V2 Warden 2018) audio classification (d) AudioMAE, (Huang et al. 2022a) is
evaluated on ESC-50 (Piczak) environmental sound classification. Similar trends can be observed in other modalities, often even on smaller models.
36

---

Published as a conference paper at ICLR 2025
F RELATION TO NNCLR
Figure shows the difference between NNCLR (Dwibedi et al. 2021) and NNA. NNCLR uses the NN-swap also for the negatives, resulting in worse signal due the NNS being retrieved from
FIFO queue of features from previous model states.
Sample Batch
Sample Queue
Nearest Neighbor
Altracting Force
Repelling Force
NNCLR
NNA
Stop Gradient
Figure 13: The NN-swap NNCLR introduces inter-sample correlations between positives but uses features from previous state of the model. Using the NN-swap only for the positives preserves the samere correlations while using features from the current state of the model as negatives to improve the loss signal.
G PRACTITIONER'S S GUIDE
find MIM-Refiner to be easy to tune. By freezing the encoder and training multiple ID heads attached to the encoder with different hyperparameters, one can get quick and cheap evaluation of suitable hyperparameters. We mainly use two metrics judge the performance of an head:
Accuracy of k-NN classifier trained on a subset of the data (e.g. 10% of the data). This is relativley cheap to compute and can be done periodically during training. For the k-NN classifier, one can use either features of encoder block or features of intermediate blocks in an II) head to judge the representation at the respective location in the network.
The accuracy the NN-swap, i.e. how often is the NN from the NN-swap from the same class the query sample. This metric is essentially free compute the NN-swap is required for training anyways.
37


Reviews:
[{"review_id": "QLBisF4bfd", "reviewer": "Reviewer_Auz1", "summary": "The paper focuses on bridging the gap between large MIM pre-trained models and SOTA methods. The paper first discovers that  MIM models have different types of blocks: those that mainly improve the pre-training objective and others that are responsible for abstraction. Then, the paper proposes a method MIM-Refiner, which adds Instance Discriminator heads on pre-trained MIM models for refinement. The ID heads exploit the intermediate representations to consistently improve the performance of MIM pretrained models. While the performance gains on large dataset full-finetuning are small, the proposed methods show remarkable gains on few-shot settings.", "questions": "Please refer to the weakness. I believe a clear description of the method and experimental setup is one of the most important things when writing a paper (weakness 1).\n\nAdditional question: what does the “relative” in Figure 3(d) mean? Does the value calculated by the performance of ( the i+1 th layer - the i-th layer)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper first points out the influence of the lightweight decoder on the feature learning of the encoder in MIM methods.\n2. The analyzing part is well-written.", "weaknesses": "1. The description of the method and experimental setup needs to be clarified. (a) Which blocks need to be fine-tuned during refinement, or do all blocks need to be fine-tuned? (b) How many epochs are needed to refine different models? (c) What is the structure of the ID head? Answers to all these questions should be contained in the manuscript.\n2. Unfair comparison. The paper misses an important baseline - train the original model with 0 heads with the same epochs to demonstrate the importance of refinement (instead of just training more epochs).\n3. Some typos. L267-269, see Table 1 instead of Figure 3b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OZUkCxG9Ns", "reviewer": "Reviewer_BbVs", "summary": "This paper presents a contrastive learning boosting method called MIM-Refiner to refine the features of pre-trained MIM models. MIM-Refiner leverages multiple instance discrimination heads (ID) which are connected to different immediate layers. Each ID head contains a contrastive loss that captures semantic information to improve the quality of learned representations. By training a few epochs, the features of MIM-Refiner surpass the current MIM models on multiple experiments: on ViT-H with data2vec 2.0 on ImageNet-1K, the accuracy of the proposed method reaches state-of-the-art performance on linear probing and low-shot classification.", "questions": "Please refer to weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a detailed analysis of the blocks of MIM models in which different blocks extract features with a specific focus and the most efficient features learned by MIM are from the middle blocks.\n\n2. A contrastive learning-based method called MIM-Refiner is proposed to refine the representation of current MIM models by attaching the middle layers with ID objective. \n\n3. Experimental results show the effectiveness and generalization ability of MIM-Refiner on downstream tasks.", "weaknesses": "1. As the discussion of end-to-end training, the proposed method MIM-Refiner seems to be a two-step training method, with first step training MIM models and fine-tuning the updated models by incorporating ID heads to middle layers. Practically, this might increase the complexity of the training paradigm and deployment. Is it possible to improve the proposed method with end-to-end training on MIM and ID? If not, what are the potential bottlenecks to circumvent this goal?\n\n2. There is no overview diagram that shows the detailed architecture of MIM-Refiner or how the training diagram goes. The diagram in Figure 4 provides partial information but does not clearly illustrate these points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ByCfGLf5L", "reviewer": "Reviewer_HaAb", "summary": "---\n\n## **Summary**\n\nThe paper identifies the representation degradation issue in Masked Image Modeling (MIM)-pretrained large foundation models. To address this, the authors propose a simple yet effective method to prevent degradation and further improve the representation quality of MIM methods by adding auxiliary contrastive losses to the last layers of Vision Transformers (ViTs) on top of the MIM objective. The paper provides improved performances with large margins over current state-of-the-art (SOTA) methods through extensive experiments and rigorous analysis, demonstrating the success of the proposed approach.\n\n---", "questions": "---\nSince D2V2 is used as a baseline, does the representation degradation issue also appear in the audio and language domains?\n\n---", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "---\n\n## **Strengths**\n\n- The paper is well-written, with clear observations, a well-developed motivation, a straightforward idea, a clearly-stated method, extensive experiments, and comprehensive analysis.\n  \n- It effectively identifies the representational degradation phenomenon in large visual foundation models pre-trained with MIM self-supervised learning (SSL), providing evidence through multiple experiments.\n  \n- The proposed method offers a simple and effective solution to prevent this issue and improve the representation quality of MIM SSL.\n\n- Rigorous experiments and analysis are conducted to show the success of the proposed method, with large improvements over current SOTA.\n\n---", "weaknesses": "---\n### **Limitations**\n\n1. To prevent representation quality degradation in the last layers of ViTs, the authors experiment with contrastive loss, which requires constructing a queue/pool for positive and negative samples. I noticed the proposed method uses a top 20-NN approach to retrieve positive samples in the queue, which could contribute significantly to the increased training time per step. What's the queue size used? how much does it contribute to the increased training time per step?\n\n2. Since the paper emphasizes preserving the richness of representations, evaluation on dense prediction tasks such as object detection and instance segmentation (OD/IS) would be valuable, in addition to the provided segmentation probing on ADE20K.\n\n   - It would be meaningful to compare the performance of MIM-refiner-pretrained ViT-L on COCO object detection against MAE-pretrained ViT-L following the ViTDet framework [1].\n\n   - [1] Li, Y., Mao, H., Girshick, R., & He, K. (2022, October). *Exploring plain vision transformer backbones for object detection.* In European Conference on Computer Vision (pp. 280-296). Cham: Springer Nature Switzerland.\n\n---\n\n### **Recommendation**\n\nConsidering the strengths and weaknesses discussed above, my recommendation for this paper is **ACCEPT**. This is a strong paper with a clear contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hiyCoIvps9", "reviewer": "Reviewer_JDMo", "summary": "This paper introduces MIM-Refiner, which leverages contrastive learning to boost MIM models. The proposed method is simple and has demonstrated effectiveness in few-shot image classification tasks.", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: This paper is well-written and easy to follow.\n\nS2: This paper is not the first to point out that the encoder in MIM methods partially performs image encoding and representation learning. A similar conclusion is also discussed in [A], highlighting that MIM methods using a single ViT structure tend to face this issue. The reviewer previously conducted experiments on MAE-B, showing that introducing an additional decoder can effectively alleviate this problem. This paper demonstrates that, for methods like MAE that use an asymmetric encoder-decoder architecture， especially in larger models， a small decoder cannot fully decouple encoding and decoding, providing academic insights.\n\nS3: This paper proposes a simple and effective MIM-Refiner method, refining the later blocks of MIM models to enhance MIM representations effectively.\n\n[A] Context Autoencoder for Self-Supervised Representation Learning.", "weaknesses": "W1: Existing work [A] has shown that fine-tuning MIM models can enhance their representation capability (for image classification), but the improvement under full fine-tuning is minimal. Additionally, MAE has demonstrated significant transfer performance on dense prediction tasks [B] (object detection/instance segmentation). Fine-tuning MIM models with contrastive learning methods is unlikely to bring substantial improvement and may even negatively impact performance.\n\nW2: Current vision foundation models, such as DINOv2, exhibit strong patch-level representation learning capabilities and combine MIM and CL. Their learned representations have shown effectiveness in tasks like image classification, pixel classification, and depth estimation. Although this paper discusses the relationship between MIM-Refiner and these models, suggesting that MIM-Refiner can build on them for further improvement, I am concerned that MIM-Refiner may degrade pixel-level representation performance for tasks like semantic segmentation or depth estimation (especially when the backbone is fixed).\n\n[A] Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label Efficient Representations.\n\n[B] Exploring Plain Vision Transformer Backbones for Object Detection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)