Cluster and Predict Latents Patches for Improved Masked Image Modeling
Authors: Timothée Darcet
Federico Baldassarre
Maxime Oquab
Julien Mairal
Piotr Bojanowski
Venue: Accepted by TMLR ()

Abstract: Masked Image Modeling (MIM) offers a promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art.
  In this paper, we systematically analyze target representations, loss functions, and architectures, to introduce CAPI -- a novel pure-MIM framework that relies on the prediction of latent clusterings.
  Our approach leverages a clustering-based loss, which is stable to train, and exhibits promising scaling properties.
  Our ViT-L backbone, CAPI, achieves 83.8\% accuracy on ImageNet and 32.1\% mIoU on ADE20K with simple linear probes, substantially outperforming previous MIM methods and approaching the performance of the current state-of-the-art, DINOv2.

Published in Transactions on Machine Learning Research (05/2025)
Cluster and Predict Latent Patches for Improved Masked Image Modeling
Timothée Darcet
timdorott@meta.com com
Meta FAIR
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
Federico Baldassarre
baldassarree@meela.ccom
Meta FAIR
Maxime Oquab
gas@meta.com com
Meta FAIR
Julien Mairal
Sulien.
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
Piotr Bojanowski
bejjsnouskiimmett.ccom
Meta FAIR
Reviewed on OpenReview: https: // openreview. forum? id= Ycma "cmm"
Abstract
Masked Image Modeling (MIM) offers promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art. In this paper, we systematically analyze target reprosentations, loss functions, and architectures, to introduce CAPI a novel pure-MIM framework that relies on the prediction of latent clusterings. Our approach leverages a clustering-
loss, which is stable to train, and exhibits promising scaling properties. Our VrT-L backbone, CAPI, achieves 83.8% accuracy ImageNet and 32.1% mloU on ADE20K with simple linear probes, substantially outperforming previous MTM methods and approaching the performance of the current state-of-the-art, DINOv2. We release all our code and models.
1 Introduction
Recent advances large-scale visual representation learning have established foundation models as a cornerstone of modern computer vision. Self-supervised representations have proven particularly effective in domains with limited annotations, such as satellite imagery (Tolan et al., 2024) and medical imaging (Xu et al., 2024; Vorontsov et al., 2024; Chen et al., 2024; Moutakanni al., 2024; Dermyer et al., 2025), while enabling breakthroughs in more fundamental vision tasks like monocular depth estimation (Yang et al., 2024a; Bochkovskii et al., 2024; Yang el al., 2024b), keypoint matching (Edstedt et al., 2024), and tracking (Tumanyan et al., 2025). This shift from small supervised models to large self-supervised generalist models mirrors the evolution natural language processing since the publication of BERT (Devlin al., 2018) and GPT (Radford al., 2018), where largo-scale models pretrained on web-scale unlabeled data have become ubiquitous foundation models.

---

Published in Transactions on Machine Learning Research (05/2025)
ADE knn
ADB linear
INIK atlentive
Compute patch
Cluster into
88
representations
pseudo calegoriie
32
36
84
top- acc
TMMA Teacher
mloU
2.8
mloU
32
80
24
76
20
21
72
111d
10
10/1
10
10
10
10
10
10
10
Student
Inference MFLOP
Inference #FLOP
Inference #FLOP
Predict calegury of missing patches
CAPI
DINOV2
image patches embedded by a EMA encoder are grouped into clusters. Their assignments are then used as the training signal for the online model. The loss is purely about prodicting the content of missing patches and does not rely on augmentations a contrastive loss. Evaluation scores: evaluate frozen reprosentations on ADE20K segmentation with k-nn and linear probe and on ImageNet-11 classification with an attentive probe. We compare to MAE, data2vec 2.0, I-JEPA, and AIM. Compared to other masked image models, CAPI achieves higher performance with fewer FLOP, scaling well with model size, and approaches the scores of DINOv2+reg.
Figure 1: CAPI Method overview:
However, the impressive scalability of language models remains unmatched in vision: the best self-supervised visual encoders contain around one billion parameters (Oquab et al., 2024), hundreds of times smaller than current language models (Liu et al., 2024). A reason for this gap may be found in the discrepancy between the pretraining tasks used in vision and in language. The success of language models stems from the generality of the language modeling task: modeling the distribution of data, conditioned on context. Naturally, researchers have attempted to adapt this approach to computer vision, which resulted in masked image modeling (MIM): the prediction of missing image content given surrounding context.
Yet, existing MIM approaches have not matched the reprosentation quality of alternative self-supervised methods. Pixel-level reconstruction objectives, e.g. MAE (He et al., 2022), provide good initialization for fine-tuning, but yield poor frozen representations, possibly because the target is too low-level to capture the task's inherent uncertainty (LeCun, 2022; Assran et al., 2023). Instead of reconstructing pixels, other works propose reconstructing in a pretrained encoder's latent space (Zhang et al., 2022; Fang et al., 2023; 2024). However, this approach requires existing encoder and cannot learn representations from scratch.
The most promising direction uses the latent representation of the online model or an exponential moving average (EMA) it as a learning target for the model, bootstrapping an informative latent space from scratch. This approach is used in the current slate-of-the-arl method for self-supervised learning of representations, DINOv2 (Oquab et al., 2024). However, these methods often suffer from poor stability (Zhou et al., 2022) and scnsitivity to hyperparameters (Assran et al., 2023), requiring additional objectives like contrastive learning to produce competitive representations (Zhou et al., 2022; Oquab et al., 2024; Alkin et al., 2024).
In this work, we focus on online latent masked image modeling. We isolate it from other stabilizing objectives, and systematically study the design choices it involves. We center the discussion around three aspects of the masked image modeling principle: the target representation, the formulation of the loss, and the architecture used perform predictions. We show that with the right implementation, a simple masked image modeling objective can lead to features competitive with the current state of the art in SSL. In brief, our method relies a pair of vision transformers, an online model trained with masked image modeling, and target encoder updated via an exponential moving average of the online model (fig. left). The training signal for the ouline model comes from the patch representations of the target encoder, which is updated through an exponential moving average (EMA). The patch embeddings of the EMA encoder are converted to soft assignments on pseudo-categories using a learned online clustering, while the student receives a partially masked image as input and is trained to predict the clustering assignments of the missing patches. Using this method, we train CAPI, 300-million parameter visual encoder whose representations approach DINOv2's performance while significantly outperforming previous masked image models (fig. 1 right).
2

---

Published in Transactions on Machine Learning Research (05/2025)
Online clustering
Maraan
Teacher
Linear prototypes)
VIT-L (24 blocks)
CB
EMA
Student
VIT-L (24 blocks)
K.V
Predictor 12 blocks
Linear
Detailed overview of our method with reference tensor sizes for a VIT-L/16 model. We denote in red the parts that are trained by the main loss, in purple the parts that are trained with the clustering loss, and in blue the parts that are updated by the EMA.
Figure 2:
2 Related Work
Self-suporvised representation lcarning. Self-supervised learning (SSL) is a pre-training paradigm which model optimized to solve pretext lask on unlabeled data, often collected at scale. As the model is not trained to match specific human annotations, the benefit of this type of training is to produce generalist model, that can be adapted to solve many different downstream tasks. Depending on the application, these tasks can be solved either by fully fine-tuning the mode, or by using the representations extracted with the frozen model. Tn the SST literature, some works have focused mainly on full fine-tuning (He et al., 2022; Huang et al., 2023), while others have shown that frozen representations can reach excellent performance on a wide range of tasks, avoiding costly fine-tuning (Oquab et al., 2024) and generalizing to annotation-seaa scarcce domains where fine-tuning not possible (Tolan al., 2024; Xu et al., 2024). Historically, early work self-supervised learning focused on hand-crafted pretext tasks such predicting the rotation of an image (Gidaris et al., 2018), the relative position of patches (Doersch et al., 2015) or the color of grayscale image (Zhang et al., 2016). Subsequent works pushed the field forward with methods based on clustering (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; 2021) and contrastive learning (Oord et al., 2018; Misra & Maaten, 2020; Chen et al., 2020b; He et al., 2020). Nowadays, the best self-supervised encoder inherits from these lamilies (Oquab et al., 2024) and complements them with a masked image modeling objective (Zhou et al., 2022). However, training with both global and MIM objectives can prove difficult, as multiple components can interact negatively'. In this work, we study the masked image modeling component in isolation, suggesting improvements to properly stabilize the optimization ion objective in the absence of global term.
Pixel reconstruction. Learning by predicting missing part of an image was first proposed in Context encoders (Pathak et al., 2016). This was envisioned conceptually similar to denoising autoencoders (Vincent et al., 2008), in the case where the noise is masking process. More recently, the success of masked language modeling (Devlin et al., 2018) and autoregressive pretraining in natural language processing (Radford et al.,
1 'Zhou et al. (2022) showed that using shared head for the DINO and iBOT losses gave better results at small scales. However, large scale, Oquab al. (2024) observed that this caused the IROT loss explode late into the training, significantly degrading performance. Untying the heads prevented the losses from competing directly, stabilizing the training.
3

---

Published in Transactions on Machine Learning Research (05/2025)
2018) brought a new wave of interest for transferring these ideas to vision. iGPT (Chen et al., 2020a) was the first effort to train a transformer (Vaswani et al., 2017) by generating pixels. Chen et al. (2020a) proposed rasterizing images to very low resolution, then training for autoregressive next-token prediction. Then, the advent of the ViT (Dosovitskiy et al., 2021) architecture sparked further interest in the field. Following the initial exploration of Dosovitskiy et al. (2021), BeiT (Bao el. al., 2021) tried using the quantized latents of a dVAE as targets for a masked image pretraining, using the tokenizer from DALL-E (Ramesh et al., 2021). Bei'T has proven useful as an initialization for further fine-tuning, but severely underperformed baselines in representation learning. To simplify BeiT, SimMIM, and MAE (Xie et al., 2021; He et al., 2022) concurrently proposed using raw pixels as targets. Thanks to clever split encoder encoder/decodee /decoder architecture, MAE proved very stable and reached interesting representations, despite its simplicity. However, it still fell short of previous SSL methods in terms of representation quality: MAE models need to be scaled to a ViT-H size to match the linear probing performance of a 25x smaller DINOv1 WTT-S/16(Carr)) et al., 2021).
Latent reconstruction. Concurrently to MAE, Zhou et al. (2022) proposed iBOT. To obtain a more semantic tokenization, iBOT used the online output of the model being trained as the reconstruction target for masked image modeling. This led to good representations, and the method was reused to obtain the current -tate-of-the-art (Oquab et al., 2024). However, the iBOT objective was very unstable and required an additional DINO (Caron et al., 2021) loss to stabilize the training. The idea of using online representations targets was then reused in data2vec (Baevski et al., 2022) and I-JEPA (Assran et al., 2023; Bar et al., 2024). I-JEPA in particular proposed reusing the split encoder perroder/deooder /decoder architecture of MAE (with the 'decoder' being called 'predictor" in I-JEPA) and removing the projection head of iBOT, to obtain a more stable objective in the latent space. The improvements in I-JEPA establishod a new tradcoff between stability and performance, but it was still both sensitive to hyperparameters (-12 points on IN-1k when changing the "target scale" from [0.15, 0.2) to [0.125, 0.2] (Assran et al., 2023)) and weaker than DINOv2 (81.1 on ImageNet-1k, 5.4 below DINOv2 while using a model twice bigger).
Clustcring in self-supervised learning. Our approach is also related to methods that use clustering for self-supervised learning. First of this line of work, DeepCluster (Caron et al., 2018) proposed using a simple k-means to obtain pseudo-labels. Subsequently, SwAV (Caron et al., 2020) introduced an online clustering to replace the offline k-means. DINO (Caron et al., 2021) built on SWaV, and made the clustering be implicitly learned by the MLP projection head of the student. In a masked image modeling sctup, iBOT (Zhou et al., 2022) proposed to reuse the DINO projection head for masked image modeling, implicitly using a clustering as target. Other recent approaches have used clustering for masked image modeling. Gidaris et al. (2024) used codebook assignments as target for a masked image modeling training, and complement it with a global loss, obtaining a formulation similar to iBOT where the clustering aspect appears more clearly. The codebook, however, is obtained by randomly sampling previous batches, whereas we attempt to learn more optimal codebook with an online clustering. Most similar to our approach is the work of Salehi et al. (2024). They proposed a masked video modeling setup where the larget encoder is a patch-wise MLP trained via gradient descent at the same time as the predicting model, via a loss comparing the target representation to the prediction of the other model. To prevent collapse, they use a Sinkhorn-Knopp algorithm to rebalance the assignments of the patches to the prototypes, obtaining a form of clustering. Both the clustering aspects and the mauhorn-KKnopp algorithm are similar to our approach, but the target encoder is significantly different. In our study, we focus on cases where the target is the latent representation of the online model or its EMA, while they learn an entirely separate MLP to encode the targets. We also focus on images, while they focus on videos.
3 Approach
At a high level, masked image modeling involves masking a part of the input, feeding the visible region to a prediction model, and optimizing it to predict the content of the missing parts. Despite the simple formulation, the effectiveness of reconstruction-basedd methods and the propertics of the trained models are dramatically influenced by a number of design choices. In this section, we discuss the three aspects of masked image modeling depicted in Figure 3: the type of patch representation used as target (fig. 4), the loss
4

---

Published in Transactions on Machine Learning Research (05/2025)
Choose loss formulation (fig. 5)
Target
Choose target
Choose a predictor
representation
architecture (fig. 6)
?
?
?
?
Figure 3: Overview of the components of reconstruction-based model. We identify three main choices involved in designing a masked image model: the choice of largets (fig. 4), the loss function (Section 3.1, fig. 5) and the architecture of the predictor (Section 3.2, fig. 6).
formulation (Section 3.1, fig. 5), and the prediction architecture (Section 3.2, fig. 6). Based on our findings, we introduce CAPI, an SSL model that enjoys stable learning and strong representation capabilities.
Overview of training. In short, our main design choices are: first, we reconstruct images in latent space with a teacher-student framework, following iBOT. As the online latent MTM framework has already shown very promising results, we restrict our focus to this specific setup. Then, we formulate our loss using a clustering component, inspired by SeLa and SwAV, and draw inspiration from the regularization methods they introduce, in particular the Sinkhorn- Knopp (Sinkhorn & Knopp, 1967) rebalancing from SeLa (Asano et al., 2020). Finally, we employ a cross-attention predictor model, separate from the encoder, to perform reconstructions, as in crossMAE (Fu et al., 2024).
Our encoder and the predictor are transformers (Dosovitskiy et al., 2021) and, during pre-training, they operate tandem as the student (fig. left). The teacher an EMA of the encoder. The typical values for one training iteration using square images of side 224 pixels and model with patch size 16 are:
We pass the full image to the teacher, collect n 14 14 = 196 patch tokens, and apply an online clustering obtain soft assignments that will be used as learning targets.
The encoder receives a partial view of the input image: we apply a patch embedding layer obtain patch tokens, we drop Parop Xn of these patches and pass the encoder the remaining Tekp (1 - pårop) xn 69 patches (párop 65%).
The encoder takes these nikeep 69 tokens along with nleg learnable register tokens (Darcet et al., 2024), and processes them to obtain Rencoded Wkeep frog
85 encoded tokens.
Target
Target
Target
Frozen encoder
EMA encoder
EMA of student
(a) Pixel targets
(b) Frozen target encoder DeeiTPPeCo,EVA)
(c) EMA encoder
(IGPT, MAE, ATM)
(IBOT, Data2Vec, I-JEPA, Ours)
Figure 4: The target reprosentations commonly usod in MIM. (a) In the simplest case, the pixel values of the masked patches are flattened and directly used as targets. (b) An alternative is use the output of frozen encoder targets, obtain more semantic representation. (c) To alleviate the need for preexisting pretrained model, the target encoder can be EMA of the online model. We focus on this case.
5

---

Published in Transactions on Machine Learning Research (05/2025)
C
PCG
PLG
EMA
MLP
MLP
Linear
Pred.
Target
Pred.
Target
Pred.
Target
(a) Direct nonparametric loss function
(b) Loss after MLP projection (iBOT, DINOv2)
(c) Loss after clustering assignments (proposed)
(MAE, I-JEPA)
Figure 5: The different loss formulations considored here. We depict in red the flow of the gradient. (a) The loss can simply be a nonparametrie function, such as MSE (in MAE) or Huber loss (in I-JEPA). (b) In iBOT and DINOv2, the prediction is passed through an MLP head, and the target passed through an EMA of this head. The loss is the cross-entropy of their outputs. (c) We propose using a clustering to obtain the targets. The loss is the cross-entropy between the predictions and the assignments.
We sample upred 7 coordinates among the dropped sel, and for each we forward a [MSK] token through the predictor, which predicts their assignment by aross-attending to the encoded view.
A detailed visual diagram of the method can be found in fig. 2.
3.1 Clustering- based loss formulation
Some self-supervised learning approaches like SwAV and DINO can be interpreted as form of latent clustering. They employ a cross-entropy loss between the student and teacher oulput distributions. These distributions, produced by a linear or MLP head, can be seen as soft cluster memberships, where the centroids correspond to the prototypes. Replicating the DINO objective, iBOT proposes to pass the student prodictions through an MLP head and pass the teacher embeddings through the EMA of this head. We argue that in iBOT, in the context of MTM, the clustering interpretation of the head is not valid any more, because the loss is not done between two versions of the same representation. While in DINO both targets and predictions are [CLS] tokens, in iBOT the targets are patch tokens while the predictions are special [MSK] tokens. This can also be seen as distribution mismatch: the MLP head of the student is trained with [MSK] imputs but is instcad applied to rogular patch tokens in the teacher. This mismatch would be even stronger in an asymmetrric architecture as in MAE or I-JEPA, where the largets and predictions come from two different networks (see the split design in fig. 6b). In practice, the iBOT loss is very unstable: without the stabilizing effect of the DINO loss, the iBOT formulation unable to bootstrap itself and results in trivial representations (see table 1c). Our proposition is simple: to come back to valid clustering interpretation and, hopefully, avoid such instabilities, we explicitly use a clustering on top of the EMA network to create the target representations. This decouples the training of the teacher projection from the student's and the training remains stable even in the absence of an additional global loss for stabilization.
Online clustcring. Inspired by the minibatch k-means algorithm and the SwaV loss, we define our online clustering process as follows. Let X E Raad be the output. of the teacher, with row vectors x; for E (I....,n.. We apply an L2 normalization and a linear projection to obtain the assignment logits 4 E IRP:
24
4 = C-
T
where C' is a matrix in Rexd, whose rows are the p "centroids" in dimension d. We then apply a softmax with temperature 7 to obtain soft assignments:
exp(l1/T)
a; = El exp(ln/7))
(2)
We wish to estimate C by solving the following problem:
log a (k)
min
(3)
6

---

Published in Transactions on Machine Learning Research (05/2025)
where dj (k) is the k-th coordinate in ag. By minimizing the entropy, force the assignments to be as close lo one-hot possible, which pushes the centroids towards their "assigned" samples. This can be seen form of clustering with logistic loss. However, simply solving this problem can result in empty clusters. This is a common problem in mini-batch k-means (Sculley, 2010), and is usually solved by adding a reassignment phase: the centroids of the empty clusters are discarded and moved next to another non-empty centroid. In our case, we do not wish the centroids to move that abruptly, as it could disturb the training of the student. Instead, we propose use the Linkhorn-Knopp (SK) algorithm (Sinkhorn & Knopp, 1967), used in SeLa (Asano et al., 2020) and DINOv2 (Oquab et. al., 2024). Using the SK algorithm, we obtain d', an assignment where the distribution of tokens over the clusters is near-uniform:
{ah,
(4)
with 7' another temperature parameter. Note that we do not backpropagate through the SK algorithm. We adapt the loss in eq- (3), and learn the centroids C by minimizing the cross-entropy between and a':
(k) log a
We minimize this loss with AdamW (Loshchilov, 2017) alongside the training of the main model. As the clustering loss and the MIM loss depend two disjoint sets of parameters, we optimize them with two distinct optimizers without interference.
Positional collapse. A crucial point of self-supervised learning is avoiding trivial solutions, i.c. situations where the model minimizes the loss without learning useful features. These failure modes, also known representation collapse, are usually addressed by adding regularization mechanisms. For example, color jittering (Chen et al., 2020b) helps prevent the model from focusing uniquely on color. While training CAPI, we observed a specific type of reprosentation collapse as the positional encoding started outweighing the content of the patch emneddinggs. In the extreme case, the model learns to predict the position of the masked tokens instead of their content, resulting in a zero loss with a trivial model. In most observed cases, both entangled in the target representation, while in the ideal scenario, the
content and position information target would consist purely of semantic features. Although this problem does not arise in all training runs, it can be significant issue when it does. We show in fig. 11 an example of feature maps of some early models affected by this phenomenon.
We propose a simple solution to alleviate the problem. By running the SK separately at each position in eq. (4), we force the joint distribution of tokens over positions and clusters be near-uniform. Uniformity of the joint distribution directly implies zero mutual information between the clustering and the targets. This way, the modified SK ensures that our targets contain positional information.
Pred.
Pred.
Pred.
Predictor
Encoder/predictor
Encoder
Encoder
?
?
?
(a) Fused predictor
(b) Split self-att. predictor (MAE, I-JEPA)
(c) Split cross-att, predictor (CrossMAE, ours)
(BeiT, iBOT, SimMIM, data2vec)
Figure 6: The different predictor architectures discussed in the paper. Here, cach box represents a transformer. The black lines represent the residual stream for a token. (a) In the fused architecture, the predictor and the encoder are the same transformer, which takes as input both the visible and mask tokens, (b) In the split architocture, the encoder processcs only the visible tokens, while the prodictor processes both the visible and the mask tokens. (c) The split architecture can be further refined by removing the visible tokens from the predictor. The predictor processes only the mask tokens, and uses cross-attention to access the context. of the visible tokens.
7

---

Published in Transactions on Machine Learning Research (05/2025)
3.2 Predictor architecture
Model architecture is another important component in reconstruction-basell SSL. Two broad calegories are widely used in previous work. Be'T, iBOT, and SimMIM use a fused architecture: a single vision transformer that takes as inputs patches and mask tokens (fig. 6a). Fused architectures are difficult to train and yield poor results, as reported in previous work (Zhou et al., 2022) and confirmed by our experiments. MAE uses a split architecture, with an encoder that only forwards the patches, saving memory and compute, and a predictor that forwards both patches and mask tokens (fig. 6b).
In this work, we et al., 2024) for pixel-based reconstruction. In this case, the predictor forwards only the mask tokens (fig. 6c), which can access the context of the encoded patches via cross attention. Using a cross-attention predictor has two main advantages. First, it allows further efficinccy gains, as we only forward a reduced set of tokens in the predictor, and we can even subsample this set of tokens. Second, mask tokens do not interact with each other in the cross-attention mechanism, i.e. each prediction independent of other positions. This alleviates the need for multiple predictor forward passes with different prediction sets, used in I-JEPA.
lighter predictor architecture, which was initially explored in cross (Fu
4 Experiments
In this section, report empirical evaluations of our model. We describe experimental details and present some ablation studies. Then we discuss thole-image understanding results and dense prediction results.
4.1 Experimental setup
Pretraining dataset. Most methods from the self-supervised learning literature choose pretrain ImageNet-lk. This dataset is usually chosen because of its relatively small size, allowing for easy experimentation, and the ability to compare to existing methods pretrained on it. However, this has led to an overspecialization of SSL methods to the type of object-centric images found in ImageNet-lk. Recent foundation models obtain state-of-the-ars results by exploiting much larger datasets, such as ImageNet- 22k (Zhou et al., 2022) and LVD-142M (Oquab et al., 2024). If we are to design a method that can produce new foundation models, we believe it is crucial to design to be able to handle such large datasets.
To this end, we carry out all ablation experiments on ImageNet-22kk. It is composed of 14M images from 22k categories taken from the WordNet ontology. Although it is close to ImageNet-IK in nature, its much larger size and diversity make it suitable to train excellent foundation models, as reported by Oquab et al. (2024).
For our longer experiments, we train on multiple datasets: ImageNet- for comparability with previous works, ImageNet-222 to test scaling, Places205, to test training on more diverse and less object-centric data, and finally LVD-142M, a large-scale automatically curated dataset used in previous SSL foundation models. We refer the reader Oquab et al. (2024) for more details on the curation process.
Model architecture. We do all our experiments with a Vision Transformer (Dosovitskiy et al., 2021) of 300M parameters (ViT-L). The model has patch size of 14, a hidden size of 1024, and 24 transformer blocks. Note that in ablation experiments, the patch size used is 16. This architecture is widely used in various computer vision tasks, and most basclines provide model of comparable size. We equip the vision transformer with registers (Darcet et al., 2024). These additional tokens were recently proposed as way to add an information buffer, which enabled the model to produce smoother feature maps. For the predictor, we use 12 transformer blocks that cross-attend to the output of the encoder. This is similar to a standard transformer predictor (Vaswani et al., 2017), with the difference that we do not include self-attention layers. In this predictor, every token is forwarded independently and separately attends to the encoder output. When using a different encoder size, we align the embedding dimension, MLP ratio, and number of attention heads of the predictor to those of the encoder, and use predictor depth equal to half that of the encoder.
Implementation Details. The learning rate follows a linear warmup followed by a cosine annealing. We truncate out the last, 20% of the cosine, proposed in I-JEPA (Assran al., 2023). To simplify the choices
8

---

Published in Transactions on Machine Learning Research (05/2025)
of parameters and schedules, set the teacher EMA momentum to ş 1 Ir, and we set the learning rate for the clustering to half' of the backbone learning rate. The impact of the most important hyperparameters will be discussed in section 4.2. All our pretraining hyperparameters are summarized in table 7.
Evaluation protocol. All the evaluations reported in this paper fall into two categories: image classification and semantic segmentation. For all classification tasks, we use an attentive probe (Assran et al., 2023; El-Nouby et al., 2024; Bardes el al., 2023). We use this evaluation protocol because our model does not. learn a single global image representation, preventing the use of a linear probe. In this evaluation, we train an attentive pooling to extract a global vector and use this vector as input to a lincar layer. The parameters of this probe are trained in a supervised fashion, and we report accuracy on the validation set. For segmentation tasks, we use lightweight classifiers on top of frozen local features. Previous works used a linear head trained with gradient descent on features of images augmented with various augmentations (Oquab et al., 2024). To obtain more lightweight evaluation, we simply extract the features from all images in the dataset without augmentations, then train a linear logistic regression on these features with L-BFGG (Byrd et al., 1995) using an off-tho-shelf library (Raschka et al., 2020). Although this results in lower mIoU numbers, the simplicity of the evaluation allows us to grid over different hyperparaneterrs, producing very robust results. For an even more lightweight classifier, we also consider a non- parammemtt k-NN segmentation evaluation. For each test patch, we retrieve k most similar patches in the training data and pool the segmentation label for that patch. We chose the optimal regularization parameters by doing grid search using 10% of the training set. For all segmentation lasks, we measure performance using mloU.
Baselines. We compare to the performance of previous models trained using missing context prediction: BeiT (Bao et al., 2021), MAF (He et al., 2022), data2vec 2.0 (Baevski et al., 2023), I-JEPA (Assran et al., 2023), and AIM (EI-Nouby et al., 2024). Although AIM is not masked image modcling method, we believe the similarity between masked image modeling and autoregressive modeling makes it a relevant comparison. To provide additional points of comparison, we report in grey the performance of iBOT (Zhou et al., 2022), MTM-refiner (Alkin et al., 2024) (dBOT-refined) and DINOv2+reg (Oquab et al., 2024; Darcet et al., 2024), who use DINO or contrastive loss to complement MIM objective. When multiple model sizes are available, we report the model closest in size to our models (ViT-I./14) In the specific case of MIM-refiner, multiple models were published with different MIM pretraining bases. We select the best-performing model our evaluations, which turns out to be dBOT-refined in all cases. Finally, we include DINO, MoCov3 and MSN (Caron et al., 2021; Chen et al., 2021; Assran et al., 2022) as representative of non-MIM SSL methods.
4.2 Ablation Studies
We conduct extensive ablation studies to study the effect of design choices on performance. To make the ablation study more tractable, we train on the ImageNet-22 dataset for 100k iterations with a patch size of 16. To provide slightly more robust results, the default sctting was run twice with different sceds, and the results of the two runs were averaged. All results are presented in table 1.
Predictor architecture. We evaluate the different predictor architectures. The split self-attention predictor produces better representations while training 32% faster than the fused predictor (table 1a). Using pure cross-attention in the predictor obtains even better representations and allows an additional 18% speedup by avoiding a forward on patch tokens. It also removes the dependency between different predictions, alleviating the need for repeated forwards of the predictor as in I-JEPA.
Masking strategy. The most common masking strategies in the literature are random masking, block masking (inpainting), or inverse block masking (outpainting). Inverse block masking induces bias the position of the masked patches: most often, the model will see the center of the image, and predict the edges. To prevent. this, we propose applying a random circular shift to the mask before using il (+rol1). This ensures that all positions in the image are equally likely to be masked. We ablate the type of masking in table 1b. The masking ratio is 65% for all strategics except random masking, where is set 90%, which increases performance. Random masking is much less effective than the other strategics, and inverse block masking works best, with clear improvement when using +roll.
9

---

Published in Transactions on Machine Learning Research (05/2025)
ADE
IN1k
head
loss
ADE
IN1k
ADE
IN1k
random
23.6
76.4
Ø
IJJPA
23.7
79.3
Fused
23.8
73.1
block
25.6
79.9
MLP
IOOT
1.7
11.1
Split, self-attn
27.9
77.7
inv. block
27.2
80.7
MLP
CAPI
26.4
80.8
Split, cross-atin
29.1
81.4
inv. block +roll
29.1
81.4
Linear
CAPI
29.1
81.4
(a) Predictor architecture
(b) Masking strategy
(c) Loss formulation
ADE
IN1k
ADE
IN1k
depth
width
ADE
TN1k
[0.2, 1.0
27.9
81.4
55%
28.0
81.1
5
1536
30.9
81.5
00.6. 1.0
29.1
81.4
65%
29.1
81.4
12
1024
29.1
81.4
[1.0,1.]
28.9
80.9
75%
28.1
81.2
21
768
28.3
81.3
(d)
Crop range
(e) Masking ratio
(I) Predictor shape
ADE
IN1k
ADE
IN1k
ADE
IN1k
25.9
79.3
learnable 30.0
81.6
Standard
28.5
81.3
16
29.1
81.4
RoPE
29.1
81.4
Proposed
29.1
81.4
(g) Number of registers
(h) Positional encoding
(i) Sinkhorn-Kuopp algorithm
Table 1: Ablation study of the main paramelers and design choices in our algorithm. We report both image segmentation and classification. We highlight the default setting in gray, and bold the best-performing solution. We report ADE20K k-nn segmentation and ImageNet-1k attentive probing classification scores. An in-depth analysis of these results is provided in section 4.2.
Loss formulation. We evaluate the different strategies for computing a loss function discussed in section 3.1. We compare the performance direct Huber loss, an iBOT loss with an MLP head, as well as our clustering- based loss with a linear or MLP head. With no head and direct loss, the model starts well but quickly regresscs to worse representations. The iBOT head does not work in our setup, probably because of the split predictor design (since it implies that target tokens and prediction tokens do not come from the same network). Finally, the proposed clustering head alleviates all these issues and allows for stable training and good representations.
Crop range. To prevent overfitting, we use random cropping and flipping augmentations. We tweak the bounds of the cropping scale to [c, 1.0] and try various We observe that the method can train well without any augmentation, resizing the training images to a fixed 224x224 resolution (80.9 on ImageNet with 1.0). We get better scores with minor cropping augmentation with a range of [0.6, 1.0).
Masking ratio. In our model, we need to set the ratio m of image patches that are masked, and reconstructed. We train our model for various m and check the final performance. The optimal masking ratio seems to be around 0.65, but the algorithm seems quite robust to the choice of this parameter.
Predictor shape. We study the impact of the predictor depth on performance. We train the model with predictors of various depths and adapt the width to match the total number of parameters. Shallow networks will have a larger width. We see that a more shallow predictor leads to better performance, but in our informal experiments, we have observed that such archilectures are less stable long schedules. For this reason, we stick to the predictor with 12 layers and a width of 1024.
Registers. In our model, the local feature maps serve as a supervisory signal to train the model, so high-quality feature maps are crucial. To this end, we use register tokens, which were proposod improve the quality of feature maps. We see that using registers has large effect on performance, with an improvement of 3.2 points on ADE and 2.1 points ImageNet when using 16 registers (table 1g). This is coherent. with
10

---

Published in Transactions on Machine Learning Research (05/2025)
ADE knn
INIk attentive
#prototypes
ADE
IN1k
1024
14.9
73.8
28
dataset
ADE20K
IN1k
2048
19.8
77.4
80
IN
28.0
81.7
4096
27.5
80.7
8192
28.5
81.3
24
78
IN22k
29.1
81.4
IXD-142M
30.6
81.2
16384
29.1
81.4
23
32768
29.1
81.7
25k
506
100k
200K
2.k
50k
100k
200k
Iteration
Iteration
Figure 7: Additional ablation experiments. (Left) Influence of the number of prototypes. (center) Influence of the training length. Each point here is an independent training. (right) Influence of the training dataset.
recent. observations tabular data, for which registers were crucial to learn good representations. (Thimonier et al., 2024).
Positional encoding. In our experiments, we consider using different versions of positional encoding, We try the classic learnable position embeddings well relative ones such RoPE (Su et al., 2024). Because of the case of use and transferability to higher resolutions, settle on using RoPE.
Sinkhorn-Knopp algorithm. We describe the problem of positional collapse in section 3.1. Changing the set of points considered in the Sinkhorn solves it, improving stability and granting small performance increase (table 1i). We did not observe any positional collapse when using it.
Number of prototypes. We evaluate the effect of varying the number of prototypes p in our clustering-based loss. The performance generally increases with the number of prototypes, at the cost of higher memory footprint. We use K = 16384, which strikes a good balance between memory and performance.
Scaling. We study the effect of three scaling axes on the performance of our model: number of parameters, training length, and dataset size. To study the scaling potential of our algorithm, we train additional ViT-S and ViT-B models. We report the performance of the family of models in fig. 1. We see that performance consistently improves with model size, and for all models, our algorithr outperforms the state-of-the-art In fig. 7 (center), we report the effect of the number of training iterations and pretraining dataset on performance. We train models using the ablation configs, on ImageNet-22N) and with patch size 16. In fig. 7 (right), we show the influence of the training data on model performance. Using larger dataset has positive effect on segmentation, while only slightly deteriorating the ImageNet-Ik accuracy.
4.3 Results
Image classification. We evaluate our model and compare it to state-of-the-art reconstruction-base SSL models. We run the evaluation on four datasets including object recognition, fine-grained classification, and scene recognition. We use ImageNet-1k (Russakovsky et al., 2015), iNaturalist 2021 (Van Horn et al., 2021), Places205 (Zhou al., 2017), and SUN397 (Xiao et al., 2010). For ImageNet, we also report OOD robustness by rumning inference on additional test sets: ImageNet-Vi V2 (Rocht et al., 2019), ImageNet-Rect 2020), ImageNet-A (Hendrycks et. al., 2021), and ObjectNet (Barbu al., 2019). For each model, resize the image to 224x224 and collect the patch tokens output by the model. We feed those to attentive probe implemented as single layer of multi-head cross-attention with a single query (head size d/ /64, no residual). For classes and an embedding size d, the probe contains 2d + (3 + c)d parameters, which are trained with AdamW for 10 epochs, selecting the best learning rate for each model/tast on a held-out split of the training set. More details on the protocol are available in appendix H. All the results are summarized in table 2.
(Beyer et al.,
We see that our model outperforms all previous state-of the-art models, by large margin. When training ImageNet-1k, observe very good results, outperforming all other reconstruction-based models of
11

---

Published in Transactions on Machine Learning Research (05/2025)
ImageNet
Model
Arch.
Dataset
val
ReaL
v2
A
ObjectNet
iNat
Places205
SUN3977
DINO!
VTT-B/16
INIK
76.9
83.6
65.3
14.3
34.8
64.1
59.0
61.2
MoCov3"
VIT-B BB16
INIk
76.6
83.1
65.3
14.0
35.9
61.4
57.8
63.2
MSN
VIT-L/16
INIK
70.8
78.7
59.3
12.1
27.7
3.6
40.7
60.5
IBOT
VIT-L/116
INIK
80.9
86.5
70.3
41.9
28.9
70.5
62.0
64.6
dBOT-refined
VIT-L/116
INIK
82.8
87.9
73.4
44.7
49.0
72.9
63.2
68.7
I-JEPA
VIT-I1/14
INIK
79.5
85.3
68.7
37.0
22.6
64.2
59.9
61.9
MAE
VIT-L/16
INIk
79.4
85.3
68.9
38.3
19.3
69.3
60.6
61.8
Duta2Vec 2.0
VIT-L/16
INIk
80.1
85.7
69.4
42.1
24.6
65.4
61.9
64.4
CAPT
VIT-L/14
INIk
82.9
87.6
72.9
47.5
43.7
76.8
65.4
70.9
DINOv2+reg (distilled)
VIT-L/14
LVD-142M
86.5
89.9
79.1
75.5
64.5
86.6
08.3
79.6
DINOv2+reg
VIT-g/14
LVD-142M
87.4
90.3
80.1
67.0
81.7
88.3
08.8
79.3
BelT
VIT-L/16
IN22k
40.8
46.1
30.7
8.7
2.1
26.5
36.8
29.9
I-JEPA
VYT-H/14
IN22k
78.1
84.3
67.4
38.6
25.1
67.7
60.2
65.5
ATM
VIT-600M/14
DFN-2B+
79.0
84.8
67.9
41.0
20.4
73.5
62.5
66.1
CAPI
VIT-L/14
IN22k
83.6
88.1
74.3
55.2
52.4
82.0
66.3
74.5
CAPI
VIT-L/14
Places205
79.2
84.7
68.4
39.1
33.0
73.4
68.6
77.5
CAPI
VIT-L/14
LVD-142M
83.8
88.2
74.8
55.3
56.8
81.2
67.1
75.6
Table Image classification results. We train an attentive probe on top of frozen features. For each baseline, we report the model closest in size ViT-L/14. We separate models trained on ImageNet- 1k only from models trained on larger dataset. We report top-1 accuracy on all datasets, and for ImageNet we report some additional test sets. For reference, we report in grey the performance of other SSL models using DINO or costrastiv ive losses. This shows that CAPI narrows the gap using only a MIM approach. t: Non-MIM SSL approaches. f; We report linear probing results (CLS token only) as they are better than attentive probing.
comparable size. CAPI excels particularly on out-of-distribution generalization, outperforming all basclines by more than 19 points on ObjectNet. Additionally, while the gap with other methods is somewhat limited ImageNet, the difference in scene classification (SUN397) is much larger. Compared to iBOT, which uses DINO loss to stabilize MIM loss, CAPI performs better across the board. MIM-refiner, which uses a contrastive loss to refine pretrained MIM model, obtains great scores when applied to dBOT. Our CAPI is comparable on most. ImageNet, benchmarks (+0.1 on IN-val, -0.3 on IN-ReaL, -0.5 on IN-v2) while being slightly better on ImageNet-A (+2.8) and falling behind on ObjectNet (-5.3). CAPI however significantly outperforms MIM-refiner out of domain, with gap of 3.9, 2.2 and 2.2 points on iNaturalist, Places and SUN respectively.
Interestingly, observe that CAPI works particularly well when trained larger and more diverse datasets. In second category, compare models trained on datasets larger than ImageNet--1k Our three CAPI models outperform all baselines on all datasets, except the CAPI-Places205 which is slightly below AIM on ImageNet-A. It should be noted, however, that ATM was trained on more than 2 billion images, with a sampling distribution tailored towards ImageNet. When trained on bigger datasets, CAPI significantly reduces the gap between reconstruction- based methods and our topline DINOv2+reg: the gap on ImageNet. goes from 8.4 points to 3.6, and on SUN397 goes from 13.2 1.8.
Dense image understanding. As we have seen above, our model allows training high-quality local features that can be successfully pooled solve imago-level tasks. We also want to evaluate our model on densc prediction problems such image segmentation. To this end, we run k-NN and linear segmentation following the protocol described in the experimental details. We run this for all the baselines reported above three datasets. We use ADE-20k (Zhou et al., 2017), Pascal VOC 2012 (Everingham et al., 2010), and Cityscapes (Cordts et al., 2016). We report mloU for all configurations in table 3.
As in the classification evals, CAPI trained on ImageNet- 1k outperforms all reconstruction based baselines by quite a wide margin on all evaluation setups. Compare to dBOT-refind,, interestingly, CAPI obtains similar score in linear probing while having worse scores in k-nn probing. It possible that the contrastive refining process is helpful to obtain good k-nn score. When training larger datasets, the conclusion is similar:
12

---

Published in Transactions on Machine Learning Research (05/2025)
ADF-20k
Pascal-VOC
Cityscapes
Model
Arch.
Dataset
k-NN
linear
k-NN
linear
k-NN
linear
DINO
VIT-B/16
INIK
20.1
25.2
48.8
56.1
32.0
35.4
MoCov3!
VIT-B/16
IN1k
21.8
26.6
54.4
62.2
31.6
34.7
MSN
VIT-L/16
INIK
18.3
22.7
17.4
55.8
28.8
34.2
IOOT
VIT-L/16
INIK
26.0
30.7
60.2
68.8
35.7
39.8
dBOT-refined
VIT-L/16
INIK
30.8
32.8
66.7
70.2
38.4
41.7
1-JEPA
ViT-H/14
IN1k
20.8
25.7
56.7
63.6
26.4
34.5
MAE
ViT-I,/16
IN1K
21.5
27.4
53.7
61.5
32.8
38.5
Data2Vec 2.0
VIT-L/16
INIK
24.2
27.6
57.5
58.1
32.8
38.2
CAPI
VII-L/14
INIk
29.2
34.4
60.7
69.7
35.6
41.7
DINOv2+rreg (distilled)
ViT-I/11
IVD-142M
34.8
38.3
66.7
71.3
42.3
45.9
DINOv2+reg
VTT-g/14
LVD-142M
34.0
39.0
63.0
72.8
42.0
46.8
BefT
VIT-L/16
IN22k
3.5
8.3
6.9
19.1
15.6
24.0
I-JEPA
VIT-H/14
IN22k
18.9
26.3
55.0
64.2
23.2
34.2
AIM
ViT-600M/14
DFN-2B+
31.4
60.2
67.0
32.1
CAPI
VIT-L/14
IN22k
29.7
35.2
61.1
70.4
35.2
41.0
CAPТ
ViT-I/114
Places205
35.2
39.1
61.7
69.4
39.5
44.6
CAPT
VIT-L/14
IVD-142M
32.1
37.2
63.8
72.7
38.9
44.3
Table 3: Semantic segmentation results. We train a k-NN or a logistic regression probe on top of frozen features. For each baseline, we report the model size closest to ViT-L/14. We separate models trained on ImageNet--k only, and the ones trained on a larger dataset. We report mloU on all datasets. For reference, also report the performance of some other SSL models using DINO contrastive losses complement the MIM loss. This shows that CAPI narrows the gap using only MIV approach. i; Non-MIM SSL approaches.
CAPI outperforms all basclines by a wide margin and even beats DINOv2+rog in some setups. When trained on Places205, CAPT achieves mIoU 1.2 points higher than DINOv2+reg on ADE20K k-nn segmentation. To our knowledge, this the first time DINOv2+reg is bested on segmentation with frozen features on ADE20K. DINOv2+rog, however remains the most versatile model, with good results across the board and the best results on Cityscapes.
4.4 Additional explorations
As final set of experiments, we investigate some additional propertics of our model. We investigate its robustness to change of input resolution and try to obtain global representations using the predictor.
High-rosolution image understanding: Our model was trained on 224 x 224 images. To compare with the I-JEPA model trained natively at high resolution, we try to evaluate our model on 448 x 448 images. In table 4, we see that our model does not require high-resolution training or evaluation to achieve the best performance. Our model trained at 224 and evaluated at 224 outperforms the large I-JEPA model trained 448 and evaluated at 448. Moreover, using RoPE embeddings, CAPI more robust to resolution changes: it loses only 0.3% (versus 1.0% for I-JEPA) when increasing the resolution.
Method
train res.
eval@224
eval@44t
I-JEPA-H
224
79.4
78.4
I-JPA-H
448
79.6
82.5
CAPI-L
224
83.8
83.5
Table 4: ImageNet-lk 1k attentive probing accuracy of I-JEPA and CAPI at different input resolutions.
13

---

Published in Transactions on Machine Learning Research (05/2025)
CAP1
DINOv2+00g
Data2Ncc
I-EEA
Figure 8: Visualization of the features of CAPI and baseline models. We apply a PCA to the features and map the three first components to RGB. The features produced by CAPI are discriminative and smooth.
IN1k
iNat21
SUN397
Representation
k-NN
Lincar
Lincar
Lincar
avg. pooling
57.1
77.1
49.1
73.3
predictor pooling
73.8
81.1
69.6
77.4
Table 5: Classification using predictor representations, compared to the average pooling of the patch tokens.
Qualitative feature analysis. We propose a qualitative assessment of the dense features in fig. The dense features computed with CAPI are amongst the most discriminative and smooth. We see the emergence of distinct objects, without much noise in uniform regions. We observe that the CAPI features are less noisy than the ones from DINOv2+reg, while being more focused on semantics and less colors than the MAE features. For example, the shaded building in the first image has CAPI features similar to the other buildings, while in MAE the features are closer to other dark areas of the image.
Obtaining global reprosentations. In all the previous evaluations reported in the experimental section, we trained a head on top of local features. Our model does not provide an aggregate representation like the [CLS] token in DINOv2. In this experiment, we try to exploit the predictor to obtain global image representations. We forward the whole image through the encoder, and then pass the same amount mask tokens through the first attention layer of the predictor, cross-attending to the patch tokens. We obtain a global representation by average-pooling the output of this predictor altention layer. We train a linear model this representation several classification datasets and compare it with the average pooling of the patch embeddings in table 5. We see that using the attention pooling learned by the predictor provides better representations than averaging local features from the encoder.
5 Broader impact
As with any machine learning model, the impact of our work depends on how it used. This work particular focuses on improving the quality of self-supervised visual features, which can pose specific privacy issues if model is trained on identifiable pictures, and pose fairness issues if the model absorbs biases from the dataset it is trained on. Additionally, the training of models and the related exploratory experiments
14

---

Published in Transactions on Machine Learning Research (05/2025)
directly use energy and computational resources, which can have environmental impact. We discuss environmental impact in more detail in the appendix.
6 Discussion and Concluding Remarks
In this paper, we have proposed novel reconstruction based self-supervised learning algorithm. Our algorithm is based on an online clustering of dense features computed with a teacher network. The latent assignments are used as largets to train the student. We propose to implement the student as an encoder followed by a cross-attention predictor. The teacher is updated as an EMA of the encoder. The proposed algorithm is simple and allows the training of a state-of-the-art model in the category of reconstruction based approaches. Our ViT-L outperforms all available reconstruction-based models, including much larger architectures. We have shown promising scaling trends until the 300M model sizes of the ViT family, opening up potential for further scaling in future work. Some questions remain open: in particular, we believe it would be interesting to explore the scaling capabilitics beyond the scale we explored, and to study the optimal training dataset for this sort of training. The question of whether such pure reconstruction- baase approach can reach the performance of contrastive learning methods remains open. We hope that our work will inspire further research in this direction.
Acknowledgments
We thank Momo for fig. 8. Julien Mairal was supported by ERC grant mumber 101087696 (APHELEIA project).
References
Benedikt Alkin, Lukas Miklautz, Sepp Hochreiter, and Johannes Brandstetter. Mim-refiner: A contrastive learning boost from intermediate pre-trained representations, 2024.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020. 3, 5, 7
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022. 9
Mahmoud Assrn, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. CVPR, 2023. 2, 4, 8, 9, 25
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022. 4, 25
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In ICML, 2023. 9
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021. 4, 9, 19, 25
Amir Bar, Florian Bordes, Assaf Shocher, Mido Assran, Pascal Vincent, Nicolas Ballas, Trevor Darrell, Amir Globerson, and Yann LeCun. Stochastic positional embeddings improve masked image modeling. In ICML, 2024. 4
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.
NewrIPS, 2019. 11
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yanı LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning, 2023. 9
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? CoRR, abs/2006.07/2 07015, 2020. URL https: //arxvv...///////aaeaa org/abs/2006. 07159. 11
Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second, 2024.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained oplimization. SIAM, 1995. 9, 22
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In BCCV, 2018. 3,
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020. 3, 4
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers. In ICCV, 2021. 3, 4, 9, 19
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML, 2020a. 4
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose loundation model for computational pathology. Nature Medicine, 2024. 1
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b. 3, 7
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Euzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 12
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In JCLR, 2024. 5, 8, 9, 25
Patrick Dermyer, Angad Kalra, and Matt Schwartz. Endodino: A foundation model for gi endoscopy, 2025. 1
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; pre-training of deep bidirectional transformers for langnage understanding. CoRR, 2018. 1,3
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 4, 5,
Johan Edstedt, Qiyu Sun, Georg Bökman, Märten Wadenback, and Michael Felsberg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19790-
19800, 2024.1 1
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large antoregressive image models. In ICML, 2024. 9, 22, 25
Mark Everingham, Lue Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 2010. 12
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: visual representation for neon genesis. Image and Vision Computing, 2024. 2
Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders, 2024. 5, 8
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Spyros Gidaris, Audrei Bursuc, Oriane Simeoni, Antonin Vobecký, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Moca: Sell-supervised representation learning by predicting masked online codebook assignments. TMLR, 2024. 4
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 2, 3, 4, 9, 19
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 11
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hon, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. TPAMI, 2023. 3
J. D. Hunter. Malplotlib: A 2d graphics environment. Compuling in Science & Engineering, 2007. 26
Diederik P Kingma. Adam: A method for stochastic optimization. In ICLR, 2014. 22
Yann LeCun. path towards autonomous machine intelligence, 2022. 2
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Cheugda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report, 2024. 2
Loshchilov. Decoupled weight decay regularization, 2017. 7, 22
maintainers and TorchVision contributors. Torchvision: Pytorch's computer vision library. https://github. htm/ pytorch/vision, 2016. 22
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR, 2020. 3
Théo Moutakanni, Piotr Bojanowski, Guillaume Chassagnon, Céline Hudelot, Armand Joulin, Yann LeCun, Matthew Muckley, Maxime Oquab, Marie-Pierre Revel, and Maria Vakalopoulou. Advancing human-centric ai for robust x-ray analysis through holistic self-supervised learning, 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. preprint wrXX=:1807.03745 2018. 3
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024. 2, 3, 4, 7, 8,
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting, In CVPR, 2016. 3
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training, 2018. 1,3
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot Lext-lo-inage generation. In ICML, 2021.
Sebastian Raschka, Joshua Patterson, and Corey Nolet. Machine learning in python: Main developments and technology trends in data science, machine learning, and artificial intelligence, 2020. 9, 22
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Hnang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015. 11
Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammmad Thoker, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Sigma: Sinkhorn-guided masked video modeling. In ECCV, 2024.
17

---

Published in Transactions on Machine Learning Research (05/2025)
David Sculley. Web-scale k-means clustering. In WWW, 2010. 7
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 1967. 5, 7
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 2024.
Hugo Thimonier, José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, and Bich-Liên Doan. T-jepa: Augmentattion-free self-supervised learning for tabular data. arXie preprint arXie:2410 05016,, 2024. 11
Jamie Tolan, Hung-I Yang, Benjamin Nosarzesski, Guillaume Couairon, Huy V Vo, John Brandt, Justine Spore, Sayantan Majumdar, Daniel Haziza, Janaki Vamaraju, et al. Very high resolution canopy height maps from rgb imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar. Remote Sensing of Environment, 2024. 3,
Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised poinl tracking in a single video. In ECCV, 2025.
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In CVPR, 2021. 11
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017. 4, 8
P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, 2008.
Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine, 2024. 1
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010. 11
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiu, 2021. 4
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 2021. 3,
Lihe Yang, Bingyi Kang, Zilong Huang, Xiangang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a.
Like Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2, 2024b.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In ECCV, 2016.
Xinyu Zhang, Jiahui Chen, Junkun Yuan, Qiang Chen, Jian Wang, Xiaodi Wang, Shumin Han, Xiaokang Chen, Jimin Pi, Kun Yao, et al. Cae v2: Context autoencoder with clip target, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba, Scene parsing through ade20k dataset. In CVPR, 2017. 11, 12
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022. 2, 3, 4, 8, 9, 19
18

---

Published in Transactions on Machine Learning Research (05/2025)
ADE knn
ADE linear
mioU
mloU
oudent
texcher
INlk avg. pool knn
INlk avg. pool logreg
top-1 acc
top-] ace
100k
200k
200k
400k
500k
100%
200%
300%
4001
500K
100k
2H0
3001
4000
500K
Iteration
Truii iter.
Train iler.
Figure 9: The loss curve of our CAPI VIT-I during training.
Figure 10: Comparative downstream scores of the teacher model and the student model throughout train- ing.
A Loss curve
We report in fig. the loss curve of our CAPI VIT-L model. After an initial adjustment period, the loss trends smoothly downwards, with no sign instability or plateauing. Compared to other latent masked image modeling methods such I-JEPA iBOT, this trend reassuring, and might indicate good potential for further scaling.
B Blockwise masking strategy
The so-called "block masking" strategy used in many masked image modeling methods is by no means standardizes and can actually refer to several different implementations. The most common block masking implementation was proposed in BeiT (Bao et al., 2021), then reused iBOT (Zhou et al., 2022) and MAE (He et al., 2022). involves sampling many rectangular regions and doing multiple attempts to mask out approximately the right number of patches. Another implementation was proposed in I-JEPA, adding multiple constraints on the masks, to obtain similar multi-block mask. Additionally, some methods postprocess the proposed mask to obtain a constant number of masked patches, in order to keep the same sequence length in all batch elements.
In CAPI, we propose simpler heuristic; we sample a single rectangular mask, and truncate out the excess patches at the lower right end. Conversely, our implementation of inverse block masking to sample a block mask, then simply invert it.
C Batch size
In all our experiments, we batch size of 16384. Although the memory footprint is slightly offset by the high proportion of tokens not processed by the encoder and predictor, recognize that this may be limiting factor for some users. For the ViT-I, this batch size fits in nodes of A100 80GB GPUs. We believe that the batch size could be reduced to 4096 without significant loss in performance, but the recipe may need adjustments, and we do not have results at smaller batch size.
D Self-distillation interpretation
It was observed in DINO (Caron al., 2021) that the downstream scores of the EMA model were consistently higher than the ones of the online model during training. This led to the interpretation of DINO
19

---

Published in Transactions on Machine Learning Research (05/2025)
IN1k
Representation
Learnable head
val
ReaL
V2
A
R
Sketch
Objectnct
avg pooling
Linear
76.8%
82.5%
64.4%
21.8%
43.5%
33.9%
33.7%
pred pooling
Linear
81.8%
86.9%
71.6%
46.7%
55.8%
47.0%
48.3%
cat(pred pooling, avg pooling) feature map
Linear
81.9%
87.0%
71.5%
45.8%
55.6%
46.5%
47.1%
Attentive
83.8%
88.2%
74.8%
56.5%
58.8%
47.9%
54.9%
Table 6: Comparison of linear probing scores on ImageNett using different representations and classification heads. In the first three rows, the pooling is fixed function, while in the fourth row, the pooling is Iearned specifically or ImageNet via an attention layer. We observe that, while the average pooling of CAPI does not produce a great global vector, the predictor pooling trick allows CAPI to be competitive on the ImageNet linear probing benchmark. The pooling, however, is not the optimal pooling for ImageNet since an attentive pooling manages get better score.
input
PCA
PCA 2
PCA 3
PCA
PCA 5
PCA 6
Figure 11: PCA visualization of the patch embeddings of some early CAPT models, showing partial positional collapse. The models did learn some useful nontrivial features (around 60% accuracy on ImageNet), but the positional information was very present and dominated the trainingn, preventing the model from learning better features.
self-distillation method, where the EMA model, the 'teacher' distilled its slightly better represcntations into the online model, the "student". We observe that this interpretation still seems to hold in CAPI, albeit to a lesser extent, evidenced by the comparison of teacher and student performance in fig. 10.
E Predictor pooling versus attentive probing
We report in table 6 the comparison of linear probing scores ImageNet using different representations, with the topline being the attentive pooling results. We observe that the predictor learns much better pooling than the baseline average pooling, although it does not reach the topline whore the pooling is supervised (attentive probing).
F PCA visualizations of early models suffering positional from collapse
We show in fig. 11 the PCA visualization of the patch embeddings of an early CAPI model that showed some partial positional collapse.
20

---

Published in Transactions on Machine Learning Research (05/2025)
3 M: Tensor b d
M: Tensor
# b
4 M = N. ex
M = M. M.exp()
for in range(3)
for range (3)
6- M /= sum dim= (0, 1)
6+
susum (dim=0)
7
M / sum dim=2)
M /= sum (dim=2)
(a) Standard SK
(b) Modified algorithm
Figure 12: PyTorch pseudo-code for the proposed modified Sinkhorn-Knopp algorithm. We normalize by the sum of the tokens for every given position, instead of normalizing across all positions.
Hyperparameter
Value
Batch size
16384
Optimizer
AdamW
Learning rate
1e-3
Teacher momentum
Clustering Ir Ir schedule
1r
linear warmup + Irunc. cosine
Warmup length
10%
cosine truncation
20%
Weight decay
0.1
AdamW /
(0.9, 0.95)
Number of prototypes
16384
Student temperature
0.12
Teacher temperature
0.06
Num SK iter
3
Stochastic depth
0.2
Weight init
xavier uniform RMSnorm
Norm layer
Norm E
1c-5
Patch embed Ir
0.2 Ir
Norm layer wd
0.1 wd
Image size
224
Angmentations
RRCrop, HFlip bf16
Training dtype
Parallelism
FSDP
Pred. / im
7
Layerscale
No
Biases
No
Rope frequencies
(7epa 7), axial inverse rloc
Masking type
Masking ratio
65%
Table 7: CAPI pretraining recipe
G Modified Sinkhorn-Knopp
We provide the pseudo-code for the standard Sinkhorn-Knopp and for our modified version in fig. 12. Both the original code and the proposed change are very simple. The actual code additionally contains an initial additive shift prevent numerical instabilities in the exponential, as well as collective alduceduce for distributed training.
21

---

Published in Transactions on Machine Learning Research (05/2025)
H Detailed evaluation protocol
In all cases, evaluations are performed with frozen model, and use only the patch tokens outputted by the vision transformer. The input images are always at resolution 224x224.
H.1 Classification
The backbone model is kept frozen, and extract only the patch tokens from its output. On top of these features, train attentive pooling classifier, consisting of learned query, two learned k anv projections, and a final projection to the number of classes. The attention is multi-head, with the head dimension being fixed at 64 and the number of heads being Amugted This head is optimized with a cross-entropy loss and the AdamW optimizer (Kingma, 2014; Loshchilov, 2017) for 12500 iterations at batch size 1024 (10 ImageNet epochs). The learning rate is warmed up linearly for 1250 iterations then annealed with a cosine schedule. We grid the weight decay over (5e-4, 1e-3, 56-2) and the peak learning rate over (1e-5, 2c-5, 5e-5, 1c-4, 2c-4, 5e-4, 1e-3, 2e-3, 5e-3, 1e-2), training one classifier head for each pair of hyperparameters (30 in total). We choose the optimal hyperparameters using the accuracy finally report the accuracy of this classifier on the test set. The training dataset is lightly augmented using a torchvision RandomResizedCrop (maintainers & contributors, 2016) with default hyperparameters and a random horizontal flip. During evaluation, the images are resized to 256 then center cropped to 224x224 pixels.
10% held-out part of the training set, then
H.2 Segmentation
We compute the features for the train and test set considered at resolution 224, and hold out 10% of the train set as a validation set. Using these frozen features, the segmentation problem is reduced to a simple classification problem, on which we can use simple k-NN and linear classifiers. The linear classifier is trained for logistic regression with L-BFGS (Byrd et al., 1995) regularized with L2 penalty, as implemented in the cuml library (Raschka et al., 2020). In both cases, a grid of hyperparameters tested, and the ones performing best the validation set are retained. For the k-NN classifier, grid the number of neighbors over (1, 3, 10, 30), and the distance used over (L2, cosine). For the linear classifier, grid the regularisation parameter C, testing 8 values along a log-space between 10 and 10
H.3 Feature standardization
Some of the baselines suffer from poor conditioning of their features, which cause very bad results when fitting a logistic regression over these features. To reduce this issue, in the segmentation evaluation we standardize features by substracting their mean and dividing them by their standard deviation. These statistics are computed using the features from the training set only. This significantly improves the scores of pixel reconstruction-bat based methods, while the other mothods are mostly unaffocted. We report a comparison of segmentation results with and without standardization in table 8. In the rest the paper, all segmentation results are obtained with standardization.
H.4 Baselines
All baselines are vision transformers, allowing us to use the same evaluation protocol. We feed the 224x224 image to the model after imagenet normalization of the pixel values, and extract the patch tokens after the last transformer block. For the specific case of AIM (El-Nouby al., 2024), we follow the advice from the original paper and extract the patch tokens after before the end of the ViT, specifically after layer 18.
Compute cost and environmental footprint
We mcasure the training of a CAPI VIT-L model to take 180h on 32 A100 GPUs, amounting to 5763 A100 hours. This consumed around 2651 kWh of electricity, which we estimate amount to approximatcly 928 kgCO2eq. The entire project used 3. 3.75M A100 hours, which we similarly estimate to have emitted 604
22

---

Published in Transactions on Machine Learning Research (05/2025)
knn
logreg
model
standardization
ADE
Cityscapes
VOC2012
ADE
Cityscapes
VOC2012
CAPI
Falsc
32.5
39.2
64.9
37.9
44.7
73.2
CAPI
True
33.0
39.2
65.2
37.7
44.3
73.3
aim 600M
False
14.3
27.8
38.5
7.1
28.3
61.3
aim 600M
Truc
nan
32.1
60.2
31.5
38.2
67.0
dinov2 vitgl4+reg
False
33.8
42.0
63.1
38.9
46.8
72.9
dinov2 vitg14+reg
True
34.0
42.0
63.0
39.0
46.8
72.8
ijepa vith14 inlk
False
20.1
25.9
57.4
25.2
33.5
63.0
ijepa vith14 inlk
True
20.8
26.4
56.7
25.7
34.5
63.6
jeppa vith14 in22k
False
17.9
22.9
56.2
24.1
32.8
63.6
ijepa vith14 in22k
Truc
18.9
23.2
55.0
26.4
34.2
64.2
mae vill16
False
7.8
25.3
17.3
27.4
38.4
61.5
mae vitl16
True
21.5
32.8
53.7
27.4
38.5
61.5
Table 8: Comparison of segmentation results with and without standardization
Table 9: Summary models mentioned in the paper.
We associate to each a unique uid, and detail the hyperparameters which are not constant across all runs.
uid
dataset
depth
Ir
muciiigg
ratio
roll
leacue heal
land
loss
scalo
BVD-1422
G006
66%
Tru
mudilaad
[0015
LVD-142M
500G
708
66%
Thu
modilied
[0055
chatering
mge
100G
Thise chatering
chatering
10ge
LONG
16
nusdom
50%
Faha: chaterias
2000
A9hda
101R
16
1091
9--03
0:500
1-:33
bork
0536
Falle chaterrng
E/nar
10050
AZzdl
IN223
16
24
1024
1024
2:-03
0.596
10.03
Rov. bloc
Fabe chateria
Lince:
CE
1000
modited
10
(0155 10075
A3bbd
IN22X
10TE
16
24
12
1924
1024
10103
0.506
16:00
Rev. blok
15%
Thue Sdatitty
ideasity
Habes
1000
standacd
10
(0155
A265
1N221
100G
16
24
12
1024
1021
2.-03
0.506
1c-03
lsv. block
66%
The EMAA
MLP
CE
2000
mudiaal
16
10009
Arb-8
IN22
100R
16
3.8
12
1091
1091
9--11
0.500
1-:03
Irv. block 0036
Thee chatering
MLPP
CP
rage
madtifee
10
10055 1009
A745
IN221
16
24
12
1924
1024
2:311
0.596
1::03
Icy. block
6536
The chateria
Lince:
CE
poge
modited
16
DOINS 1000
A6763
IN221
16
24
12
1924
1021
21:11
0.596
1::0
Iey. block 853
Thoe chiserrng
Linee:
CE
Dage
moditee
10
2015, 100/5
A4188
IN22L
100G
16
24
12
1924
1021
26-03
0.506
1c:00
lov. block
55%
The chatering
Limaz
CE
muditid
16
(6155 1000
Arthe
1001
1-40
block
153
Thon chatering
Liner
CF
modiffed
1015,
AFSSO
65%
The clusering
modiled
(015, 1007
A2dul
703
85%
Thee chutering
modifod
(015, 1007
ASk18
66%
mudilas
(015
chatering
A723
24
1024
lsy.
The diateria
2000
standaed
16
loy.
chittria
1090
10
INUk
RING
24
5-01
lsy.
65%
Thu chatering
mudiliad
16
1007
MS31
P205
14
24
1954
1091
1e401
0:58
S-:01
Iny. blck
183
Thoe chatering
Linoe
CB
rage
moditfed
10
1015) 1009
Abe00
18223
23k
16
24
12
1024
1024
2:003
0.506
1-:01
2sy. blck
66%
The disteris
Liare:
CE
1000
modifad
16
Add44
1N225
16
24
12
1024
1024
2:30
0.596
1::03
Rsy. back
15%
The dizterang
Liuces
CE
1000
modided
16
10055,1005
A2ta8
IN222
2706
16
24
12
1021
1021
2--13
0.506
1-00
1sy black
66%
Tru chatering
Limar
CE
mudiial
16
(005.10005
Aahi4
INIK
16
24
12
1024
1021
2--11
035
1-40
RoV. block
103
Thee chatering
Lineer
CB
moditfe
16
1005,100/5
An425
LVD-1123
1002
16
24
12
1924
1024
26-03
0.596
1::03
1oy. baock
65%
The dlaserrng
Lintez
CE
modildl
16
(00%)10005)
cCO2eq for the electricity consumption. Note that the carbon footprint estimations here are purely scope 2 estimations, i.e. limited to electricity consumption, and are further limited to the electricity consumption of the GPUs. A full carbon accounting should additionally include many other harder to estimate emissions, such as the electricity consumption of the other server components and the rest of the datacenter appliances, and scope 3 emissions from the component manufacturing, datacenter construction, and their respective end-of- life. 2
J List of models used
We provide in table the list all models presented in this paper, along with a unique hash identifier and the relevant hyperparameters. Non-listed hyperparameters are detailed table 7. To disambiguate any possible unclarities in the presented results, table 10 provides the mapping from tables and figures to model identifiers.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Table 10: Reference of models used in different figures and tables.
Fig
Models used
fig. 1
Meb2b, Mc2dd, M8c4d
table la
Adcab, Ae319, Aa5a3, A7d26
table lb
Adcab, Ae319, A0dd4, A9b4a, A7cc0
table lc
Adcab, Ac319, A3bb3, A2fcb, Acb48
table ld
Adcab, Ae319, A7419, Ae7b3
table
Adcab, Ae319, A41b8, AcSbe
table 1f
Adcab, Ac319, Af989, A2da0
table lg
Adcab, Ac3f9, A9ce8
table 1h
Adcab, Ae319, A1177
table li
Adcab, Ae319, A72fb
table 2
Meb2b, M5c2e, M2d34, M8319
table
Meb2b, M5e2e, M2d34, M8319
table
Meb2b
fig. 8
Meb2b
table
Meb2b
fig. 7
Adcab, Ac319, Abc05, Acd44, A2ca8, Adcab, Ac319, Aab94, Aa428
fig. 10
Meb2b
fig. 14
Meb2b
fig. 13
Meb2b
K
Visualisations
In fig. 13 and fig. 14, we provide visualisations of the feature maps of CAPI compared to other state-of-the-ane self-supervised vision models.
24

---

Published in Transactions on Machine Learning Research (05/2025)
CAP1
DINOv2+reg
MAB
DataZVec 2.0
AIM
IJEM
Boulouard FRANKIZ
Bealouard FRANKIZ
STERE
FROTIC
Figure 13: Visualization of the features produced by CAPI and other vision models at various resolutions: CAPI VIT-L/14, DINOv2+rog ViT-g/14 (Darcct et al., 2024), BETT VIT-L/16 (Bao et al., 2021), AIM VIT-3B 14 (El-Nouby et al., 2024), MAE VIT-H/14 (El-Nouby et al., 2024), I-JEPA ViT-H/14 (Assran et al., 2023), and data2vec2 VIT-I./16 (Baevski et al., 2022). We apply a PCA decomposition to the dense outputs produced by each model for each image individually, and rescale the three first components to the RGB range for visualization.
25

---

Published in Transactions on Machine Learning Research (05/2025)
Input
PCA
Channel Channel
Channel 2
Channel 3
Channel 4 Channel 5
Channel 6 Channel
Figure 14: Visualization of the features produced by CAPI ViT-L/14 applied to images at 560 pixel resolution. We apply a PCA decomposition to the dense outputs produced by the model across all images. The first. column shows the first 3 components as RGB, The next eight columns show the first eight channels individually using a coolwarm colormap from Matplotlib (Hunter, 2007).
26


Reviews:
[{"review_id": "eATNA2Z332", "reviewer": "Reviewer_ogTS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The results of this MIM approach for k-NN and linear evaluation are promising.\n- The paper is well written and has good motivation in the introduction.\n- Proposed approach seems stable across hyperparameters like crop range, etc.\n- Detailed hyperparameters are released to allow for the reproduction of the work.\n\nWeaknesses:\n1.  **Unclear effectiveness of method**: The paper directly compares to previous works, e.g. MAE, in a linear and kNN setup. However, this paper introduces many minor things, e.g., changing masking strategy, using registers, predictor shape, and predictor architecture. From the ablations (Tab 1. ) the biggest improvements one can observe are: predictor architecture + 6 %, masking strategy 5.5%, registers + 3.2%. These are minor details (including predictor pooling), which I acknowledge to be important, but make the comparison with previous works like MAE impossible. The paper frames to be about cluster and predict, so how impactful is the cluster and predict part actually w/o the add-ons? How would the model perform if it replaced the sinkhorn loss with a normal L2 loss? How would MAE perform with this masking strategy, registers, predictor architecture, etc.? Without these results, one can't tell the effectiveness of the proposed approach. \n\n2. **Missing full finetuning results:** It is commonly known that MAE performance for non-full finetuning achieves limited performance. As this paper introduces an improved MIM approach, it also needs to compare with the previous MIMs in the full finetuning setting to see if it also improves there or at least does not degrade.\n\n3. **Missing discussion to highly relevant works**: The paper does not cite nor acknowledge the contributions from previous works that \nsuch as: \n   -  [1] Masked Siamese Networks (MSN) use \"cluster and predict\" like this paper, but on a course (CLS token), not a dense level for MIM. This paper can be phrased as an extension, just in a dense manner.  \n    - [2] Sela: the first paper introducing predicting Sinkhorn assignments for SSL that has been later used in methods like DINO and SWAV. As the whole approach relies on this method, it should be referenced and acknowledged. \n   - [3] SIGMA is very similar in idea (see title of [3]), predicting Sinkhorn-Knopp cluster assignments in the dense feature space for masked video modeling. As far as I see, [3] does not use a symmetric architecture like in this paper, which uses a symmetric one with EMA, thus being more elegant. *However, the overarching idea of cluster and predict in a dense manner for masked modeling has been implemented and proposed in [3]* for videos, which limits the novelty of this paper and needs to be correctly referenced and acknowledged.\n\n   -> In summary, this work can be phrased as a combination of [1] and [3], predicting sinkhorn cluster assignments [2] for images [1] but in a dense manner like [3]. \n\n4. **Missing results on various backbones:** Results are shown solely on ViT-Large, however, a new approach must also be evaluated on various backbones to show effectiveness not just in one setting. I recommend including results on ViT-Base at least as it is common in MIM.\n\n5. **Positional collapse exists?!** The paper introduces a fix to the problem of positional collapse (end of 3.1). However, when looking at the ablation study, does this positional collapse exist? Without the proposed solution for this problem, the performance only degrades by 0.1% and 0.6% on IN1k and ADE. \n\n6. **Feature evaluation via unsupervised part segmentation**: Rather than visualizing the features and qualitatively calling them the most smooth and discriminative, one could opt for quantitative results through the evaluation of unsupervised part segmentation like in [4].\n\n[1] Assran, M., et al. Masked Siamese Networks for Label-Efficient Learning. ECCV 2022.\n\n[2] Asano, Y. M. et al. Self-labelling via simultaneous clustering and representation learning. ICLR 2020.\n\n[3] Salehi, M., et al. SIGMA: Sinkhorn-Guided Masked Video Modeling. ECCV 2024.\n\n[4] Ziegler, A., et al. Self-Supervised Learning of Object Parts for Semantic Segmentation. CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CAPI, a masked image modeling (MIM) framework that improves upon current methods in the k-NN and linear evaluation scheme. The framework predicts dense cluster assignments from masked patches using an EMA architecture. This work adds many add-ons to the typical MIM scheme, such as masking strategy and registers, to achieve promising results for MIM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following suggested changes and experiments must be made (more details see above):\n- The suggested experiments above to measure the effectiveness of the proposed approach inside the MIM literature.\n- Sufficient acknowledgement of relevant works [1-3] beyond the related works section as the proposed method is an extension of [1] in a dense manner and [3] for images rather than videos.\n\nThese changes are critical for securing my recommendation. Given those, I believe that the paper is of interest to the communities based on the good results for MIM using k-NN and linear. I am looking forward to the authors' response.", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mXXxQbirZr", "reviewer": "Reviewer_SGLo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The systematic analysis of masked image modeling with illustrative figures helps clarify the shortcomings of previous approaches and naturally highlights the advantages of CAPI.\n- The paper provides extensive ablation studies to validate the design choices made for CAPI, presenting key findings and revealing critical factors for effective MIM training.\n- The proposed CAPI outperforms previous MIM approaches on both high-level semantic (image classification) and dense prediction (segmentation) tasks.\n\nWeaknesses: No major weaknesses warranting rejection. Please address the minor concerns below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes the limitations of existing masked image modeling (MIM) approaches from three perspectives -- target representation, loss formulation and predictor model architecture. Based on this analysis, it proposes a novel clustering-based MIM training framework, CAPI. CAPI outperforms previous MIM models on both image classification and segmentation tasks, significantly narrowing the performance gap with DINO v2, the current state-of-the-art self-supervised learning method. Extensive ablation studies validate the authors' design choices and provide several key insights for MIM training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have a few concerns regarding missing details and the clarity of some explanations in the paper. I would like to hear the authors' responses on these issues:\n- Strictly speaking, AIM is not an MIM model. It is an autoregressive model using a prefix masking strategy. That's why AIM is able to scale up to 7B parameters. Could the authors clearly distinguish these two objectives or explain why AIM was categorized as an MIM model?\n- The paper clearly demonstrates that CAPI outperforms previous MIM approaches. However, it seems less evident in what aspects CAPI outperforms DINO v2, if any. Are there specific advantages of CAPI over DINO v2 that the authors would like to emphasize? If relevant experimental results are available, please include them in the response.\n- Please explicitly state the final total loss including both backbone training loss and clustering loss (Eq. 5). Are these two losses simply summed with equal weights?\n- If possible, please elaborate further on the curation process of Loca.\n- Could the authors compare the attentive probe (Table 1-2) and attention pooling with predictor (Table 5) under the same training configuration, such as the same model size and training set?\n- In Table 1, are the evaluation results on ADE obtained using a KNN classifier, like the plot in Figure 6 (center)?\n- In Table 1 (c), the authors observed that a linear head performs better than an MLP head under the CAPI loss formulation. Could the authors elaborate on possible reasons or insights behind this observation?\n- How many training iterations were conducted for CAPI models reported in Table 3?\n\nMinor typo: In Section 4.4, the performance drop when increasing the resolution is 0.3%, not 0.5%, for CAPI.", "broader_impact_concerns": "I don't believe there are ethical concerns about this paper. Please discuss the limitation of the proposed method and add the Broader Impact Statement section in the final draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3AXO8ATdBg", "reviewer": "Reviewer_zw92", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The work introduces several mechanisms for improving efficiency for SSL in image data, achieving strong results. The focus on scalable SSL is a welcome objective that can potentially benefit the field as a whole, and could help democratise AI by enabling better training paradigms for researchers with limited compute budgets.\n2. The work shows that latent MIM approaches yield competitive results. This seems to be a potentially fruitful area of novel research, and the paradigm on latent versus autoencoded masked image modelling serves as productive discourse with high interest and potential for advancing the field.\n3. The authors use existing modelling techniques in a novel manner, and several of the presented ideas and proposals come across as intuitive and well aligned with explicit (and implicit) objectives. Individual contributions, such as custom SK normalisation to prevent positional collapse, cross-attention decoders as prediction / representation heads, and batch-wise clustered targets, serve as novel architectural designs in their own right.\n4. There is a level of methodological rigour in the experimental results. Experimental ablations come across as meaningful and well constructed, and the results are presented clearly.\n\n**Weaknesses**:\n1. A central weakness in the current draft is that the work is somewhat opaque about the ideas, taxonomy, nomenclature, and contributions. The ambiguity in presentation and wording around how the work is placed in the greater taxonomy of SSL approaches for images weakens the clarity of the proposal, and makes the presented ideas less clear to the reader. More specifically;\n    - (a). In the presentation, the exact nature of the “split” architecture is unclear. In CrossMAE, it seems that the decoder is designed via unidirectional cross-attention blocks to inform the predictions from latents without incurring the cost of predicting unmasked tokens, which are not aggregated as part of the loss function in autoencoding MIM approaches. As this reviewer understands the setup of the current work, the proposed “split” is applied as part of the encoder, since the predictions are made in the latent space directly without reconstruction. As such, the cross attention is now seemingly a part of an encoding process, distinct from how a decoder is viewed in the autoencoding setting (e.g., MAE). This could be made clearer by introducing Figure 8 earlier, or by clarifying the presentation of these ideas in context.\n    - (b). Fig. 3 seems to be included to distinguish distillation from reconstruction, while Fig. 4 seems to look to distinguish training objectives, but without the distinction of latent / reconstruction and instead categorise approaches by design of trainable modules. The distinction comes across as somewhat arbitrary.\n    - (c). It is implied that DINO is a latent clustering method (Sec. 3.1), but is not formulated as such in Fig. 4. This seemingly looks to setup the idea that clustering in the form of batch wise clustering “on-the-fly” is different from a \"slow\" EMA clustering approach used in DINO, but this distinction is not necessarily clear to the reader at a cursory glance. Again, this comes across as somewhat arbitrary.\n    - (d). A minor nitpicky detail; presenting Sinkhorn-Knopp iterations for clustering as fundamentally based off SwAV (and DINOv2) is disputable, and it would seem that DeepCluster is a more fitting reference. Particularly as this work can -- at least in this reviewers opinion -- be seen as a continuation of DeepCluster.\n2. The choice of baselines are missing certain key points of comparison. The choice of comparing with MAE and Data2Vec 2.0 when the current manuscript explicitly cites MIM-refiner comes across like a glaring omission. These results should be included and discussed to a greater extent, as they represent a compelling alternative in efficient SSL modelling approaches. Hence, positing the work in the broader discourse on representational MIM versus autoencoding MIM seems very much aligned with the objectives of the authors.\n3. The method operates with a prohibitively large batch size (Table 6). While this improves stability in cluster assignments, it has a significant impact on efficiency and scalability, particularly in terms of memory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors outline an SSL learning framework for images which applies masked image modelling (MIM) in the latent space, positioning itself alongside I-JEPA as a contrastive approach for representations. Centrally, the authors are proposing a mini-batch clustering approach with modified Sinkhorn-Knopp (SK) iterations for doubly stochastic normalisation for the targets, with the main idea of applying latent clustering from epoch based assignments proposed in DeepCluster (Caron et al. 2018) to batch-based assignments via an EMA teacher. The work applies a CrossMAE decoder as a unidirectional prediction / representation head, which improves efficiency. The authors propose a custom patchwise / batchwise SK approach to prevent trivial correlations from positional encodings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Adding baselines for MIM-Refiner and adding more discussion around latent versus autoencoding paradigms for MIM is necessary. This reviewer is aware that playing the \"asking for more baselines\"-card is straight out of the playbook of the \"unreasonable reviewer\". However in this case it would seem to be not only appropriate, but also serves to improve transparency and discourse in the field. I am not of the opinion that including these results (even if they are comparatively stronger) would lessen the contribution. On the contrary, it would strengthen the work.\n2. The reliance on a massive batch size appears critical for stabilising the online clustering via SK iterations, but it could also be a limitation in resource-constrained settings. A more in-depth discussion how the models performance and stability might be affected if smaller batch sizes were used would be prudent. Adding a discussion whether there are strategies to mitigate this reliance would strengthen the work.\n3. The patch dimension used in the model seems to be implied to be 16 x 16 (Overview of Training, Sec. 3), and 14 x 14 (ViT-L/14) in results (Tables 2, 3). For 224 x 224 images, a patch size of 14 would result in 256 patches, not 196. This leaves the reader unsure that the authors are consistent with previous works on their use of architectural notation, i.e. if the examples used in calculations in Sec. 3 are mistaken or hypothetical, to be taken outside the context of the modelling approach in the paper. This needs to be clarified.\n4. The draft should define a clearer nomenclature for clustering on different levels, e.g. EMA clustering with DINO, direct epoch-based clustering in DeepCluster, and the batch-wise clustering approach presented in the paper. \n5. The taxonomy outlined for different SSL approaches and clustering could be tightened, which would strengthen the clarity of the work. \n6. Figure 8 comes across as much more informative than Figures 3, 4, by virtue of showing how the \"split\" decoder is applied as a variant of a representation head (e.g., SimCLR, DINO, etc.). This should perhaps be promoted to feature in the main paper, as opposed to a mention in the appendix. The roles of the two CE's are a little unclear, particularly the one used in the clustering process.", "broader_impact_concerns": "This work advances the field of self-supervised modelling for images, and the ethical concerns related directly to the paper are arguably not specific enough to warrant a broader statement. The method does advance training of models on unlabelled data, but while there are potential concerns on privacy and intellectual property, there are existing methods that already enables self-supervised and unsupervised learning.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper systematically investigates three core components of Masked Image Modeling (MIM)—self-supervision targets, loss functions, and predictor architectures—within the context of self-supervised learning. Building upon these analyses, the authors propose CAPI, a more effective MIM framework. All reviewers agree that the experimental results support the authors' claims, and the topic will likely appeal to TMLR’s audience interested in self-supervised learning. Therefore, AE recommends acceptance of this submission.

That said, one common concern raised by the reviewers is the lack of comparison with recent self-supervised learning (SSL) baselines. AE suggests addressing this concern in the revision to strengthen the paper’s contributions and better demonstrate the effectiveness of the proposed framework.

Revision Suggestions:
- **Include missing Figure 2.** The current manuscript refers to Figure 2, but it is not present in the submission.
- **Incorporate comparisons with key SSL baselines.** Specifically, please consider including methods such as DINOv1, MoCo v3, MSN, and DINOv2 (ViT-L). Although some of these methods do not rely on MIM, comprehensive comparisons across a diverse set of SSL approaches are essential in the field of self-supervised learning.
Decision: Accept (poster)