DINOv2: Learning Robust Visual Features without Supervision
Authors: Maxime Oquab
Timothée Darcet
Théo Moutakanni
Huy V. Vo
Marc Szafraniec
Vasil Khalidov
Pierre Fernandez
Daniel HAZIZA
Francisco Massa
Alaaeldin El-Nouby
Mido Assran
Nicolas Ballas
Wojciech Galuba
Russell Howes
Po-Yao Huang
Shang-Wen Li
Ishan Misra
Michael Rabbat
Vasu Sharma
Gabriel Synnaeve
Hu Xu
Herve Jegou
Julien Mairal
Patrick Labatut
Armand Joulin
Piotr Bojanowski
Venue: Accepted by TMLR ()

Abstract: The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP on most of the benchmarks at image and pixel levels.

Published in Transactions Machine Learning Research (01 (204))
DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab",, Timothée Darcet"*, Théo Moutakanni",
Huy V. Vo", Marc Szafraniec', Vasil Khalidov', Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Symnaeve, Hu Xu, Hervé Jegou, Julien Mairal', Patrick Labatut', Armand Joulin', Piotr Bojanowski"
Meta Al Research
Inria
core leam
"quul contribution
Reviewed on OpenReview:
Abstract
The recent breakthroughs in natural language processing for model pretraining on large quantitics of data have opened the way for similar foundation models in computer vision. Those models could greatly simplify the use of images in any system by producing general- purpose visual features, i.e, features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially superviedd methodss, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim accelerating and stabilizing the training at scale. In terms of data, propose automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the supe-suue literature. In terms of models, we train VIT model (Dosovitskiy et al. 2021) with IB parameters and distill it into a series of smaller models that surpass the best available general-purposs leatures, OpenCLIP (Ilbarco et al, 2021 on most the benchmarks at image and pixel levels.
1 Introduction
Learning task-agnostic pretrained representations have become the standard in Natural Language Proccss- ing (NLP) (Radford et al. 2019 Raffel et al. 2020 Chowdhery al., 2022 Hoffmann al. 2022 Touvron let al. 2023). One can use these features "as they are", i..., without fine-tuning, and achieve performances on downstream tasks that are significantly better than those produced by task-specific models (Brown et al. 2220)) This success has been fueled by pretraining large quantities of raw text using pretext objectives, such as language modeling [Radford et al.] 2017 or word vectors (Devlin et al. 2019 that require no supervision.
Following this paradigm shift in NLP, we expoct similar "foundation" models to appear in computer vi- sion (Bommasani et al. 2021p. These models should generate visual features that work out of the box on any lask, both the image level, e.g., image classification, and pixel level, e.g., segmentation. Most promis- ing efforts towards these foundation models foeus on text-guided pretraining, i.e., using form of textual supervision to guide the training of the features (Joulin al. 2016 Mahajan et al.] 2018 Radford et al.)
All the authors are affiliated to Meta, except Julien Mairal who is affiliated to Inria. Timothée Darcet and Pierre Fernandez have a co-affillation with Inria. Théo Moutakanni has a co-afflliation with Université Paris Saclay. Alaaeldin El-Nouby has a cnaffiliatinn with Inria PSL. Correspindence: (gas, timdarcet, theonontakanni, ajoulin, bojanowski) (ajannwski)@mmett..cmm Coneta.com

---

Published in Transactions Machine Learning Research /2024)
(b)
(c)
(d)
Figure 1: Visualization of the first PCA components. We compute PCA between the patches of the images from the same column (a, b, and d) and show their first components. Each component matched to different color channel. Same parts matched between related images despite changes of pose, style or even objects. Background is removed by thresholding the first PCA component.
This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information images, and complex pixel-level information may not surface with this supervision. Purthermore, these image encoders require aligned text-image corpora and hence, do not offer the flexibility of their text counterparts, that is, to learn from raw data alone.
2021).
An alternative to text-guided pretraining is self-supervised learning (Caron et al., 2018 Chen et al. 2020 He et al. (022) where features are learned from images alone. These approaches are conceptually closer to pretext tasks such as language modeling and can capture information at the image and pixel level (Caron
al.
2021 Additionally, the features output by self-supervised models have been shown to exhibit various
useful properties, and have enabled enabled a wide variety of applications (Amir et al. 2022 Tumanyan
al. 2022 Ofri-Amar al., 2023; [Hamilton el. al. 2022). However, despite their potential learn general-
purpose features, most of the advances in self-supervised learning were made in the context of pretraining on small curated dataset, ImageNet-Ik (Russakovsky
al., 2015). Some efforts on scaling these approaches beyond ImageNNet-lk have been attempted (CCaron et al. 22019 Goyal et al.] 2021 2022a). but they focused on uncurated datasets, which typically lead to significant drop in the quality of the features. This is explained by the lack of control over the data quality and diversity, which are essential produce good features.
In this work, we explore if self-supervised learning has the potential to learn general-purpont visual features if pretrained on a large quantity of curated data. revisit existing discriminative self-supervised approaches that learn features at both the image and patch level, such iBOT (Zhou et al. 2022a), and reconsider some of their design choices under the lens of a larger dataset. Most of our technical contributions are tailored toward stabilizing and accelcrating discriminative self-supervisd lcarning when scaling in model and data sizes. These improvements make our approach around 2x faster and requiie 3x less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.
Regarding pretraining data, we have built an automatic pipeline to filter and rebalance datasets from an extensive collection of uncurated images. This pipeline is inspired by pipelines used in NLP (Wenzek el al. 2020), where data similarities are used instead of external metadata and do require manual annotation. A major difficulty when dealing with images in the wild is to rebalance concepts and avoid overfitting a few dominant modes. In this work, a naive clustering approach works reasonably well to resolve this issuc. We gathered small but diverse corpus of 142M images to validate our approach.

---

Published in Transactions Machine Learning Research (01 /2024)
Tnet
Segmentation
Monoculat Depth
Classification
857
64
1.8
92
84
Accuracy
56
R-MSE
Accmracy
81
1.5
8.4
78
40
80
1010
10
1012
10
10 flops
10
10
10
10
10t0
10"
10
Blpss
flops
flogss
Finegraized Classifcation
Instance Retrioval
ImaggeNet- Skkich)
Video Understanding
80
75
70
Accuracy
72
60
70
Acentacy
65
64
45
:00
SSL
56
55
30
40
50
DINO22
10
10
101
10
10"1
10
10th 10"1
10
10"
10
Dloss
flops
Hops
Dlops
Figure 2: Evolution of performance when scaling in parameters. We show performance on eight types of vision tasks, presented in Sec. 7 and average metries with cach type. Features are extracted from our self-supervised encoders, DINOv2 (dark blue), and we compare them with self-supervised met (pale orange), as well as wcakly-supervised methods (dark pink). We report the best-performing weakly- supervised model's performance as a dashed horizontal line. Our family of models drastically improves over the previous state of the art. in self-supervised learning and reaches performance comparable with weakly- supervised features. See Sec. for detailed analysis.
Finally, we provide a variety of pretrained visual models, called DINOv2, trained with different Vision Transformers (ViT) (Dosovitskiy et al. 2016) architectures on our data. We release all the models and the code to retrain DINOv2 on any data. We validate the quality of DINOv2 on various computer vision benchmarks at both image and pixel levels as we scale them, as summarized in Fig. 2 We conclude that self- supervised pretraining alone is good candidate for learning transferable frozen features that are competitive with the best openly available weakly-supervieet models.
2 Related Work
Intra-image self-supervised training. first family of self-supervised methods focuses on pretext tasks built. from the image, ie., extracting signal from the image to be predicted from the rest of the image. This idea has become prevalent with the work of Doersch et al. (2015), where they train by predicting the context of a given patch. Many other pretext tasks were introduced based on, for example, re-colorizing
images [Zhang et al.) 2016), prodicting transformations (Gidaris al. 2018). impainting (Pathak et al.
2016) or patch re-ordering (Noroozi & Favaro 2016 Misra & Maaten 2020). Recently, the emergence of
patch- based architoctures, like ViIs, has led a revisit of inpainting for pro-training (He et al. 2022 Bao
al. 2021 El-Nouby et al. 2021), potentially in feature space (Assran et al. 2023 Baevski et al. 2022
Of particular interest, He el. al. (2022) show that a masked aulo-encoder (MAE) learns features that provide substantial improvements when finetuned on downstream tasks. This property of MAEs has been further
2022 audio (Xu et al. 2022). and across other modalitics (Girdhar et al.) 2023 However, their features require supervised finetuning, while our features perform well out of the box.
validated on vidco (Tong et al.
Discriminative self-supervised learning. The second line of work, closer ours, using discriminative signals between images or groups of images to learn features. This family of methods has roots in carly deop learning work (Hadsell et al. 2006) but bocame popular with the emergence of instance classification Wu et al. 2018). Several improvements
methods (Dosovitskiy et. al. 2016 Bojanowski & Joulin 2017

---

Published in Transactions Machine Learning Research (01 /2024)
Uncurated Data
Augmented Curated Data
Curated Data
Embedding
Deduplication
Retrieval
Figure 3: Overview of our data processing pipeline. Images from curated and uncurated data sourees first mapped embeddings. Uncurated images are then deduplicated before being matched curated images. The resulting combination augments the initial dataset through a self-supervised retrieval system.
wore made based cither on instanco-level objectives Hénaff et al. 2019 He al.] 2020
Chen & Ho 2021
Chen et al. 2020 Grill et al. 2020; Caron et al. 2021) or clustering (Caron et al. 2018 Asano et al.
2020 Caron et al. 2020p. These methods provide performant frozen features on standard benchmarks like
ImageNet (Russakovsky et al., 2015), but they hard to scale larger model sizes (Chen al. 2021] b. In this work, we revisit the training of these approaches in the context of large pretraining dalasels and models. In particular, we build on top of Zhou et al. (2022a] that we find particularly suited for scaling.
growing body of work has focused on the scaling abilities of self-superviedd learning in terms of data and model size (Caron et al. 2019 Goyal et al. 2019 Tian et al. 2021 Goyal et al.] 2022a). Most these works use large quantities of uncurated data to train models without supervision. They show evidence that discriminative methods scale with data, but because of the poor quality of the pretraining data, most of the results are obtained by finetuming the features. Of particular interest, Goyal al. (202T) have also shown that these methods benefit. from scaling in model size given enough pretrained data. This line of work questions the ability of self-supervised methods to work any data while we focus on producing the best pretrained encoders.
Scaling self-supervised pretraining.
Automatic data curation. Our datasel construction borrows from the image retrieval community (Wein- zaeptel al. 2021 Radenović et al.) 2018b Berman et al. 2019 Douze et al. 2009 Tolias al., 2016 Revaud et al. 2019). In particular, the use of retrieval to augment the training set has been studied in the context of semi-supervised learning (Yalniz al. 2019). Similarly, others have used hashtags other metadata Ma-
hajan et al. 2018 Radford et al. 2021) b pretrained vision encoders (Schuhmann et al. 2021 2022) to filter uncurated datasets. Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images. Our approach is inspired by text curation pipelines (Wenzek et al.] 2020), where a language model is trained on Wikipedia to score texts extracted from an uncurated source.
Data Processing
We assemble our curated LVD-142M dataset by retrieving, from large pool of uncurated data, images that are close to those several curated datasets. We describe below the main components in our data pipeline including the curated/ uneurated data sources, the image deduplication step and the rotrieval system. Our
pipeline does not require any metadata text and directly works with images, as shown in Fig. g We refer the reader to appendix for more details on our approach.
Data sources. Our selection of curated datasots is detailed in the appendix (Table 15 and contains ImageNet-222k, the train split of ImageNet-Ik: Google Landmarks and several fine-grained datasets. For the

---

Published in Transactions on Machine Learning Research (01 (214))
uncurated data sourec, we colloct raw unfiltered dataset of images from a publicly available repository of crawled web data. From each web page in the repository, we extract URL links of images from <img> tags. We discard URLs that are unsafe or restricted by domains, and post-process the downloaded images (PCA hash deduplication, NSFW filtering, and blurring identifiable faces). This results in 1.2B unique images.
Deduplication. We apply the copy detection pipeline of Pizzi al. (2022) to the uncurated data and remove near-duplicate images. This reduces redundancy and increases diversity among images. We also remove near-duplicales of images contained in the lest or validation set of any benchmark used this work.
Self-supervised image retrieval. We build our curated pretraining dataset by retrieving images from uncurated data source that are close images in our curated sources. In order to do this, first compute an image embedding using a self-superviee ViT-H/16 network pretrained on ImageNet-22k,, and use cosine-similarity as distance mcasure between images. Then, we perform k-means clustcring of the uncurated data. Given query dataset for retrieval, if it is large enough we retrieve N (typically 4) nearest neighbors for each query image. it is small, we sample M images from the cluster corresponding to each query image. Although visual inspection seemed to indicate good retrieval quality for N much larger than 4, this leads to more collisions (images that are nearest-neighbor retrievals of multiple queries). We choose N = 4 as provides a good tradeoff in that sense.
Implementation Details. The deduplication and retrieval stages of our pipeline rely on the Faiss li- brary (Johnson et al.] 2019) to efficiently index and compute batch searches of nearest embeddings. In particular, we heavily leverage its support for GPU-accelerated indices, using inverted file indices with prod- uct quantization codes [Jegou al., (2010). The whole processing is distributed on a compute cluster of 20 nodes equipped with 8 V100-32GB GPUs and takes less than two days to produce the IVD)-12M dataset.
Discriminative Self-supervised Pre-training
We learn our features with a discriminative sele-supervised method that can be seen combination of DINO and iBOT losses with the centering of SwAV (Caron et al. 2020). We also add a regularizer to spread features and a short high-resolution training phase. We rapidly introduce each of these approaches, but more details can be found in the related papers, or in our open-sourced code.
Image-level objective (Caron et al., 2021). We consider the cross-entropy loss between the features extracted from student and a teacher network. Both features are coming from the class token of a ViT, obtained from different crops of the same image. We pass the student class token through the student DINO head. This head is MLP model outputting vector of scores, that we call 'prototype scores'. We then apply softmax to obtain p.. Similarly, we apply the teacher DINO head to the teacher class token obtain leacher protolype scores. We then apply a soft followed by centering with moving average (or Sinkhorn- Knopp centering as detailed thereafter) to obtain pt. The DINO loss term corresponds to:
LDIOO = --==--E-- log
We learn the parameters of the student and build the teacher head with an exponential moving average of past iterates (He et al. 2020
Patch-level objective (Zhou et al. 2022a). We randomly mask some of the input patches given the student, but not to the teacher. We then apply the student iBOT head the student. mask tokens. Similarly, we apply the teacher iBOT head to the (visible) teacher patch tokens corresponding to the ones masked in the student. We then apply the softaax and centering steps above, and obtain the iBOT loss term:
CIBOT =
Σ Pri log

---

Published in Transactions Machine Learning Research (01 /2024)
where are patch indices for masked tokens. Similarly to above, we learn the paramoters of the student, and build the teacher head through exponential moving average.
Untying head weights between both objectives. Both the DINO and the iBOT loss use a learnable MLP projection head. It is applied to the output tokens and the loss is compute atop. In Zhou et al. (2022a), an ablation study shows that sharing paramcters between the DINO and iBOT heads leads better performance. At scale, we observed that the opposite is true, and we therefore two separate heads in all our experiments.
Sinkhorn-Knopp centering (Caron et al., 2020). Ruan et al. 2023 recommend to replace the teacher softmax- centeriin step of DINO and iBot by the Sinkhorn- Knopp (SK) batch normalization of Sw (Caron et al. 2020). We run the Sinkhorn-Knopp algorithm steps for 3 iterations. For the student, we apply the softmax normalization.
KoLeo regularizer (Sablayrolles et al., 2019). The KoLeo regularizer derives from the Kozachenko- Leonnekkoo differential entropy estimator (see Beirlant et al. (1997); Delattre & Fournier (2017)) and encourages a uniform span of the features within a batch. Given set of n vectors (I1,..., Tn), it is defined
Luolee Σ log(dn,))
where dn,s = minj ||: - æyyl is the minimum distance between 2; and any other point within the batch. We also by-normalize the features before computing this regularizer.
Adapting the resolution (Touvron et al. 2019). Increasing image resolution is key to pixel- level downstream tasks such as segmentation or detection, where small objects disappear low resolutions. However, training at high resolution is time and memory demanding, and instead, we increase the resolution of images to 518 518 during a short period at the end of pretraining. This also similar UniViT training from Likhomanenko et al. 2021 and FlexiViT training from Beyer et al. (023).
Efficient implementation
We consider several improvements to train models at larger scale. We train models on A100 GPUs using PyTorch 2.0. The code and pretrained models are made available under Apache 2.0 license The details models in the appendix, Table [7] With the same hardware, compared the iBOT implementation, the DINOv2 code runs around 2x faster using only 1/3 of the memory.
Fast and memory-efficient attention. We implemented our own version of FlaehAtt (Dao al. 2022 to improve memory usage and speed the self-attention layers. Our version is par with or better than the original on all cases considered, while covering more use-cases and hardware. Duc to the GPU hardware specifies, the efficiency is best when the embedding dimension per head is multiple of 64, and the matrix operations are even better when the full embedding dimension is a multiple of 256. As a consequence, our ViT-g archilecture slightly differs from the archilecture proposed by [Zhai al.] (2022) in order to maximize compute efficiency, and we use embedding dimension of 1536 with 24 heads (64 homeh)ea),, rather than 1408 with 16 heads (88 (hmn/hed).. Our experiments did not show significant differences final accuracy, and our ViT-g backbone counts 1.1B parameters.
Sequence packing. The DINO algorithm requires forwarding both large crops (at resolution 224) and small crops (resolution 98). When split. into patches, these two groups are represented by token sequences of different lengths and cannot be forwarded together. In order to accelerate training, use trick called 'sequence packing," which originates from NLP (Krell et al, 2022). The idea is simple: we concatenate the

---

Published in Transactions on Machine Learning Research (01 (2122)
sequences we must forward through the transformers into single long sequence. We pass this sequence through the transformer blocks as usual. However, block-diagonal mask is applied to the self-attention matrix in attention layers, preventing attention between different sequences. This way, the forward is strictly equivalent forwarding each sequence separately. 'This trick gives us significant compute efficiency gains compared to using separate forward and backward passes, as in prior implementations. The lower-level components our setup are available in the xFormers library Lefaudeux et al. (022)).
Efficient stochastic depth. We implement improved version of stochastic depth (Huang et al 2016 that skips the computation of the dropped residuals rather than masking the result.. This saves memory and compute in proportion approximately equal to the drop rate, thanks to specific fused kernels. With high drop rates (d = 40% in this work), this allows drastic improvement in compute efficiency and memory usage. The implementation consists of randomly shuffling the B samples over the batch dimension, and slicing the first (1 d) x B samples for the computations in the block.
Fully-Sharded Data Parallel (FSDP). Minimizing our objective with the AdamW optimizer requires 4 model replicas in float32 precision student, teacher, optimizer first moments, optimizer second moments. This sums to 16 GB of memory for a billion-parameter model such our ViT-g. In order to roduce this memory footprint per GPU, we split the model replicas across GPUs, i.e. sharding 16 GB across GPUs using the PyToorc implementation of FSDP. Consequently, the model size is not bounded by the memory single GPU but by the total sum of GPU memory across compute nodes. The Pytorch implementation of FSDP brings a second advantage, which is lo save on the cross-GPU communication costs: the weight shards are stored in float32 precision as required by the optimizer, but broadcasting weights and reducing gradients is done in float16 precision for the backbonn (MLP heads gradients are reduced in float32 to avoid training instabilitis)). This leads to approximately 50% reduction in communication costs compared to the float32 gradient all-reduce operation used in DistributedDData Parallel (DDP), which is used in other self-supervised pretraining methods (Caron et al.) 2021 Zhou et al.] 2022a). As a consequence, the training procedure scales more elficiently than DDP with float16 autocast when scaling the number of GPU nodes. Overall, Sitorch-FSDE preccssin is superior to DDP with autocast in virtually all cases we encountered.
Model distillation. Most of our technical improvements to the training loop aim at improving the training of large models over large quantities of data. For smaller models, we distill them from our largest model, the ViT-g, instead of training them from scratch. Knowledge distillation (Hinton et al. 2014) aims at reproducing the output of a large model with a smaller model by minimizing some distance between both outputs for network to the student network, we leverage the same training loop with few exceptions: we use larger model as a frozen teacher, keep a spare EMA of the student that we use as our final model, remove the masking and stochastic depth, and, apply the iBOT loss on the two global crops. In ablations, we observe that this approach achieves better performance than training from scratch, even for a ViT-L. Our distillation method ends up close to the one described by Duval al. (2023), except we do not modify the loss terms for distillation and evaluate the EMA of the student.
of given inputs. Since our objective function is form of distillation from the teacher
Ablation Studies
We present a set of ablations to empirically validate different components of our pipeline: the technical modifications described in Sec. 4 the pretraining data and the impact of model distillation. We consider various downstream tasks that are described in Sec. 7
6.1 Improved Training Recipe
Our approach improves over the iBOT method by combining it with several existing components described in Sec. g To evaluate their importance, we train multiple models where we successively add components to a baseline iBOT model. We report the Top-1 accuracy on the validation set ImageNet- Ik with a N-NN
https///tttha.cm/ccccchrrrrrrere

---

Published in Transactions Machine Learning Research (01 /2024)
INel-1k k-NN
INet-1k linear
BOT
72.9
82.3
+ (our reproduction)
74.5 1.1.6
83.2 1 0.9
+ LayerScale, Stochastic Depth
75.4 0.0.
82.0 4 1.2
+28k prototypes
76.6 1.2.2
81.9 0.1
KoLco
78.9 1 2.3
82.5 1 0.6
+wiGILU FFN
78.7 0.2
83.1 0.6
+ +aatch size 14
78,9 1 0.2
83.5 1.4
+ Teacher momentum 0.994
79.4 0.5
83.6 0.1
Tweak warmup schedules
80.5 11
83.8 0.2
+ Batch size 3k
81.7 122
84.7 0.9
Sinphoru-Knopp
81.7
84.7
+ +Untying heads DINOv2
82.0 0.3
84.5 4 0.2
Table Ablation study of training differences between IBOT and DINOv2. We optimize for k-NN performance, in experience, the lincar probe performance is lower-bounded by the k-NN performance. Some modifications, like LayerScale and a high Stochastic Depth (rate=0.4), incur a decrease linear probe performance, but have the benefits of increasing the stability of training by avoiding NaN loss values during training (Touvron et al. 2022). Overall, these modifications allowed for the next set of improvements to be added. Experiments are run using the ViT-Large architecture on ImageNet-222
Training Data
INet-1k
Im-A
ADE-20k
Oxford-M
iNat2018
iNat2021
Places205
INet-22k
85.9
73.5
46.6
62.5
81.1
85.6
67.0
INet-22k INet-1k
85.3
70.3
46.2
58.7
80.1
85.1
66.5
Uncurated data
83.3
59.4
48.5
54.3
68.0
76.4
67.2
LVD-142M
85.8
73.9
47.7
64.6
82.3
86.4
67.6
Ablation of the source of pretraining data. We compare the INet-22k dataset that was used in iBOT to our dataset, LVD-142M. Each model is trained for the same number of iterations, that is smaller than in our final run, without high-resolution adaptation. Pretraining on LVD-142M maintains the performance INct-1k 1k while leading to models that perform better in other domains.
Table 2:
Generally, observe that each component improves the performance on either k-NN or linear probing and even both in most cases. Only LayerScale and Stochastic Depth incur a performance drop in lincar probing but significantly improve the training stability our experience.
and linear probe in Table 1
6.2 Pretraining Data Source
The quality of features is directly related to the quality of the pretraining data. In this experiment, we probe the impact LVD-142M compared ImageNet-22k,, a commouly used pretraining dataset, or using directly raw and uncurated data. For the uncurated dataset, randomly sample 142 million images from the same data source as LVD-142M. We train a ViT-g/14 on cach dataset for the same number of iterations. We also include variant of ImageNet-22k obtained by removing the synsets of ImageNet-1k (INet-22k \ INet-Ik) for completeness. We report the comparisons in Table 2
The most salient observation is that training on a curated set of images works better on most benchmarks than training uncurated data. This confirms the benefit of curating data, even in the case of self- supervised pretraining. When compared with models trained on ImageNet-22k, training on LVD-142M is also superior on all the benchmarks but ImageNet-1k, This confirms that training on a more diverse of images improves the quality of the features in domains that are not covered by ImageNet-22k. We also see that training on our curated data increases the performances on domains that are not used for the curation process (INaturalist 2018 2021 and Places205), proving that scale and diversity can benefit unseen domains.

---

Published in Transactions Machine Learning Research (01 /2024)
ImaggeNet-IB Ik
ImtageNet-V V2
Image Skeetch
Food101
Cars
AmsterTime
Oxford H
86
78
94
40
85
76
LXD1122
Diee
50
92
30
35
84
74
0
H
1
H
1
H
1
H
H
H
1
Evolution of performance as function of model size for two different. pretraining datasets: ImageNeet2224 (14M images) and LVID-142M (142M images). The ViT-gg trained LVD-142M surpasses the ViT-g trained on ImageNet-22k most benchmarks.
Figure 4: Model scale versus data scale.
KoLeo
INet-1k
Im-A
ADE-20k
Oxford-M
MIM
INet-1k
Im-A
ADE-20k
Oxford-M
85.3
70.6
47.2
55.6
X
85.3
72.0
44.2
64.3
85.8
72.8
47.1
63.9
V
85.8
72.8
47.1
63.9
(a) Koleo loss
(b) MIM objective iBOT
Table 3: (a) Effect of the KoLeo loss term. (b) Effect of the iBOT Masked Image Modeling (MIM) loss term. Evaluation performed on ImageNet- (1k,A) (classification with linear probe, accuracy %), ADE-20k (segmentation with linear layer, mIoU) and Oxford-M (image retrieval, mAP). Each model is trained on the same number of iterations, that is smaller than our final run. The KoLeo loss term improves nearest-ncighhle search tasks (e.g. retrieval), and the MIM loss improves patch-level tasks (e.g. segmentation).
Overall, the conclusion of this ablation is that our dataset provides good balance of different types of images that leads to the best performance overall.
6.3 Model Size and Data
We quantify the importance of scaling data with the model size Fig. As the size of models grow, training on I.VD-142M becomes more beneficial than training ImageNet-22k. For instance, ViT-g trained on LVD-142M matches the performance on ImageNet-lke model trained on ImageNet-222 while significantly outperforming it the other benchmarks.
6.4 Loss Components
We validated the proposed Lechnical improvements in Sec. 6.1 by adding them incrementally. This section analyzes the performance hit observed we ablate specific loss terms, starting from our best pest-performin model. We ablate the importance of the KoLeo loss and the impact of the masked image modeling term. For both, we report performance on ImageNet-Ile -k using a linear classifier, ADE-20k segmentation using a linear classifier, and neirest image retrieval on Oxford-M. Table shows the impact using KoLeo loss. We see that the instance retrieval performance improves by more than 8%, confirming that this term helps spread features in the output space. At the same time, the other metrics do not suffer from this regularization. In Table 3b we show the impact of using the masked image modeling term from iBOT. This term is critical for dense prediction tasks, leading to almost 3% performance improvement.
6.5 Impact of Knowledge Distillation
For small architectures, we distill larger models instead of training them from scratch. We use the distillation procedure described in Sec. 5 We evaluate the effectiveness this approach by comparing ViT-L/14 trained from scratch with one distilled from a VIT-g/ 14 over 12 benchmarks in Fig. 5 We also report the performance the ViT-g/ 14 used for distillation as a topline. The distilled model outperforms the one trained from scratch all 12 benchmarks, validating our pretraining approach for small models.

---

Published in Transactions Machine Learning Research (01 /2024)
IN
1992
N01-g/14 Sonvra
PANOn 2015
#5.7
514
Arch
Method
INet-Ik
Segm.
Depthy
Classif.
07.9 47.55
24.
VIT-g/14
Scratch
86.5
73.4
1.00
92.1
83.1
VIT-L/14
Scratch
81,5
72.2
1.10
90.2
56.0
13A
VIT-L/14
Distill
86.3
73.3
1.08
91.2
01-
2.57
138
Arch
Method
Finegr.
Retriev.
ARSketch
Video
74
6:35
VIT-g/14
Scratch
78.3
75.2
77.0
69.3
VIT-L/14
Scratch
75.8
71.3
69.5
67.3
750
VIT-L/14
Distill
77.6
76.3
74.5
67.5
1001
DN
(a) Comparison individual metrics
(b) Averaged metrics on 8 vision tasks
Figure 5: Effectiveness of knowledge distillation. Comparison between a ViT-L trained from scratch distilled from DINOv2 using ViT-g/14. For reference, we also report the performance of the ViT-g/14 teacher. We show that a ViT-I, model distilled from a frozen ViT-gg outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.
ImsgeNa-1k
ADE-20K
84
8.3
82
Accuncy
miou
43
81
80
41
224
416
79
39
224
416
78
224
336
512
640
768
224
336
512 resolition
640
768
resolution
Figure 6: Role of resolution. Performance of ViT-L,/16 trained on ImageNet-Ik fixed resolution ("224" and "416") or trained at 224 then 416 for a short duration ("224~+416") We train lincar classifiers top of frozen features at different resolutions and report Top-1 accuracy on Image.Net and mloU on ADF-20k. We observe that performing SSL training at high resolution for short duration achieve behavior and results close to training at the same high resolution for the full training, al a fraction of the cost.
6.6 Impact of Resolution
We measure the impact of changing the resolution during the pretraining on the performance of image and patch-level features. We consider models trained from scratch using fixed resolution of either 224 224 416 x 416, and a model trained from scratch at 224 x 224, then resumedd for 10k more iterations at 416 x 416. High-resolution training is compute-intensive, so we conduct this ablation on a small setup: a VIT-L, 16 trained ImageNetlk. In Fig, we report the performance of linear probe ImageNet-1k and ADE- 20k, evaluated at various resolutions. The model trained on high-resolution images performs best across resolutions, but this comes at high cost: training 416 is approximately 3 x more -ompute-intensive than training at 224. On the other hand, training at high resolution for only 10k iterations at the end of the training is almost as good and only requiring a fraction of the compute. As a consequence, we include this step at the end of the training her than training high resolution from scratch.
10

---

Published in Transactions on Machine Learning Research (01 /2044)
7 Results
In this section, we present the empirical evaluation of our models on many image understanding tasks. We evaluate both global and local image representations, on category and instance-level recognition, semantic segmentation, monocular depth prediction, and action recognition. We detail the list of benchmarks in Appendix C The goal of this evaluation is twofold. First, we show that our self-supervised features outper- form the current state of the art by a very large margin. Second, we show that they match, or surpass the performance of teakly-supervised ones substantial mumber of tasks.
Basclines. In our comparisons, we use two kinds of models baselines. We compare to the best performing self-supervised models that are openly available. First, we rum our evaluations for MAE (He et al.] 2022).
DINO (Caron al.) ) SEERv2 (Goyal et al. 2022a MSN (Assran al. 2022), EvV (Li et al.,
2022a Mugs (Zhou et al. 2022b) and iBOT (Zhou et al, 2022a). When several architectural variants were proposed for a given mothod, we report results for the one that leads to best top-1 accuracy on ImageNet- Ik. Second, we report performance of open-source seakky-supervised models such as CLIP (Radford et al.)
2021 OpenCLIP (Ilharco et al., 2021 Cherti et al., 2023), and SWAG (Singh ct al. 2022). When evaluating models on ImageNet-Tlk we report the performance for each the aforementioned methods. For all other evaluations, we report the four best-performing models amongst SSL ones. Also, for reference, we report the best performing OpenCLIP-G for weakly supervise ones.
7.1 ImageNet Classification
As a first evaluation, we probe the quality of the holistic image representation produced by the model on the ImageNett-Ik classification dataset. We evaluate the quality of features by training simple classifier over a frozen backbone, and do not perform finetuning of the backbone weights. Following previous work, we use a linear model for simplicity, ensuring a reproducible evaluation, despite the fact that classes may not be linearly separable. Because most SSL methods were developped using ImageNet-1k validation performance as a debugging signal, we also report the top-1 accuracy on ImageNeetRRell and ImageNet-V2 V2. In order to report this additional validation performance, for all models, we run the evaluation with our code. We compare our frozen features to the best publicly available SSL features in Table 4 regardless of architecture or pretraining data. We see the components proposed in this work lead to a very significant. improvement (+4.2%) over the previous state of the art (iBOT ViT-L/16 trained on ImageNet-22kk on linear evaluation. At the same time, we also see that the performance increase on the alternative test sets is larger for our method, indicating stronger generalization. We describe details our linear evaluation Appendix B3.3
How far are we from wcakly-supervised models? We also want to validate that our features are com- petitive with state-of-the-art open-source weakly supervised models. To this end, we compare on ImageNet- 1k, using the linear evaluation, to three off-the-shelf methods with several architectural variants. For all models, we run the linear evaluation using our code, after making sure that our numbers match those re- ported in technical reports and papers. We show the result of this evaluation in Table 4 We see that our backbone, surpases the performance of OpenCLIP with a ViT-G/14 14 architecture (+0.3%) and EVA-CLIP with a ViT-g/14 (+0.1%). At the same time, we also observe that our performance the ImageNet-VV3 test set is significantly better (+1.1% versus EVA-CLIP), indicating better generalization. For the remainder of this section, we report OpenCLIP-G as a reference for weakly-supervised models.
Can we finetune the encoders? We question if the ability of our models to produce high quality frozen features impact their performance when finetuned with supervision on a specific dataset. While this is not core to this paper, this experiment indicative of whether we have involuntarily specialized our models to the setting of linear evaluations of frozen features. To run this sanity check, apply the finetuning pipeline from Touvron el al. (2022). without tweaking Top-1 accuracy on the validation set of ImageNet-1k 1k improves by more than +2% when the backbone is finetuned. This is truc both when using models at resolution 224 and 448. Further gains can be obtained by tuning the hyper-parameters of the finetuning, but this is beyond the goal of this sanity check. Nonotheless, our best finetuned performance (88.9%) is only couple of percent below (-2.2%) the absolute state of the
In Table 5] we show that the
11

---

Published in Transactions Machine Learning Research /2024)
KNN
linear
Method
Arch.
Data
Text sup.
val
val
ReaL
V2
Weakly supervised
CLIP
VIT-L/14 14
WIT-400M
79.8
84.3
88.1
75.3
CLIP
VIT-L/I44)
WTT-400M
V
80.5
85.3
88.8
75.8
SWAG
ViT-H/14
IG3.6B
82.6
85.7
88.7
77.6
OpenCLIP
VIT-H/14
LAION-2B
81.7
84.4
88.4
75.5
OpenCLIP
VIT-G/14
LAION-2B
83.2
86.2
89.4
77.2
EVA-CLIP
ViT-g/14
custom
83.5
86.4
89.3
77.4
Self-supervised
MAE
VIT-H/14
INet-1k 1k
X
49.4
76.6
83.3
64.8
DINO
VIT-S/8
INet-kk
78.6
79.2
85.5
68.2
SEERv2
RG10B
IG2B
X
79.8
MSN
ViT-L/7
INet- 1k
X
79.2
80.7
86.0
69.7
ESVIT
Swin-B/W=14
INet-1k 1k
X
79.4
81.3
87.0
70.4
Mugs
VIT-L,/16
INet- Ik
X
80.2
82.1
86.9
70.8
BOOT
ViT-L/16 16
INet-22k
X
72.9
82.3
87.5
72.4
ViT-S/14
IVDD-14M
X
79.0
81.1
86.6
70.9
VIT-B/14
LVD-142M
X
82.1
84.5
88.3
75.1
DINOv2
VIT-L/14
IVD-142M
X
83.5
86.3
89.5
78.0
VIT-g/14
LVD-142M
X
83.5
86.5
89.6
78.4
Table 4: Linear evaluation ImageNet-1k of frozen pretrained features. We report Top-1 accuracy on the validation set. for publicly available models trained on public or privale data, and with or without text supervision (text sup.). For reference, we also report the kNN performance on the validation set. We compare across any possible architectures (Arch.), at resolution 224 x 224 unless stated otherwise. The datasot used for training EVA-CLIP a custom mixture, see paper for details (Fang et al. 2023).
arts 19), obtained by Chen al. 2023a )
As DINOv2 leads features that strong in both the linear and finetuning settings, a strong property of our approach is that finetuning is optional.
Arch.
Res.
Linear
Finetuned
A
224
86.5
88.5
+2.0
ViT-g/14
448
86.7
88.9
+2.2
Table 5: Supervised finetuming Image/Net-1k. We the pipeline of Touvron et al. (2022) finelune our encoders on ImageNet-lk at resolutions 224 224 or 448 448. We compare with the accuracy obtained with linear probing and observe only modest improvements with fine-tuning: this suggests that DINOv2 features already perform well out-of-the-box.
Robustness analysis. To complement our study, and probe the gemeralization our features, we evalnate our ImageNet-Ik models trained with linear classification heads on domain generalization benchmarks. We the best performing linear classifier described above and simply run inference those benchmarks. Please note that most results in the literature are obtained with models that are finetuned end-to-end on ImageNet-lk: We show the result of this experiment in Table G When comparing with state-of-the-art SSL methods, our models shows drastically better robustncss (+ (+-29.6% on A (Hendrycks et al. 2021b +22.1% on R ((endrycks et al. (021aa) and +23.0% on Sketch (Wang et al. 2019) compared to iBOT). Our model also improves upon the best weakly-supervised model ImageNet-A while lagging behind and Sketch.

---

Published in Transactions Machine Learning Research (01 /2024)
Method
Arch
Data
Im-A
Im-R
Im-C4
Sketch
OpenCLIP
ViT-G/14
LAION-2B
63.8
87.8
45.3
66.4
MAE
ViT-H/14
INet-Ik
10.2
34.4
61.4
21.9
DINO
VTT-B/8
INet-Ik
23.9
37.0
56.6
25.5
BOT
VIT-L/16
INet-22k
41.5
51.0
43.9
38.5
VIT-S/14
LVID-142M
33.5
53.7
54.4
41.2
ViT-B/14
LVD-142M
55.1
63.3
42.7
50.6
DINOv2
VIT-L/14
LVD-142M
71.3
74.4
31.5
59.3
V/T-g/14
LVD-142M
75.9
78.8
28.2
62.5
Table 6: Domain Ceneralization with a linear probe on top of frowen features al resolution of 224.
Higher numbers are better for all benchmarks except Im-C.
Image classification
Video classification
Feature
Arch
iNat2018
iNat2021
Places205
K400
UCF-101
SSv2
OpeICLIP
ViT-G/14
73.0
76.0
69.8
78.3
90.7
35.8
MAE
ViT-H/14
31.0
32.3
52.4
54.2
70.6
29.2
DINO
VIT-B/8
59.6
68.3
60.4
64.5
85.0
32.6
iBOT
ViT-L/16
66.3
74.6
64.4
72.6
88.6
38.7
ViT-S/14
69.0
74.2
62.9
67.8
87.0
33.1
VIT-B/14
76.4
81.1
66.2
73.2
89.1
34.4
DINOv2
VIT-L/14
80.4
85.1
67.3
76.3
90.5
35.6
ViT-g/14
81.6
85.7
67.5
78.4
91.2
38.3
Table 7: Linear evaluation on other image and video classification The image benchmarks contain large quantity of fine-grained examples about objects scenes. The video benchmarks classification and human-objee interaction. All the features are frozen with a linear probe on top.
action
7.2 Additional Image and Video classification Benchmarks
In this section we study the generalization of features downstream classification benchmarks. We consider two sets of evaluations in that context. On one hand, we use large and finegrained datasets such as iNaturalist and Places205. On the other, we use the image classification tasks originally proposed in SimCLR (Chen et al.) 2020). For iNaturalist. 2018, iNaturalist 2021, and Places205, train lincar classifier with data augmentations as in Sec. 7. We report top-1 accuracy for those three datasets in Table Interestingly, our model significantly outperforms OpenCLIP ViT-G/14 on both variants of iNaturalist (+8.6% and +9.7% for 2018 and 2021 respectively), and lags slightly behind on Places 205 (-33%))
In a second set of evaluations, we measure the performance our model on video action recognition even though our features were not trained videos.. We evaluated features three datasets, namely UCF- 101 (Soomro et al. 2012). Kinetics-400 (Kay et al. 2017) and Something-Sonae Somiing-Smmehinggg v2 (Goyal et al, 2017 For this ovaluation, we pick 8 evenly spaced frames in vidco and train linear classifier on the average of the features for UCF and K-400. For SSv2, we opt for concatenation to retain more temporal information than with feature averaging. For each dataset, we measure average accuracy and report the results in Table 7 We see that amongst. self-supervised approaches, our model clearly sets new state of the art. Moreover, our model matches the accuracy of the OpenCLIP features on UCF and Kinetics (+0.1% and +0.5% respectively) and clearly outperforms them on SSv2 (+2.5%). This is particularly interesting, as SSv2 requires much richer understanding of the video frames.
Finally, in Table g we compare selected frozen features on 12 transfer classification benchmarks initially proposed by Chen et al. 2020). This benchmark covers scenes, objects (food, cars, planes), and textures.

---

Published in Transactions Machine Learning Research (2024)
Feature
Arch
Food
C10
C100
SUN
Cars
Airer
VOC
DTD
Pots
Cal101
Flowers
CUB
Avg
OpenCLIP ViT-G/14
94.5
98.7
91.0
84.0
96.1
80.2
89.3
86.0
95.7
98.1
99.5
89.9
91.9
MAE
VIT-H/14
78.4
96.1
83.9
63.9
56.1
63.4 4
84.3
75.4
89.4
95.9
92.3
57.2
78.0
DINO
VIT-B/8
85.1
97.2
86.9
70.3
76.6
70.6
86.7
79.6
93.2
95.4
97.6
81.7
85.1
BOT
ViT-L/16
91.0
99.0
92.8
75.6
71.8
72.4
89.0
80.7
87.7
97.5
99.6
82.1
86.6
ViT-S/14
89.1
97.7
87.5
74.4
81.6
74.0
87.8
80.6
95.1
97.0
99.6
88.1
87.7
ViT-B/14
92.8
98.7
91.3
77.3
88.2
79.4
88.2
83.3
96.2
96.1
99.6
89.6
90.1
DINOv2
VIT-L/14
94.3
99.3
93.4
78.7
90.1
81.5
88.3
84.0
96.6
97.5
99.7
90.5
91.2
ViT-g/14
94.7
99.5
94.4
78.7
91.4
87.2
89.0
84.5
96.7
97.6
99.7
91.6
92.1
Table 8: Linear evaluation of frozen features on fine-graines benchmarks. Accuracy on 12 bench- marks covering objects, scenes and textures following the evaluation protocol praposed in Chen et al. 2020
Oxford
Paris
Met
AmsterTime
Feature
Arch
M
GAP
GAP-
ACC
mAP
OpenCLIP
ViT-G/14
50.7
19.7
79.2
60.2
6.5
23.9
34.4
24.6
MAE
ViT-H/14
11.7
2.2
19.9
4.7
7.5
23.5
30.5
4.2
DINO
ViT-B/8
40.1
13.7
65.3
35.3
17.1
37.7
43.9
24.6
iBOT
VIT-L/16
39.0
12.7
70.7
47.0
25.1
54.8
58.2
26.7
VIT-S/14
68.8
43.2
84.6
68.5
29.4
54.3
57.7
43.5
ViT-B/14
72.9
49.5
90.3
78.6
36.7
63.5
66.1
45.6
DINOv2
VIT-L/14
75.1
54.0
92.7
40.0
68.9
71.6
50.0
VIT-g/14
73.6
52.3
92.1
82.6
36.8
73.6
76.5
46.7
Table 9: Evaluation of frozen features on instanco-level recognition. We consider different bench- marks and report their main metrics.
We replace the Birdsnap dataset with CUB because the former was not publicly available in its entirety. We follow the experimental protocol outlined by Chen al. (2020), namely training logistic regression precomputed features. Our model significantly outperforms state-of-thh-art SSL models, with most notable diffcrences on Stanford Cars (+14.8% versus DINO ViT-B/8) and FGVC Aircraft (+14.8% versus iBOT ViT-L/16). Even though these benchmarks favor text-guided pretraining, features still competitive with OLeMCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (-5.3%) and Cars (-4.7%).
7.3 Instance Recognition
In this experiment, probe model the task of instanco-lovel recognition using non-parametric approach. Images from a database are ranked according to their cosine similarity with a query image. We evaluated our model and compare to baselines on Paris and Oxford, that are landmark recognition bench- marks. We also evaluated on Met, dataset of artworks from the Metropolitan museum, and Amster Time, containing street view images matched to archival images of Ansterdam. We measure performance by com- puting the mean average precision and report our results Table g We that our features significantly outperform both SSL (+41% mAP on Oxford-Hard) Hard), and ceakly-uuperiiedd (+34% mAP on Oxford-Hard) ones. It is interesting to see that our features perform well across task granularitics, both at the category-levdl and instance-level. This is desirable property for strong off-the-shelf computer vision features.

---

Published in Transactions Machine Learning Research (01 /2024)
ADE20k (62.9)
CityScapes (86.9)
Pascal VOC
(89.0)
Method
Arch.
lin.
+ms
lin.
+
lin.
+ms
OpenCLIP
VIT-G/14
39.3
46.0
60.3
70.3
71.4
79.2
MAE
ViT-H/14
33.3
30.7
58.4
61.0
67.6
63.3
DINO
ViT-B/8
31.8
35.2
56.9
66.2
66.4
75.6
BOT
VIT-L/16
44.6
47.5
64.8
74.5
82.3
84.3
VIT-S/14
44.3
47.2
66.6
77.1
81.1
82.6
ViT-B/14
47.3
51.3
69.4
80.0
82.5
84.9
DINOv2
VIT-L/14
47.7
53.1
70.3
80.9
82.1
86.0
ViT-g/14
49.0
53.0
71.3
81.0
83.0
86.2
Table 10: Semantic segmentation on ADE20K, CityScapes and Pascal VOC with frozen features and a linear classifier (lin.) and with multiscale (+ms). The absolute state of the art from Wang et al. (2022), Liu et al. (021] and Chen et al. (0188) respectively are mentioned at the lop of the Table. For reference, using the Mask2Former pipeline (Steiner ot al. 2021) with a ViT-Adapter (Chen et al. 2023b on top of our frozen ViT-g/14 1/4 backbone gives 60.2 mloU on ADE-20k.
7.4 Dense Recognition Tasks
We probe the quality of patch-level features extracted from network several dense downstream tasks. We consider semantic image segmentation and monocular depth estimation in several seltings and we conduct. evaluations on multiple datasets for each.
Semantic segmentation. For semantic segmentation evaluation, we consider two different setups. Linear: linear layer is trained to predict class logits from patch tokens. It is used to produce a low- resolution logil map (eg 32x32 for a model with patch size 16), which is then upsampled to full resolution (512x512) to obtain segmentation map. This procedure is extremely simple but cannot easily produce high-resolution segmentations. +ms: a boosted version of the linear setup. We concatenate the patch tokens of the last layers, use a larger image resolution of 640, and use multiscale test- augmentations to improve the predictions. We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table 10
Our models show very good performance all datasets and for all setups. Interestingly, evaluation using +ms is on par with fully finetuning MAE with an Upernet decoder (53.0 versus 53.6 mloU). This is surprising because use significantly simpler predictor. Also, our best model, when evaluated using the boosted recipe, almost matches the state of the art Pascal VOC (86.2 versus 89.0 mloU).
Frozen backbonn a SOTA pipeline. In a final experiment, we freoze our backbone, and plug it into ViT ViT-Adapter Chen al. (202bb) with Mask2former head (Cheng et al. 2022 We tune the weights of the adapter and head, but keep the backbone frozen, meaning 66% of the weights are frozen. This allows for a lighter segmentation training than full end-to-end fine-tuning. With this setup, we reach 60.2 mloU on ADE20k, close to the competitive state of the art, standing at 62.9 mIoU (Wang et al. 2022). Although our setup for this experiment doesn't 't makes use of the optimisations described in Sec. the segmentation training in this experiment took hours on 16 V100 GPUs
Depth estimation. In this experiment, evaluate our patch-level features three monocular depth estimation benchmarks: NYUd, KITTI and zero-shol transfer from NYUd to SUN3d. We follow the evalu- ation protocol of et al. 2022b). We consider three different setups for this evaluation. lin. 1: we extract the last layer of the frozen transformer and concatenate the [CLS] token to cach patch token. Then we bi-lincarly upsample the tokens by a factor of 4 to increase the resolution. Finally we train a simple lincar layer using classification loss by dividing the depth prediction range in 256 uniformly distributed bins and

---

Published in Transactions Machine Learning Research (21/))
NYUd (0.330)
KITTI
NYUd - SUN RGB-D
(2.10)
(0.421)
Method
Arch.
lin.
lin. 4
DPT
lin.
lin. 4
DPT
lin.
lin. 4
DPT
OpenCLIP
VIT-G/14
0.541
0.510
0.414
3.57
3.21
2.56
0.537
0.476
0.408
MAE
VIT-H/14
0.517
0.483
0.415
3.66
3.26
2.59
0.545
0.523
0.506
DINO
ViT-B/8
0.555
0.539
0.492
3.81
3.56
2.74
0,553
0.541
0.520
BOT
VIT-L/16
0.417
0.387
0.358
3.31
3.07
2.55
0.447
0.435
0.426
VIT-S/14
0.449
0.417
0.356
3.10
2.86
2.34
0.477
0.431
0.409
ViT-B/14 0.399
0.362
0.317
2.90
2.59
2.23
0.448
0.400
0.377
DINOv2
VIT-L/14
0.384
0.333
0.293
2.78
2.50
2.14
0.429
0.396
0.360
ViT-g/14
0.344
0.298
0.279
2.62
2.35
2.11
0.402
0.362
0.338
Depth estimation with frozen features. We report performance when training a linear classifier on top of one (lin. 1) or four (lin. 4) transformer layers, as well, as the DPT decoder (DPT) of Ranfu et al. (2021). We report the RMSE metric on the 3 datasels. Lower is better. For reference, we report statc-of-tho-art results taken from Li ct al. 2022bb on each benchmark on top of the Table.
Table 11:
use a linear normalization lollowing Bhat et al. 2021 lin. 4: we use the same protocol that we use with one layer, but concatenate the tokens from layers 7 {3,6, 6, 9, 12} for ViT-S/B | {5, 12, 18, 24) for ViT-L, and I = {10, 20, 30, 40} for ViT-g. DPT: we use the DPT decoder (Ranftl et al. 2021 on top of our frozen models and setup a regression task. We scale the size of the head following the dimension of the features for each architecture. We show results for all baselines, all datasets and all setups in Table II
From this lable, we see that our features clearly surpass the best SSL and WSL features available. It interesting see that iBTT features extracted from a ViT-L outperform the ones of OpenCLIP with a VIT-G. This observation supports the intuition that caption-based feature learning fails to learn subtle patterns like this one. Also, our model, with the DPT decoder and frozen backbone, matches or exceeds the performance of the recent work of Li et al. (2022b). Finally, the out-of-domain generalization result on SUN-RGBd shows that our features allow very good transfer between domains. A depth prediction module trained on indoor scenes from NYUd generalizes pretty well to the outdoor examples SUN-RGBd.
7.5 Qualitative Results
In this final section of the empirical evaluation of our features, we propose few qualitative analyses.
Semantic Segmentation and Depth Estimation. We show some qualitative results for our dense prediction evaluations: segmentation on ADE20K in Fig. 7 and depth estimation on NYUd, KITTI and SUN RGB-D in Fig. 7 We compare DINOv2 with OpenCLIP with lincar classifier on each datasct. While not perfect, the linear segmentation model using our DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup. Indeed, the segmentation mask produced by OpenCLIP-G shows many artifacts and disconnected components. The qualitative results on depth estimation clearly illustrate the quantitative gap between OpenCLIP and DINOv2. These results highlight that our features, well as the features extracted from OpenCLIP, are able to linearly separate complex information such as depth, even though neither was trained with this type of information. However, our features lead to a much smoother depth estimation, with less artifacts. Some objects such as the chair on the SUN RGB-D image are completely ignored by OpenCLIP and correctly positioned using our features.
cat-of-distribution generalization. We show few examples of applying the depth prediction and segmentation linear classifiers to out-of-distribution examples in Fig. g The qualitative results support our claim that our features transfor between domains. The quality the depth and segmentation predicted for pictures of animals, or paintings is very good, even though the domains very different.
16

---

Published in Transactions Machine Learning Research (01 /2024)
ADEZAN
POAAA
PONN--RS
KITTI
Input
OpeNCLIP-G
DINOv2-g
Input
OpenCLIP-G
DINOv2-g
Figure 7: Segmentation and depth estimation with linear classifiers. Examples from ADE20K,
NYUd, SUN RGB-D and KITII with linear probe on frozen OpenCLIP-C and DINOv2-g features.
Figure 8: Examples of out-of-distrriattio examples with frozen DINOv2-g features and a linear probe.
PCA of patch features. We show the results of the principal component analysis (PCA) performed on the patch features extracted by our model. We keep only patches with a positive value after we threshold the first component. This procedure turns out to separate the image's main object from the background. compute second PCA on the remaining patches across three images depicting the same category. We color the three first components with three different colors and present the results in Fig. [ and g There are two interesting observations: first, our unsupervised foreground / background detector, based on detecting the highest variance direction, performs very well and is capable of delincating the boundary of the main object in the picture. Second, the other components correspond to 'parts" of objects and match well for images the same category. This is an emerging property model not trained to parse parts of objects.
17

---

Published in Transactions Machine Learning Research (01 /2024)
HELLO
Figure 9: More visualization of the first PCA components. We compute the PCA between the patches from all the images and show their first 3 components. Each component corresponds a specific color channel. Same parts matched between related images depsite changes of pose, style Background is removed by removing patches with a negative score of the first PCA component.
objects.
Patch matching. Finally, we explore what lype of information our patch-level features contain by match- ing them across images. We start by detecting the foreground object using the procedure described above. Then, we compute the cuclidean distance between patch features extracted from two images and map them by solving an assignment problem. In order to reduce the number of matches, we then apply a non-maximum suppression to keep only the salient ones. In Fig. 10
show
examples of such matchings.
We observe that the features seem to capture information about semantic regions that serve similar purpose different objects or animals. For instance, the wing of plane matches the wing of bird. We also observe that the model is robust to style (image versus drawing), and to large variation of poses (see the elephant).
8 Fairness and Bias Analysis
We conduct two fairness evaluations of our models. We probe for geographical fairness and potential harmful label associations. For both evaluations, we experiment with our largest ViT-g model.
8.1 Geographical Fairness
We evaluate geographical fairness the Dollar Street dataset introduced in De Vrics et al. (209)) using the evaluation protocol of Goyal et al. (2022b). This benchmark compares performance across countries and income levels. contains 16,073 images from 289 households across countries. The task recognize 94 concepts that vary visually between households based on income location. In Table 12 we compare our model with SEERv2 (Goyal et al. 2022a), a model trained on a geographically diverse set images. Our model slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised bascline reported by Goyal et al. (022aa). However, still observe significant difference betweon regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe. This shows that model is still biased toward Western countries. Similarly, our model performs
18

---

Published in Transactions Machine Learning Research (01 /2024)
(Vehicles)
(Birds / Airplanes)
(Elephants)
(Drawings / Animals)
Figure 10: Matching across images. match patch-level features between images from different mains, poses and even objects that share similar semantic information. This exhibits the ability of our model to transfer across domains and undersland relations between similar parts of different objects.
Income buckets
Regions
Method
Arch.
Data
low
medium
high
Africa
Asia
Americas
Europe
SEERv2
RG-10B
IG-1B
59.7
78.5
86.6
65.9
76.3
81.1
85.6
DINOv2
VIT-g/14
LVVD-142M
67.4
83.3
90.5
74.0
81.6
86.2
89.7
Table
Geographical fairness and diversity analysis across income buckets and regions.
significantly better high-income households than low-income ones, with difference of 31.7%. Despite improvements, we observe significant biases in our models toward wealthy households from Western countries.
8.2 Gender, Skintones and Age
In a second set of evaluations, we question how our model classifies images of people of different gender, skin tone, and age (all self-reported).
follow the protoool Goyal et al. (2022b), where we train a multiclass classifier subset of 619 classes of ImageNet-22k. We group the 619 classes into four broader categories: Human, Possibly Human, Nou-Human, or Crime. Non-Human and Crime are considered barmful. Using this classifier, run inference on 2955 images from the Casual Conversations dataset (Hazirbas al. 2021 and keep all labels in the top-5 that are assigned probability of 0.1 or more. Because of that, we can assign multiple classes to each image. We make one modification to the original ovaluation protocol: we do not backpropagate gradients to the backbone and keep it frozen. We compare our model to SEERRv in Table

---

Published in Transactions Machine Learning Research (01 /2024)
Gender Skintone
Age Groups
female darker
female lighter
male darker
male lighter
Model
Assoc.
18-30
30-45
45-70
70+
SEER
Non-Human
0.0
0,0
0.0
0.0
0.0
0.0
0.0
0.0
RG-10B
Crime
0.0
0.0
0.0
Human
94.9
95.8
86.6
79.0
90.5
88.3
91.9
82.3
Possibly-Humer
13.6
6.7
65.0
60.2
32.8
37.2
29.4
6.5
DINOv2
Non-Human
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
ViT-g/14
Crime
0.0
0.0
0.2
0.0
0.0
0.1
0.0
0.0
Human
97.3
97.7
86.1
84.0
91.2
90.2
93.2
88.7
Possibly-Human
15.8
17.2
52.2
48.1
35.3
37.3
23.0
9.7
Table 13: Label association fairness evaluation across gender, skintones and age groups. We follow the protocol proposed by Goyyl et al. (2022b) with a slight modification. Instead of finetuning the backbone, we simply learn a linear classifier on the subset 619 classes of ImageNet-22k.
Model to Reproduce
GPU Power
Total power consumption
Carbon emitted
GPU Type
GPU-hours
PUE
consumption
(ICO2eg)
DINOv2-g
A100-40GB
400W
22,016
1.1
9.7 MWh
3.7
Table 14: Carbon footprint of reproducing DINOv2. We report the potential carbon emission of reproducing DINOv2-g when assuming a power consumption for the A100-40GB 400W, a PUE of 1.1 and carbon intensity factor of 0.385 kg CO2e per KWh.
Our model often classifies images of all groups as Human without large deviations across skin tones. Neither SEERv2 nor DINOv2 predict harmful labels from the Non-Human or Crime meta-categories (except for two instances where the background contains bars visually similar to prison bars). We see that our model triggers the Possibly-Humas classes often. This class is constructed from objects in ImageNet-22kk that often related to Humans, such as Scarf, Glasses, or Beard. Our model often predicts the Possibly-Huma class for men because of the prevalence of the Beard class. clear pattern indicates bias against. a particular group in this study, While this is encouraging, also acknowledge that more thorough evaluation of biases may reveal flaws in our model.
Estimating the Environmental Impact of Training Models
Training foundation models consumes a significant amount of energy, resulting in carbon dioxide emissions. Patterson al. (2021) propose methodology report an estimation of the carbon emitted during the training of a model based on the specifics of the data center and its power grid. This computation informs the design of the data center used for the training models and the choice location for data centers. This methodology requires to know the specifics the data center used for training, which can be complex when multiple data centers are involved over time. Additionally, these specifies are most often not in the control of the AI practitioner, and hence, this methodology less helpful when practioners make technical decisions about future trainings. Instead, in this section, we follow alternative that reports the potential carbon emission of retraining a similar model in an average data center located in the US. This met hodooogy was used in previous work natural language processing (Strubell et al. 2019: Touvron establish apple-to-apple comparison between pretraining schemes. More precisely, we fix the value of all exogenous variables, i.e. the Power Usage Effectiveness (PUE) and carbon intensity factor of a power grid the same values in Touvron al. (2023), that is, PUE of 1.1 and the carbon intensity factor to the US average of 0.385 kg CO2eq/Ka Wh. We use the same formula as in Patterson et al. (2021) to estimate the potential energy consumption and the carbon emission. For the power consumption of an A100-80GB, we take the thermal design power for NVLink systems, which 400W. We report the potential carbon emission
al. 2023)
20

---

Published in Transactions on Machine Learning Research (01 (214))
of retraining a DINOv2 ViT-g in Table [4 For comparison, retraining an OpenCLIP VIT-L or OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, if run in the same data center. This is 10x more carbon cmission. Note that this comparison is not fair to them, since they also train a text encoder in parallel, and we thus do not report them in the table. However, it gives a reasonable guideline for those who are interested in training only visual features: in this context, training self-supervised model is preferable in terms carbon emission. Training a text-guided model still makes sense when planning to reuse the text encoder.
Carbon footprint of the whole project. Additionally, we estimate the footprint of the whole project to be between 0.5k and 1k tCOzeq using the same grid as presented above [ This carbon footprint represents in the order of 200k GPU-days. The primary sources of emissions are the sell-supervised pre-trainings of the models. For example, a single pre-training of ViT-g model (22k GPU-hours) emits 3.7 tons of CO2eq, while a linctuning on ImageNet-1k (Ik GPU-hours) emits 0.2 lons. This estimate only considers the GPUs' electricity consumption and ignores other emissions, such as their manufacturing and disposal.
10 Future work and Discussion
In this work, we present DINOv2, new series of image encoders pretrained large curated data with no supervision. This is the first SSL work on image data that loads to visual features that close the performance gap with (weakly) supervised alternatives across wide range of benchmarks and without the need for finetuning, We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table [], ii) a larger model scale with improved results regardless of the data used for training (Fig. A. iif) larger dataset (Fig. and iv) the distillation process that makes smaller models benefit from the performance of the strongest ViT-g model (Fig. 5). few properties emerge from these models, such as an understanding of object parts and scene geometry regardless of the image domains. We expect that more of these properties will emerge at larger scales of models and data, akin to instruction emergence in large language models, and plan to continue scaling along these axes. This paper also demonstrates that these visual features are compatible with classifiers as simple as linear layers meaning the underlying information is readily available. In future work, we plan to leverage this ability to train a a language-enabled AI system that can process visual features as if they were word Lokens, and extract the required information to ground the system.
Acknowledgments.
We thank Mathikde Caron for initial discussions that led to this work. Julien Mairal was supported by the ERC grant number 101087696 (APHELAIA project) and by ANR 3IA MIAI@Grenoble Alpes (ANR-19- 19- P31A-0003). We thank Olivia Joulin for the horse drawing used in Fig. IO We thank Madelcine and Léon for posing for Fig. g We also thank the rest of FAIR and Meta AI for feedback on this work through the entire project.
References
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. In ECCV workshop on 'What is Motion For?", 2022.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vodaldi. Self-labelling via simultancous clustering and representation learning. In ICLR, 2020.
Mahmoud Assran, Mathikde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning, In ECCV, 2022.
Por context, a full Boring 777 return flight between Landon and New York corresponds approximately 1COgmg.
21

---

Published in Transactions Machine Learning Research (01 (2024)
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embeddinng predictive archi- tecture. In CVPR, 2023.
Alexci Bacvski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Maxim Berman, Hervé Jégou, Vedaldi Andrea, lasonas Kokkinos, and Matthijs Douze. MultiGrain: unified image embedding for classes and instances. ar preprint Xrx: 05509, 2019.
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint drXiv2006.001559, 2020.
Lucas Beyer, Pavel Izmailov, Alexander Kolcsnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In CVPR, 2023.
Shariq Farooq Bhat, Ibrahcom Alhashim, and Peter Wonka. AdaBins: bins. In CVPR, 2021.
Dopth estimation using adaptive
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In ICML, 2017.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunitics and risks of foundation models. arxio preprint Xrx::21088. 07258, 2021.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Ne 2020.
Mathildo Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustcring for unsupervisod learning of visual features. In ECCV, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image
features on non-curated data. ICCV, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, et al. Symbolic discovery of optimization algorithms. arXiv preprint Xrx:22330 06675, 2023a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
22

---

Published in Transactions on Machine Learning Research (01 (2044)
Xinlci Chen, Saining Xie, and Kaiming Hc. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun Ho, Tong Lu, Jifeng Dai, and Yu Qiao, Vision transformer adapter for dense predictions. In ICLR, 2023b.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, Mastod-attention mask transformer for universal image segmentation. In CVPR, 2022.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharc,, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-inage learning. In CVPR, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:204.00811 2022.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamod, and A. Vedaldi. Describing textures the wild. In CVPR, 2014.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Euzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understand- ing. In CVPR, 2016.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact altention with io-awareness. In NewrlPSS 2022.
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. Does object recognition work for everyone? In CVPR workshops, 2019.
Sylvain Delattre and Nicolas Fournier. On the kozachenko- leonenko entropy estimator. Journal of Statistioal Plamning and Inference, 185:69 93, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Carl Docrsch, Abhinav Gupta, and Alexoi Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox Dis- criminative unsupervised feature learning with exemplar convolutional neural networks. TPAMI, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. In JCLR, 2021.
Matthijs Douze, Hervé Jégou, Harsimrat Sandhawalia, Laurent Amsaleg, and Cordelia Schmid. Evaluation of gist descriptors for web-scale image search. In CIVR, 2009.
Quentin Duval, Ishan Misra, and Nicolas Ballas. A simple recipe for competitive low-compute self supervised vision models. arXio preprint orXiv:2901. 094511,991111.. 2023.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? ar preprint arxiv:1 12. 10740, 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
23

---

Published in Transactions on Machine Learning Research (01 /2024)
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR, 2004.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kilti
dataset. IJRR, 2013.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and vidcos. In CVPR, 2023.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self supe-supe visual representation learning. In ICCV, 2019.
Priya Goyal, Mathilde Caron, Benjamin Lefandeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Sele-supervised pretraining of visual features the wild. preprint arXiv:2103.0111 2021.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Mannat Singh, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint 2rx:::2202.08360 2022a.
Priya Goyal, Adriana Romero Soriano, Caner Hazirbas, Levent Sagun, and Nicolas Usunier. Fairness indi- cators for systematic assessments of visual feature extractors. In FAcct, 2022b.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joamna Materzynska, Susanne Westphal, He- una Kim, Valentin Hacnel, Ingo Fruend, Peter Yianilos, Moritz Mucller-Freitag, et al. The "somcthing something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Bann-Bstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcnoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning. In NewrIPS, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In ICLR, 2022.
Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, and Cristian Canton Ferrer. Towards measuring fairness in ai: the casual conversations dataset. 1EEE Transactions on Biometrics, Behavior, and Identity Science, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Docrsch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. PMLR, 2019.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR 2019.
24

---

Published in Transactions on Machine Learning Research (01 /2024)
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. In CVPR, 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NewrIPS Deep Learning Workshop, 2014.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint (ii::2203.155000 2022.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sodra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightaman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. 2021.
Herve Jogou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. TPAMI, 2010.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transac- tions on Big Data, 2019.
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data, In ECCV, 2016.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video datasel. arXivv preprint
2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cate- gorization. In 3DRR, 2013.
Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https: https://gitan //githu....tt.... http///ttub..cooaaaeooooooooooo/ womm//accebookeeerrcc///ae wwwoerrrrrooooooooooooooooooosre 2022.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning, In ICLR, 2022a.
Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang- Binsformer: Revisiting adaptive bins for monocular depth estimattio.. arXiv preprinl arXi::204.00088. 2022b.
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnacve, Ronan Collobert, and Alex Rogozhnikov. Cape: Encoding relative positions with continuous augmented positional embeddings. In NewrIPS, 2021.
Huajun Liu, Fuqiang Liu, Xinyi Fan, and Dong Huang. Polarized self-attention: towards high-quality pixel- wise regression. arXiv preprint NXix:2107.0078. 2021.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits weakly supervised pretraining, In ECCV, 2018.
25

---

Published in Transactions Machine Learning Research (01 /2044)
S. Maji, Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fino-grained visual classification of aircraft. Technical report, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR, 2020.
Mllina-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Dolev Ofri-Amar, Michal Geyer, Yoni Kasten, and Tali Dekel. Neural congealing: Aligning images to a joint semantic atlas. In CVPR, 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahne, Trevor Darrell, and Alexei Efros. Context, encoders: Feature learning by inpainting, In CVPR, 2016.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Mand Texier, and Jeff Dean. Carbon emissions and large neural network training. arXin preprint arXiv:2104. 2021.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self-supervised descriptor for image copy detection. In CVPR, 2022.
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. CVPR, 2018a.
Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning enn image retrieval with no human anno- tation. TPAMI, 2018b.
Alec Radford, Rafal Jozefowicz, and llya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arxiv: orXiv:1704.014 2017.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, Peter Liu, et al. Exploring the limits of transfer learning with unified lext-to-text transformer. JMLR, 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto Souza. Learning with average precision: Training image retrieval with listwise loss. In ICCV, 2019.
Yangjun Ruan, Saurabh Singh, Warren Morningstar, Alexander A Alemi, Sorgey loffe, Ian Fischor, and Joshua V Dillon. Weighted ensemble self-supervised learning. In ICLR, 2023.
26

---

Published in Transactions Machine Learning Research (01 (2024)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjoev Sathoesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégon. Spreading vectors for similarity search. In ICLR, 2019.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NewrIPS Data Centric AI Workshop, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wighlman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
Noam Shazeer. Glu variants improve transformer. arxiv preprint wxXi:2002.052000 2020.
Nathan Silberman, Derek Hoiem, Pushmect Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra. Godik, Raj Pratock Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting Weakly Supervised Pre-Trree nee-Trainnng of Visual Perception Models. CVPR, 2022.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao, Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arviv preprinl 1212 0402, 2012.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer How to train your vit? data, augmentation, and regularization in vision transformers. TMLR, 2021.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. ACL, 2019.
Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised lcarning from uncurated data. In ICCV, 2021.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrioval with integral max-pooling of cnn activations. In ICLR, 2016.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Vidcomac: Masked autoencoders data-efficient learners for self-supervised video pre-training. In NewrlPS, 2022.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test, resolution discrep- ancy. In NewrIPS, 2019.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In ECCV, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In CVPR, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shopard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
27

---

Published in Transactions Machine Learning Research (01 /2044)
Grant Van Horn, Elijah Cole, Sara Beery, Kimborly Wilber, Serge Belongie, and Oisin Mac Aodha. Bench- marking representation learning for natural world image collections. In CVPR, 2021.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongshong Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2022.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019.
Frederik Warburg, Soren Hauberg, Manuel Lopez-Antequera, Pau Gargallo, Yubin Kuang, and Javier Civera. Mapillary street-level sequences: dataset for lifelong place recognition. CVPR, 2020.
Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Learning super-features for image retrioval. In ICLR, 2021.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR--000-000,, 2010.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chandhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datascts from web crawl data. In LREC, 2020.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2 a large-scale benchmark for instance-level recognition and retrieval. In CVPR, 2020.
Zhirong WU, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- con-parametric instance discrimination. In CVPR, 2018.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to 200. In CVPR, 2010.
Hu Xu, Juneheng Li, Alexci Bacvski, Michael Auli, Wojeiech Galuba, Florian Metze, Christoph Feichtenhofer, et al. Masked autoencoders that listen. arXiv preprint orXiv:2207.06400 06400, 2022.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXie preprint Xr:: 00546, 2019.
Burak Yildiz, Seyran Khademi, Ronald Maria Siebes, and Jan van Gemert. Amstertime: A visual place recognition benchmark dataset for severe domain shift. In ICPR, 2022.
Tikolaas-Antonias Ypsilantis, Noa Garcia, Guangxing Han, Sarah Ibrahimi, Nanne Van Noord, and Giorgos Tolias. The met. dataset: Instance-level recognition for artworks. In NewrIPS Datasets and Benchmarks Track, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lueas Beyer, Scaling vision transformers. In CVPR, 2022.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NewrIPS, 2014.
Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhon, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022a.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi- granular self-supervised learning framework. arxin preprint Xrx::2203 144 15, 2022b.
28

---

Published in Transactions Machine Learning Research (01 /2024)
A Data Processing
A.1 Data selection
Our selection of datasets for buikding LVD-142M is detailed in Tab. 15 This collection is intended to provide images covering well various downstream vision tasks both for imago-level and dense recognition.
A.2 Image similarity
We employ cosine similarity compare image features (whether feature generated for deduplication) with the following similarity function m.
f(s) f(r)
m(s,r) r) cosine-similarity (s)) = II(()))lll((()
where and are pair of images to compare and f is the model generating features.
A.3 Deduplication
Self-decuullicaioon. To deduplicate our uncurated data source of 1.3B images, we compute and use the embeddings generated by Pizzi et al. (2022) and retrieve the A 64 nearest neighbors of each image (using cosine similarity). Considering only neighbors with a similarity >0.6, we extract the connected components of the associated k-NN graph thanks to scalable disjoint set data structure implementation. We then only keep one representative for each component of duplicate images. This results in self-deduplicated data source of 1.IB images.
Relative deduplication To reduce redundancy and also properly evaluate the performance of our features, we discard remaining images of our self-deduplicated data source that are too similar to train and test splits for self-deduplication, with
of our evaluation datasets. To achieve this, we apply a similar procedure stricter similarity >0.45, this time identifying the duplicate components (if any) to which each reference image belong and discarding entircly. This results self- and glatively-doduplican data souree of 744M images.
A.4 Retrieval
We employ two approaches augment dataset retrieval: sample-based and blaster-based. The first one, sample-based, applies to dalasets larger than 1M images and consists in collecting fixed number k of nearest images for each sample image of the dataset to retrieve, effectively trying to multiply by the size of the dataset. We k = 4 for Google Landmarks v2 and ImageNett222k but larger k = 32 make this specific retricval core part of our LVD-142M dataset. For smaller datasets, the second approach, cluster- consists in first clustering our uncurated data source into 100, 000 separate clusters thanks to a distributed k-means implementation. Each cluster should capture different types of image concept and contents. We then pick 10, 000 images from each cluster associated with more than 3 images of the retrieved dataset. As this can result in a very large number reirieved images for some dataset, we restrict such retrievals to a maximum of 1M images to maintain the balance between the different datasets within LVD-142M.
B Implementation Details
B.1 Unsupervised pre-training
For unsupervised pre-training we build on the DINO and iBOT codebases. We use hyperparameters shown in Table 16 ViT architectures described in Table 17
KoLco regularization. We apply the KoLco regularizer with a weight of 0.1 betweon the class tokens the first global crop, for all samples within GPU without cross-commmication for this step.

---

Published in Transactions Machine Learning Research (01 /2024)
Task
Datasct Split
Images
Retricval
Retrieved
Final
classification
ImageNet-222 /
14,197,088
as is sample
14,197, 086
classification
ImageNet-
/-
14,197,00 076
56, 788,344
56, 784,3444
classification
ImageNet-1k train
1,281,167
sample
40,997,344
40,997,344
iine-grained classif.
Caltech 101 train CUB-200-2011 train DTD / train1
3,030
cluster
2,630,000
1,000,000
ine-grained classif. fine-grained classif. fie-grained classif. fine-grained classif. fin-grrined classif.
5,994
cluster 1,300,000
1,000,000
1,880
cluster 1,580,000
1,000,000
(GVC-Aircraff train
3,334
cluster 1,170,000
1,000,000
Floerrs-12 train
1,020
cluster 1,060,000
1,000,000
Food-101 / train
75,750
cluster 21,670,000
1,000,000
fine-grained classif.
Oxford-IIIT Pet trainval
3,680
cluster 2,750,000
1,000,000
fine-grained classif.
Stanford Cars / train
8,144
cluster
7,220,000 000
1,000,000
fineggraind classif.
SUN397 / train]
19,850
cluster 18,950,000
1,000,000
fin-grained classif.
Pascal VOC 2007 train
2,501
cluster 1,010,000
1,000,000
segmentation
ADE20K / train
20,210
cluster 20,720.000
1,000,000
segmentation
Cityscapes / train
2,975
cluster 1,390,000
1,000,000
sogmentation
Pascal VOC 2012 (seg.) / trainaug
1,464
cluster 10,140,000
1,000,000
depth estimation
Mapillary SLS train
1,434,262
as is cluster
1,434,262
depth estimation
KITTI/ / train (Eigen)
23,158
3,700,000
1,000,000
depth ostimation
NYU Depth V2 train
24,231
cluster 10,850,000
1,000,000
depth estimation
SUN RGB-D / train
4,829
cluster 4,870,000
1,000,000
retrieval
Google Landmarks v2 train (clean)
1,580,470
as is sample
1,580,470
retrieval
Google Landmarks v2 train (clean)
1,580,470
6,321,880
6,321,880
retrieval
AmsterTime new
1,231
cluster
960,000
960,000
retrieval
AmsterTime / old
1,231
cluster
830,000
830,000
retrieval
Met train
397,
cluster
62,860,000
1,000,000
retrieval
Revisiting Oxford / base
4,993
cluster
3, 3,680,000
1,000,000
retrieval
Revisiting Paris base
6,322
cluster 3,660,000
1,000,000
142,109,386
Table 15: Composition of our IVD-142M dataset. We report the list datasets and associated splits used build the datasel, how they were included (as is without retrieval or via sample-based cluster-based retrieval). For retrievals, we indicate the actual number of retrieved images and the final number included the dataset. We chose to include as many datasets as possible the pretraining data order to cover many domains possible. We kept few datasets aside in order to evaluate performance outside of the pretraining domain. More details about dataset usages can be found in Table
30

---

Published in Transactions Machine Learning Research (01 /2024)
Arch.
Drop-rate
LR
Hatch size
DINOv2-8 (distilled)
ViT-S/14
le-3
2048
DINOv2-B (distilled)
ViT-B/14
1e-3
20488
DINOv2-I. (distilled)
VIT-L/14
1e-3
2048
DINOv2-L (from scratch)
VIT-L/14
0.4
3.5e-4
3072
DINOv2-g (from scratch)
VIT-g/14
0.4
3.5e-4
3072
Table 16: Training hyperparameters for DINOv2-S, DINOv2-B, DINOv2-L and DINOv2-g. All models run for 625k iterations with optimizer AdamW, an imitial LayerScale value of 1e-5, weight decay cosine schedule from 0.04 to 0.2, a learning rate warmup 100k iterations, a teacher momentum cosine schedule from 0.994 to 1, and we train in float16 precision in all cases (except for the DINO heads where reduce the gradients in Hloat22).
Arch.
Embed dim
Heads
Blocks
FFN layer
ViT-S/14 (distilled)
384
12
VIT-B/ /14 (distilled)
768
12
18
MLP
VIT-L/14 (distilled)
1024
16
24
MLP
VTT-L, (from scratch)
1024
24
SwiGLU
ViT-g/14 14 (from scratch)
1536
24
40
SwiGLU
Table 17: Architecture details of the ViT-S/B networks used in this work. We MLP
feed-forward networks for distilled models, and SwiGLU (Shazeer 2020) when training from scratch.
EMA update for the teacher. The teacher is initialized with the same state as student, and an exponential moving average of the student network, with momentum value in [0.994, 1.0] following cosine schedule. It is updated at the end of every training step.
B.2 High- Resolutin adaptation
We initialise the model with the pretrained weights then train it for 10k iterations with the same procedure as the original pretraining. All the schedules are kept the same as in the original training, but compressed to fit in 10k iterations. All the hyperparameters kept the same in the first pretraining, except the base learning rate which is reduced.
B.3 Linear probing evaluation
For linear probing we define evaluation parameters: the learning rale, how many oulput layers use, whether we concatenate the average-pooled patch token features with the class token (or use only the class token). We train linear layer with SGD for 12500 iterations, using randeoo-reszzed-cropp data augmentation, and perform the following grid search:
learning rate in (0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005. 0.01, 0.10000000000020055000000,
output layers in {1,4}
concatenate average-pooled tokens in no)
We then report the highest accuracy value obtained the validation set is common practice. Note that this grid search is not expensive, because at each iteration we perform inference on the backbone only once, then feed the output to all linear classifiers (cach performing a single matrix multiplication).
C
List of Datasets used
We show in Table 18 the list of benchmarks and datasets used and their purposes.
31

---

Published in Transactions Machine Learning Research (01 /2024)
Dataset
Pretraining Retrieving
Eval.
Task
Citation
(as is)
pretraining
data
ImageNet-Ie 1k
x
Classif.
Russakovsky et al.
2015
Image.Nel-222
/
Dong et al. 2009
ImageNet-
V2
X
Classif.
Recht et al. 2019
ImageNel-Real.
X
Classif.
Beyer et al.
2020
ImageNet-A
X
X
Classif.
Hendrycks et al 2021b
Image.Ne--4
X
Classif.
Hendrycks & Dietterich
2019
ImageNet-Hi
X
Classif.
Hendrycks
2021a
ImageNet-Sil
X
Classif.
Wang et al. 2019)
Food-101
X
Classif.
Bossard ct al. 2014
CIFARA R-
X
Classif.
Krizhevsky et
2009
CIFAR-100
X
Classif.
Krizhevsky a.
2009
SUN397
X
Classif.
Xiao et al. 2010
StanfordCars
X
Classif.
Krause et al.
2013
FGVC-Aireraft
X
Classif.
Maji et al.
2013
VOC 2007
X
Classif.
Everinghari ham et al.
2010
DTD
X
Classif.
Cimpol
2014
Oxford Pets
X
Classif.
Parkhi et al.
2012
Caltech101
X
Classif.
Fei-Fei et al.
2004]
Flowers
X
Classif.
Nilsback & Zisserman
2008
CUB200
X
Classif.
Welinder et al. 2010
INaturalist. 2018
X
Classif.
Van Horn et al. 2018
iNaturalist 2021
X
Classif.
Van Horn el al.
2021
Places-205
X
Classif.
Zhou et al.
2014
UCF101
X
Video
Soomro el. al. 2012)
Kinetics-400
X
Video
Kay et a. 2017
SSv2
X
Video
Goyal ct al. 201
GLD v2
Weyand et al. 2020)
R-Paris
X
Retrieval
Radenović et al.
201
R-Oxford
X
Retricval
Radenović et al.
2018a
Met.
X
Retrieval
Ypsilantis et al.
2021
Amstertime
X
Retrieval
Yildiz et 2022
ADE20k
X
Seg.
Zhou et al. 2017
Cityscapes
X
Seg-
Cordis et. al. 2016
VOC 2012
X
Seg.
Everingham et al.
2010
Mapillary SLS
Warburg et. al. 2020
NYU-Depth V2
X
Depth Depth
Silberman et a.
KITTI
X
Geiger
2013
SUN-RGBD
X
Depth
Song et al. 2015
StlletStreet
X
/
Fairness
De Vries et al.
2019
Casual Conv.
X
Fairness
Hazirbas ot al.
2021
Table 18: List of datasets used.


Reviews:
[{"review_id": "4kMqaDCevi", "reviewer": "Reviewer_7CGu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this is a well polished paper containing an exhaustive set of evaluations.  It’s probably better to view this as a systems paper rather than an algorithmic contribution as there is limited technical novelty.  This being said, I found many of the implementation details in the paper quite interesting and the results are very strong.\n\nThe major contribution of this paper will be a set of high quality and versatile features that are publicly available and usable out-of-the-box, with good out-of-distribution generalization, beats OpenCLIP on many tasks, that are also very good for instance recognition and dense problems in general. This will be useful to the community and based on these strengths, I recommend acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of this paper is to extract pretrained image features that are useful “out-of-the-box” (without fine tuning).  The proposed approach combines a number of recent ideas from the literature and trains on a large dataset curated in a particular way detailed in the paper.  \n\nThe main model is primarily based on a combination of recent iBot and Dino approaches.  The authors also describe a number of techniques for efficient training.  Ablations demonstrate the value of the curation process and each component of the model/\n\nThe final “Dino v2” model is evaluated in many ways : some highlights include\n* Outperforming Open CLIP and EVACLIP on the Imagenet linear probing benchmark, \n* Strong generalization to out of domain data as shown on a number of domain generalization benchmarks, and\n* Outperforming other SSL and weakly supervised approaches on instance recognition, semantic segmentation and depth estimation.  On some datasets, the performance approaches that of fully supervised SOTA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some questions/suggestions to the authors below.  My vote for acceptance is not conditioned on addressing these points.\n\n* I’m curious to understand why the gap to competitors is so big on instance recognition?  What about this problem is hard for the other approaches that Dino v2 solves?  And what is absolute SOTA on this task?\n* There is no ablation of the data pipeline components but this would also be valuable to know — for example, how much of an impact does the de-duplication part of the pipeline have on the final results?\n* In a few places in the paper there are claims about improving the stability of training, but this doesn’t ever seem to really be quantified in any way.\n* I’m somewhat confused about the curation process for the dataset — is the reason why we want to sample from the uncurated set to boost the representation of the underrepresented parts of the curated set?  \n* I did not understand the Nested Tensors paragraph\n* For completeness I would recommend summarizing the iBot losses in more detail since this is such an important part of the approach taken in this paper\n\nSmall nits:\n* Misspellings:\n** Page 1, “Additionnally”\n** Page 11, “litterature”\n* Data sources section on page 4: can you say what “publicly available repository of crawled web data” was used?\n* Table 1: what is the “128k prototypes” line?\n* Fig 4 axis labels are cut off (possibly from overaggressive vspacing)\n* Table 4: there are different versions of the LAION dataset — can you say which one was used here?\n* Robustness analysis section on page 11: add citations for imagenet A,R, sketch\n* Semantic segmentation section on page 14: the third paragraph of this section wants to have a bold header for readability.  This sounds like a very nice result - can you add more specifics here?  What fraction of weights is fine tuned? how much time does it take?\n* Class labels in the qualitative examples are quite difficult to read.", "broader_impact_concerns": "Sufficiently addressed in paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3RcI6OMkpI", "reviewer": "Reviewer_6PH8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well written, easy to understand and follow. The paper makes multiple meaningful contributions, such as efficient implementations, dataset contribution. The code and model claims to be open sourced (will the data be released too?). The experiments are thorough, show the effects of all the components and have useful insights.\n\n\nQuestions:\n\nIs the deduplication done between the uncurated images and the initial training sets (e.g. imagenet) or only on the uncurated images?\n\nThe uncurated dataset is filtered to include images similar to those in the training data, e.g., imagenet. Does this harm generalization to other tasks that have data not like imagenet? For example, OCR, understanding graphs and image data of other types? Table 2 indicates that the uncurated data hurts performance on the chosen tasks, which makes sense, but does the curated version hurt generalization?\n\n\"Untying head weights between both objectives\" Could this be clarified? A bit unclear exactly what is meant by this and seems significant to do.\n\n\"The distilled model outperforms the one\ntrained from scratch on 10 out of 12 benchmarks\" But figure 5(a) shows it outperforming the ViT-L scratch on all 12?\n\nWhy does the small distilled model outperform the larger model on paris and oxford? Any insights on that result?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a pre-training dataset and pre-training method for a large ViT model. The paper proposes and studies the effects of different data sources/filtering, training loss components/tasks, and makes an efficient training implementation. The approach is evaluated on many standard image datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall the paper is good. Answering the above questions would be great.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mE0RaZ09S1", "reviewer": "Reviewer_SnYd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow. Clearly, a lot of work and effort went into writing this paper as well as into generating and presenting the results. All in all, this is a very solid submission.\n### Strengths:\n**Introduction and Related Work**: The motivation of this paper is clearly written in the introduction. The related work section is well structured and comprehensive. I appreciate the authors putting their paper into context at the end of each paragraph in the related work section.\n \n**Methodology**: The authors provide a very detailed description of their method and the loss functions they included in Section 4.  In Table 1, they ablate the different contributions separately, showing that they all improve the final result. The authors also describe the data processing pipeline in a lot of detail, as well as the steps they took to make the approach more efficient.\n\n**Results**: This paper shows a ton of results on various benchmarks, both qualitative and quantitative. The Figures and Tables are all very well designed. Clearly, a lot of time went into how the results should be displayed and presented, and the general quality of the presentation is commendable. For example, the design of Fig. 5 is pretty cool. The structure of the results Section(s) is comprehensive, which is also commendable, given how many results there are.\n\n**Fairness and Bias Analysis**: Given that the authors propose a new foundation model, I appreciate their effort on evaluating its fairness. The choice of benchmark as well as the baseline model they compare to looks reasonable to me.\n\n**Estimating the Environmental Impact of Training our Models**: Similar to the point above, I appreciate the detailed analysis of the environmental cost of retraining a DINOv2 ViT-g model, as well as the cost of the whole project. I find the latter point particularly interesting. The authors write they estimate the cost of the whole project between 0.5k and 1k tCO2eq. Maybe providing some context for this number would be helpful for a reader to have a better understanding of this number. For comparison, a seat on one return flight between London and New York emits around 1.8 tonnes of carbon dioxide (tCO2) (co2.myclimate.org). This means, this project cost an equivalent of between 278 and 556 single person return flights. A Boeing 777 carries 312 passengers, so the cost of the project is equivalent to one-two return flights of a full Boeing 777 between London and New York. There are somewhere between 10 and 36 daily flights from London to NY, which probably makes the environmental impact of this project negligible overall. -> Not sure if this suggestion/ calculation is helpful, but at least for me, putting the total project cost in relation to single-person flights helped me better understand this cost. This may or may not be a helpful addition to this section of your paper.\n \n### Weaknesses:\n\n[major] „We release all the models and the code to retrain DINOv2 on any data.” How about the curated dataset LVD-142M? If the dataset will be released, please explain in what way. Most (if not all?) of the constituent datasets (Table 15, Appendix) are publicly available, but not in a convenient pipeline where it would be easy to build LVD-142M from scratch. Thus, releasing the curated dataset in some form would allow the research community to reproduce the presented results. Further, the released foundation models will be used by a lot of people for a multitude of tasks. We know that dataset biases affect the trained models. Not being able to access the training data also prevents the community from assessing the biases inherited by the models. **I would like to also get input from my fellow reviewers and the Action Editor, but to me, releasing the training data is a necessary requirement to recommend accepting this paper.**\n\nI have listed further issues I found in this paper in the field below.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work is a follow-up and an expansion of the Dino method with several improvements. It scales Dino to a larger dataset, and accelerates and stabilizes training at scale. The authors also propose a new pretraining dataset (LVD-142M), which is diverse and curated in contrast to the uncurated datasets commonly used in SSL training. The authors benchmark their models on a wide range of datasets and report SotA results on many of them. Notably, their method compares favorably to models using images and captions, e.g. CLIP, while DinoV2 only uses images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have written down several questions to the authors in the Requested Changes section. Many points are rather questions than requested changes, and I tried to mark them accordingly. \n\n- RQ: requested change\n- Q: question\n\n[RQ, please provide intuition for the results if possible] In general, I think it is not very clear what it is about DinoV2 that leads to its strong performance. Why does it perform so well?  \n- “A few properties emerge from these models, such as an understanding of object parts and scene geometry regardless of the image domains.” -> Do you have an understanding which part of the loss / the architecture/ the training paradigm is responsible for this emergent behavior? \n- Looking at Tables 6 and 7, do the authors think the superior performance stems from the better dataset (Inet-22k vs LVD-142M) or from using a better objective compared to e.g. iBOT?\n\n[major, RQ]: section 6.2, page 7: “When compared with models trained on ImageNet-22k, training on LVD-142M is also superior on all the benchmarks but ImageNet-1k. This confirms that training on a more diverse set of images improves the quality of the features in **domains that are not covered by this dataset.**” -> The latter statement is very questionable since both ImageNet1K and ADE-20k have been used to curate LVD-142M. LVD-142M has been designed to contain images that are most similar to these datasets, thus, the claim that these domains are not covered by LVD-142M is wrong, in my view. Please reformulate this sentence or use a holdout set of test datasets that were not used for curation. Also, what is Oxford-M? I couldn’t find it in the Tables 15/18.\n\n[Q]: Following up on the last point, we know that IN-22k was also used to filter the uncurated dataset to build LVD-142M. The performance when training on LVD-142M is fairly similar to when training on IN-22k. I wonder whether the images that are most similar to IN-22k are causal for the increase in robustness compared to the uncurated dataset.\n\n### Data preprocessing: \n\n[Q] “For the uncurated data source, we collect a raw unfiltered dataset of images from a publicly available repository of crawled web data. From each web page in the repository, we extract URL links of images from <img> tags. We discards URLs that are unsafe or restricted by domains, and post-process the downloaded images (PCA hash deduplication, NSFW filtering, and blurring identifiable faces). This results in 1.2B unique images.” -> 1) Will these URL be made publicly available? 2) If yes, will the code to download them will be made publicly available? 3) Will the deduplication code for the dataset obtained from the image URLs be made publicly available? 4) Do the authors have an idea how much the uncurated dataset intersects with LAION2B? Why didn’t the authors use LAION2B as their starting point?\n\n[Q/ RQ] “We build our curated pretraining dataset by retrieving images from our uncurated data source that are close to images in our curated sources.” Several questions here:\n\n1)\t[**important**, RQ] Does it matter for the evaluation performance whether a dataset has or has not been used for curation? In the results section, it is very hard to see whether a dataset has or has not been used for curation. For example, in Table 2, as far as I understand, all datasets have been used for curation except for ImageNet-A. Same for Figure 4 and other Tables/ Figures. Intuitively, the performance should improve much more on datasets which were used for curation compared to those that were not. Please investigate whether this seems to be the case or not. Maybe reporting averaged accuracies on datasets that were vs were not used for curation on the models shown in Table 2 would be sufficient. If indeed, the performance increase on the datasets used for curation is stronger than on datasets not used for curation, it should be clearly visible in all Tables / Figures which datasets have and have not been used for curation. And then it should also be discussed as a limitation of this method. \n\n2) [RQ, not relevant for acceptance] Would it work to test DinoV2 on the WILDS benchmark? This should be OOD compared to the image datasets used for curation. In general, I don’t expect this to be a huge issue, the results in Table 7 on i-Naturalist and Places-205 which were not used for curation, are strong. This point is not crucial for acceptance because the authors already present a lot of results on different benchmarks.\n\n3) [**important**, RQ, please provide details in the paper] How did you choose which datasets should be part of the curation pipeline and which should only be evaluated on? Was this an iterative process where more and more datasets were added or was the list final before any models were trained?\n\n[RQ, please provide details in the paper] “Given a query dataset for retrieval, if it is large enough we retrieve N (typically 4) nearest\nneighbors for each query image.” -> Why 4? Has this choice been validated or is it an arbitrary number?\n\n[RQ, not important for acceptance] Please evaluate on ObjectNet. This dataset should not overlap with neither IN-22k nor with LVD-142M and might provide another interesting point estimate of generalization performance.\n\n\nThere are some typos in the text which I outlined below. I have likely missed a few. A round of proof-reading could help eliminate all typos.\n\n\n#### Typos [RQ]:\n-\tIntroduction: „additionnally“\n-\tIntroduction: „all-purposed features“ sounds weird to me. I think “general purpose features” sounds better and would suggest to replace all instances of „all-purposed features“ with my suggestion.\n-\tPage 4: “We discards URLs” -> discard\n-\tPage 7: “We report the Top-1 accuracy on the validation set of ImageNet-1k with a k-NN\n-\tand a linear linear in Table 1.”\n-\tPage 9: “approximate” -> approximately\n-\tPage 17: “This show that” -> shows\n-\tPage 19: “it gives a reasonable guidelines” -> guideline\n\n", "broader_impact_concerns": "I think the training data (the curated LVD-142M dataset) should be released such that dataset biases can be assessed by the community. The foundation models released by the authors will be used by many researchers for many different tasks, and as a community, we should have the ability to study all possible sources of biases and un-fairness inherited by these models.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As acknowledged by all reviewers, it is a solid, high quality paper that should be accepted at TMLR. The main outcome of the paper is a foundation model accompanied by the training dataset (with the code for curation) that the authors have promised to release. Both pretrained features and the dataset are expected to be of high interest to the community.
Decision: Accept (poster)