You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning
Authors: Théo Moutakanni
Maxime Oquab
Marc Szafraniec
Maria Vakalopoulou
Piotr Bojanowski
Venue: NeurIPS 2024 poster (2024)
Keywords: dinov2
self-supervised learning
data-augmentations
scaling law
Primary area: machine_vision
TL;DR: Self-Supervised Learning (SSL) doesn't need hand-crafted data-augmentations, given enough amount of data. It's possible to train a powerful DINOv2 model with less augmentations than all other modern SSL models.

Abstract: Self-Supervised learning (SSL) with Joint-Embedding Architectures (JEA) has led to outstanding performances. All instantiations of this paradigm were trained using strong and well-established hand-crafted data augmentations, leading to the general belief that they are required for the proper training and performance of such models. On the other hand, generative reconstruction-based models such as BEIT and MAE or Joint-Embedding Predictive Architectures such as I-JEPA have shown strong performance without using data augmentations except masking. In this work, we challenge the importance of invariance and data-augmentation in JEAs at scale. By running a case-study on a recent SSL foundation model -- DINOv2 -- we show that strong image representations can be obtained with JEAs and only cropping without resizing provided the training data is large enough, reaching state-of-the-art results and using the least amount of augmentation in the literature. Through this study, we also discuss the impact of compute constraints on the outcomes of experimental deep learning research, showing that they can lead to very different conclusions.

You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning
Théo Moutakanni
Maxime Oquab FAIR at Meta
FAIR at Meta
MICS, Université ParisySaclay t theomoutakanni @meta. com
Mare Szafraniec
Maria Vakalopoulou MICS, CentraleSupélce, Université Paris- Paris-Saclay
Piotr Bojanowski FAIR at Meta
FAIR at Meta
Abstract
Self-Supervised learning (SSL) with loint-Emeedding Architectures (JEA) has led to outstanding performances. All instantiations of this paradigm were trained using strong and well-established hand-crafted data augmentations, leading to the general belief that they are required for the proper training and performance of such models. On the other hand, generative reconstruction based models such as BEIT and MAE or Loint-Embedeen
Predictive Architectures such as I-JEPA have shown strong performance without using data augmentations except masking. In this work, we challenge the importance of invariance and data-augmennation in JEAs at scale. By running case-study on a recent SSL, foundation model DINOv2. we show that strong image representations can be obtained with JEAs and only cropping without resizing provided the training data is large enough, reaching sfac-off-the-ar results and using the least amount of augmentation in the literature. Through this study, we also discuss the impact of compute constraints on the outcomes of experimental deep learning research, showing that they lead to very different conclusions.
Introduction
Self Self-supervised learning (SSL) has sigmificantly improved performance across various tasks. Despite requiring supervision, SSL heavily depends on carefully selected data augmentations (9
35). Previous experimental literature has shown that removing even one augmentation such as random
rescaling or color jittering reduces linear evaluation performance on ImageNetlk [15 41 by least
But reliance image-specific data augmentations limits the generalizability of self-supervised approaches. How can this be applied to graph, time-series or, for example, medical imaging with totally different channels and characteristics? According to multiple studies (1 21
10 points
[ 11 39)
37). the best data augmentations are dependent on the target tasks, and no set hand-crafted data augmentations can solve them all. But is this the case?
The problem the following: SSL methods that do build on such augmentations are based on image reconstruction, and are able to achieve competitive results, though only through fine- tuning 44 25)). On the other hand, jointdembedding architecture (JEA) methods, delivering strong results without fine-tuning and producing linearly separable features, seem largely reliant on data augmentations in light of the seminal SimCLR [TT] 12 study. I-JEPA (2) was the first method to challenge this assumption, but the authors still show significant performance gap when compared to their augmentation- based altermatives. Therefore, one can reasonably believe that the core principles driving JEA methods towards great performance are inherently dependent on data augmentation,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table Comparison of our model trained using RandomCrop, without random resizing nor other photometric augmentations against SSL models that do not leverage hand-crafted augmentations. All other models are reconstruction based, in the pixel space or in the latent space, and use more augmentations than our setup. We only use RandomCrop without resizing, and masking.
ImageNet-Ik lk
Other Classif.
Segm.
Depth
Method
Data
Arch.
val
V2
real
Places205
iNat18
ADE20k
NYUd
AIM
DFN-2B+
H/14
70.0
43.7
64.4
51.7
64.0
13.2
0.862
BEIT
INet-Ik
L/16
73.5
59.0
78.3
57.9
40.8
6.4
0,544
MAE
INet-1k
H/14
78.0
66.5
83.7
56.0
42.2
33.3
0,482
IJJEPA
INet-Ik
H/14
79.3
66.0
83.8
58.4
47.6
31.1
0,537
IJJEPA
INet-22k
g/16
71.3
59.7
78.6
59.1
55.3
32.0
0.521
Ours
INet-22k
L/14
84.0
74.8
88.6
65.1
75.1
43.5
0.411
and the underlying invariance learning that it entails. As side-effect, JEA setups have been cementing this belief in the community. In this context, the present paper successfully disproves the assumption that these augmentations are necessary. We show that it is possible to train state-of-the-art joint embedding architectures without augmentations other than crops without random rescaling, and optionally masking. To the best of our knowledge, train such model using the least amount of daga-angmentattons in the literature. Data cropping without rescaling, along with masking, might be considered two of the only universal data augmentations that can be applied the properties of the data. We show that building such SoTA model simply requires a larger amount of diverse data, longer training schedule, and more careful optimization.
extensively tuned to work with augmentations [9 13 35
broad number of modalities by just using
For most methods, data augmentations prevent model collapse on trivial solutions such as color histogram 7 They can also enhance performance by artificially increasing the diversity and size of datasets using multiple crops per image (9] T0). While preliminary work has attempted to explain the theory behind data augmentations and sele-super learning have conducted experimental analysis of their impact and necessity in SSL [
11 21 37 such analysis has not been conducted on large-scale SSL pretraining with newer methods, restricting the conclusions that can be drawn from their results.
In this work, we aim to determine the role of data augmentations at scale by training models using DINOv2 (35) on multiple datasets of varying sizes (from 1M to 140M images), on multiple model sizes (from Small to Large), and using diverse data augmentation combinations.
Contributions. By doing rigorous experimental analysis through this paper, the first to prove two counter-intuitive claims that go against most previous assumptions:
(i) We show that the impact of data autiaaaammattiionns invariance enforcement in self- superviued learning secondary the impact of their artificial increase in dataset size and distribution.
(ii) We explore when and why we can remove hand-crafted data augmentations with minor impact on performance by looking all scaling aspects of deep learning: data, compute and model size.
In the process, we also provide two by-products on top of our claims:
(iii) We show through our experimental setup that scaling laws are ethereal and that practitioners optimizing for different compute and data budget might find very different conclusions.
(iv) We achieve state-of- satt--of-the-are hand-crafted augmentations. This result is existence proof that joint-embedding architecture methods do not inherently rely on learning invariances defined by data augmentations, challenging the prior beliefs in the community and prompting theoretical understanding of SSL.
the-art art performance on extensive evaluations for a model trained without
2 Related Work
Data augmentation in SSL practice. Data augmentations have been a key component of most self-supervised works, including the early Exemplar-CNN [18). The network would be trained

---

IBOT
Local Crops
Original
Global Crop 1
Global Crop
Data Aug. Strategies
(a) Image
(b) Original
(c) Shared
(d) Crop Resize
Figure Top: Visual description of pretraining losses. In blue: the local to global DINO loss, in red: the global to global DINO loss and in green the latent masked token predic- tion (iBOT) loss. Bottom: Our different augmentation strategies. 'Original' uses sev- eral augmentations (RandomResizedCrr))
ColorJitter: RandomGrayscale Gaussi anBlur, RandomHor zontalFlip and RandomSolarize), 'Shared' uses the same augmentations but shares them between each view of the same image obtained with RandomResizedCie setting only uses wwwdoooReszzedrooo We also introduce a 'Crop' setup which uses RandomCrop without random rescaling and that is visually similar to 'Crop Resize'.
The "Crop + Resize'
using a classification objective, with each class corresponding data augmentations of image in the dataset. In the context of early pretext-task based SSL, when training convolutional neural networks by predicting relative location of patches, Doersch, Gupta, and Efros (16) have observed trivial solutions. The network was overfitting to chromatic aberrations, and the authors used data augmentations to counter this effect. In a similar way, Bojanowski and Joulin (7) have observed that their discriminative model would emphasize trivial factors of variation such as colour. The solution proposed in this work was discard color information and take the output of Sobel filtering as an input to the algorithm, along with image jitter and flipping. In 2018, the community observed jump in performance when evaluating on Pascal VOC 2007 image classification (8] 22 J. the implementation of both works leveraging RandomResizedCrr carried out the first extensive ablation of the impact of data augmentation on performance, and many of the conclusions from this work serve standard in follow-up up SSL work [10] 24 26 35
instead of RandomCrop. The authors of SimCLR [11
most modern joint- coint-embedding self-supervised models learn using at least one loss term that pushes together the representation of two augmented views of the same image. By applying different data augmentations to the two views, they force the model to map the augmented characteristics to the same point, enforcing invariance in the model representation.
Sensitivity of learning to data augmentation. With the widely accepted view of the importance of invariance learning for SSL models, several works have studied the impact of the choice of data augmentations on downstream performance: Xiao al. (52) show that the wrong choice of way to reduce the mutual information between different views an image to improve results. Bendidi et al. [5] show through experiments that the choice, amplitude and combination of transformations effeets the learnt representation performance in downstream tasks and that the benefit aataeauumentttions depend each specific class. Purushwalkam and Gupta 37 show that poor choice of cropping parameters can harm performances.
augmentations can have detrimental effect; Tian et al. [46] consider data augmentations
At the same time, self-supervised learning algorithms based on image reconstruction don't 't use many data augmentations. Models trained reconstruct missing patches directly the pixel space 4 25 33 have led to significantly lower linear-evaluation performance but did not leverage any photometric or hand-crafted augmentations. El-Nouby et (32) show that BEIT more robust to smaller dataset size. Recently, I-JEPA [2]. based on reconstruction in the feature space obtained strong performance

---

without using photometric augmentations. The authors split the comparison against other SSL frameworks according to the type of augmentations used (see Table in (2). suggesting data augmentations make for an unfair advantage in SSI. training. It is worth noting that reconstruction based methods still use random resizing in their augmentations.
Theoretical studies. Furthermore, a series of theoretical works have been studying the apparently critical data augmentation for SSL algorithms. For example, Kügelgen et al. (30) prove, under some assumptions, that data augmentations allow isolating content from style, when using InfoNCE as an
objective, as in SimCLR. Eastwood al. [19] follow up on this work, and propose using structured data augmentation framework in order to disentangle content from style in the learnt representations: the goal is avoid discarding style information during the learning process. Zhang and Ma (53) propose splitting the multiple operations involved in data augmentation pipelines in order to produce a hierarchy of augmentations: invariance each augmentation translated into a loss applied at different stages of the backbone, leading to accuracy improvements. Notably, they mention that "The most indispensable process in contrastive learning is the data augmentation module." " Saunshi inductive biases applied to the learning algorithm,
et al. (22 propose seeing data augmentations and derive theorems valid in the case of linear representations, in order to further understand the principles driving the success of contrastive learning.
In this work, we provide critical result regarding the importance of data augmentation, that dramatically improves our understanding of the core principles of self-supervised learning, prompting renewed theory: data augmentations are not necessary and only cropping without resizing is sufficient to train strong SSL joint-embedding architecture models.
Approach
The goal of this work is to study the importance of data augmentation for training joint embedding architectures. In this section, we provide a quick recap of the SSL training algorithm that we use for our study. We then provide a precise description of the experimental setup, with details about the training regimes and data augmentation strategies.
3.1 Short description of the DINOv2 algorithm
We tailor our study around the algorithm that was used for training the recently proposed DINOv2 (35) models. We hypothesize that learnings from this study should apply to other SSL methods, but we chose to focus our experimental work on the best modern JEA method. That way, we can provide strong counter-argument
the assumptions used in previous literature. We briefly summarize the components of the DINOv2 learning algorithm, and for more details. we point to the original publication 35). Figure [ provides schematic diagram of the loss terms used for training the model.
Global DINO loss. The DINOv2 training loop is largely based on the DINO algorithm (9). Given a student (s) and teacher (t) network, and views of the same image ei and 22, the student network is trained by minimizing loss between the two representations:
((((1), ((())))
The teacher network, is taken as moving average of the student. We refer the reader to 9 for more details about the definition of l.
Local reconstruction-basee iBOT loss. DINOv2 models were trained using additional local masked-image reconstruction loss. As opposed to MAE or BEIT, the reconstruction task in iBOT happens in the latent space, not in the pixel input space. Given view 21 as used in the DINO loss, we create an impaired version ún by replacing a fixed percentage of patches with a [MASK] token. The target is to reconstruct the value of masked features using the remaining context:
(s((1), ((())))
If several views are used, the iBOT loss can be applied to every such view. The strategy for choosing the masked patches is
of the method.
Multicrop and data augmentations. practice, one does not use only two views vi and 02- Following [10]. DINOv2 uses Multicrop, with two global crops and eight local crops. Global crops are obtained using RandomResizedCCro with scale parameter of [0.32, 1.0). then resized 224 pixels. For local crops used [0.05, 0.32] resized to 98 pixels instead. Each crop, be it global

---

or local, undergoes series of photometric augmentations: ColorJitter, RandomGrayscale, GaussianBlur, Randomifo11zontaaFPL and RandomSolarize. We refer the official DINOv2 repository for the exact implementation details
3.2 Experimental setup
Hyperparameters and training regimes. The original DINOv2 repository proposes sets of hyperparameters for training SSL models. The first set (that we refer to as the low-compute regime) corresoonds to setup for fast experimental iterations, designed to run for 100 epochs (125k iterations) with a batch size of 2048. This setup is optimized for performance on the ImageNet- kk dataset (corresponding the low-data regime). The second set (high-compule regime) is designed for longer training runs 500 epochs (625k iterations) and is optimized for performance on larger datasets, such as ImageNet-224. (the high-data regime). These two sets differ in many values, including notably the learning rate schedule and warmup, weight decay, batch size, patch size. In our experiments, we use both sets compare the behavior of the algorithm against data augmentation. We provide detailed empirical discussion of the differences in Sec.
Modifications applied data augmentation. In our study, apply different configurations of data augmentations. We provide here the details the four configurations that we consider.
we use all the data augmentations, exactly like the original DINOv2 work. Each crop is passed through series of random photometric augmentations independently.
Original:
Shared: each view of an image is obtained using different crop, but apply the exact same photometric augmentation to all views. Two images in the batch will have different photometric augmentations applied to them.
Crop Resize: we create views with no photometric augmentations all, only RandomResizedCrop
Crop: we create views without any photometric augmentations at all. We also don't use BandomResizzedroop Instead, we use a Resize to 256 x 256 pixels and Crop to 224 or 98 pixels, leading to views with the exact same aspect ratio and zoom, with random resizing. This setup is very close a masking strategy where different sets of tokens would be dropped, up to pixel alignment patch boundaries.
Figure I provides an illustration of the images that we obtain by applying these configurations. We omit the 'Crop' setting from the illustration given that the 'Crop' and 'Crop+Resize" provide images that are hard to qualitatively differentiate with a naked eye, even if they differ in term of final performances. the end, our 'Crop' strategy only applies two augmentations between generated views: cropping without random resizing and random patch masking, which are two very similar augmentations (the cropping drops pixels outside of a field-of-views and masking inside of it). We chose to keep those two specific augmentations as we think that they can be generalized to nearly all kind of data, from graphs to 3D medical images and time-serics.
Implementation details. For our study, we use the standard ImageNet-IK Ik [4]] and ImageNet- 22k (13] datasets, well the LVD-142M dataset originally used DINOv2 [35] The selection of these datasets allows us to benchmark the impact of their size in the trainings. The pre-training code performs 100 epochs in 27 hours on compute nodes of 8 A100-80GB GPUs each, where epoch is set to 1250 iterations. For evaluations, we follow DINOv2 linear evaluation protocol on
multiple different tasks including classification (ImageNet- Ik (41) Places205 (53). ImageNet-v2 (38).
ImageNet-Real (6]. iNaturalist' 18 [49]. Flickrl 2ogos32 40). GTSRB [45). RP2K (36). Products-
10k (3), FireRisk [3),, RESISC [14], MIMIC (28), CheXpert (27. VinDr (31] NTH-14 (T)).
segmentation (ADE20k 55) and depth estimation (NYU-Depth v2 44
5

---

Table 2: New domains classification results of DINOv2 VIT-L trained on ImageNet- when
varying data augmentations. None of those domains were used in the pretraining data of the models.
Domain (Metric)
Remote Sensing (Acc.) FireRisk
Medical Imaging (AUROC)
Dataset
RESISC
MIMIC
CheXpert
VinDr
NIH-14
Original
59.0 =
91.9
75.3
83.3
80.0
72.5
Shared
60.8 1.8
91.6 0.2
75.8 0.5
85.6 1 2.3
81.0
$1.00..0 1.0
74,4 T 1.9
Crop+Resize
60.7 f 1.7
91.1 0.8
75.8 0.5
84.1
f 0.8
80.8 0.8
73.7
1.2
Crop
60.9 1 1.9
92.3 0.4
75.5 0.2
83.0
82.5 2.5
74.2 1 1.7
Experiments and discussion
4.1
Why do need augmentations?
The first question want answer is: why do need data augmentations? Are they systematically needed, because they constitute a core component the modeling? Are they a trick that facilitates the optimisation of deep neural networks, exactly like in supervised learning?
The role data-augmentations.
the context of supervised learning, data augmentations have been used as a means of virtually augmenting the training dataset (2T). That way, additional training samples could be obtained, reducing the risks of overfitting and improving the robustness of the models trained. In self-supervised learning however, data-augmentations have been usually modeled as a way to enforce representations be invariant toward specific image's characteristics, such as
color or scale [46 52 (3).
Invariance can be harmful. Most of the self-supervised methods and their data-aur augmentations have been optimised on ImageNet. While DINOv2 has probably been optimised on more benchmarks, some tasks and domain are missing. In Table we show the results of removing hand-crafted aatt-auummetttionns on new domains (not used for training) and in Table 3 (left) new tasks (in the training set but not used for hyperparameter search). We can see that data- aata--agmmetations are actually harmful on most of those, meaning that understanding the real impact of augmentations and invariance while removing them is very important to increase SSL robustness.
Disambiguating the impact of data augmentation in self-supervised learning, We developed the 'Shared' configuration that still applies augmentations to artificially increase the number of training samples but doesn't i enforce invariance to those augmentations. We compare invariance in Table (right) and show that the 'Shared' setup has effectively lower invariance than the 'Original' one. We quantify it using the average cosine similarity of 16 augmented views of the same image, repeated on 100 images from different classes. Higher cosine similarity means higher invariance. We also report a mean/std normalized similarity, computing the metrics using negative pairs' cosine similarity.
Enforcing invariance losses its usefulness at scale. By comparing the 'Shared' and 'Original' results in Fig. 2 we can see that both setups reach way above 80% linear-probe accuracy on ImageNet when trained all three datasets. While the 'Original' setup gets better performances, can
http://gihhbbbommocccccokeeresre (Apache-2.0 license)
Table 3: (left): New task classification results of DINOv2 ViT-L trained on ImageNet-224, when varying data augmentations. None of those tasks were used to tune DINOv2's hyperparameters. (right): Measure of invariance toward augmentation. Higher cosine similarity means higher invariance the model embeds multiple augmentations of the same image to closer vectors.
Task
Logos
Signs GTSRB
Products
Cosine Normalized Similarity Cos. Sim.
Metric
Dataset
FlickrLogos32
RP2K
Products-10 100k
Original
80.1
80.3
92.8
66.5
Original
0.69
2.69
Shared
78.2 1.9
78.9 | 1.4
1
68.1 f 1.6
Shared
0.64
2.62
C+R
80.8 1 0.7
76.3 1 4.0
93.3
67.7
1.2
C+R
0.58
2.47
Crop
76.9 3.2
85.1 1 4.8
93.4 0.6
67.3 0.8
Crop
0.56
2.44

---

ImageNetIk
Places 205
NYU-Depth
85
Top-1 Acc.
65
RMSC
60
.45
Top-1 Acc.
.4
80
55
INaturalist 18
ADE20K
Top-1 Acc.
45
70
Stared Crep
mloU
75
Resize
60
40
-50
35
IN IN22KLVD-142M,
IN IN22k LVD-142M
INIK IN22k LVD-142M
Figure 2: Impact of dataset size when varying data augmentations. Results of ViT-L linear evaluation benchmarks, including classification (ImageNetlk, Places 205 and Naturalist18), depth estimation (NYU-Depth) and segmentation (ADE20k). Cropping without resizing (" ('Crop') reaches very high performances on a wide variety benchmarks, given that the dataset size is large enough.
that the gap decreases with the amount of data, from - -1.3% when trained on ImageNetlk and 110% with ImageNet22k,, to -0.4% with LVD- 142M. As show in Fig. 3 (left), the gap between the 'Shared' and 'Original' setups also decreases with the number of training epochs. According to experiments in prior literature [1 39) data augmentations enforce invariance and obtain competitive performance. For example, when trained and evaluated on ImageNetlk, SimCLR and BYOL respectively lose 27.6 and 13.1 points of accuracy in the "Crop*Resize"" setup [39] In contrast, our 'Shared' setup only loses 1.2%, disproving this view.
required the sample level to
Increasing the sample count is the key to strong SSL. We know that ecconstruction-base models like BEIT, MAE or I-JEPA don't 't need handcrafted autaettgmmttttionn on ImageNetlk, k, and according to El-Nouby et al. [32]. "denoising autoencoders
more sample efficient than joint embedding techniques". Those findings led us to conjecture that the real impact of lata-augmentation is to artificially increase the number of samples and allow JEA reach good performance with less data.
To assess this, we compare the "Crop + Resize" and 'Shared' settings. Both settings do not enforce invariance, but they differ because the 'Shared' method artificially increases the sample count of the pretraining dataset. We show in Fig. 2 that the gap with respect to the 'Original' setting is way bigger for CCrop + Resize' setting the compared to the 'Shared' one when use ImageNet to pretrain our models. Without the sample count increase data-augmentations. DINOv2 reaches lower performance on all five benchmarks when pretrained
small dataset (Fig. 2
Data-augmentations play the same role in supervised and self-supervised learning. All those results lead to the same conclusion: invariance is useful but not necessary for the learning of JEA, and augmentations artificially increase the number of samples the pretraining dataset. This also explains why reconstruction-bant based methods do not need as many augmentations: they are more data efficient But this not their only role, as data-augmented setups perform better in shorter trainings Fig left). Since setups eventually converge to the same result with sufficient data, this
ImageNetlk
ImageNe122k
LVVD-142M
LVD-1 142M
Top-1 Acc.
Original
75
Shared
Crop Resize
70
Crop
100
300
500
100
300
500
100
300
500
H
Figure 3: Impact of data augmentations when we scale the number of training epoch (left) or the VIT architecture size (right) on the accuracy of linear probe on ImageNetIk ViT-L when pretraining on ImageNetlk, k, ImageNet22k and 142M. The 'Original' and 'Shared' setups scale with the number of epochs for all datasets, but the 'Crop" and "Crop*Resize" setups only scale with larger datasets.

---

suggests data augmentations are likely linked to easier optimization rather than to core learning signal. They are just another
4.2 Can we remove hand-crafted augmentations totally?
We focused on what was the impact of data augmentations in modern self-supervised learning using the 'Original", 'Shared' and 'Crop Resize' settings. We will now show that we can reach state-of-the-art performance without hand-crafted data-augmentations and resizing.
Scaling dataset size. Given that DINOv2 is more robust than other methods when training on ImageNet Ik without handcrafted data-augmat auggeemattntti (39), and knowing from Sec. 4.1 that data augmentations mainly increase artificially the dataset size, we hypothesize that self-supervised JEA can reach SOTA performance by increasing the size of the pretraining dataset used, even when removing the random resizing. We can see in Fig. 2 that increasing the dataset size has a notable property: all four settings converge to the same performances and the gap almost disappears between settings with and without hand-crafted augmentations, displaying non-decreasing scaling curve with the number of samples. The linear- -robbe accuracy gap decreases below 1% on ImageNetlk k and remains small on other benchmarks.
Scaling pretraining epochs. The highest performances were obtained when we pretrained our models for 500 epochs on larger datasets and 100 on the smaller one. However, a question arises: what happens if we train for longer on ImageNetlk, and faster on ImageNet22k and LVD-142M. In Fig. 3 (Ieft) we show that the behavior of low and high data regimes differ. While the larger datasets show non-decreasing scaling curves with respect to the number of epochs, ImageNet1 k has an uncoupling between the settings with and without autta-augmenttiios.. We notice that training longer on smaller scale dataset is harmful for performances when we don't 't use the dataset size increase property of tataaaaugmentttions. This is probably a consequence of overfitting and another proof of our first claim in Sec. What is more compelling is that, following the 'Shared' setting scaling curve, the "Crop+Resize" and 'Crop' settings' performance increases with longer pretraining, reducing the gap with the 'Original' one and converging to the same performance.
Data augmentation and model capacity. In Fig. (right), report ImageNetlk linear results when varying both fata-augmentation strategies and model size. We can see that the gap between the best setup without hand-crafted data-augmentations and the classical setup (' ("Crop+Resize" versus 'Original') increases when we use bigger model size, going from -0.6% for a ViT-S and ViT-B to 1.1% for ViT-L and -01-1. 11.5% for ViT-H. This means that as we scale our model, we need more augmentation to increase the amount of data and reduce overfitting. This is in line with the most common understanding of scaling laws in experimental deep learning (29]: need to scale compute, model size and data at the same time. This also explains why previous work [1] (39] had a hard time scaling self-supervised JEA without data-augmentattons. They needed larger dataset, more compute and larger models, but all previous experimental studies on the impact of fata-auumentations in SSL were mainly applied to ImageNetIk with smaller ResNets. This also explains why anta-augmentation strategies became more aggressive during the evolution of self-supervised methods from its inception to modern methods like DINOv2, when people were scaling models but not datasets.
'Crop'
Role of the 'reconstruction-ben based' iBOT loss. While DINO and iBOT use the exact same loss, they differ in terms of target. With DINO, the teacher and the student are looking at two different views of the same image, with either different data augmentations in the 'Original' setting, the exact same in the 'Shared' one, augmentation at all with the 'Crop' one. In the iBOT case however, the student and the teacher see the same exact augmented view, but the student has some masked tokens.
Table 4: Impact of the iBOT loss on linear evaluation for multiple datasets for VIT-L trained for 500 epochs on SVD-142M We compare results with and without using masking and the iBOT loss.
INIk
18
ADE20k
NYU-Depth +
iBOT + masking
X
X
X
X
Data Aug.
Original
84.0
84.4
74.7
73.6
41.4
45.6
0.430
0.413
Shared
82.6
84.0
74.1
73.6
40.7
44.8
0.448
0.414
Resiee
82.6
83.5
71.3
71.1
40.3
44.4
0.437
0.415
Crop
82.1
83.4
69.8
71.4
40.6
43.5
0.445
0.417
8

---

It means that regardless of the augmeenggtnttoo strategy we use, both the teacher and the student see the same thing in the larger crops. Both settings are similar, and despite iBOT using a conditional latent variable, it is possible to see it as masked image modeling task in latent space, which is more akin to joint embedding predictive architecture (JEPA) than joint embedding architecture (JEA). To remove all confounding factors, we ablate the iBOT loss by setting its weight to zero and by removing the masking strategy Table We can see that it is still possible to train a very good model without iBOT and masking even if we "Crop' settings increases sli slightly, eg. from 1.0% with iBOT to 1.9% without, for ImageNet linear probing, from 0.04 to 0.015 RMSE for NYUd, but the performances remain competitive; more importantly, it in fact possible to train DINO alone without hand-crafted data-angmentations.
stricter definition of JEA. The gap between the "Original' and
Comparison against other models. We compare our model trained on ImageNet22k with the 'Crop' setting against other SOTA models trained without hand-crafted augmentations in Table T It's worth noting that our setup uses the smaller amount of augmentations: AIM (34) uses "Crop+Resize" and RandomHorizon tww.iiootallllaaa. talFlip, BETT 50 uses "Crop+Resize" and ColorJitter, MAE (25) uses "roop+ and I-JEPA (2 uses Crop+Resizee Our model outperforms alternatives wide variety tasks by significant margin without hand-crafted data-augmentations.
4.3
Is scaling all you need in research?
ImageNetlk
Param.
INIk
INat18
ADE20k
NYUd
85
High Compute
49.6
28.5
20.0
0.650
75
Layer Decay
47.6
26.8
20.7
0.693
Prototypes
48.8
27.6
20.7
0.661
Head Dim.
49.0
27.8
17.7
0.654
Momentum
49.7
28.0
18.2
0.654
ADH20K
Warmup Epoch
51.3
26.0
24.4
0.605
Warmup Temp
51.3
28.5
22.1
0.627
mloU
Drop Path Rate
52.3
30.6
22.9
0.631
Shared
Weight Decay
53.4
30.5
21.7
0.646
Crop
Kelize
LR
74.2
47.6
33.5
0.517
Crop
Low Compute
78.7
54.2
37.2
0.486
Low Compute
High Compute
Figure 4: (left): Impact of hyper parammtrr optimisation's target compute on the accuracy of linear probe on ImageNetlk and ADE20K for models trained on ImageNetlk: We can see that optimising for high compute leads to poor performances on the 'Crop" and 'Crop+Resize" settings, which the opposite of findings when we optimize for low compute. (right): Impact of hyperparameter tuning using 'Crop+Resize" on ImageNetlk for 100 epochs. For each line, we switch only one hyper- paremmeeeee from the configuration optimized on ImageNet22k for 500 epochs (High compute) to the one optimized for ImageNetlk k for 100 epochs (Low).
Same experiment, different outcomes. During this study, we used two sets of aaper-parameters optimized for different use cases. One optimized for training models on the smaller scale dataset ImageNetlk during 100 epochs (low compute) that we used for all trainings on ImageNet k, and one for training those models on the larger datasets ImageNet22k and LVD-142M during 500 epochs (high compute) that we used for all trainings on ImageNet22k and LVD-1422 142M. Indeed, training on ImageNetlk using pyper-parameters optimized the larger datasets gives different conclusions.
We show in Fig. (left) the ImageNetlk linear-probe accuracy and ADE20K linear segmentation mloU obtained after training ViT-L on ImageNettk k when we use the 'Low Compute' and 'High Compute' sets
Interestingly, DINOv2 doesn't work when trained on ImageNetlk using the 'Crop' or "CoopRResize" settings with the 'High Compute' hyper-parameterr only 46% linear accuracy, despite obtaining SoTA performances when trained on ImageNe122k and LVD- 142M with the exact same hyper-parameters, providing nice scaling curves in the process. However, when we optimize the hyper-parameter for the specific scale of data (ImageNet1k and compute (100 epochs),
It reaches
that it reaches 73.5% with finear-probe presented before.

---

With the 'High Compute' hyperparameters, going from the 'Original' to 'Crop+Resize" setups makes DINOv2 lose 29.19 19.1% on the Linear-probe evaluation, which is phase with previous literature (39). This means that if our only goal was to get scaling curve leading to SoTA models, without optimizing for lower compute and data, we wouldn't have been able to disprove the assumption from previous studies that pretraining strong JEA without is impossible when using ImageNetIk [II (9).
with minimal loss in performances
The issue with high-seale experimental studies. In Fig. (right), we show that it's not trivial to go from one hyper-paratee parammer setup to the other, gains being non linear when tweaking parameters one at a time. Such issue already arose previously in the literature. For example, it was originally thought that ViT needed hundreds of millions of images
reach good performances [17]. but such claim has been proven false in later work (20, 48). The same happened with Large Language Models, with
work scaling them hundred of billions of parameters like OPT [54] reaching 175B before newer
methods showed improved performances with smaller sizes as in 23 47
Conclusion
In this work, we challenged the common belief that joint-emhedding architectures require data augmentations to deliver strong performance. The underlying interpretation of that belief, followed by theorists, is that the core learning signal in these methods relies on mapping differently augmented samples the same embedding knowledge in the design of data augmentations. Also, the photometric augmentations of images (such as blurring and color jittering) cannot be easily mapped to other modalities (such as speech or text). As such, this belief restricts the scope of high-perfoneee perrhhrmmrraccee JEA methods to images.
the latent space. In particular, this reasoning implies expert
Our experiments show that this is not true, as is possible to achieve performance similar data- augmented pipelines with the DINOv2 SSL algorithm without using what is referred to, in prior literature, hand-crafted data augmentations. Therefore, our results prove that joint-embedding architectures do not necessarily require domain knowledge during training to deliver ofte-offthhear performance. The data augmentations merely influence training by increasing the dataset size, as shown by our 'Shared' setup, which does not enforce the learning of invariances.
Additional experiments with even weaker data augmentations, such as the 'Crop' setup, show that given enough training iterations and data, the models will converge to comparable performance as their stronger data-ted counterparts. Our results suggest that prior conclusions largely depended on the smaller scale of experiments. However, we note that strong data augmentations lead to stronger performance in the case of shorter training, suggesting that they also help ease the optimization process during learning. We hope future work can shed more light on that observation.
Limitations. Given the cost of pretraining models, we focused on the leading self-supervised joint-embedding architecture, DINOv2, that provides strong performance at different data scales. While the results might be different for other algorithms, in principle, the existence proof that we offer in this work still holds. We noted that data augmentations tend to ease optimization, but the underlying mechanism not known at this point. We merely claim that handcrafted photometric augmentations are optional for successful learning. While it has measurable effect on some setups, this does not invalidate our conclusions. For large architectures, there is still a tiny gap in performance between augmented and non -ugmented setups for long training. While hypothesize that this might be due to slight variations in the optimal hyperparameters, there could also be a deeper reason that is not covered within the scope of this study.
Statement of Broader Impact. The total compute cost. for this study was approximately 1.50k GPU-hours. We used strong models to blur all human faces in our web-based image data pool.
10

---

References
[T] Y. M. Asano, C. Rupprecht, and A. Vedaldi. A critical analysis of self-supervision, or what can learn from single image. 2020. arXiv: 13132 [v]
[2] M. Assran et al. SuperSuuee Learning from Images with a Joint-Embedding Predictive Architecture" In: arXiv preprint arXiv:2301.08243 (2023).
[3] Y. Bai et al. Products-IOK: A Large-scale Product Recognition Dataset. 2020. arXiv: 2008
10545 [cs.CV]
[4] H. Bao, L. Dong, and F. Wei. "BEiT: BERT Pre-Training of Image Transformers". In: arXiv
preprint arXiv:2106. 08254 (2021).
[5] 1. Bendidi et al. No Free Lunch in Self Supervised Representation Learning. 2023, arXiv:
2304. 11718 [cs
[6] I.. Beyer et al. "Are we done with ImageNet?" In: CoRR abs/2006.07159 (2020). arXiv:
2006 07159)
[7] P. Bojanowski and A. Joulin. "Unsupervised learning by predicting Noise". In: JCML. 2017.
[8] M. Caron et al. "Deep clustering for unsupervised learning of visual features". In: ECCV.
2018.
[9] M. Caron et al. "Emerging properties in self-supervised vision transformers". In: ICCV. 2021.
[10] M. Caron al. "Unsupervised learning of visual features by contrasting cluster assignments".
In: NeurlPS. 2020.
[11] T. Chen et al. "A simple framework for contrastive learning of visual representations". In: preprint arXiv:2002. 05709 (2020).
[12] T. Chen et al. "Big self-supervised models are strong semi-supervised learners". In: Advances
in neural information processing systems 33 (2020), pp. 22243-22255,
[13] X. Chen al. "Improved baselines with momentum contrastive learning". In: preprint arXiv:2003.0423 04297 (2020).
[14] G. Cheng. J. Han, and X. Lu. "Remote Sensing Image Scene Classification: Benchmark and State of the Art". In: Proceedings of the IEEE 105.10 (Oct. 2017), pp. 1865-1883, ISSN:
1558-2256. DOT: 10. 1109/ jproc 2017 2.....01..2277 2675998
[15] J. Deng et al. "ImageNet: A Large-Scale Hierarchical Image Database". In: CVPR09. 2009.
[16] Doersch, Gupta, and A. A. Efros. "Unsupervised visual representation learning by context prediction". In: ICCV. 2015.
[17] A. Dosovitskiy et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition
at Scale". In: preprint arXiv:2010.1 1929 (2020).
[18] A. Dosovitskiy et al. "Diseriminative unsupervised feature learning with exemplar convolu-
tional neural networks". In: 7'PAMI (2016).
[19] Eastwood al. "Self-Supervised Disentanglement by Leveraging Structure in Data Aug-
mentations". In: arXiv preprint arXiv:2311.000 1.088/5 (2023).
[20] H. Gani, M. Naseer, and M. Yaqub. How to Train Vision Transformer Small-scale Datasets?
2022. arXiv: 2210. 07240 [cs.CV]
[21] J. Geiping et al. How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization. 2023. arXiv: 2210..0441 [cs LG]
[22] S. Gidaris, Singh, and N. Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. 2018. arXiv: 1803. 07728 (cs. CV]
[23] F. Gloeckle et al. Better & Faster Large Language Models via Multi-token Prediction. 2024.
arXiv: 2404 19737 [cs.CL]
[24] J.BB. Grill et al. "Bootstrap your own latent: A new approach to self-supervised learning". In: NeurIPS. 2020.
[25] K. He et al. "Masked autoencoders are scalable vision learners". In: arXiv preprint arXiv:2111.0637 (2021).
[26] K. He et al. "Momentum contrast for unsupervised visual representation learning". In: CVPR. 2020.
[27] J. Irvin et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. 2019. arXiv: 1901. 07031 [cs.CV]

---

[28] A. E. W. Johnson et al. "MIMIC-CXR, de-identified publicly available database of chest radiographs with free-text reports". In: Scientific Data 6.1 (Dec. 2019), p. 317. ISSN: 2052-
4463. DOT: 10. 1038/ 9111///11979--9-33333
[29] J. Kaplan et al. Scaling Laws for Neural Language Models. 2020. arXiv: 2001 08361
[cs .G]]
[30] J. von Kügelgen et al. Self-Supervised Learming with Data Augmentations Provably Isolates Content Style. 2022. arXiv: 22006 04619 [stat. .KL]
[31] H. Q. Nguyen et al. VinDr-CXR: An open dataset of chest X-rays with radiologist's 's annotations.
2020. arXiv: 2012. 15029 [V]]
[32] A. El-Nouby al. Are Large-scale Datasets Necessary for Self-Supervised Pre-training? 2021.
Xiv: 2112. 10740 [cs.CV]
[33] A. El-Nouby et al. "Scalable Pre-training of Large Autoregressive Image Models". In: arXiv preprint arXiv:2441.08855 (2024).
[34] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models. 2024. arXiv:
1401.08544 [cs.CV]
[35] M. Oquab et al. "Dinov2: Learning robust visual features without supervision". In: arXiv preprint arXiv:2304.0710
(2023).
[36] J. Peng, C. Xiao, and Y. Li. RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classification. 2021. arviv: 22066. 12634 [cs
[37] S. Purushwalkam and A. Gupta. Demystifying Contrastive Self-Supervised Learning: Invari- ances, Augmentations and Dataset Biases. 2020. arXiv: 2007 13916 [cs.CV]
[38] B. Recht et al. "Do ImageNet Classifiers Generalize to ImageNet?" In: Proceedings of the 36th International Conference on Machine Learning. Ed. by K. Chaudhuri and R. Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, June 2019, pp. 5389-5400.
[39] P. H. Richemond et al. "BYOL works even without batch statistics". In: preprint
ar arXiv:2010.10244 (2020).
[40] S. Romberg et al. "Scalable logo recognition in real-world images". In: Apr. 2011, p. 25. DOI:
10. 1145/1991996. 1992021
[41] O. Russakovsky et al. "Imagenet large scale visual recognition challenge". In: IJC (2015).
[42] N. Saunshi et al. "Understanding contrastive learning requires incorporating inductive biases".
In: International Conference on Machine Learning. PMLR. 2022, pp. 19250-1999 19286.
[43] S. Shen et al. FireRisk: A Remote Sensing Dataset for Fire Risk Assessment with Benchmarks Using Supervised and Self-supervised Learning. 2023. arXiv: 2303. 07035 [cs.Cu) cV]
[44] N. Silberman et al. "Indoor segmentation and support inference from rgbd images". In: ECCV. 2012.
[45] J. Stallkamp et al. "Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition".. In: Neural Networks 0 (2012), pp. ISSN: 0893-6080. DOr: 10. 1016/] neunet. 2012.022200 02.016.
[46] Y. Tian al. "What makes for good views for contrastive learning?" In: Advances in neural information processing systems 33 (2020), pp. 6827-6839.
[47] H. Touvron al. "LLaMA: Open and Efficient Foundation Language Models". In: arXiv preprint arXiv:2302. 13971 (2023).
[48] H. Touvron et al. Training data-efficient image transformers & distillation through attention.
2021. arXiv: 2012. 12877 [cs.Cv]
[49] G. Van Horn et al. "The inaturalist species classification and detection dataset".. In: CVPR.
2018.
[50] W. Wang et al. "Image as a Foreign Language: BEiT Pretraining for All Vision and Vision- Language Tasks". In: arXiv preprint arXiv:2208. 10442 (2022).
[51] X. Wang et al. "ChestX -Ray8: Hospital- Scale Chest X-Ray Database and Benchmarks on Suukev--e Classification and Localization of Common Thorax Diseases". In: 2017
IEEF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017. DOI:
10. 1119/////// .vpr.2017..... 201773666
[52]
T. Xiao et al. What Should Not Be Contrastive in Contrastive Learning. 2021. arXiv: 2008 05659 [V]
12

---

[53] J. Zhang and K. Ma. "Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 16650-16659.
[54] S. Zhang al. "Opt: Open pre-trained transformer language models". In: arXiv preprint arXiv:2205.0r 200600068888. (2022).
[55] B. Zhou et al. "Scene parsing through ade20k dataset". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 633-641.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide an existence proof that possible to train a self-supervised joint-emhedding architecture (based on DINOv2) that reaches state-of-the-ar performance without relying on invariance to data augmentations, as was previously deemed necessary. Table]ccompne with alternative methods not using domain-specift data augmentations, while Figures 2 show that the performance almost matches the data-augmented baseline.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: "Limitations" paragraph is present at the end of the conclusion section. The main limitation is in the generality of the claim regarding alternative methods. The existence proof still holds true though, as the example we provide sufficient to disprove prior community belief that SSL joint-embedding architectures are based learning invariance between different domain-specific (hand-or craffddr data augmentations.
Guidelines:
The answer NA means that paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
14

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This is an experimental paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The hyperparameter sets and code available publicly in the official DINOv2 repository. The ImageNet- Ik and Imagenet-22K experiments are sufficient to support all the claims in the paper, and these datasets available publicly well as the linear-probing evaluation code and documentation to reproduce results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
15

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code, hyperparameters, and data sufficient support our claims available for the main experimental results as mentioned above.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
iublic/guides http///eueeesioosoooooo.oeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We employ the base settings provided in the DINOV2 repository, that we used produce the results in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
16

---

Answer: [No]
The claims following our experimental results rely on the presence, or not, of large differences in performance between the different setups that we used. During the study, the authors did not observe variability in the evaluation results large enough to justify the cost of running multiple model trainings to provide such bars, and believe that this tradeoff is acceptable.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide a description of the setup (5 compute nodes of A A100-80GB GPUs each) well its speed (125k iterations in 27 hours) in the Experimental Setup section 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://:////uuu.eeeeeeeeee http://rrrrpprrcooplil/////ataat
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
17

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The authors believe this work does not have societal impact, as it focuses on the understanding of SSL algorithms and builds on prior literature and code that is already publicly available.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't release such high risk information.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: We did include the licenses of the datasets, However, proper citations are referenced for all datasets and code; in particular the code is Apache-2.0 as mentioned in the corresponding footnote.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't 't release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't do such thing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "ckogDzXmXd", "reviewer": "Reviewer_hWoD", "summary": "This paper demonstrates the possibility of training JEA SSL encoders using limited augmentations (not 0 augmentations as in the title) compensated by significantly increasing the size of the training data. This is built on the claim that most SSL works have relied on augmentations for SOTA performance. The experiments show that training with crop+resize augmentation alone, with much larger training data can match the performance of ImageNet-1K + hand-crafted augmentations. More experiments demonstrate the benefit of scale and how augmentations' role diminishes as scale increases.", "questions": "See weaknesses section", "limitations": "Discussed but more raised in the Weaknesses section.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written and easy to follow. Motivation and experimental results/takeaways are presented in a clear format.\n- The case study, while only focusing on DINOv2 and iBOT, is thorough with enough supporting ablations to the claims \n- Discussion on the role of scale throughout the paper is appreciated and certainly useful to conduct further research", "weaknesses": "- Misleading title - The title \"you don't need data augmentations\" is certainly misleading since cropping (an augmentation) is still applied under a large dataset regimes. The authors have also not demonstrated results on other methods to make a general claim about the un-usefulness of augmentations in SSL as a whole.\n\n- (Crop + Resize) and (Crop) are both still augmentations - I am skeptical about the main claims because I disagree with the main assumption that \"crop\" and \"crop+resize\" (1) do not increase the dataset size and (2) do not induce invariance. In \"Crop\" randomly removing 32 pixels (256 - 224) from the height and width, very much changes the spatial information compared to the original image and certainly induces an invariance when used in multi-view SSL. Have the authors studied the impact of reducing cropping strength? What happens if you resize to say 230 and crop to 224? \nWith the above concern in mind, Crop+Resize is an even stronger augmentation and surely increases the dataset size artificially which questions several explanations of results (for example - line 220). A simple visual examination of RandomResizedCropped ImageNet-1K samples could show several low-overlapping copies of the same image since the main content need not always be a single object at the center of the image. \n\n- Masking is also an augmentation - Similar to the above discussion on cropping, masking is also a form of augmentation because it fundamentally alters input information and training in such a manner forces an invariance to such alterations. \n\n- Need results on other models - While the studies on DINOv2 are extensive, the bold claim of not needing augmentations for SSL needs to be studied across the realm of SSL because it contradicts the SimCLR paper (as discussed by authors in Introduction section) and a recent work [1].\n\n[1] Morningstar et al, Augmentations vs Algorithms: What Works in Self-Supervised Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for recognizing the importance of the role of scale in self supervised learning and for their questions that made us think more deeply about the semantics and experiments we used throughout our paper. In the following we will be addressing each of your comments. \n\n**W1**) Please refer to the general rebuttal. Thanks to your comment, the title of our paper will be changed to: “Self-supervised learning doesn’t need domain-specific data augmentations at scale” to highlight more the claims of the paper.\n\n**W2-3**) Hand-crafted data-augmentation is the term used in the original I-JEPA paper [1] to talk about most augmentations but masking and RandomResizedCrop. Hand-crafted data augmentations are mentioned in I-JEPA as augmentations that can’t be used on other modalities (thus hand-crafted for the specific modality in use). In our paper, we used the same term but included the Resizing as a hand-crafted augmentation as it isn’t usable in text or modalities where resampling is impossible or doesn’t make sense.\n\nWe propose to change “hand-crafted” to “domain-specific” to make it more explicit. We think that this change in semantics (we used hand-crafted data augmentation in the paper but not the original title) answers both of your points about the boldness of our claim (that was, we think, only in the title) and your question about the fact that cropping and masking are augmentations. We will also make sure to update the paper with the correct wording when necessary.\n\nTo support our claims, we actually don't need our model to be trained with *exactly* zero invariance or data augmentations. A lot of papers are claiming that data augmentations are necessary to achieve good performance with classical SSL [1,2,3] and that it’s not the case for reconstruction-based methods like MAE, I-JEPA or AIM [4,5]. However, MAE, I-JEPA, and AIM still use data augmentations (masking and RandomResizedCrop). It’s also worth noting that all the data-augmentation ablations in [1] are only done on what we call “domain-specific augmentations” and that all their setup include RandomResizedCrop. We show that their results don't hold at scale with DINOv2, which we think is novel and interesting.\n\nWe claim that we can remove all domain-specific augmentations and still train a very powerful model, using fewer augmentations than all the other methods in the literature.\nWhen we analyze the impact of data-augmentation and invariance (Table 3 in the rebuttal), we don’t need it to be exactly zero for our claims to hold. We just need the Shared and Crop+Resize setup to enforce less invariance than the Original setup. The performance gap that is created by enforcing less invariance is compensated by increasing the amount of data, showing that the main bottleneck was the artificial increase of data, not the invariance making. The impact of the RandomCrop and masking augmentations doesn’t change between all our setups, and we only analyze the difference relative to all the other augmentations.\n\nWe also want to add that we have a table in the original paper (Table 2) that ablates the masking augmentation, training a strong model with only one augmentation (RandomCrop, without resizing and without masking). This model still beats all the other baselines from Table 1 in our paper by a large margin.\n\n**W4**) Please refer to the general rebuttal. We add an analysis on MOCOv3 (a more modern version of the SimCLR paper) and explain why our choice is constrained to DINOv2 (the Crop setup was unstable with MOCOv3 and we didn't manage to train a model with it). Our claim might hold for other methods as long as it’s possible to scale them effectively to ImageNet-22k (which is not an easy task). In the meantime, our paper can already impact people training DINOv2 models on other modalities [6,7,8,9,10,11].\n\n\n[1] Morningstar et al, Augmentations vs Algorithms: What Works in Self-Supervised Learning \n\n[2] T. Chen et al. A Simple Framework for Contrastive Learning of Visual Representations\n\n[3] J.-B. Grill et al. Bootstrap your own latent: A new approach to self-supervised Learning\n\n[4] M. Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture\n\n[5] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models\n\n[6] R. J. Chen et al. Towards a general-purpose foundation model for computational pathology\n\n[7] H. Xu et al. A whole-slide foundation model for digital pathology from real-world data\n\n[8] E. Vorontsov et al. Virchow: A Million-Slide Digital Pathology Foundation Model\n\n[9] T. Moutakanni et al. Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning\n\n[10] F. Perez-Garcia et al. RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision\n\n[11] J. Tolan et al. Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1AsOAn2fX8", "reviewer": "Reviewer_Jk8X", "summary": "Traditionally, it is believed that the effectiveness of Joint-Embedding Architectures (JEAs) such as SimCLR lies in their ability to map augmented views of the same image to the same representation in the latent space, thus requiring specific data augmentations that lead to superior downstream performance.  Moreover, such augmentations, like blurring and color jittering, are not directly applicable to non-visual data like speech or text, potentially limiting the utility of JEAs to image-only applications.\n\nThis work challenges the necessity of hand-crafted data augmentations for training Self-Supervised Learning (SSL) models with JEAs. Specifically, the authors provide a rigorous study using DINOV2, demonstrating that effective image representations can be achieved using minimal augmentation—specifically, only cropping without resizing—as long as the dataset is sufficiently large. Further, augmentations mainly serve to expand the effective size of the training dataset rather than enforcing the learning of invariances. Role of augmentations is studied on the three critical scaling dimensions in deep learning: data volume, computational power, and model size. To support these claims, experiments are conducted across a range of dataset sizes, from smaller datasets like ImageNet1k to larger ones such as LVD142, and included multiple model sizes, from small to large.", "questions": "1. What is “AIM” in Table 1? The abbreviations for algorithms used in Table 1 are not defined or mentioned in the text. \n\n2. “DFN-2B+” is undefined and used in Table 1. What is it?", "limitations": "Mentioned in the Weaknesses section", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality.** A lot of previous works focus on studying the impact of different data augmentations both from a theoretical and empirical perspective. This paper additionally studies this impact at scale, which to the best of my knowledge, is completely novel. It is clear how this work differs from previous contributions, and the analysis shows new insights. \n\n**Quality.** The paper is well written and the claims are mostly well-substantiated with extensive experimentation supporting them. \n\n**Clarity.** The paper is well-organized and clear, with understandable figures. Different data augmentation approaches and hyperparameter choices are clearly outlined. \n\n**Significance.** \n- The work provides evidence that one could train state of the art SSL approaches with *DINOv2* without on hand crafted data augmentations except cropping and resizing. This could be useful for scaling SSL applications to non-image domains such as time series etc.  \n- The idea of studying the use of data augmentations to enforce invariance through “Shared” augmentation setting is very interesting\n- The role of compute and non linear gains with changing hyperparameters is important for large scale training and is well highlighted", "weaknesses": "1.  My major concern is with respect to the usefulness and impact of this work. The authors highlight training of large scale SSL models with just cropping and resizing. As also mentioned in the paper, the key use case of this study would be generalizability of SSL approaches to domains such as medical imaging with totally different channels and characteristics. However, cropping may not be an effective augmentation strategy in these domains—for instance, in medical imaging, key features like cells or tumors often occupy a small portion of the image. The paper lacks evidence with respect to their stated motivation– Does their learnt representation actually generalize better to non-vision domains as compared to the representations learnt through traditional hand crafted augmentations ? \n\n2. Despite using data augmentations such as color jitter in JEPAs, it is unclear if the model is actually learning invariance to them. In some works [1][2], it has been shown that despite training a predictor in the latent space, the model can ignore the augmentations and merely learn invariance. There is no discussion from this perspective, which I believe is very relevant to the work. \n\n- [1] Garrido, Quentin, Laurent Najman, and Yann Lecun. \"Self-supervised learning of split invariant equivariant representations.\" arXiv preprint arXiv:2302.10283 (2023).\n\n- [2] Gupta, Sharut, et al. \"In-Context Symmetries: Self-Supervised Learning through Contextual World Models.\" arXiv preprint arXiv:2405.18193 (2024).\n\n3. I would like to see more insights into the gaps in performance observed in low and large data regimes. By the claims made in the paper, with long enough training, training without data augmentations should achieve the same performance as its counterpart. However, a gap still remains. More clarity on this would benefit the work.\n\nI am willing to change my score if the authors address the above comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for recognizing the importance of scaling SSL on other domains. In the following we extended our experiments, including also other imaging domains, making our conclusions more solid.\n\n**W1**)  You indicated that the main motivation for this work, which is to allow using SSL models on different domains, was not supported by experimentation. Indeed, generalization to new domains is very important given that multiple papers used DINOv2 on medical imaging, remote sensing, and many other ones, either directly as is or as a pretrained model to finetune [1,2,3,4].  We decided to test (Table 1 in the rebuttal) our models on such domains where invariance toward domain-specific augmentations could be harmful: \n- logos, signs and product’s packaging where for example text can be degraded by blurring and zooming, and arrows can be inverted if the features are invariant to the flipping;\n- remote sensing, where colors and zooming can have a big impact on the performances;\n- medical imaging where classes differ in type (shape vs texture) and size (small nodule, broken ribs, enlarged heart etc.).\n\nTable 1 in the rebuttal shows that a model trained without domain-specific augmentations tends to generalize better on most of these domains, notably on products (RP2K and Products-10k), remote sensing (FireRisk) and medical dataset (MIMIC, CheXpert, Vindr and NIH-14).\n\nWe gained 4.8% acc. on GTSRB when switching from the original DINOv2 to the Crop only one, showing the impact of resizing on this task.\n\nOn RP2K, Products-10k, and medical imaging datasets, all methods without domain-specific data-augmentations get, in general, better performances, proving that those augmentations were tailored for ImageNet-like tasks but can be detrimental to other ones, as stated in [8].\n\n**W2**) Thank you for the comment. Indeed, models might not learn invariance even when trained with data-augmentations. However, we quantitatively show in Table 3 in the rebuttal that the Original setup is more invariant than the Shared, Crop+Resize and Crop setup toward data-augmentations. For that, we follow the prior work of [5, 6] and use as a metric the average cosine similarity between augmented views of the same image. We also normalize based on the average similarity between random images to take into account potential model variability.\n\nAnalyzing the quantitative results of our empirical study in Table 3, we observe an interesting pattern, with the invariance decreasing between the Original,  Shared, Crop+Resize and Crop settings. It was expected that the most invariant setup would be “Original”, and the least invariant would be “Crop”, but it’s interesting to see that the Shared and Crop+Resize setup have different levels of invariance despite having the same “invariance enforcement” in the loss. A more detailed analysis of data augmentations and equivariance would be a good follow-up to this paper.\n\n**W3**) The gap between low and large data regimes cannot be recovered only by training for longer on the smaller dataset. As it is discussed in this study, scaling SSL is harder than only changing the dataset’s size. Increasing the number of epochs alone will lead to overfitting. This happens less with data-augmentations as they artificially increase the dataset size. Increasing the number of data alone will help, but the training will be suboptimal as the model will see each image for a shorter amount of time.\nThus, scaling both training data and length is what can make models with and without data augmentations reach the same performances.\n\nAnother remark about the remaining gap for our 500 epochs setup with LVD-142M with and without augmentations. We shortly mentioned this in the limitations, but here is a more detailed explanation:\n- Hyperparameters were tuned by DINOv2’s author using all data-augmentations. We used the exact same parameters for all our setups, and thus are probably using suboptimal parameters when removing augmentations.\n- The larger the model used, the more data we need to close the gap. You can see this phenomenon in the paper, Figure 3 on the right. We might be in a setup (ViT-L)  where more data is needed to fill the gap (for example, CLIP ViT-L models are trained using billions of images [7]).\n- We don’t use data-augmentations in our linear classifier training to remove confounding factors when evaluating our models. Adding data-augmentations here isn’t necessary to reach best performances using the original DINOv2. However, some invariances are actually good on ImageNet-like domains [8], and the ones used in the original setup are tailored exactly for this. The gap might close even more if we optimized data-augmentations during the classifier training, transferring the invariances from the pretraining step to the classification step.\n\nQuestions:\nSorry for the confusion about AIM and DFN-2B+. These abbreviations come from the paper [9]. We will add all citations to Table 1.\n\n\n[1] X. Song, General Purpose Image Encoder DINOv2 for Medical Image Registration\n\n[2] M. Baharoon et al. Towards General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks\n\n[3] B. Cui et al. Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery\n\n[4] X. Bou et al. Exploring Robust Features for Few-Shot Object Detection in Satellite Imagery\n\n[5] Q. Garrido et al. Learning and Leveraging World Models in Visual Representation Learning\n\n[6] A. Devillers & M. Lefort, EQUIMOD: an equivariance module to improve visual instance discrimination\n\n[7] M. Cherti, Reproducible scaling laws for contrastive language-image learning\n\n[8] I. Bendidi et al. No Free Lunch in Self Supervised Representation Learning\n\n[9] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dC2u1Eq81h", "reviewer": "Reviewer_PAfm", "summary": "This paper examines the question of whether data augmentations, specifically hand-crafted domain-specific augmentations, are needed for self-supervised learning.  This question has relevance as the existing state of the art methods use such augmentations in the standard natural image domain, which leaves open the question of what one should do when trying to apply self-supervised learning to a new domain, eg medical imaging.\n\nThrough experimental analysis, this paper demonstrates that the primary impact of data augmentation is in increasing the effective dataset size, and that with sufficient data, compute, and model complexity, the gap between performance with and without such data augmentations can be largely closed.  In particular, the paper shows state of the art results under the condition of not using hand-crafted augmentations, while using a joint-embedding architecture.", "questions": "As a more open-ended question, do your results have any implications for reconstruction-based methods for SSL?  Ie, in looking at Table 1, considering the proposed method is able to achieve better results than existing reconstruction-based methods, while being similarly constrained to not leverage hand-crafted augmentations, do you have thoughts on why this gap existing, or put another one, what kind of information would need to be added to the reconstruction-based methods to close the gap?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ paper is well-written and well-reasoned, making it easy to follow\n+ the question examined is of interesting fundamental importance, in giving insight into self-supervised learning and what one should take into account when applying SSL to new data modalities\n+ convincing experimental results to support the argument that the main effect of data augmentation in SSL is for increasing effective training set size, and that by scaling data set size, the performance gap with and without hand-crafted augmentations closes", "weaknesses": "- the experiments were all using DINOv2 - this is a secondary weakness since the paper offers the results as an existence proof that hand-crafted augmentations are not necessary.  however, to have a more complete understanding of the effect of such hand-crafted augmentations, it would be interesting to know whether the results found hold in general for SSL, even if the specific learning algorithm/architecture, were changed, to know whether this is a general phenomenon or if there is an aspect that is specific to something in DINO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive reception of our work. In the following we will be addressing each of your comments. \n\n**W1**) Please refer to the general rebuttal. We add an analysis on MOCOv3 (a more modern version of the SimCLR paper) and explain why our choice is constrained to DINOv2 (the Crop setup was unstable with MOCOv3 and we didn't manage to train a model with it). Our claim might hold for other methods as long as it’s possible to scale them effectively to ImageNet-22k (which is not an easy task). In the meantime, we think that our paper can already impact people training DINOv2 models on other modalities [1,2,3,4,5,6].\n\n**Q1**) It’s an interesting question about insights on reconstruction-based SSL. Our paper shows that the constraint on hand-crafted data-augmentations is not the main factor in the existing gap, but multiple factors can still explain it. First, our model uses DINOv2, which is carefully optimized to work on large scale datasets. I-JEPA, however fails to train good models on ImageNet-22k, but according to the results in our paper, this could be explained by the use of suboptimal hyper-parameters. MAE also fails to scale effectively, in line with a new paper explaining that learning by reconstruction forces the model to learn uninformative features [7]. At the end, the issue might be that we add too much unused information to learn for reconstruction-based methods, and that scaling data increases even more this amount of information.\n\n[1] R. J. Chen et al. Towards a general-purpose foundation model for computational pathology\n[2] H. Xu et al. A whole-slide foundation model for digital pathology from real-world data\n[3] E. Vorontsov et al. Virchow: A Million-Slide Digital Pathology Foundation Model\n[4] T. Moutakanni et al. Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning\n[5] F. Perez-Garcia et al. RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision\n[6] J. Tolan et al. Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar\n[7] R. Balestriero et Y. LeCun, Learning by Reconstruction Produces Uninformative Features For Perception", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful questions.

To recapitulate our paper, we summarize our contributions:
- We can train a powerful and scaled self-supervised model with no domain-specific data-augmentations, in contrast to all alternative approaches. This model provides an existence proof that it is possible to train a model with a similar performance to its domain-specific data-augmented counterpart. This is true only at the ImageNet-22k scale. 
- We reproduce the results of prior literature [1,2,3] at the ImageNet-1k scale, showing that domain-specific data augmentations are necessary (if the amount of data is too small).
We conclude that SSL does not rely on enforcing representational invariance to domain-specific data augmentations in order to perform well: it is possible to obtain strong models (within 1% of the state-of-the-art) as long as we scale training data.

We never claimed that data-augmentations were un-useful, nor that removing them would lead to better performances on all benchmarks (we actually get a performance boost on some domains and show this in rebuttal table 1). In this work, we discuss that we don’t need most of them to get great performances at scale, disproving the belief that they are the key to modern SSL.

Given that our title might be less specific than the claims in our paper, we will rename our article “Self-supervised learning doesn’t need domain-specific data augmentations at scale”.

Our conclusion doesn’t contradict but extends the prior knowledge present in the SSL literature (e.g. the SimCLR work [2], and more recent work such as Morningstar et al. [1]), proving that enforcing domain-specific invariances is not the core learning mechanism of SSL, and that dataset scale was the issue. The data augmentations that we use, i.e. cropping and masking, are very general by nature, suggesting it is possible to extend SSL approaches (in particular DINOv2) to other domains, given proper scale, while data-augmentation can also be harmful to specific domains or classes [4].

The reviewers notably complained that our experiments were limited to only the DINOv2 and DINO methods. There are several reasons for this:
- Our claim about not needing domain-specific augmentations only works with larger scale datasets, but to the best of our knowledge, DINOv2 is the only method that scales to the ImageNet-22k scale. We provide additional experiments in Table 2 of the rebuttal, where we trained MOCOv3 [5] (an improved version of SimCLR) on ImageNet-1k and -22k, with and without domain-specific data augmentations. First, we see that we obtain the same results as DINOv2 when training on ImageNet-1k: removing augmentations has a big impact. We also observe that the method does not scale well to more data - at least not with the hyperparameters provided by the authors - and that ImageNet classification performance decreases when using the larger dataset. The same scaling issue happens with the official I-JEPA models [6] that we tested in Table 1 of our paper. 
- Scaling SSL to more data is harder than just changing the dataset. Tweaking hyperparameters to scale up a method is extremely compute-intensive (3-6M GPU-hours for DINOv2 [7]), and we believe that this is not necessary to support the claim that SSL does not need domain specific augmentations to get good performances: a single existence proof, that we provide in this work, appears sufficient. Given the popularity of DINOv2 as the state-of-the-art self-supervised methods on different domains [8,9,10,11,12,13], we think our work can have an important impact on more powerful and scaled SSL methods. In the Rebuttal Table 1, we analyze the generalization capabilities of our different data-augmentation setups and show that removing augmentations can be beneficial in some domains.

We also add results in Rebuttal Table 3 to quantify the invariance toward augmentation between the different models, proving that removing data-augmentations indeed reduce the learned invariance.


[1] Morningstar et al, Augmentations vs Algorithms: What Works in Self-Supervised Learning

[2] T. Chen et al. A Simple Framework for Contrastive Learning of Visual Representations

[3] J.-B. Grill et al. Bootstrap your own latent: A new approach to self-supervised Learning

[4] I. Bendidi et al. No Free Lunch in Self Supervised Representation Learning

[5] X. Chen et al. An Empirical Study of Training Self-Supervised Vision Transformers

[6] M. Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

[7] M. Oquab et al. DINOv2: Learning Robust Visual Features without Supervision

[8] R. J. Chen et al. Towards a general-purpose foundation model for computational pathology

[9] H. Xu et al. A whole-slide foundation model for digital pathology from real-world data

[10] E. Vorontsov et al. Virchow: A Million-Slide Digital Pathology Foundation Model

[11] T. Moutakanni et al. Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning

[12] F. Perez-Garcia et al. RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision

[13] J. Tolan et al. Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar

Meta-review: The paper, originally titled "You Don’t Need Data Augmentations in Self-Supervised Learning," addresses a important question in the field of Self-Supervised Learning (SSL): whether strong, hand-crafted data augmentations are essential for effective model training, particularly in Joint-Embedding Architectures (JEAs). Running empirical simulations with the DINOv2 model, the authors show that effective image representations can be achieved with minimal augmentation (eg. cropping without resizing) provided the training dataset is sufficiently large. The study uses a range of dataset sizes, from ImageNet1k to the large-scale LVD142 dataset, and studies models of various sizes. The results indicate that augmentations primarily serve to expand the effective dataset size rather than enforce invariance learning, especially when scaling up.

The authors clarified during the rebuttal period that they do not claim data augmentations are universally unnecessary. Instead, they argue that at scale, most augmentations are not required to achieve strong performance. The minimal augmentations used in this study, i.e. cropping and masking, are general by nature, suggesting that SSL approaches could be extended to other domains without relying on domain-specific augmentations. This point is particularly relevant for fields like medical imaging, where such augmentations may not be well-defined or applicable. The authors propose a more accurate title for the final version: "Self-supervised learning doesn’t need domain-specific data augmentations at scale."

The paper acknowledges that removing data augmentations may not lead to improved performance across all benchmarks. However, it provides valuable insights into the role of augmentations in large-scale SSL models. The findings suggest that with sufficient data and model complexity, the gap in performance between models trained with and without domain-specific augmentations can be significantly reduced.

In summary, this paper offers an interesting and useful  contribution to the current understanding of SSL methods by challenging the conventional view regarding the necessity of data augmentations. The reviewews recommend accepting the paper. The proposed title revision is also supported, as it more accurately reflects the scope of the work.
Decision: Accept (poster)