You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning

Authors: théo  moutakanni
maxime  oquab
marc  szafraniec
maria vakalopoulou mics
  centralesupélec
piotr  bojanowski
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:51:58

Abstract: Self-Supervised learning (SSL) with Joint-Embedding Architectures (JEA) has led to outstanding performances. All instantiations of this paradigm were trained using strong and well-established hand-crafted data augmentations, leading to the general belief that they are required for the proper training and performance of such models. On the other hand, generative reconstruction-based models such as BEIT and MAE or Joint-Embedding Predictive Architectures such as I-JEPA have shown strong performance without using data augmentations except masking. In this work, we challenge the importance of invariance and data-augmentation in JEAs at scale. By running a case-study on a recent SSL foundation model -- DINOv2 -- we show that strong image representations can be obtained with JEAs and only cropping without resizing provided the training data is large enough, reaching state-of-the-art results and using the least amount of augmentation in the literature. Through this study, we also discuss the impact of compute constraints on the outcomes of experimental deep learning research, showing that they can lead to very different conclusions.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) has significantly improved performance across various tasks. Despite not requiring supervision, SSL heavily depends on carefully selected data augmentations [9, [11] [12] [13] 35] . Previous experimental literature has shown that removing even one augmentation such as random rescaling or color jittering reduces linear evaluation performance on ImageNet1k [15, 41] by at least 10 points [1, 11, 39] . But reliance on image-specific data augmentations limits the generalizability of self-supervised approaches. How can this be applied to graph, time-series or, for example, medical imaging with totally different channels and characteristics? According to multiple studies [1, 5, 21, 37] , the best data augmentations are dependent on the target tasks, and no set of hand-crafted data augmentations can solve them all. But is this the case?The problem is the following: SSL methods that do not build on such augmentations are based on image reconstruction, and are able to achieve competitive results, though only through finetuning ( [4, 25] ). On the other hand, joint-embedding architecture (JEA) methods, delivering strong results without fine-tuning and producing linearly separable features, seem largely reliant on data augmentations in light of the seminal SimCLR [11, 12] study. I-JEPA [2] was the first method to challenge this assumption, but the authors still show a significant performance gap when compared to their augmentation-based alternatives. Therefore, one can reasonably believe that the core principles driving JEA methods towards great performance are inherently dependent on data augmentation, Table 1 : Comparison of our model trained using RandomCrop, without random resizing nor other photometric augmentations against SSL models that do not leverage hand-crafted augmentations. All other models are reconstruction based, in the pixel space or in the latent space, and use more augmentations than our setup. We only use RandomCrop without resizing, and masking.ImageNet and the underlying invariance learning that it entails. As a side-effect, JEA setups have been extensively tuned to work with augmentations [9, 13, 35] , cementing this belief in the community.In this context, the present paper successfully disproves the assumption that these augmentations are necessary. We show that it is possible to train state-of-the-art joint embedding architectures without augmentations other than crops without random rescaling, and optionally masking. To the best of our knowledge, we train such model using the least amount of data-augmentations in the literature. Data cropping without rescaling, along with masking, might be considered two of the only universal data augmentations that can be applied on a broad number of modalities by just using the properties of the data. We show that building such SoTA model simply requires a larger amount of diverse data, a longer training schedule, and more careful optimization.For most methods, data augmentations prevent model collapse on trivial solutions such as color histogram [1, 7] . They can also enhance performance by artificially increasing the diversity and size of datasets using multiple crops per image [9, 10] . While preliminary work has attempted to explain the theory behind data augmentations and self-supervised learning or have conducted experimental analysis of their impact and necessity in SSL [1, 5, 11, 21, 37] , such analysis has not been conducted on large-scale SSL pretraining with newer methods, restricting the conclusions that can be drawn from their results.In this work, we aim to determine the role of data augmentations at scale by training models using DINOv2 [35] on multiple datasets of varying sizes (from 1M to 140M images), on multiple model sizes (from Small to Large), and using diverse data augmentation combinations.Contributions. By doing a rigorous experimental analysis through this paper, we are the first to prove two counter-intuitive claims that go against most previous assumptions:(i) We show that the impact of data-augmentations invariance enforcement in self-supervised learning is secondary to the impact of their artificial increase in dataset size and distribution.(ii) We explore when and why we can remove hand-crafted data augmentations with minor impact on performance by looking at all scaling aspects of deep learning: data, compute and model size.In the process, we also provide two by-products on top of our claims:(iii) We show through our experimental setup that scaling laws are ethereal and that practitioners optimizing for different compute and data budget might find very different conclusions.(iv) We achieve state-of-the-art performance on extensive evaluations for a model trained without hand-crafted augmentations. This result is an existence proof that joint-embedding architecture methods do not inherently rely on learning invariances defined by data augmentations, challenging the prior beliefs in the community and prompting a new theoretical understanding of SSL.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We provide an existence proof that it is possible to train a self-supervised joint-embedding architecture (based on DINOv2) that reaches state-of-the-art performance without relying on invariance to data augmentations, as was previously deemed necessary. Table 1 compares with alternative methods not using domain-specific data augmentations, while Figures 2 show that the performance almost matches the data-augmented baseline.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Data augmentation in SSL practice. Data augmentations have been a key component of most self-supervised works, including the early Exemplar-CNN [18] . The network would be trained using a classification objective, with each class corresponding to data augmentations of one image in the dataset. In the context of early pretext-task-based SSL, when training convolutional neural networks by predicting relative location of patches, Doersch, Gupta, and Efros [16] have observed trivial solutions. The network was overfitting to chromatic aberrations, and the authors used data augmentations to counter this effect. In a similar way, Bojanowski and Joulin [7] have observed that their discriminative model would emphasize trivial factors of variation such as colour. The solution proposed in this work was to discard color information and take the output of Sobel filtering as an input to the algorithm, along with image jitter and flipping. In 2018, the community observed a jump in performance when evaluating on Pascal VOC 2007 image classification [8, 22] , the implementation of both works leveraging RandomResizedCrop instead of RandomCrop. The authors of SimCLR [11] carried out the first extensive ablation of the impact of data augmentation on performance, and many of the conclusions from this work serve as standard in follow-up SSL work [10, 24, 26, 35] : most modern joint-embedding self-supervised models learn using at least one loss term that pushes together the representation of two augmented views of the same image. By applying different data augmentations to the two views, they force the model to map the augmented characteristics to the same point, enforcing invariance in the model representation.Sensitivity of learning to data augmentation. With the widely accepted view of the importance of invariance learning for SSL models, several works have studied the impact of the choice of data augmentations on downstream performance: Xiao et al. [52] show that the wrong choice of augmentations can have a detrimental effect; Tian et al. [46] consider data augmentations as a way to reduce the mutual information between different views of an image to improve results. Bendidi et al. [5] show through experiments that the choice, amplitude and combination of transformations effects the learnt representation performance in downstream tasks and that the benefit of data-augmentations depend on each specific class. Purushwalkam and Gupta [37] show that a poor choice of cropping parameters can harm performances.At the same time, self-supervised learning algorithms based on image reconstruction don't use many data augmentations. Models trained to reconstruct missing patches directly in the pixel space [4, 25, 33] have led to significantly lower linear-evaluation performance but did not leverage any photometric or hand-crafted augmentations. El-Nouby et al. [32] show that BEIT is more robust to smaller dataset size. Recently, I-JEPA [2] , based on reconstruction in the feature space obtained strong performance without using photometric augmentations. The authors split the comparison against other SSL frameworks according to the type of augmentations used (see Table 1 in [2] ), suggesting data augmentations make for an unfair advantage in SSL training. It is worth noting that reconstruction based methods still use random resizing in their augmentations.Theoretical studies. Furthermore, a series of theoretical works have been studying the apparently critical data augmentation for SSL algorithms. For example, Kügelgen et al. [30] prove, under some assumptions, that data augmentations allow isolating content from style, when using InfoNCE as an objective, as in SimCLR. Eastwood et al. [19] follow up on this work, and propose using a structured data augmentation framework in order to disentangle content from style in the learnt representations; the goal is to avoid discarding style information during the learning process. Zhang and Ma [53] propose splitting the multiple operations involved in data augmentation pipelines in order to produce a hierarchy of augmentations: invariance to each augmentation is translated into a loss applied at different stages of the backbone, leading to accuracy improvements. Notably, they mention that "The most indispensable process in contrastive learning is the data augmentation module." Saunshi et al. [42] propose seeing data augmentations as inductive biases applied to the learning algorithm, and derive theorems valid in the case of linear representations, in order to further understand the principles driving the success of contrastive learning.In this work, we provide a critical result regarding the importance of data augmentation, that dramatically improves our understanding of the core principles of self-supervised learning, prompting renewed theory: data augmentations are not necessary and only cropping without resizing is sufficient to train strong SSL joint-embedding architecture models.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: A "Limitations" paragraph is present at the end of the conclusion section. The main limitation is in the generality of the claim regarding alternative methods. The existence proof still holds true though, as the example we provide is sufficient to disprove prior community belief that SSL joint-embedding architectures are based on learning invariance between different domain-specific (hand-crafted) data augmentations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
approach: 
 The goal of this work is to study the importance of data augmentation for training joint embedding architectures. In this section, we provide a quick recap of the SSL training algorithm that we use for our study. We then provide a precise description of the experimental setup, with details about the training regimes and data augmentation strategies.We tailor our study around the algorithm that was used for training the recently proposed DINOv2 [35] models. We hypothesize that learnings from this study should apply to other SSL methods, but we chose to focus our experimental work on the best modern JEA method. That way, we can provide a strong counter-argument to the assumptions used in previous literature. We briefly summarize the components of the DINOv2 learning algorithm, and for more details, we point to the original publication [35] . Figure 1 provides a schematic diagram of the loss terms used for training the model.Global DINO loss. The DINOv2 training loop is largely based on the DINO algorithm [9] . Given a student (s) and a teacher (t) network, and two views of the same image v 1 and v 2 , the student network is trained by minimizing a loss between the two representations:EQUATIONThe teacher network, is taken as a moving average of the student. We refer the reader to [9] for more details about the definition of ℓ.Local reconstruction-based iBOT loss. DINOv2 models were trained using an additional local masked-image reconstruction loss. As opposed to MAE or BEIT, the reconstruction task in iBOT happens in the latent space, not in the pixel input space. Given a view v 1 as used in the DINO loss, we create an impaired version ṽ1 by replacing a fixed percentage of patches with a [MASK] token.The target is to reconstruct the value of masked features using the remaining context:EQUATIONIf several views are used, the iBOT loss can be applied to every such view. The strategy for choosing the masked patches is a hyper-parameter of the method.Multicrop and data augmentations. In practice, one does not use only two views v 1 and v 2 . Following [10] , DINOv2 uses Multicrop, with two global crops and eight local crops. Global crops are obtained using RandomResizedCrop with a scale parameter of [0.32, 1.0], then resized to 224 pixels. For local crops we used [0.05, 0.32] resized to 98 pixels instead. Each crop, be it global or local, undergoes a series of photometric augmentations: ColorJitter, RandomGrayscale, GaussianBlur, RandomHorizontalFlip and RandomSolarize. We refer to the official DINOv2 repository for the exact implementation details. 1 Hyperparameters and training regimes. The original DINOv2 repository proposes two sets of hyperparameters for training SSL models. The first set (that we refer to as the low-compute regime) corresponds to a setup for fast experimental iterations, designed to run for 100 epochs (125k iterations) with a batch size of 2048. This setup is optimized for performance on the ImageNet-1k dataset (corresponding to the low-data regime). The second set (high-compute regime) is designed for longer training runs of 500 epochs (625k iterations) and is optimized for performance on larger datasets, such as ImageNet-22k (the high-data regime). These two sets differ in many values, including notably the learning rate schedule and warmup, weight decay, batch size, patch size. In our experiments, we use both sets to compare the behavior of the algorithm against data augmentation. We provide a detailed empirical discussion of the differences in Sec. 4.3.Modifications applied on data augmentation. In our study, we apply different configurations of data augmentations. We provide here the details of the four configurations that we consider.• Original: we use all the data augmentations, exactly like in the original DINOv2 work. Each crop is passed through a series of random photometric augmentations independently.• Shared: each view of an image is obtained using a different crop, but apply the exact same photometric augmentation to all views. Two images in the batch will have different photometric augmentations applied to them.• Crop + Resize: we create views with no photometric augmentations at all, only RandomResizedCrop.• Crop: we create views without any photometric augmentations at all. We also don't use RandomResizedCrop. Instead, we use a Resize to 256 × 256 pixels and Crop to 224 or 98 pixels, leading to views with the exact same aspect ratio and zoom, with no random resizing. This setup is very close to a masking strategy where different sets of tokens would be dropped, up to pixel alignment to patch boundaries.Figure 1 provides an illustration of the images that we obtain by applying these configurations. We omit the 'Crop' setting from the illustration given that the 'Crop' and 'Crop+Resize' provide images that are hard to qualitatively differentiate with a naked eye, even if they differ in term of final performances. In the end, our 'Crop' strategy only applies two augmentations between generated views: cropping without random resizing and random patch masking, which are two very similar augmentations (the cropping drops pixels outside of a field-of-view, and masking inside of it). We chose to keep those two specific augmentations as we think that they can be generalized to nearly all kind of data, from graphs to 3D medical images and time-series.Implementation details. For our study, we use the standard ImageNet-1k [41] and ImageNet-22k [15] datasets, as well as the LVD-142M dataset originally used in DINOv2 [35] . The selection of these datasets allows us to benchmark the impact of their size in the trainings. The pre-training code performs 100 epochs in 27 hours on 5 compute nodes of 8 A100-80GB GPUs each, where 1 epoch is set to 1250 iterations. For evaluations, we follow DINOv2 linear evaluation protocol on multiple different tasks including classification (ImageNet-1k [41] , Places205 [55] , ImageNet-v2 [38] , ImageNet-Real [6] , iNaturalist'18 [49] , FlickrLogos32 [40] , GTSRB [45] , RP2K [36] , Products-10k [3] , FireRisk [43] , RESISC [14] , MIMIC [28] , CheXpert [27] , VinDr [31] , NIH-14 [51] ), segmentation (ADE20k [55] ) and depth estimation (NYU-Depth v2 [44] ). The first question we want to answer is: why do we need data augmentations? Are they systematically needed, because they constitute a core component of the modeling? Are they a trick that facilitates the optimisation of deep neural networks, exactly like in supervised learning?The role of data-augmentations. In the context of supervised learning, data augmentations have been used as a means of virtually augmenting the training dataset [21] . That way, additional training samples could be obtained, reducing the risks of overfitting and improving the robustness of the models trained. In self-supervised learning however, data-augmentations have been usually modeled as a way to enforce representations to be invariant toward specific image's characteristics, such as color or scale [46, 52, 53] .Invariance can be harmful. Most of the self-supervised methods and their data-augmentations have been optimised on ImageNet. While DINOv2 has probably been optimised on more benchmarks, some tasks and domain are missing. In Table 2 we show the results of removing hand-crafted data-augmentations on new domains (not used for training) and in Table 3 (left) on new tasks (in the training set but not used for hyperparameter search). We can see that data-augmentations are actually harmful on most of those, meaning that understanding the real impact of augmentations and invariance while removing them is very important to increase SSL robustness.Disambiguating the impact of data augmentation in self-supervised learning. We developed the 'Shared' configuration that still applies augmentations to artificially increase the number of training samples but doesn't enforce invariance to those augmentations. We compare invariance in Table 3 (right) and show that the 'Shared' setup has effectively lower invariance than the 'Original' one. We quantify it using the average cosine similarity of 16 augmented views of the same image, repeated on 100 images from different classes. Higher cosine similarity means higher invariance. We also report a mean/std normalized similarity, computing the metrics using negative pairs' cosine similarity.Enforcing invariance losses its usefulness at scale. By comparing the 'Shared' and 'Original' results in Fig. 2 , we can see that both setups reach way above 80% linear-probe accuracy on ImageNet1k when trained on all three datasets. While the 'Original' setup gets better performances, we can see that the gap decreases with the amount of data, from -1.3% when trained on ImageNet1k and -1.0% with ImageNet22k, to -0.4% with LVD-142M. As we show in Fig. 3 (left), the gap between the 'Shared' and 'Original' setups also decreases with the number of training epochs. According to experiments in prior literature [1, 11, 39] , data augmentations are required at the sample level to enforce invariance and obtain competitive performance. For example, when trained and evaluated on ImageNet1k, SimCLR and BYOL respectively lose 27.6 and 13.1 points of accuracy in the 'Crop+Resize' setup [39] . In contrast, our 'Shared' setup only loses 1.2%, disproving this view.Increasing the sample count is the key to strong SSL. We know that reconstruction-based models like BEIT, MAE or I-JEPA don't need handcrafted data-augmentations on ImageNet1k, and according to El-Nouby et al. [32] , "denoising autoencoders are more sample efficient than joint embedding techniques". Those findings led us to conjecture that the real impact of data-augmentation is to artificially increase the number of samples and allow JEA to reach good performance with less data.To assess this, we compare the 'Crop + Resize' and the 'Shared' settings. Both settings do not enforce invariance, but they differ because the 'Shared' method artificially increases the sample count of the pretraining dataset. We show in Fig. 2 that the gap with respect to the 'Original' setting is way bigger for 'Crop + Resize' setting the compared to the 'Shared' one when we use ImageNet1k to pretrain our models. Without the sample count increase of data-augmentations, DINOv2 reaches lower performance on all five benchmarks when pretrained on a small dataset (Fig. 2 ).Data-augmentations play the same role in supervised and self-supervised learning. All those results lead to the same conclusion: invariance is useful but not necessary for the learning of JEA, and augmentations artificially increase the number of samples in the pretraining dataset. This also explains why reconstruction-based methods do not need as many augmentations: they are more data efficient [32] . But this is not their only role, as data-augmented setups perform better in shorter trainings (Fig. 3 , left). Since setups eventually converge to the same result with sufficient data, this suggests data augmentations are likely linked to easier optimization rather than to a core learning signal. They are just another hyper-parameter.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This is an experimental paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
can we remove hand-crafted augmentations totally?: 
 We focused on what was the impact of data augmentations in modern self-supervised learning using the 'Original', 'Shared' and 'Crop + Resize' settings. We will now show that we can reach state-of-the-art performance without hand-crafted data-augmentations and resizing.Scaling dataset size. Given that DINOv2 is more robust than other methods when training on ImageNet1k without handcrafted data-augmentations [39] , and knowing from Sec. 4.1 that data augmentations mainly increase artificially the dataset size, we hypothesize that a self-supervised JEA can reach SOTA performance by increasing the size of the pretraining dataset used, even when removing the random resizing. We can see in Fig. 2 that increasing the dataset size has a notable property: all four settings converge to the same performances and the gap almost disappears between settings with and without hand-crafted augmentations, displaying a non-decreasing scaling curve with the number of samples. The linear-probe accuracy gap decreases below 1% on ImageNet1k and remains small on other benchmarks.Scaling pretraining epochs. The highest performances were obtained when we pretrained our models for 500 epochs on larger datasets and 100 on the smaller one. However, a question arises: what happens if we train for longer on ImageNet1k, and faster on ImageNet22k and LVD-142M. In Fig. 3 (left) we show that the behavior of low and high data regimes differ. While the larger datasets show non-decreasing scaling curves with respect to the number of epochs, ImageNet1k has an uncoupling between the settings with and without data-augmentations. We notice that training longer on smaller scale dataset is harmful for performances when we don't use the dataset size increase property of data-augmentations. This is probably a consequence of overfitting and another proof of our first claim in Sec. 4.1. What is more compelling is that, following the 'Shared' setting scaling curve, the 'Crop+Resize' and 'Crop' settings' performance increases with longer pretraining, reducing the gap with the 'Original' one and converging to the same performance.Data augmentation and model capacity. In Fig. 3 (right), we report ImageNet1k linear results when varying both data-augmentation strategies and model size. We can see that the gap between the best setup without hand-crafted data-augmentations and the classical setup ('Crop+Resize' or 'Crop' versus 'Original') increases when we use bigger model size, going from -0.6% for a ViT-S and ViT-B to -1.1% for a ViT-L and -1.5% for a ViT-H. This means that as we scale our model, we need more augmentation to increase the amount of data and reduce overfitting. This is in line with the most common understanding of scaling laws in experimental deep learning [29] : we need to scale compute, model size and data at the same time. This also explains why previous work [11, 39] had a hard time scaling self-supervised JEA without data-augmentations. They needed larger dataset, more compute and larger models, but all previous experimental studies on the impact of data-augmentations in SSL were mainly applied to ImageNet1k with smaller ResNets. This also explains why data-augmentation strategies became more aggressive during the evolution of self-supervised methods from its inception to modern methods like DINOv2, when people were scaling models but not datasets.Role of the 'reconstruction-based' iBOT loss. While DINO and iBOT use the exact same loss, they differ in terms of target. With DINO, the teacher and the student are looking at two different views of the same image, with either different data augmentations in the 'Original' setting, the exact same in the 'Shared' one, or no augmentation at all with the 'Crop' one. In the iBOT case however, the student and the teacher see the same exact augmented view, but the student has some masked tokens. 4 : (left): Impact of hyper-parameter optimisation's target compute on the accuracy of a linear probe on ImageNet1k and ADE20k for models trained on ImageNet1k. We can see that optimising for high compute leads to poor performances on the 'Crop' and 'Crop+Resize' settings, which is the opposite of our findings when we optimize for low compute. (right): Impact of hyperparameter tuning using 'Crop+Resize' on ImageNet1k for 100 epochs. For each line, we switch only one hyper-parameter from the configuration optimized on ImageNet22k for 500 epochs (High compute) to the one optimized for ImageNet1k for 100 epochs (Low).Same experiment, different outcomes. During this study, we used two sets of hyper-parameters optimized for different use cases. One optimized for training our models on the smaller scale dataset ImageNet1k during 100 epochs (low compute) that we used for all trainings on ImageNet1k, and one for training those models on the larger datasets ImageNet22k and LVD-142M during 500 epochs (high compute) that we used for all trainings on ImageNet22k and LVD-142M. Indeed, training on ImageNet1k using hyper-parameters optimized on the larger datasets gives different conclusions.We show in Fig. 4 (left) the ImageNet1k linear-probe accuracy and ADE20k linear segmentation mIoU obtained after training a ViT-L on ImageNet1k when we use the 'Low Compute' and 'High Compute' sets of hyper-parameters. Interestingly, DINOv2 doesn't work when trained on ImageNet1k using the 'Crop' or 'Crop+Resize' settings with the 'High Compute' hyper-parameters. It reaches only 46% linear accuracy, despite obtaining SoTA performances when trained on ImageNet22k and LVD-142M with the exact same hyper-parameters, providing nice scaling curves in the process. However, when we optimize the hyper-parameters for the specific scale of data (ImageNet1k) and compute (100 epochs), we can see that it reaches 73.5% with a linear-probe, as presented before.With the 'High Compute' hyperparameters, going from the 'Original' to 'Crop+Resize' setups makes DINOv2 lose 29.1% on the linear-probe evaluation, which is in phase with previous literature [39] . This means that if our only goal was to get a scaling curve leading to SoTA models, without optimizing for lower compute and data, we wouldn't have been able to disprove the assumption from previous studies that pretraining a strong JEA without data-augmentations with minimal loss in performances is impossible when using ImageNet1k [11, 39] ).The issue with high-scale experimental studies. In Fig. 4 (right), we show that it's not trivial to go from one hyper-parameter setup to the other, gains being non linear when tweaking parameters one at a time. Such issue already arose previously in the literature. For example, it was originally thought that ViT needed hundreds of millions of images to reach good performances [17] , but such claim has been proven false in later work [20, 48] . The same happened with Large Language Models, with work scaling them to hundred of billions of parameters like OPT [54] reaching 175B before newer methods showed improved performances with smaller sizes as in [23, 47] .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The hyperparameter sets and code are available publicly in the official DINOv2 repository. The ImageNet-1k and Imagenet-22k experiments are sufficient to support all the claims in the paper, and these datasets are available publicly as well as the linear-probing evaluation code and documentation to reproduce results. 
conclusion: 
 In this work, we challenged the common belief that joint-embedding architectures require data augmentations to deliver strong performance. The underlying interpretation of that belief, followed by theorists, is that the core learning signal in these methods relies on mapping differently augmented samples to the same embedding in the latent space. In particular, this reasoning implies expert knowledge in the design of data augmentations. Also, the photometric augmentations of images (such as blurring and color jittering) cannot be easily mapped to other modalities (such as speech or text). As such, this belief restricts the scope of high-performance JEA methods to images.Our experiments show that this is not true, as it is possible to achieve performance similar to dataaugmented pipelines with the DINOv2 SSL algorithm without using what is referred to, in prior literature, as hand-crafted data augmentations. Therefore, our results prove that joint-embedding architectures do not necessarily require domain knowledge during training to deliver state-of-the-art performance. The data augmentations merely influence training by increasing the dataset size, as shown by our 'Shared' setup, which does not enforce the learning of invariances.Additional experiments with even weaker data augmentations, such as the 'Crop' setup, show that given enough training iterations and data, the models will converge to comparable performance as their stronger data-augmented counterparts. Our results suggest that prior conclusions largely depended on the smaller scale of experiments. However, we note that strong data augmentations lead to stronger performance in the case of shorter training, suggesting that they also help ease the optimization process during learning. We hope future work can shed more light on that observation.Limitations. Given the cost of pretraining models, we focused on the leading self-supervised joint-embedding architecture, DINOv2, that provides strong performance at different data scales.While the results might be different for other algorithms, in principle, the existence proof that we offer in this work still holds. We noted that data augmentations tend to ease optimization, but the underlying mechanism is not known at this point. We merely claim that handcrafted photometric augmentations are optional for successful learning. While it has a measurable effect on some setups, this does not invalidate our conclusions. For large architectures, there is still a tiny gap in performance between augmented and non-augmented setups for long training. While we hypothesize that this might be due to slight variations in the optimal hyperparameters, there could also be a deeper reason that is not covered within the scope of this study.Statement of Broader Impact. The total compute cost for this study was approximately 150k GPU-hours. We used strong models to blur all human faces in our web-based image data pool.NeurIPS Paper Checklist 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The authors believe this work does not have societal impact, as it focuses on the understanding of SSL algorithms and builds on prior literature and code that is already publicly available.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We don't release such high risk information.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: The claims following our experimental results rely on the presence, or not, of large differences in performance between the different setups that we used. During the study, the authors did not observe variability in the evaluation results large enough to justify the cost of running multiple model trainings to provide such bars, and believe that this tradeoff is acceptable.Guidelines:• The answer NA means that the paper does not include experiments. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper, originally titled "You Don’t Need Data Augmentations in Self-Supervised Learning," addresses a important question in the field of Self-Supervised Learning (SSL): whether strong, hand-crafted data augmentations are essential for effective model training, particularly in Joint-Embedding Architectures (JEAs). Running empirical simulations with the DINOv2 model, the authors show that effective image representations can be achieved with minimal augmentation (eg. cropping without resizing) provided the training dataset is sufficiently large. The study uses a range of dataset sizes, from ImageNet1k to the large-scale LVD142 dataset, and studies models of various sizes. The results indicate that augmentations primarily serve to expand the effective dataset size rather than enforce invariance learning, especially when scaling up.

The authors clarified during the rebuttal period that they do not claim data augmentations are universally unnecessary. Instead, they argue that at scale, most augmentations are not required to achieve strong performance. The minimal augmentations used in this study, i.e. cropping and masking, are general by nature, suggesting that SSL approaches could be extended to other domains without relying on domain-specific augmentations. This point is particularly relevant for fields like medical imaging, where such augmentations may not be well-defined or applicable. The authors propose a more accurate title for the final version: "Self-supervised learning doesn’t need domain-specific data augmentations at scale."

The paper acknowledges that removing data augmentations may not lead to improved performance across all benchmarks. However, it provides valuable insights into the role of augmentations in large-scale SSL models. The findings suggest that with sufficient data and model complexity, the gap in performance between models trained with and without domain-specific augmentations can be significantly reduced.

In summary, this paper offers an interesting and useful  contribution to the current understanding of SSL methods by challenging the conventional view regarding the necessity of data augmentations. The reviewews recommend accepting the paper. The proposed title revision is also supported, as it more accurately reflects the scope of the work.

Reviews: review: paper_summary: paper_summary: This paper demonstrates the possibility of training JEA SSL encoders using limited augmentations (not 0 augmentations as in the title) compensated by significantly increasing the size of the training data. This is built on the claim that most SSL works have relied on augmentations for SOTA performance. The experiments show that training with crop+resize augmentation alone, with much larger training data can match the performance of ImageNet-1K + hand-crafted augmentations. More experiments demonstrate the benefit of scale and how augmentations' role diminishes as scale increases.
strength_weakness: strength_weakness: - Misleading title - The title "you don't need data augmentations" is certainly misleading since cropping (an augmentation) is still applied under a large dataset regimes. The authors have also not demonstrated results on other methods to make a general claim about the un-usefulness of augmentations in SSL as a whole.

- (Crop + Resize) and (Crop) are both still augmentations - I am skeptical about the main claims because I disagree with the main assumption that "crop" and "crop+resize" (1) do not increase the dataset size and (2) do not induce invariance. In "Crop" randomly removing 32 pixels (256 - 224) from the height and width, very much changes the spatial information compared to the original image and certainly induces an invariance when used in multi-view SSL. Have the authors studied the impact of reducing cropping strength? What happens if you resize to say 230 and crop to 224? 
With the above concern in mind, Crop+Resize is an even stronger augmentation and surely increases the dataset size artificially which questions several explanations of results (for example - line 220). A simple visual examination of RandomResizedCropped ImageNet-1K samples could show several low-overlapping copies of the same image since the main content need not always be a single object at the center of the image. 

- Masking is also an augmentation - Similar to the above discussion on cropping, masking is also a form of augmentation because it fundamentally alters input information and training in such a manner forces an invariance to such alterations. 

- Need results on other models - While the studies on DINOv2 are extensive, the bold claim of not needing augmentations for SSL needs to be studied across the realm of SSL because it contradicts the SimCLR paper (as discussed by authors in Introduction section) and a recent work [1].

[1] Morningstar et al, Augmentations vs Algorithms: What Works in Self-Supervised Learning
questions: questions: See weaknesses section
limitations: limitations: Discussed but more raised in the Weaknesses section.
paper_summary: paper_summary: Traditionally, it is believed that the effectiveness of Joint-Embedding Architectures (JEAs) such as SimCLR lies in their ability to map augmented views of the same image to the same representation in the latent space, thus requiring specific data augmentations that lead to superior downstream performance.  Moreover, such augmentations, like blurring and color jittering, are not directly applicable to non-visual data like speech or text, potentially limiting the utility of JEAs to image-only applications.

This work challenges the necessity of hand-crafted data augmentations for training Self-Supervised Learning (SSL) models with JEAs. Specifically, the authors provide a rigorous study using DINOV2, demonstrating that effective image representations can be achieved using minimal augmentation—specifically, only cropping without resizing—as long as the dataset is sufficiently large. Further, augmentations mainly serve to expand the effective size of the training dataset rather than enforcing the learning of invariances. Role of augmentations is studied on the three critical scaling dimensions in deep learning: data volume, computational power, and model size. To support these claims, experiments are conducted across a range of dataset sizes, from smaller datasets like ImageNet1k to larger ones such as LVD142, and included multiple model sizes, from small to large.
strength_weakness: strength_weakness: 1.  My major concern is with respect to the usefulness and impact of this work. The authors highlight training of large scale SSL models with just cropping and resizing. As also mentioned in the paper, the key use case of this study would be generalizability of SSL approaches to domains such as medical imaging with totally different channels and characteristics. However, cropping may not be an effective augmentation strategy in these domains—for instance, in medical imaging, key features like cells or tumors often occupy a small portion of the image. The paper lacks evidence with respect to their stated motivation– Does their learnt representation actually generalize better to non-vision domains as compared to the representations learnt through traditional hand crafted augmentations ? 

2. Despite using data augmentations such as color jitter in JEPAs, it is unclear if the model is actually learning invariance to them. In some works [1][2], it has been shown that despite training a predictor in the latent space, the model can ignore the augmentations and merely learn invariance. There is no discussion from this perspective, which I believe is very relevant to the work. 

- [1] Garrido, Quentin, Laurent Najman, and Yann Lecun. "Self-supervised learning of split invariant equivariant representations." arXiv preprint arXiv:2302.10283 (2023).

- [2] Gupta, Sharut, et al. "In-Context Symmetries: Self-Supervised Learning through Contextual World Models." arXiv preprint arXiv:2405.18193 (2024).

3. I would like to see more insights into the gaps in performance observed in low and large data regimes. By the claims made in the paper, with long enough training, training without data augmentations should achieve the same performance as its counterpart. However, a gap still remains. More clarity on this would benefit the work.

I am willing to change my score if the authors address the above comments.
questions: questions: 1. What is “AIM” in Table 1? The abbreviations for algorithms used in Table 1 are not defined or mentioned in the text. 

2. “DFN-2B+” is undefined and used in Table 1. What is it?
limitations: limitations: Mentioned in the Weaknesses section
paper_summary: paper_summary: This paper examines the question of whether data augmentations, specifically hand-crafted domain-specific augmentations, are needed for self-supervised learning.  This question has relevance as the existing state of the art methods use such augmentations in the standard natural image domain, which leaves open the question of what one should do when trying to apply self-supervised learning to a new domain, eg medical imaging.

Through experimental analysis, this paper demonstrates that the primary impact of data augmentation is in increasing the effective dataset size, and that with sufficient data, compute, and model complexity, the gap between performance with and without such data augmentations can be largely closed.  In particular, the paper shows state of the art results under the condition of not using hand-crafted augmentations, while using a joint-embedding architecture.
strength_weakness: strength_weakness: - the experiments were all using DINOv2 - this is a secondary weakness since the paper offers the results as an existence proof that hand-crafted augmentations are not necessary.  however, to have a more complete understanding of the effect of such hand-crafted augmentations, it would be interesting to know whether the results found hold in general for SSL, even if the specific learning algorithm/architecture, were changed, to know whether this is a general phenomenon or if there is an aspect that is specific to something in DINO.
questions: questions: As a more open-ended question, do your results have any implications for reconstruction-based methods for SSL?  Ie, in looking at Table 1, considering the proposed method is able to achieve better results than existing reconstruction-based methods, while being similarly constrained to not leverage hand-crafted augmentations, do you have thoughts on why this gap existing, or put another one, what kind of information would need to be added to the reconstruction-based methods to close the gap?
limitations: limitations: yes

Comments: comment: We thank the reviewer for the positive feedback. We will include the discussion to the revised manuscript.
We thank the reviewer for the positive feedback and valuable suggestions that helped us improve our paper. We will of course include the contents of the rebuttal in the camera ready.
Thank you for your detailed response.
Many thanks for conducting additional experiments and providing a detailed rebuttal that addresses many of the weaknesses identified and questions raised. I appreciate the addition of results on medical imaging, remote sensing and logos and signs datasets. I emphasize that all clarifications made during this rebuttal should be added to the revised manuscript to improve clarity of the work. 

Given the rebuttal addressed most of my concerns, I have increased my score to 5.