FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed
Authors: Jiaqi Zhang
Juntuo Wang
Zhixin Sun
John Zou
Randall Balestriero
Venue: NeurIPS 2025 poster ()
Keywords: Self-Supervised Learning
Vision Foundation Model
Robustness
Efficient Training
Curriculum Learning
Primary area: deep_learning
TL;DR: In this work, we show that vision foundation models such as DINOv2 can achieve fast convergence and maintain high robustness by applying data curriculum and integrating data augmentation in the frequency domain during pretraining.

Abstract: Large-scale vision foundation models such as DINOv2 boast impressive performances by leveraging massive architectures and training datasets. The expense of large-scale pre-training puts such research out of reach for many, hence limiting scientific advancements. We thus propose a novel pretraining strategy for DINOv2 that simultaneously accelerates convergence–and strengthens robustness to common corruptions as a by-product. Our approach involves a frequency filtering curriculum–low-frequency being seen first–and the Gaussian noise patching augmentation. Applied to a ViT-B/16 backbone trained on ImageNet-1K, while pre-training time is reduced by 1.6×–from 16.64 to 10.32 NVIDIA L40S days–and FLOPs by 2.25×, our method still achieves matching robustness in corruption benchmarks (ImageNet-C) and maintains competitive linear probing performance compared with the DINOv2 baseline. This dual benefit of efficiency and robustness makes large-scale self-supervised foundation modeling more attainable, while opening the door to novel exploration around data curriculum and augmentation as a means to improve self-supervised learning models robustness.

F Frequency Based Curriculum Learning Improves Robustness and Training Speed
Jiaqi Zhang'
Juntuo Wang
Zhixin Sun
John Zou'
Randall Balestriero
Brown University ?Comell University
(jisqi_2hang6, juntuo_ wang, john_ zou, randall_ bagaall__blessrrrrr))ooooroo edu
284548cornell.cm
Abstract
Large-scale vision foundation models such as DINOv2 boast impressive perfor- mances by leveraging massive architectures and training datasets. But numerous scenarios require practitioners to reproduce those pre-training solutions, such as on private data, new modalities, or simply for scientific questioning-wiie currently extremely demanding computation-wisc. We thus propose novel pre- training strategy for DINOv2 that simultaneously accelcrates convergence-ann strengthens robustness to common corruptions by-product. Our approach involves a frequency filtering
being seen first-and the Gaussian noise patching augmentation. Applied VIT-B/16 backbone trained on ImageNet-IK, while pre-training time and FLOPs are reduced by 1.6x and 2.25 benchmarks (ImageNet-C) and maintains competitive linear probing performance compared with baseline. This dual benefit of efficiency and robustness makes large-scale self-supervised foundation modeling more attainable, while opening the door novel exploration around data curriculum and augmentation as means to improve self-supervised learning models robustness. The code is available at https://githut. cww www./evin/021///aattt...mo
method still achieves matching robustness in corrpption
112 x 112
224 224
Full Resolution Training Gaussian Patch
Low-Frequency Training Stage
Epoch
75%
Figure I: The FastDINOv2 training pipeline comprises stages, In the first stage, the initial 75% of training epochs utilize only low-frequency features extracted via downsampling. In the second stage, the remaining 25% of epochs employ full-resolution images with Gaussian noise patching.
Table 1:
Comparison of training costs, evaluation on ImageNet-C, and linear probing on the ImageNet- 1K validation set using frozen ViT-B DINOv2 backbone trained on ImageNet- 1K.
Training Method
Training Time (4)ys)(
ImageNet-IK(1))
ImageNet-C (L)
GFLOPs (L)
DINOv2
16,64 (NVIDIA L40S)
77.8%
56.5%
493.76
FastDINOv2
10.32 (NVIDIA L40S)
76.2%
56.7%
219.92
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Models such as DINOv2 [22] and CLIP [25], built on Vision Transformer (ViT) backbones [ achieve remarkable performance, generalization, and even upstream robustness [39 224]. These advances stem largely from self-supe learning (SSL) paradigm in which models learn useful representations from unlabeled data by solving proxy tasks (c.g.. contrastive matching, predicting
masked patches) rather than relying on costly manual annotation [2]. SSL's appeal lies in its ability to leverage the massive, ever-growing pools of raw images available online; by discovering structure in data itself, SSL yields features that transfer effectively to downstream tasks such as classifica- tion, objection detection, and segmentation, often matching or exceeding their fully supervised counterparts 9 [13].
However, reproducing these recent breakthroughs typically demands huge compute: large ViT vari- ants (e.g., ViT-G) trained for hundreds epochs on billions of images 4., multi-GPU clusters, and sophisticated optimization recipes. This resouree barrier put state-of-the-ar SSL out of reach for many academic labs and startups, limiting both reproducibility and further innovation. Moreover, while pre-trained SSL models frequently demonstrate excellent performance benchmarks, robustness to real-world distribution shifts commmn corruptions, sensor noise, weather rrfects-remain crucial for safety-critical applications such medical imaging [2] 30 tonomous driving [4) Although some studies have shown that SSL can improve robustness compared to supervised pre-training [8] 3 21]. most SSL methods do not explicitly optimize for robustness,
clean
and existing robust pre-training often compounds the compute demands [26 [35].
In particular, recent large-scale SSL models exhibit an emergent robustness only when trained at extreme scales: for instance, DINOv2 and related methods require model sizes upward of 86 millions of parameters for ViT-B and datasets of similar magnitude, such as LVD- 142M and LAION-5B (28), to yield strong resistance to corruptions. This scale-driven robustness is appealing in principle, but is prohibitively expensive in practice for researchers without access to massive compute. Consequently, there is a pressing need to design compute-efficient SSL training method that still deliver robustness guarantees.
To address this gap, we propose a two-stage curriculum for DINOv2 pre-training that both speeds up convergence and enhances robustness to frequency-based corruptions-all without resorting to prohibitively large models or datasets. Our approach is motivated by the observation that high- frequency and low-frequency corruptions pollute different spectral bands of images, and that data- driven curricula can steer learning to emphasize or de-emphasize particular frequency bands at different training phases. The training curriculum consists of two stages:
Stage Low-frequency training. We begin by downsampling images, emphasizing their low- frequency components. This encourages the model to quickly learn broad, coarse features and accelerates convergence on clean data.
Stage Full-resolution with high-frequency augmentation. We then transition to full-resolution inputs while introducing Gaussian-noise patching, where random image patches are replaced with noise. This forces the model to learn invariances to high-frequency perturbations and improves robustness.
With extensive experiments, our method achieves not only faster convergence, but also higher robustness models and datasets with various scales. This work demonstrates that robustness need not be an emergent by-product of extreme scale but can instead be built into SSL through careful curriculum design and augmentation. We believe this opens the door more accessible, reproducible, and robust supe-vupe training. In summary, our contributions are the following:
We conduct comprehensive robustness and frequency- -based analysis on models pretrained with a low-frequency data curriculum- an underexplored direction. We identify the low-frequency bias introduced by this training scheme and propose Gaussian noise patching complementary augmentation to enhance robustness.
The proposed curriculum accelerates convergence on ImageNet-1K 1K ViT-B backbone, reducing pretraining time by 1.66x and FLOPPs by 2.25x, while maintaining matching robustness and competitive clean linear probing accuracy.
(27 with DINOv2 and

---

2 Related Work
Frequency-Guiin Guided Curriculum Learning By ordering the example from the easiest the hard- est, curriculum learning yields a monotonic increase in the rate of convergence under stochastic gradient descent (4 It has also been proven that an ideal curriculum can effectively smooth the optimization landscape without changing the location of global minima [14]. Thus, defining the level of difficulty effectively is essential for a powerful curriculum. Gradient- based training of deep networks systematically fits the low-frequency components of target function first, and gradually capture high-frequency features as training proceeds [36]. For vision transformer, specifically, the multi-head self-attention acts as a low-pass filter, attenuating high-frequency signals while preserving low-frequency components [23].
The coarse-to-fine toneine paradigm in computer vision motivates training models first in images with reduced information, then on their full-resolution counterparts. Therefore, is natural to define low-frequency components from an image as easy examples, while full image with both low- and high- freeureyy features as more difficult ones. From Fourier perspective, natural image content is concentrated in the low-frequee frwfereuency domain [37], so downsampling preserves most of the information while reducing computational cost. Several works incorporate small images or low-frequency components into the training pipeline to accelerate ViT convergence: RECLIP[99 applies this curriculum to pre-train 33 generalizes for various ViT-based models, including MoCo [TS] and MAE [16. However, the impact of this approach model robustness remains unclear. Moreover, as a general method, Efficien'Trrin++ evaluated on DINO 77]. but did not observe convergence speedup. In this paper, we revisit the curriculum learning applied to DINOv2. Besides the acceleration in training convergence, we also discover that this data curriculum unexpectedly biases model toward high-frequency information.
CLIP, and EfficienfTrain++
Frequency Bias to Robustness-Aceuran Tradeoffss Robustness research often adopts a frequency- domain perspective to examine how image corruptions correspond to the frequency spectrum (37). In natural images, cowffreuenccy components dominate, carrying most structural information, so nigh-frequeccy corruptions, such as Gaussian shot noise, primarily disrupt fine details like edges, while low-frequency corruptions, including contrast shifts, brightness changes, or fog, modify broader patterns. Recent works have studied the need to model the noise part of the data augmentation caattgies--ustifyyyee the use of such corruptions as means to improve downstream robustness [II. Different data augmentations introduce frequency biases that enhance robustness specific corrup- tions but reveal distinct strengths and weaknesses [8] For example, Gaussian noise augmentation strengthens robustness to high-frequency perturbations, producing www-frequency-biase model, but often degrades performance on medium-frequency distortions like motion blur or Gaussian blur [37]. Similarly, CutOut [10] augmentation, which masks image regions to emphasize high- freuueryy cues such as edges or textures, encourages models to rely on features that are easily corrupted by noise or blur, limiting robustness gains. Adversarial training, on the other hand, biases models toward low-frequency features, enhancing resilience to certain perturbations but compromising performance on low-frequency corruptions [8
6). While these augmentations offer specific strengths, they often impair generalization by prioritizing certain frequency bands over others. In this work, we address this tradeoff by integrating Gaussian noise patching (20,, which applies noise to localized image patches to balance robustness high-frequency corruptions with preserved accuracy clean images. This approach complements the high-frequency bias induced by our data curriculum, fostering a more balanced model across the frequency spectrum, though careful evaluation is needed to mitigate potential weaknesses on medium- feruuuuccrruueyyy corruptions.
3 FastDINOv2: Recipe for Fast and Robust Pretraining
This section presents the proposed curriculum learning pipeline. At high level, the pipeline incre- mentally introduces high-frequency features through a structured learning curriculum. Additionally, Gaussian noise patching is incorporated into the training process to enhance robustness.
3.1 Low-Frequency Feature Extraction
foeefreyyyncy features can be extracted through multiple approaches. While methods like Efficient- Train++ (33 employ Fourier transform with high-frequency filtering for this purpose, opt for
3

---

simpler and more computationally efficient strategy. In our pipeline, we use downsampling as lightweight proxy for low-frequency extraction, applied after DINOv2's standard image cropping preprocessing. In DINOv2's 's teacher-student framework, the teacher network processes global crops, which preserve low-frequency structural information, while the student network handles local crops, which retain high-frequency details. The cropping operation is defined as follows:
8 ~ Ul Smax),
v = h V H W
(1)
where Smia (>= 0.08) and (=ax 1.0) define the crop's area ratio relative to the original image dimensions (H, W). We adopt DINOv2's original scaling parameters: global crops use (Smin, 9max) = (0.32, 1.0), while local crops use (0.05, 0.32).
After cropping, downsampling is performed extract low- wwwfrequency features. This operation uses bicubic interpolation, a resampling method that computes a weighted average of the nearest 4 x 4 neighborhood of input pixels through a cubic convolution kernel. The specific definition for cubic kernel function is in A.3 We reduce global crop sizes from 224 x 224 to 112 x 112 and local crops from 96 96 to 48 x 48.
3.2 Frequency Based Curriculum Learning
Training Curriculum Curriculum learning S improves model performance by progressively training on simpler samples before advancing to more complex ones. In our approach, we define low- frequency components of representing coarse, large-scale patterns as the easier samples, gradually progressing to harder examples represented by the original images. Our curriculum consists of two stages. For the first 75% of training epochs, the model is trained exclusively on these low- frequency components. Subsequently, we apply a restarting mechanism by resetting the Adam optimizer's training dynamics. In the second stage, we train DINOv2 on original images containing both low- and gig-frequency information for the remaining epochs. To ensure training stability, we maintain fixed batch size across both stages.
Balancing Frequency Bias via Gaussian Noise Patching As previously mentioned, training with the low-frequency curriculum biases the model toward high-frequency features, enhancing robustness against low-frequency corruption compared to the standard DINOv2 training approach. To balance this bias and improve robustness to low-frequency features, we introduce Gaussian noise patching in the second stage of training. Combined with cut-out augmentation and Gaussian noise augmentation, this technique effectively encourages the model focus performance on clean data. For implementation, we randomly select a square patch for each image, with side length h = w < min(H, H, W), where H and W are the image's 's height and width before transformation. We then apply Gaussian noise this patch, perturbing each pixel value with noise value ë independently sampled from normal distribution ş NV11, scale"). (N1ssccale") The noise intensity is controlled by the parameter scale.
frequeeyy features while maintaining
4 Experiments
4.1 Datasets and Training Setup
Datasets For faster experimentation, we primarily use ImageNet-100 332 subset of ImageNet- IK (27]. The training set consists of 100 randomly selected classes from ImageNet- -K with the first 500 images from each class. Similarly, the validation set contains the corresponding 100 classes from the original validation set, with 50 images per class. This results in a total of 50,000 training images and 5,000 validation images. For robustness evaluation, we use -mageNet-100-CC derived from ImageNet-C (17), which benchmarks model resilience to common corruptions. We maintain the exact image selection from the ImageNet- validation set across all corruption levels and types. Additionally, we employ ADE20K for semantic segmentation tasks. Finally, we scale our approach full ImageNet-1K and evaluate robustness on ImageNet-C.
Model and Training Details For ImageNet-100 experiments, we use VIT-S/16 as the DINOv2 backbone with a total batch size of 40, distributed across 4 GPUs (10 per GPU). Experiments for large batch size in first stage of training are in Appendix A.I We adopt positional embedding with

---

interpolation for adapting to image resolution shift between two training stages. The results of applying positional embedding with or without interpolation across different embedding sizes are in Appendix A.2 We train baseline models for 500 epochs and training curriculum experiments for 200 epochs, ensuring the baseline converges to optimal performance. All ImageNet- 100 experiments use fixed epoch length of 1, 1,250 iterations. For ImageNet- K experiments, we employ ViT-B/16 with a total batch size of 512 (128 per GPU), with epoch length of 2,500 iterations. The baseline and FastDINOv2 are trained for 250 and 200 epochs on ImageNet- IK respectively. Following the official DINOv2 implementation, we use Adam W optimizer with square root learning rate scaling based batch size, yielding base learning rate of 7.9 10-4. For linear probing evaluation, use a batch size of 128 with 12.5k total iterations. All training runs are distributed across NVIDIA L40S GPUs, while evaluations use either NVIDIA A6000 or NVIDIA A5500 GPUs.
4.2 Linear Probing Reveals Fast Convergence Without Compromising Accuracy
We evaluate our freufrey data curriculum against the DINOv2 baseline through linear probing, training linear classifiers on frozen backbones. The baseline ViT-S/16 model follows standard DINOv2 training for 500 epochs with consistent 224 x 224 resolution. Our curriculum employs 112 x 112 images for the first 75% of training (150 epochs), then transitions to 224 224 images for the remaining epochs (denoted as 112-224 curriculum).
Notably, the 112-224 curriculum achieves the baseline's 's 250-epoch accuracy by epoch 200, demon- strating 20% faster convergence in training epochs and 1.44x acceleration in training time. We additionally validate the framework's 's flexibility by testing alternative first-stage resolutions in Table 2 observing consistent efficiency gains across configurations.
The 112 x 112 resolution in our curriculum's 's initial phase reduces input tokens per forward pass by 75% compared baseline (Table () yielding substantial computational savings. However, we identify a critical tradeoff: excessively small first-stage resolutions (e.g., 64 x 64 or 96 x 96) degrade linear probing performance due insufficient learning signal. While these ultra-low resolutions marginally improve training speed, they exhibit diminishing returns in efficiency gains compared to 112 x 112. Thus, we select 112 x 112 the optimal first-stage resolution balancing computational efficiency with preserved information.
Table Linear probing accuracy on ImageNet- Our data curriculum pipeline matches DINOv2 performance while achieving 1 1.44x convergence speed-up. 112-224 means that the FastDINOv2 training utilizes the 112-224 data currieulum; same applies to the remaining training methods. The training time is measured in NVIDIA L40S hours single GPU.
Training method
Accuracy
Training epoch
Training time
DINOv2
78.6%
250
24h
112-224 FastDINOv2 w/o GP
78.44%
200
13.9h
128-224 FastDINOv2 w/o GP
77.74%
200
13.6h
96-224 FastDINOv2 w/o GP
77.2%
200
12.9h
64-224 FastDINOv2 w/o GP 70.6%
200
13.48h
4.3 Data Curriculum Induces High-Frequency Feature Bias
While previous sections analyzed the curriculum design and its training acceleration benefits, we now investigate how low-frequency data curriculum impacts model robustness against common corruptions. Using ImageNet-C as our corruption robustness benchmark, we construct ImageNet- 100-C by preserving corresponding classes and images from the ImageNet-100 validation set.
Our evaluation protocol trains linear classifiers exclusively on clean ImageNet- 100 validation data using frozen backbones, then tests them on the corrupted ImageNet- MONNet--10-CC images. This ensures neither the model nor classifier encounters corrupted data during training. We compute average error rates across all corruption types and severity levels to quantify robustness.
In T our curriculum-trained models exhibit greater robustness to low-frequency corruptions compared to baseline, indicating stronger reliance on high-frequency features for classification. This emerges because high-frequency exposure during later training epochs forces the model to develop invariant representations of these components, thereby amplifying sensitivity to high- high-frequency
5

---

corruptions. The results confirm that our curriculum induces high-frequeene bias an interesting product of the training strategy.
To better interpret this shift in robustness, we group corruption types in Table
based on their
dominant frequency characteristics, following Fourier analysis approach 37
Low-frequency corruptions: Brightness, contrast, fog, frost
Mid-frequency perturbations: Motion blur, defocus blur, glass blur, Gaussian blur, snow, zoom blur
High-frequeney distortions: Gaussian noise, impulse noise, shot noise, speckle noise
Hybrid effects: Elastic transform, JPEG compression, pixelate, saturate, spatter
Table ImageNet- 100-C testing accuracies for DINOv2 baseline trained with 250 epochs, 112-224 data curriculum trained with 200 epochs, DINOv2 baseline trained with 200 epochs, and DINOv2 with Gaussian noise patching trained by 200 epochs. Comparison between DINOv2(250Ep) and 112-224(200Ep) suggests low-frequency bias induced by data curriculum, while the comparison between DINOv2(200EE)) and DINOv2 w/ GP reveals the high-frequency bias from Gaussian noise patching.
Test 112-224 Curriculum
Test Gaussian Patching(200 Ep)
Corruption type
DINOv2(250Ep)
112-224 (200Ep)
DINOv2
DINOv2 w/ GP
Brightness
71.23%
71.60% %
64.76%
66.43%
Contrast
51.49%
55.24
45.83%
47.75
Fog
47.06%
48.22 %
38.42%
40.18% %
Frost
43.80%
45.45
36.58%
39.95% %
Defocus blur
42.01 %
40.24%
33.80%
32.69%
Gaussian blur
44.38%
41.94%
37.15%
35.36%
Glass blur
36,85%
36.94% %
29.36%
30.56 %
Motion blur
43.64%
45.88%
37.37%
38.21 %
Snow
37.25%
38.08% %
30.22%
28.84%
Zoom blur
47.26%
47.40
42.12%
41.54%
Gaussian noise
32.11
29.59%
21.72%
48.72%
Impulse noise
26.97% %
25.62%
17.34%
44.89 %
Shot noise
31.06%
29.04%
21.30%
45.68
Speckle noise
40.87 %
39.30%
31.25%
50.46 %
Elastic transform
62.92%
62.07%
56.70%
57.17%
JPEG compression
58.58 %
58.02%
51.71%
52.44%
Pixelate
51.79%
50.46%
44.98%
44.47%
Saturate
65.59%
64.89%
57.39%
58.74%
Spatter
55.12%
55.34% %
48.58%
49.60
Corruption accuracy
46.84%
46.60% 78.42%
39.29%
44.93 %
Clean accuracy
78.60%
73.46%
74.05 %
4.4 Spectral Balance:
Combining Curriculum and Noise Patching
Building the figh-frequency feature bias observed in the training curriculum, demonstrate how Gaussian noise patching, low- now-frequency-iiee biased augmentation, can counterbalance this effect. Importantly, integrating these approaches retains their individual strengths while addressing their respective limitations.
We identify Gaussian noise augmentation [ as a method to introduce low-frequency bias, demon- strated by enhanced robustness against high-frequency noise corruptions. However, direct applica- tion decreases clean accuracy due to excessive low-frequency emphasis. Gaussian noise patching mitigates this by selectively applying localized noise injection alongside preserved clean regions, thereby maintaining discriminative feature learning.

---

Table 3 compares the DINOv2 baseline with DINOv2 trained using Gaussian noise patching, The patched model exhibits low-frequency bias, demonstrating greater robustness against high-frequency corruptions, including all noise corruptions. This frequency bias directly opposes that of our data curriculum, prompting the question: can Gaussian noise patching be integrated balance the curriculum's ş frequency bias?
When applied solely during the high-resolution phase the curriculum, this hybrid approach achieves balanced robustness improvements. As shown in Table 4 most corruption types show enhanced resilience, with minor reductions confined specific mid-frequency distortions (e.g.. zoom blur, elastic transform) and high-frequency artifacts (e.g., pixelate). Notably, the combined method eliminates conflicting vulnerabilities observed individual applications, significantly improving robustness to defocus blur and Gaussian blur compared to either technique alone.
The synergy stems from complementary frequency interactions: the curriculum's early low-resolution training builds robust low-frequency representations, while subsequent Gaussian noise patching mitigates overfitting to artificial high-frequency pattern.. This integration achieves spectral balance, ensuring neither frequency domain overly dominates feature encoding.
These findings underscore the value of aligning augmentation strategies with curriculum stages. Although trade-offs remain for certain corruption types, this framework offers a pathway to balance spectral biases while maintaining core model performance.
Table 4: ImageNet- 100-C test accuracy for DINOv2 baseline versus DINOv2 with data curriculum and Gaussian noise patching. Accuracy improves for most corruption types, demonstrating the effectiveness of combining data curriculum and Gaussian noise patching.
Corruption type
DINOv2
FastDINOv2
Brightness
71.23%
71.92%
+0.69%
Contrast
51.49%
56.09%
+4.60%
Fog
47.06%
47.90%
+0.84%
Frost
43.80%
47.52%
+3.72%
Defocus blur
42.01%
42.31 %
+0.30%
Gaussian blur
44.38%
44.44%
+0.06%
Glass blur
36.85%
40.24%
+3.39%
Motion blur
43.64%
45.43%
+1.79%
Snow
37.25%
37.96%
+0.71%
Zoom blur
47.26%
47.00%
-0.26%
Gaussian noise
32.11%
57.51
+25.40%
Impulse noise
26.97%
54.62%
+27.65%
Shot noise
31.06%
55.34%
+24.28%
Speckle noise
40.87%
61.59%
+20.72%
Elastic transform
62.92%
62.36%
-0.56%
JPEG compression
58.58%
61.15%
+2.57%
Pixelate
51.79%
49.18%
-2.61%
Saturate
65.59%
65.74%
+0.15%
Spatter
55.12%
56.35%
+1.23%
Corruption accuracy
46.84%
52.88%
+6.04%
Clean accuracy
78.6% %
78.4%
-0.20%
4.5 Instance Recognition Indicates Instance-Level Effectiveness
Building on the demonstrated performance in linear probing, we further evaluate FastDINOv2 on an instance-level task through instance recognition. Specifically, we assess three difficulty levels on the Oxford and Paris datasets and report the mean average precision (mAP) in table On the Oxford dataset, FastDINOv2 outperforms DINOv2 by 3.73% on the easy split and 1.22% on the medium split, with slight drop the hard split. Both models exhibit comparable performance on the Paris dataset. These results indicate that FastDINOv2 not only accelerates convergence but also enhances performance across variety of tasks.

---

Table Mean Average Precision (mAP) on instance-level recognition tasks using Oxford and Paris datasets (E: Easy, M: Medium, H: Hard)
Oxford
Paris
Training method
M
H
E
M
H
FastDINOv2 DINOv2
32. 1 %
22.26%
3.99%
56.38%
40.71 %
14.56% 14.94%
28.38%
21.04%
4.56%
56.88%
40.76%
4.6 Semantic Segmentation Suggests Intact Pixel-Level Performance
Section 4.2 and 4.5 show that the FastDINOv2 achieves faster convergence while maintaining linear probing and improving instance recognition accuracy, proving that image and instance-level task remains strong. However, since semantic segmentation requires fine fine-grained pixel understanding, we needed to verify whether our approach- - which spends 75% of training time on now-frequency features could still perform well such tasks. To test this, conducted semantic segmentation experiments using ADE20K. We froze the DINOv2 backbone (trained with our curriculum) and trained only a 2-layer convolutional decoder, evaluating performance using both mean Intersection-over-Uue Union (mloU) and mean class accuracy (mAcc) metrics. Table T demonstrates that our 112-224 curriculum with Gaussian patching matches the DINOv2 baseline's segmentation performance (mloU/mAcc) in just 200 epochs. Despite initial low-frequency training, the second stage successfully recovers fine-grained pixel understanding needed for segmentation tasks.
Table Semantic segmentation evaluation with frozen DINOv2 backbone. A 2-layer convolutional decoder is trained on top of frozen DINOv2 using ADE20K dataset for 200 epochs.
Training Method
Pre- nee-training Epochs
Clean Accuracy
mAce
mloU
DINOv2 FastDINOv2
250
78.6%
25.09%
19.2
200
78.4%
25.21%
19.16
4.7 Frequency Based Analysis and Grad-CAM Visual Explanation
100-CC In addition to the evaluations using ImageNet- 100-C and linear probing, we explore whether different training paradigms influence the error sensitivity of models across all frequency bands-not only those in ImageNet- 100-00-00a and the regions the model prioritizes during predictions. To this end, we generate Grad-CAM visualizations
29 and Fourier spectral heatmaps following the methods in (37.
Figure shows the error sensitivity of the resulting model across various frequency ranges. Compared to the DINOv2 baseline, our approach achieves lower error rates in high- and medium-frequent bands. low-to-mid frequency ranges, particularly in the central region, our method reduces the error rate by significant margin. However, the model shows increased susceptibility to perturbations in the mid-to-high frequency range, indicating potential trade-offs robustness across frequency bands.
Figure 3 presents Grad-CAM visualizations that highlight further differences. While the DINOv2 baseline distributes its focus across both the target object and background areas during predictions, the model trained with FastDINOv2 emphasizes object contours more strongly. This suggests that the initial stage of pre-training provides an effective initialization, enabling the model to focus on coarse image components.
4.8 Extending to VIT-B and ImageNet- Balancing Efficiency and Robustness
To validate scalability, we extend our framework VTT-B architectures and ImageNet- pretraining. The baseline DINOv2 model is pretrained for 200 epochs using standard protocols, while our FastDINOv2 method combines a 112-224 curriculum with Gaussian noise patching over an equivalent number of apoocs-allocating 150 epochs to downsampled inputs followed by 50 epochs of full- resolution training. Evaluation includes linear probing on clean ImageNet- K validation data and robustness assessment using ImageNet-C with frozen backbones.

---

26
96
90
DINOv2 baseline
FastDINOv2
Figure 2: Fourier error sensitivity heatmap of model trained with our method and DINOv2 baseline. The heatmap is generated with subset of Imagener- validation set with 5 images sampled from each class. Color indicates the error sensitivity to that specific frequency range. Low-frequency features are mainly concentrated into center area, while the region further away from center represents feature with high frequency.
DINOv2 baseline
FastDINOv2
Figure 3: Grad-CAM maps for DINOv2 baseline and FastDINOv2. The first row images are from class "house finch, linnet, Carpodacus mexicanus" and the second row from "goose". With the data curriculum, model can better capture the contour the object. See more examples in A.7
Table reports the evaluation results and training costs for the baseline and our method. Although training time is reduced by 1.6x and FL.OPs by 2.25x, our method achieves competitive linear probing accuracy. Moreover, testing on ImageNet-C with a frozen backbone and linear classifier shows comparable error rate despite slight drop in clean accuracy, demonstrating the effectiveness of our method in enhancing robustness.
Huge memory consumption for GPU is critical bottleneck for scaling up the pretraining. However, with oww-resouution image in the first training stage, FastDINOv2 allows a much lower memory requirement. As in t during the first 75% of training epochs the maximum memory consumption of FastDINOv2 is 9.47GB baseline with same batch size. This reduction in memory requirement shows possibilities for cost-effective model by leveraging GPUs with lower memory capacity for most of the pretraining epochs.
batch size of 128 per GPU, significantly less than 33.5GB for DINOv2

---

Table 7: Comparison of training epochs and max memory consumptions on a single NVIDIA L40S GPU with 48GB memory. Both DINOv2 and FastDINOv2 apply ViT-B backbone trained on ImageNet-1K. IK.
Training Epochs
Max Memory Consumption(GBB
Training Method
Low-Res
Full-Res
Low-Res
Full-Res
DINOv2 F
200
33.5
150
50
9.47
33.5
5 Limitations, Future Work, and Conclusion
In this work, we presented an efficient training framework for the vision foundation model DINOv2, demonstrating its scalability and robustness across diverse datasets and model backbones. Our method achieves substantial improvements in training efficiency, computational cost reduction, and model performance. However, one limitation of our current approach lies in the fixed schedule for transition- ing from low-resolution to handard-resolution images during the final 25% of training epochs. This heuristic, while effective in experiments, may not be optimal across different hyperparameter settings or training regimes. Future work will focus on developing adaptive scheduling strategies that dynamically determine the transition point based training signals such convergence metries or validation performance. Additionally, we plan to investigate how our resolution-awars curriculum can be combined with other training enhancements such adversarial training paradigms.
References
[1] M. Eren Akbiyik. Data augmentation in training cnns: Injecting noise to images. arXiv preprint arXiv:2307.06855. 2023.
06855
[2] Mohammed Baharoon, Waseem Qureshi, Jiahong Ouyang, Yanwu Xu, Abdulrhman Aljouie, and Wei Peng. Evaluating general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. arXiv preprint arXiv:23 02602366, //doi doi. org 10. 48550/ arXiv. 2312.02366
2023.
[3] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari S. Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self supervisd learning. arXiv preprint arXiv: 10, 2023. URL http::t/t///r//
org/ abs/ 2304. 12210
[4] Merve Rabia Barın, Görkay Aydemir, and Fatma Güney, Robust bird's eye view segmentation by adapting dinov2. In Proceedings of the 2nd Workshop on Vision-Centric Autonomous Driving at ECCV 2024, 2024.
[5] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, 2009.
[6] Qingwen Bu, Dong Huang, and Heming Cui. Towards building more robust models with frequency bias. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in sulf-super vision transformers. arXiv preprint arXiv:2104. 14294, 2021. URL https://arxiv www.ooog/////.......o...ooooe/.e /2044. 14294
[8] Alvin Chan, Yew-Soon Ong, and Clement Tan. How does frequency bias affect the robustness of neural image classifiers against common corruption and adversarial perturbations? In Proceedings of the Thirty-Fira International Joint Conference on Artificial Intelligence, 2022.
[9] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
10

---

[10] Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. URL htttps:///xxx org/
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[12] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M. Hospedales. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 2022. doi: 10. 10.110000000 10..1099/2P.221111 3134344.
[13] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self supervised learning. In Advances in Neural Information Processing Systems, 2020.
[14] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[17] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[18] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self supervupe learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, 2019.
[19] Runze Li, Dahun Kim, Bir Bhanu, and Weicheng Kuo. RECLIP: Resource-efficient CLIP by training with small images. arXiv preprint a7X4v:2304.06028, 2023. URL https://arxiv org/ abs/ 2304. 06028
[20] Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D. Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint wwXiv:1906002611 2019. URL https://///ax org/ abs/ 1906. 02661
[21] Wenquan Lu, Jiaqi Zhang, Hugues Van Assel, and Randall Balestriero. Ditch the denoiser: Emergence of noise robustness in self-supervised learning from data curriculum. arXiv preprint arXiv:2505.1219) May 2025. URL httpps///rxxi. org/ abs/ 12191
[22] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[23] Namuk Park and Songkuk Kim. How do vision transformers work? International Conference on Learning Representations, 2022.
[24] Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. In Proceedings of the Thirty-Sixth AAAI Conference Artificial Intelligence, 2022.
11

---

[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, 2021.
[26] Colorado J. Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li. Shanghang Zhang, Devin Guillory, Sean Metzger, Kurt Keutzer, and Trevor Darrell. Self-supervised pretraining improves self-supervised pretraining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
[28] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image- text models. arXiv preprint arXiv:2210.08400 2022. URL http:htttt///aat battps://arxiv wwwxiv.org/oioo httggabb/22210 08402
[29] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
[30] Xinrui Song, Xuanang Xu, and Pingkun Yan. Dino-reg: General purpose image encoder for training-free multi-modaa deformable medical image registration. In Medical Image Computing and Computer Assisted Intervention, 2024.
[31] Hugues Van Assel, Mark Ibrahim, Tommaso Biancalani, Aviv Regev, and Randall Balestriero. Joint-embedding vs reconstruction: Provable benefits of latent space prediction for self- supervised learning. arXiv preprint arXiv:2505. May 2025. URL htttps//arxii: //axivv org/
12477
[32] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
[33] Yulin Wang, Yang Yue, Rui Lu, Yizeng Han, Shiji Song, and Gao Huang. Efficienttrain++: Generalized curriculum learning for efficient visual backbone training. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[34] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer learning: Theory and experiments with deep networks. In Proceedings of the 35th International Conference on Machine Learning. 2018.
[35] Cong Xu, Dan Li, and Min Yang. Adversarial wwweeaammooonoonnnooossiiiie hatp://mmmmmtntttttteeeteerrnnnn arXiv preprint 2012 2020. URL.littppe//arrvvoor///bb//
13154
[36] Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06522, 2019. URL hhtpp::///// /tttpps://rxxiii org/ abs/ 11/.1990.000 06523
[37] Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems, 2019.
[38] Kejia Zhang, Juanjuan Weng, Yuanzheng Cai, Zhiming Luo, and Shaozi Li. Mitigating low- frequency bias: Feature recalibration and frequency attention regularization for adversarial robustness. arXiv preprint arXiv:2407.04016, 2024. URL. lhtpp:t//// www.ai.ww.waabaa..e..e 04016)
12

---

[39] Daquan Zhou. Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng, and Jose M. Alvarez. Understanding the robustness in vision transformers. In Proceedings of the 39th International Conference on Machine Learning, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims are made explicitly both abstract and introduction: we propose efficient training pipeline for accelerating pre- pre-training and improving robustness for vision foundation model such as DINOv2. Results experiments section reflect the fast convergence, as well as how each component of the pipeline impacts model robustness.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation is discussed in section 5, where we reflect on the limitation of determining transition epochs for our curriculum during pretraining.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does not include formal theoretical result. The contribution from this work is algorithmic and experimental.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper provides comprehensive details for reproducing the experiments. The dataset and models used for experiments include ViT-S/16, ViT-B/16, Imagenet- 1 Imagenet-C, C, and ADE20K. All models and datasets above are publicly available, and more training details are included in experiment and method sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The model and training dataset are all accessible public, and we will release training code in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Sobliccguiies// CodeSubmissionoollo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https htte:/gguuddessooCcuubbiiiii www.oodeuuiiiisonnnee..
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper provides comprehensive details on hyperparameters, optimizer, epochs, and other training configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Docs the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Our work does not report error bars for large experiments due to its computa- tionally intensive nature. However, the experiment results are highly unlikely to be from chance as the findings are robust and have been observed in multiple settings with different model architecture, parameter, dataset, etc.
Justification:
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided detailed specification on the computational resources needed to run the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:: http:///ww..pp..ccpppcc/t/ttieee
Answer: [Yes]
Justification:
We carefully follow the NeurlPS Code Ethics all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
17

---

Answer: [NA]
Justification: The goal of our work is to improve model pretraining efficiency and robustness.
While there might be many potential societal consequences, of which we feel must be specifically discussed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the external datasets and pre-trained model preperly cited int the paper. For example, we have cited DINOv2, Imagenet- K and Imagenet-C: in the abstract and introduction sections.
Guidelines:
18

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapssttssppersssaaes paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation and code for reproducing the result of all the experiments are provided in supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include any crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer: [NA]
Justification:
This paper does not include any corwdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methods of this paper does not involve usage of LLMs as any important, originall, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
20

---

A Technical Appendices and Supplementary Material
A.1 Constant Batch Size Ensures Training Stability
Smaller image sizes consume less GPU memory during training, which naturally suggests using larger batch size in the first stage of the curriculum. To investigate the effect of an early large batch size, we test the 112-224 curriculum by increasing the batch size from 40 to 60 in the first stage while keeping the second stage unchanged, The base learning rate for the first stage is also scaled proportionally from 4 x 10-3 to 6 x 10-3, As shown in Table 8, while the training time is slightly reduced with the early large batch size and scaled learning rate, the clean accuracy decreases. This suggests that varying the batch size during training can introduce instability and degrade model performance. Therefore, we use a constant batch size in all subsequent experiments.
Table 8: Linear probing aceuracy for the early large batch size and learning rate. Notice that in this experiment, FastDINOv2 training utilizes 112-224 data curriculum and does not apply the interpolation for positional embedding and Gaussian patching in order to exclude their effects. In the Batch size column, 60 + 40 indicates that batch size of 60 is used in the first stage, and 40 for second stage. Same for learning rate. Training cost is measured NVIDIA L40S hours for a single GPU.
Training method
Accuracy
Training cost(hours)
Batch size
Learning rate
DINOv2
78.6%
250 epochs, 24
40
4e 3
FastDINOv2 w/o GP
78.06%
200 epochs, 12.3
60 + 40
6e 3 de - 3
FastDINOv2 w/o GP
78.42%
200 epochs, 13.9
40
4e 3
A.2 Positional Embedding for Varying Image Resolution
DINOv2 uses learnable positional embeddings, which capture the location information of input patches during training. In the first half of the training curriculum, images downsampled to 112 x 112 as input, while in the second stage, regular-sized images (224 x 224) are used. This setup requires the model to adapt to multiple input resolutions. Two approaches are available: either using separate positional embeddings for each stage or using a consistent positional embedding and interpolating it. The second approach introduces two further choices: interpolating the positional embedding based on either the first stage's 's smaller resolution (112 x 112) or the second stage's 's full resolution (224 224). In Table 9 we compare these settings and find that interpolation based on the first stage's 's smaller resolution (1 12 x 112) performs best. After the first training stage, we retain the positional embedding and apply bicubic interpolation for upsampling, adjusting it to the higher resolution in the second stage. This process mirrors our image downsampling procedure, where we also use bicubic kernel to resample pixels.
Table Linear probing accuracy on Imagenet- 100 for FastDINOv2 with 112-224 data curriculum to compare the following three design: interpolation with fixed 112 x 112 positional embedding, interpolation with fixed 224 224 positional embedding, and using separate positional embeddings(no reusing). In the third design option, positional embedding from the first stage is abandoned, while new positional embedding trained based on second-stage's image resolution. FastDINOv2 training in this experiment does not use Gaussian patching to eliminate its effect.
Training method
Accuracy
Training epoch
Positional embedding
DINOv2
78.6%
250
no interpolation
Fas w/o GP
78.8 %
200
interpolation w/ 112 x 112 interpolation w/ 224 x 224 no reusing
FastDINOv2 w/o GP
77%
200
FastDINOv2 w/o GP
78.42%
200
A.3 Down-sampling with Bicubic Interpolation
We adopt Bicubic interpolation for down-sampling to obtain low-frequency inputs for first-stage training. The cubic kernel w=t) is defined as:
21

---

(a + 2) |t/s - + +3+ 3) 11/2 + 1, 11
w1)
a |d/s - 5a | + 8a Bt] 4a, 1< < <2 (a= 00.5)
2
where represents the normalized distance from the sampling point neighboring pixel, and controls the curvature of the cubic kernel. Using this kernel function, the bicubic interpolation fzz v) computed by applying the convolution kernel w separably along both spatial dimensions:
2
f(x,g)=
Σ m=
Σ
w(m Ax) w(n Ay) f(zo m, yo + +n); m.n (-1,0,1 (3)
For an input image of size (Horig Worig) and target downsampled size (Hdoway W Wdown), the coordi- nate mapping transforms each pixel location (u, v) in the original image to (I,, y) in the downsampled space as:
(x, 3) (u Worig
Horig
(wo. vo) (L=J, Lvd), (Ax, Au) (x Jo. yo)
(4)
Wdown
Haown
A.4 Experiments on SimCLR
In order to examine the generalization ability of our method across frameworks and model archi- tectures, we conducted additional experiments by training SimCLR model with ResNet backbone using our FastDINOv2 method (denoted as "FastSimCLR"). Using LARS optimizer, the SimCLR baseline was trained for 200 epochs on full-resolution images from ImageNet-100, while FastSimCLR was trained for 150 epochs on low- www-freuency data followed by 50 epochs on full-resolution images. Despite the differences in inductive biases between ViTs and CNNs, in table I0 FastSimCLR shows faster convergence. Further, combining this curriculum with Gaussian noise patching improves robustness toward Imagenet-C.
Table 10: Linear probing accuracy on ImageNet- 100 and testing accuracy on ImageNet- 100-C for SimCLR and FastSimCI.R. Training time is measured in NVIDIA A6000 hours for single GPU,
Training Method
Epochs
ImageNet-100 (t)
ImageNet-100-CC (t)
Training Time
SimCLR FastSimCLR
200
64.48%
30.95%
14
150+50
64.82%
38.73%
1 11.2
A.5
Ablation Experiment for Transition Epoch
One critical design for FastDINOv2 pre-training framework is the transitioning epochs, where training data shifts from low-resolution images to full-resolution ones. Correctly choosing this transition point is essential if the training epoch for the first stage is insufficient, the model fails take advantage from low-resolution images to improve training efficiency; on the other hand, the model lacks exposure to the full-resolution features necessary for downstream tasks if the first stage is excessive. Thus, we determine this transition point empirically by conducting set cross validation experiments with different transition epoch in table []]
Among all the splits, the 75/25 splits consistently yielded best trade-off, achieving the highest linear probing accuracy. More extreme cases such as 0/100 would have no aceleration effect, and for 100/0 the model would only be trained on low-frequency data, which lead to notably worse performance.
22

---

Table 11: Linear probing accuracy on ImageNet- 100 for FastDINOv2 with different transition epochs proportion. The number of total training epoch is 200, and training time is measured in NVIDIA A6000 hours for a single GPU.
Training Method
Split (Stagel/Stage2)
Training Time
ImageNNet1100
vas
65/35
20.4
77.6%
FastDINOv2
75/25
20.4
78.1%
FastDINOv2
85/15
20.68
76.8%
Fast
0/100
22.3
73.9%
FastDINOv2
100/0
20.1
69.7%
A.6 Convergence Speed Comparison
In order better demonstrate the trend of convergence speed of FastDINOv2, we measure the training time gap for reaching several linear probing accuracy levels for both FastDINOv2 and DINOv2 baseline in table 12
Table 12: Training time comparison (in NVIDIA A6000 hours) required reach different accuracy
levels on ImageNet-100 100.
Training Method
55% Accuracy
65% Accuracy
75% Accuracy
78% Accuracy
DINOv2 FastDINOv2
2.6h
4.8h
12.48h
23.04h
2.64h
5.28h
11.5h
13.1h
During the low-frequency training stage (when FastDINOv2 reaches 55% and 65% accuracy). FastDINOv2 achieves similar convergence speed to the baseline trained on full-resolution images. This is attributed to the inherent low-frequency bias in deep neural networks and transformers, such that the model is more inclined to learn low-frequency information first during training. Furthermore, this low-frequency stage provides strong foundation for the subsequent full-resolution training stage (when FastDINOv2 reaches 75% and 78% accuracy), enabling FastDINOv2 more efficiently and effectively learn fine-grand graigrdi high-resolution details from full-resolution images.
A.7 GradCAM Visualization and Imagenet-C Examples
DINOv2 baseline
FastDINOv2
Figure 4: Extra Grad-CAM maps examples for DINOv2 baseline and FastDINOv2.
23

---

1000
Original
Contrast
Defocus Blur
Elastic Transform
Fog
Frost
Gaussian Blur
Gaussian Noise
Glass Blur
Impulse Noise
JPG Compression
Motion Blur
Pixelate
Saturate
Shot Noise
Snow
Spatter Figure 5: Imagenet-C examples for each corruption type.
Speckle Noise
Zoom Blur
Brightness
24


Reviews:
[{"review_id": "40DTK2WvP7", "reviewer": "Reviewer_PiGo", "summary": "The paper introduces FastDINOv2, a two-stage curriculum training approach for DINOv2, which aims to enhance both training efficiency and robustness to corruption. The framework considers training first on downsampled images for 75% of the epochs, then changing to full-resolution, together with Gaussian noise patching, for the remaining 25% of the epochs. The authors demonstrate a 1.5x reduction in training time while maintaining competitive performance on (mostly) ImageNet-based benchmarks.", "questions": "I would appreciate it if the authors could address my concerns in the weaknesses section. I would consider increasing my score if those are addressed properly.", "limitations": "The authors mention some limitations in the conclusion of the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The approach achieves substantial acceleration without a major downgrade of results.\n2. The frequency-based design of the curriculum is well-motivated and intuitive.\n3. The authors provide insightful analysis, using Frequency-based analysis and Grad-CAM to support their claims.\n\nWeaknesses:\n1. Experiments are performed solely on DINOv2 (ViT). Will the curriculum show similar results for other methods? Is it also viable for CNNs?\n2. Continuing the above, the experiments are performed mostly on an ImageNet subset (ImageNet-100). Raising concerns about how the curriculum performs on larger datasets (ImageNet-1K, ImageNet-22k, etc.)\n3. The paper does not compare to competing frameworks.", "was_revised": true, "final_justification": "I appreciate the additional experiment provided in the rebuttal. After reading the other reviews and the rebuttal I decided to increase the score to borderline accept.  \nThe proposed framework is simple and practical and shows promising results.  \nMy main concern is still the lack of evaluation on larger datasets.", "rebuttal": "We sincerely thank the reviewer PiGo for their questions and comments. Below, we answer each question accordingly:\n\n---\n\n### 1. Experiments are performed solely on DINOv2 (ViT). Will the curriculum show similar results for other methods? Is it also viable for CNNs?\n\nWe thank the reviewer for bringing up the important issue of generalization across frameworks and model architectures. To address this, we conducted additional experiments by training a SimCLR model with a ResNet backbone using our FastDINOv2 method (denoted as “FastSimCLR”). Using the LARS optimizer, the SimCLR baseline was trained for 200 epochs on full-resolution images from ImageNet-100, while FastSimCLR was trained for 150 epochs on low-frequency data followed by 50 epochs on full-resolution images.\nDespite the differences in inductive biases between ViTs and CNNs, FastSimCLR shows faster convergence. Furthermore, combining this curriculum with Gaussian noise patching improves robustness. We thank the reviewer again for this insightful suggestion–we believe this experiment further demonstrates the effectiveness of our method. \n\n\n| Training Method | Epochs | ImageNet-100 Acc | ImageNet-100-C Acc | Training Time (NVIDIA A6000 hours) |\n|-----------------|--------|----------|------------|-----------------------------|\n| SimCLR          | 200    | 64.48%   | 30.95%     | 14                          |\n| FastSimCLR      | 150+50 | 64.82%   | 38.73%     | 11.2                        |\n\n---\n\n\n### 2. The experiments are performed mostly on an ImageNet subset (ImageNet-100). Concerns about how the curriculum performs on larger datasets (ImageNet-1K, ImageNet-22K, etc.)\n\nWe thank the reviewer for this insightful question. We agree that demonstrating the effectiveness of our method on larger datasets is crucial.\nDue to limitations in computational resources, we were unable to conduct experiments on ImageNet-22K. However, we did evaluate our approach on ImageNet-1K, as presented in Table 1. In this setting, we trained a ViT-B model on ImageNet-1K, evaluated its representations via linear probing on the ImageNet-1K validation set, and tested robustness on ImageNet-C.\nOur results show that FastDINOv2 achieves a substantial reduction in computational cost while maintaining performance comparable to the baseline DINOv2. Notably, the improvements in convergence speed and robustness observed on ImageNet-100 carry over to ImageNet-1K, suggesting that our curriculum design generalizes well as dataset scale increases. These findings imply that our method may also scale effectively to even larger datasets such as ImageNet-22K. With additional computational resources, we intend to validate our approach at the ImageNet-22K scale.\n\n---\n\n### 3. The paper does not compare to competing frameworks.\n\nWe appreciate the reviewer’s concern regarding comparisons to other efficient training frameworks. Many existing approaches are tailored to specific models—such as [1] or [2]—and cannot be applied directly to DINOv2 without significant changes to training configurations. Moreover, more general frameworks like [3] do not observe a speed-up for DINO in their experiments. To our knowledge, our work is the first to propose an efficient training pipeline for DINOv2 based on a data curriculum strategy. \n\n---\n\n### References\n\n1. Li, Runze, et al. *RECLIP: Resource-Efficient CLIP by Training with Small Images*. arXiv preprint arXiv:2304.06028, 2023.\n\n2. Wu, Jiantao, et al. *DailyMAE: Towards Pretraining Masked Autoencoders in One Day*. arXiv preprint arXiv:2404.00509, 2024.\n\n3. Wang, Yulin, et al. *EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training*. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 46, no. 12, 2024, pp. 8036–8055."}, {"review_id": "xlKM7rCtO8", "reviewer": "Reviewer_PgPD", "summary": "This paper introduces \"FastDINOv2,\" a two-stage training curriculum for visual foundation models like DINOv2, which aims to improve training efficiency and model robustness simultaneously. The method begins with an initial training phase (75% of epochs) where the model is trained exclusively on down-sampled, low-resolution images to accelerate the learning of coarse, low-frequency features. In the second stage (the remaining 25% of epochs), the training transitions to full-resolution images and introduces a \"Gaussian Noise Patching\" augmentation. This augmentation is specifically designed to counteract the frequency bias induced in the first stage and to balance robustness across the entire frequency spectrum. Through extensive experiments, the authors demonstrate that their approach reduces training time by a factor of 1.6 and requires FLOPs by 2.25x, while maintaining competitive accuracy and robustness compared to the original DINOv2 model.", "questions": "- The fixed 75% / 25% split for the curriculum is a key design choice. Have you investigated other splits or considered adaptive strategies where the transition point is determined dynamically based on convergence metrics? \n - In the introduction (line 89) and elsewhere, you describe the curriculum as inducing a \"high-frequency bias\". This is not very clear. Would it be more precise to state that the model develops higher robustness to low-frequency corruptions, which implies a more substantial reliance on high-frequency features? Could you further elaborate on this causality?\n - Bicubic interpolation was used for downsampling the images. Did you study the impact of other low-pass filtering methods (e.g., a Gaussian blur before downsampling) on the efficiency and robustness of the final model?\n - Given the lack of error bars, would it be feasible to run at least a few training runs with different random seeds for a smaller configuration (e.g., ViT-S on ImageNet-100) to provide at least a sense of the variance and stability of the results?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper addresses a highly relevant problem of hugh computational costs in training foundation models. The achieved speedup are substantial.\n2. The combination of a frequency-based curriculum with a targeted augmentation (Gaussian Patching) to compensate the induced bias is novel. The paper convincingly argues how robustness can be deliberately \"built into\" the training process rather than being merely a by-product of extreme scale.\n3. The method is validated across various model and dataset sizes (ViT-S/B, ImageNet-100/1K). The evaluation is not limited to classification accuracy, but also includes robustness benchmarks (ImageNet-C), downstream tasks (semantic segmentation), and qualitative analyses (Grad-CAM, Fourier heatmaps), which makes the results very convincing.\n\n**Weaknesses:**\n\n1. The transition between the two training stages is set to a fixed value of 75% of the epochs. The authors themselves acknowledge that this heuristic approach may not be optimal. An adaptive mechanism would make the method more robust and generally applicable.\n2. The results, particularly in the tables, are presented without error bars or other measures of statistical significance. The authors justify this in the checklist by citing high computational costs, which is understandable, but remains a scientific limitation as the variance of the results is unknown.\n3. The term 'l40s' (e.g., in Abstract, Table 1) used to describe GPU resources is unclear and impairs readability. Clearly stating the GPU model (e.g., NVIDIA L40S) would enhance clarity. \n4. The rationale for using downsampling as a proxy for low-frequency extraction is somewhat unclear or oversimplified. A more explicit discussion or comparison with standard low-pass filtering methods (e.g., Gaussian blur) would significantly strengthen this argument.", "was_revised": true, "final_justification": "I appreciate the author's response. I have increased my rating to 5 because the work contains a plenty of interesting aspects, so the paper is worthy to be accepted.", "rebuttal": "We sincerely thank Reviewer PgPD for the positive review and important suggestions. Below, we would like to address each of your questions:\n\n---\n\n### Question 1. The fixed 75% / 25% split for the curriculum is a key design choice. Have you investigated other splits or considered adaptive strategies where the transition point is determined dynamically based on convergence metrics?\t\n\nWe thank the reviewer for this important question and for recognizing the limitations involved in this design choice. We have indeed experimented with different splits, including 65/35, 75/25, and 85/15, and found that the 75/25 split yields the best performance.  \nFrom a training convergence perspective, it is essential that the model can benefit sufficiently from both stages in our curriculum—low-frequency and full-resolution training. If the first stage is too short, the model may not develop a sufficiently strong initialization and gain the convergence speed up; if the second stage is too short, it may not fully learn more fine-grained details from training on high-resolution data.  \nGiven that the total number of training epochs is fixed in our current setting, we adopt a heuristic 75%/25% split for FastDINOv2. However, we agree that dynamically determining the transition point based on convergence metrics, such as online linear probing accuracy, could further optimize performance. We see strong potential in extending our method to include such adaptive curriculum strategies.\n\n| Training method | Split | Training Time (NVIDIA A6000 h) | Accuracy  |\n|-----------------|-------|------------------------|-----------|\n| FastDINOv2      | 65/35 | 20.4                   | 77.6%     |\n| FastDINOv2      | 75/25 | 20.4                   | 78.1%     |\n| FastDINOv2      | 85/15 | 20.68                  | 76.8%     |\n\n---\n\n### Question  2. In the introduction (line 89) and elsewhere, you describe the curriculum as inducing a \"high-frequency bias\". This is not very clear. Would it be more precise to state that the model develops higher robustness to low-frequency corruptions, which implies a more substantial reliance on high-frequency features? Could you further elaborate on this causality?\n\nWe thank the reviewer for bringing up this thoughtful and important question. It is indeed more precise to describe the observed effect as increased robustness to low-frequency corruptions and a greater reliance on high-frequency features. This phenomenon arises from the intrinsic frequency biases of deep neural networks (DNNs), Vision Transformers (ViTs), and property of curriculum learning.  \nPrior work has shown that DNNs exhibit a spectral bias—they tend to learn low-frequency components earlier in training and gradually incorporate higher-frequency features as training progresses [1,2]. Additionally, the self-attention mechanism in ViTs behaves like a low-pass filter [3], further reinforcing this early preference for low-frequency information.  \nIn our proposed curriculum, we exploit these inherent biases by explicitly organizing training to begin with low-frequency components, which serve as easier examples, and then transition to full-resolution images, which contain more high-frequency content and are thus harder. This progression aligns with findings in curriculum learning, where beginning with easier examples can smooth the optimization landscape and enable more effective learning of complex patterns later on [4].  \nAs a result, once the model has built a strong foundation from low-frequency features, it is better positioned to learn and rely on high-frequency components during the later stages of training. This leads to improved robustness against low-frequency corruptions, as the model has developed stronger sensitivity to the more informative high-frequency features.\n\n---\n\n### References\n\n[1] Rahaman, N., Baratin, A., Arpit, D., et al. (2019). On the Spectral Bias of Neural Networks. Proceedings of the 36th International Conference on Machine Learning (ICML).  \n[2] Xu, Z.-Q. J., Zhang, Y., & Xiao, Y. (2019). Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523  \n[3] Wang, Peihao, et al. Anti‑Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice. ICLR 2022, arXiv:2203.05962, 2022.  \n[4] Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (pp. 41–48). ACM.\n\n---\n\n### Question 3. Bicubic interpolation was used for downsampling the images. Did you study the impact of other low-pass filtering methods (e.g., a Gaussian blur before downsampling) on the efficiency and robustness of the final model?\n\nWe thank the reviewer for this insightful question. Our initial hypothesis was that reducing high-frequency content in images could improve both convergence speed and model robustness, as low-frequency information typically captures the most generalizable features hence “easier”. To this end, we applied bicubic downsampling, which inherently acts as a low-pass filter and emphasizes low-frequency components.  \nWe also considered whether further reducing high-frequency signals with additional filtering such as Gaussian blur might improve performance. However, our experiments showed that bicubic downsampling alone was sufficient to retain the relevant low-frequency information while maintaining a balance with spatial detail. Additional filtering methods, including Gaussian blur and anti-aliasing, resulted in slight loss of detail and degraded model performance.  \nTherefore, we concluded that downsampling with bicubic interpolation provides an effective trade-off between low-frequency extraction and information retention, and we adopted this approach in our final setup.\n\n| Filtering method           | Accuracy | Imagenet-C-100 (accuracy) |\n|---------------------------|----------|---------------------------|\n| Gaussian blur → Downsampling | 77.46%   | 51.24%                    |\n| Anti-Alias → Downsampling    | 76.5%    | 49.73%                    |\n| Downsampling                | 78.1%    | 52.48%                    |\n\n---\n\n### Question 4. Given the lack of error bars, would it be feasible to run at least a few training runs with different random seeds for a smaller configuration (e.g., ViT-S on ImageNet-100) to provide at least a sense of the variance and stability of the results?\n\nWe thank the reviewer for understanding our limitation on computation resources and time. We conduct experiments using multiple random seeds (specifically, seeds 0, 10, and 5) on a smaller configuration—ViT-S trained on ImageNet-100 for FastDINOv2 with the 112-224 curriculum—and tested on ImageNet-100 validation set and ImageNet-100-C. While the output naturally varies across seeds, the overall performance remains consistent. These results support the robustness of our main findings.\n\n| Seed | Accuracy | Robustness |\n|-------|----------|------------|\n| 0     | 78.1%    | 52.48%     |\n| 10    | 77.43%   | 51.75%     |\n| 5     | 77.2%    | 51.68%     |"}, {"review_id": "tjns9Z9JvZ", "reviewer": "Reviewer_Yv4e", "summary": "This paper proposes FastDINOv2, a pre-training strategy for DINOv2. The core contribution is a two-stage, frequency-based curriculum that aims to accelerate training and improve robustness to common image corruptions. The curriculum begins by training the model exclusively on downsampled, low-frequency versions of images and then transitions to full resolution images augmented for the final 25% of training. The authors demonstrate on ImageNet that their method reduces training time, while maintaining competitive linear probing accuracy and comparable robustness on the ImageNet-C benchmark.", "questions": "I would not consider my current rating to be final, but this would be contingent on your answers to the below.\n\nCould you provide any intuition or results from preliminary experiments that led to the choice of 75/25? How does the model perform with, for instance, a 50%/50%, 90%/10%, 0%/100% or 100%/0% splits? What led to your decision of 75/25?\n\nIn table 2 there is a fairly consistent trend, with one outlier. Do you have any insight into why model performance degrades as the image patches get smaller in all cases except the 128-224 which is worse than 112-224?\n\nDo you think framing your model as competitive with the standard dinov2 performance is accurate? Or is your paper better framed as a slight trade off in accuracy but improvements in speed?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe main strength of this work lies in its practical utility. The reported reduction in pre-training time from 16 days to 10 days are significant improvements. This directly addresses a major bottleneck in modern CV research and makes model training more attainable for smaller labs.\n\nThe paper's curriculum, which progresses from low-frequency easier examples to full-frequency harder examples, is intuitive and well-motivated. The authors identify that this curriculum creates a high-frequency bias and propose using Gaussian noise patching to create a more balanced model. \n\nThe authors evaluate their method across multiple ablations. They assess not only training efficiency but also performance on downstream tasks, ensuring representational quality beyond classification. Their robustness analysis on ImageNet-C breaks down performance by corruption type and frequency domain, which supports their claim.\n\nWeaknesses:\n\nThe paper's main limitation, which the authors acknowledge, is the fixed schedule for the curriculum stages. The transition stages from low-resolution to full-resolution training after 75% of epochs is unmotivated. The paper lacks an ablation study on this ratio, making it difficult to assess how sensitive the final performance is to this critical hyperparameter or whether one stage is more important than the other. It is unclear if this 75/25 split is optimal.\n\nThe method is not a strict improvement. While achieving significant speed-ups, the final model for ImageNet-1K shows a slight drop in clean linear probing accuracy (77.8% for DINOv2 vs. 76.2% for FastDINOv2). The paper frames this as \"competitive,\" but it is better framed as trade-off.", "was_revised": true, "final_justification": "Updating my score in response to the rebuttal", "rebuttal": "We sincerely thank Reviewer Yv4e for the valuable comments and positive review. We would like to address each of your questions in detail:\n\n---\n\n### Question 1. Could you provide any intuition or results from preliminary experiments that led to the choice of 75/25? How does the model perform with, for instance, a 50%/50%, 90%/10%, 0%/100% or 100%/0% splits? What led to your decision of 75/25?\n\nWe thank the reviewer for this essential question. For the ablation experiments on curriculum ratios, we used a fixed number of training epochs to ensure a fair comparison across different splits. Within this controlled training budget, our goal was to improve both learning efficiency and final performance through a two-stage curriculum.  \nThis design addresses two key aspects of our method: (a) leveraging the efficiency of low-resolution training, and (b) preserving final performance through progressive exposure to high-frequency content. The first stage uses downsampled (low-resolution) inputs that emphasize low-frequency information, while the second stage with high-resolution images enables learning of fine-grained details. If the first stage is too short, the model fails to fully benefit from this simplified setting. If it is too long, the model lacks sufficient exposure to the full-resolution features necessary for downstream performance.  \nWe experimented with multiple splits on ImageNet-100 with ViT-S, including 65/35, 75/25, and 85/15. Among these, the 75/25 split consistently yielded the best trade-off, achieving the highest linear probing accuracy. More extreme cases such as 0%/100% would have no acceleration effect, and 100%/0% would have only low-frequency data training, which performed notably worse than the above splits.\n\n| Training method | Split | Training Time (A6000 h) | Accuracy (ImageNet-100)  |\n|-----------------|-------|------------------------|-----------|\n| FastDINOv2      | 65/35 | 20.4                   | 77.6%     |\n| FastDINOv2      | 75/25 | 20.4                   | 78.1%     |\n| FastDINOv2      | 85/15 | 20.68                  | 76.8%     |\n| FastDINOv2      | 0/100 | 22.3                   | 73.9%     |\n| FastDINOv2      | 100/0 | 20.1                   | 69.7%     |\n\nThis 75/25 curriculum design also enables cost-effective pretraining. By using downsampled images during the initial stage, GPU memory usage is significantly reduced, making it feasible to perform the majority of pretraining on smaller GPUs.\n| Training Method  | Training Epochs (Low-Res) | Training Epochs (Full-Res) | Max Memory Consumption (GB) Low-Res | Max Memory Consumption (GB) Full-Res |\n|------------------|---------------------------|----------------------------|------------------------------------|-------------------------------------|\n| DINOv2           | -                         | 200                        | -                                  | 33.5                                |\n| **FastDINOv2**   | 150                       | 50                         | 9.47                               | 33.5                                |\n\n---\n\n### Question 2. In table 2 there is a fairly consistent trend, with one outlier. Do you have any insight into why model performance degrades as the image patches get smaller in all cases except the 128-224 which is worse than 112-224?\n\nWe thank the reviewer for this careful observation. Indeed, Table 2 shows that as the patch size of the low-resolution images decreases, model performance tends to degrade. This is expected, as smaller image patches discard more high-frequency information, limiting the model's ability to learn fine-grained features during the early phase of training.  \nThe 128-224 case performs slightly worse than 112-224, and we believe this minor deviation is due to a tradeoff between resolution and curriculum duration. Specifically, the 128-224 setting may not sufficiently reduce the input complexity to yield a significant efficiency benefit. In contrast, the 112-224 split achieves a better balance—providing just enough simplification to aid convergence while preserving meaningful structure.\n\n---\n\n### Question 3. Do you think framing your model as competitive with the standard dinov2 performance is accurate? Or is your paper better framed as a slight trade off in accuracy but improvements in speed?\n\nWe thank the reviewer for this important clarification. Upon reflection, we agree that describing our model as “competitive” with the standard DINOv2 performance is somewhat overstated. One of our primary focuses lies in improving training speed and reducing computational resource requirements. Thus, it is more accurate to describe our model as a slight tradeoff rather than strictly competitive. We appreciate the reviewer pointing this out and will clarify this distinction more explicitly."}, {"review_id": "GRLePe0v9v", "reviewer": "Reviewer_keDD", "summary": "This work proposes a novel pretraining strategy for DINOv2 to accelerate training process, meanwhile the strategy improves robustness of distortion. The approach is based on frequency-based curriculum learning to see low-frequency images in low-resolution first and then apply guassian patches to high-resolution images for high-frequency training. As a result, the pretraining time is reduced by 1.6 times for a ViT-B/16 on ImageNet. The probing results on ImageNet-C and ImageNet verified the benefits of model robustness and training speed from Frequency-Based Curriculum Learning.", "questions": "A similar strategy is popular in progressive training, which increases the resolution and the augmentation strength gradually during training [1,2]. As mentioned in [1], low-resolution does not mean an east task in MAE. In contrast, low-resolution images could act as regularisation to improve performance[3]. \n\nWhy use fewer training epochs? To show the speedup number, the final performance should match the original number.\n\n\n\n[1]: Wu J, Mo S, Atito S, et al. Dailymae: Towards pretraining masked autoencoders in one day[C]//European Conference on Computer Vision. Springer, Cham, 2025: 131-149.\n\n[2]: Howard, J.: fast.ai - Now anyone can train Imagenet in 18 minutes (Aug 2018), https://www.fast.ai/posts/2018-08-10-fastai-diu-imagenet.html\n[3]: Li X, Wang Z, Xie C. An inverse scaling law for clip training[J]. Advances in Neural Information Processing Systems, 2023, 36: 49068-49087.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. From Table 3 and 4, we can see clear improvement on robustness by introducing Gaussian noise patching. \n2. The idea is simple and hopefully applied in SSL families. \n\n\nWeakness:\n1. The experimental results can hardly support their claims: The final performance of the 112-224 curriculum (78.44%) is lower than the original DINOv2 (78.6%) for  linear probing on IN100. A lower performance on dense tasks in Table 5.\n2. The conclusion is made on ImageNet-100, an easy subset with less noise. For large-scale datasets full of diverse noise, the impact of Gaussian noise patching is questionable.\n3. How to combine masked image modelling and Gaussian noise patching is not clear. Does the GP applied to visible patches only? Or a replacement of masking?\n4. This work lacks a comprehensive literature review on the studies of frequency and robustness.", "was_revised": true, "final_justification": "Thank you for your updated results. I will raise my rating but won't be high. The contribution to accelerate training is good, but the weakness are also clear. First, there are some theories to explain such behaviours during training that the model prefers low-frequency components at the beginning stage [1,2]. Since the model cannot learn low-frequency components, we can just remove high-frequency components to speed up the training, which has been applied in many studies. The paper lacks a comprehensive discussion about these theories and work. Second,  the real question is how to effectively extract high-frequency components. Although this paper proposed an easy yet effective solution by Gaussian Patching, it lacks enough experimental results to demonstrate its effectiveness. Third,  the Gaussian Patching does not sound an ultimate solution to enforce models to learning high-frequency components. In fact, there is relevant study on this [4].\n\n\n[1] Balestriero R, LeCun Y. Learning by reconstruction produces uninformative features for perception[J]. arXiv preprint arXiv:2402.11337, 2024.\n\n[2] Littwin E, Saremi O, Advani M, et al. How jepa avoids noisy features: The implicit bias of deep linear self distillation networks[J]. arXiv preprint arXiv:2407.03475, 2024.\n\n[3] Wang H, Wu X, Huang Z, et al. High-frequency component helps explain the generalization of convolutional neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 8684-8694.\n\n[4] Bizeul A, Sutter T M, Ryser A, et al. COMPONENTS BEAT PATCHES: EIGENVECTOR MASKING FOR VISUAL REPRESENTATION LEARNING[J].", "rebuttal": "We sincerely thank Reviewer keDD for the insightful questions. We would like to give detailed responses to each of your comments and questions.\n\n### Question 1. The experimental results can hardly support their claims:The final performance of the 112-224 curriculum (78.44%) is lower than the original DINOv2 (78.6%) for linear probing on IN100. A lower performance on dense tasks in Table 5.\nWe thank the reviewer for this important observation. We understand the reviewer’s concern regarding the final performance of the 112–224 curriculum. While performance improvement is valuable, the central contribution of our work lies in maintaining accuracy and robustness while significantly reducing training cost, and in providing a comprehensive analysis of frequency bias in the resulting model.  \nWith only a 0.2% drop in accuracy, the 112–224 curriculum achieves both a robustness gain and a substantial reduction in training time. As noted by the reviewer, the dense task results in Table 5 further support this: FastDINOv2 achieves a 0.21% improvement in mAcc with only a 0.04% drop in mIoU, indicating comparable performance across both image-level and pixel-level tasks.  \nTo further assess the effectiveness of our method on downstream applications, we also conducted additional experiments on instance-level recognition using the Oxford and Paris datasets, each evaluated with three difficulty-level splits. On the Paris dataset, FastDINOv2 achieves comparable performance across all levels. On the Oxford dataset, our model shows substantial improvements on the easy and medium splits, further demonstrating the robustness and generalization of our approach.\n| Training method | Oxford E | Oxford M | Oxford H | Paris E | Paris M | Paris H |\n|-----------------|----------|----------|----------|---------|---------|---------|\n| FastDINOv2      | 32.11%   | 22.26%   | 3.99%    | 56.38%  | 40.71%  | 14.56%  |\n| DINOv2          | 28.38%   | 21.04%   | 4.56%    | 56.88%  | 40.76%  | 14.94%  |\n\n### Question 2. The conclusion is made on ImageNet-100, an easy subset with less noise. For large-scale datasets full of diverse noise, the impact of Gaussian noise patching is questionable.\nWe thank the reviewer for raising this important point regarding the potential impact of training on larger and noisier datasets. In Table 1, we provide results for both linear probing and robustness evaluation on ImageNet-1K and ImageNet-C. These results show that FastDINOv2 significantly reduces training time, with only a slight drop in linear probing accuracy and comparable robustness—demonstrating that the proposed data curriculum remains effective even at a larger scale. The maintained robustness further suggests that Gaussian Noise Patching continues to offer benefits on more diverse datasets.  \nDue to computational constraints, we were unable to conduct additional experiments on ImageNet-22K, but we plan to do so once we have access to sufficient resources.\n\n### Question 3. How to combine masked image modelling and Gaussian noise patching is not clear. Does the GP applied to visible patches only? Or a replacement of masking?\nWe thank the reviewer for bringing up this important aspect. To clarify, Gaussian noise is applied to a randomly selected region of the image with a random size to form a Gaussian noise patch. This process is independent of patch visibility in masked image modeling and is not intended as a replacement for masking. Instead, we introduce Gaussian Noise Patching as a complementary technique designed to help mitigate the frequency bias introduced by the data curriculum.\n\n### Question 4. A similar strategy is popular in progressive training, which increases the resolution and the augmentation strength gradually during training [1,2]. As mentioned in [1], low-resolution does not mean an east task in MAE. In contrast, low-resolution images could act as regularisation to improve performance[3].\nWe understand the reviewer’s question regarding the novelty of our work. One of our key contributions is a comprehensive analysis from the perspective of frequency and robustness, specifically focused on models trained with a low-frequency data curriculum. To the best of our knowledge, we are the first to identify and characterize the frequency bias introduced by such a training framework. Furthermore, we demonstrate that the combination of low-frequency training and Gaussian noise patching forms a non-trivial curriculum, which leads to more balanced and robust performance compared to using either technique in isolation. In addition to maintaining or improving downstream task performance and mitigating spurious correlations, our method achieves significant reductions in training time specifically for the DINOv2 model—an outcome not demonstrated in prior work.  \nThe reviewer refers to [1] and point out that low-resolution images act as a form of regularization rather than representing easier samples. From our perspective, this interpretation is not necessarily in conflict with defining them as “easy.” The self-attention mechanism in ViT inherently functions as a low-pass filter—it emphasizes low-frequency components while attenuating high-frequency signals. From this viewpoint, downsampled images align more closely with the model's internal bias and can thus be considered easier for the model to learn from.\n\n### Question 5. Why use fewer training epochs? To show the speedup number, the final performance should match the original number.\nWe thank the reviewer for pointing out this confusion. In our evaluation, the speedup is measured by comparing training time while keeping the final performance comparable. FastDINOv2 converges more quickly with the data curriculum, and in our experiments, it reaches the same level of accuracy as the baseline using fewer epochs. This faster convergence demonstrates the efficiency of our approach in reducing training cost while maintaining final performance."}]

Author rebuttal:

Meta-review: The paper proposes FastDINOv2, a two-stage frequency-based curriculum to accelerate DINOv2 pre-training. The method trains first on low-resolution images, then transitions to full-resolution images with a novel "Gaussian Noise Patching" augmentation, claiming a 1.6x training speedup while maintaining competitive accuracy and improving robustness.

The primary strength of this work is its practical impact, offering a significant reduction in the computational cost of training foundation models. The curriculum is well-motivated and its combination with a targeted noise augmentation to improve robustness is a novel contribution, supported by a thorough evaluation covering accuracy, robustness, and downstream tasks. The main weaknesses are that the method presents a trade-off—a minor drop in clean accuracy in exchange for speed—and its evaluation is limited to ImageNet-scale datasets. The paper could also better situate itself within the existing literature on spectral bias.

Despite these points, I recommend acceptance due to the paper's clear practical value in making foundation model training more efficient and robust. The discussion period was crucial in reaching this decision. Initial reviewer concerns about the method's generalizability, hyperparameter choices, and performance framing were addressed directly by the authors with new experiments and clarifications. These responses caused three of four reviewers to raise their scores, solidifying a strong consensus for acceptance. The strengths and practical utility of the work clearly outweigh its remaining limitations.
Decision: Accept (poster)