Title: FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Supervised Learning

Vision Foundation Model

Robustness

Efficient Training

Curriculum Learning
Abstract: Large-scale vision foundation models such as DINOv2 boast impressive performances by leveraging massive architectures and training datasets. The expense of large-scale pre-training puts such research out of reach for many, hence limiting scientific advancements. We thus propose a novel pretraining strategy for DINOv2 that simultaneously accelerates convergence–and strengthens robustness to common corruptions as a by-product. Our approach involves a frequency filtering curriculum–low-frequency being seen first–and the Gaussian noise patching augmentation. Applied to a ViT-B/16 backbone trained on ImageNet-1K, while pre-training time is reduced by 1.6×–from 16.64 to 10.32 NVIDIA L40S days–and FLOPs by 2.25×, our method still achieves matching robustness in corruption benchmarks (ImageNet-C) and maintains competitive linear probing performance compared with the DINOv2 baseline. This dual benefit of efficiency and robustness makes large-scale self-supervised foundation modeling more attainable, while opening the door to novel exploration around data curriculum and augmentation as a means to improve self-supervised learning models robustness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel pretraining strategy for DINOv2 to accelerate training process, meanwhile the strategy improves robustness of distortion. The approach is based on frequency-based curriculum learning to see low-frequency images in low-resolution first and then apply guassian patches to high-resolution images for high-frequency training. As a result, the pretraining time is reduced by 1.6 times for a ViT-B/16 on ImageNet. The probing results on ImageNet-C and ImageNet verified the benefits of model robustness and training speed from Frequency-Based Curriculum Learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. From Table 3 and 4, we can see clear improvement on robustness by introducing Gaussian noise patching. \n2. The idea is simple and hopefully applied in SSL families. \n\n\nWeakness:\n1. The experimental results can hardly support their claims: The final performance of the 112-224 curriculum (78.44%) is lower than the original DINOv2 (78.6%) for  linear probing on IN100. A lower performance on dense tasks in Table 5.\n2. The conclusion is made on ImageNet-100, an easy subset with less noise. For large-scale datasets full of diverse noise, the impact of Gaussian noise patching is questionable.\n3. How to combine masked image modelling and Gaussian noise patching is not clear. Does the GP applied to visible patches only? Or a replacement of masking?\n4. This work lacks a comprehensive literature review on the studies of frequency and robustness.\n\nQUESTIONS:\nA similar strategy is popular in progressive training, which increases the resolution and the augmentation strength gradually during training [1,2]. As mentioned in [1], low-resolution does not mean an east task in MAE. In contrast, low-resolution images could act as regularisation to improve performance[3]. \n\nWhy use fewer training epochs? To show the speedup number, the final performance should match the original number.\n\n\n\n[1]: Wu J, Mo S, Atito S, et al. Dailymae: Towards pretraining masked autoencoders in one day[C]//European Conference on Computer Vision. Springer, Cham, 2025: 131-149.\n\n[2]: Howard, J.: fast.ai - Now anyone can train Imagenet in 18 minutes (Aug 2018), https://www.fast.ai/posts/2018-08-10-fastai-diu-imagenet.html\n[3]: Li X, Wang Z, Xie C. An inverse scaling law for clip training[J]. Advances in Neural Information Processing Systems, 2023, 36: 49068-49087.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FastDINOv2, a pre-training strategy for DINOv2. The core contribution is a two-stage, frequency-based curriculum that aims to accelerate training and improve robustness to common image corruptions. The curriculum begins by training the model exclusively on downsampled, low-frequency versions of images and then transitions to full resolution images augmented for the final 25% of training. The authors demonstrate on ImageNet that their method reduces training time, while maintaining competitive linear probing accuracy and comparable robustness on the ImageNet-C benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe main strength of this work lies in its practical utility. The reported reduction in pre-training time from 16 days to 10 days are significant improvements. This directly addresses a major bottleneck in modern CV research and makes model training more attainable for smaller labs.\n\nThe paper's curriculum, which progresses from low-frequency easier examples to full-frequency harder examples, is intuitive and well-motivated. The authors identify that this curriculum creates a high-frequency bias and propose using Gaussian noise patching to create a more balanced model. \n\nThe authors evaluate their method across multiple ablations. They assess not only training efficiency but also performance on downstream tasks, ensuring representational quality beyond classification. Their robustness analysis on ImageNet-C breaks down performance by corruption type and frequency domain, which supports their claim.\n\nWeaknesses:\n\nThe paper's main limitation, which the authors acknowledge, is the fixed schedule for the curriculum stages. The transition stages from low-resolution to full-resolution training after 75% of epochs is unmotivated. The paper lacks an ablation study on this ratio, making it difficult to assess how sensitive the final performance is to this critical hyperparameter or whether one stage is more important than the other. It is unclear if this 75/25 split is optimal.\n\nThe method is not a strict improvement. While achieving significant speed-ups, the final model for ImageNet-1K shows a slight drop in clean linear probing accuracy (77.8% for DINOv2 vs. 76.2% for FastDINOv2). The paper frames this as \"competitive,\" but it is better framed as trade-off.\n\nQUESTIONS:\nI would not consider my current rating to be final, but this would be contingent on your answers to the below.\n\nCould you provide any intuition or results from preliminary experiments that led to the choice of 75/25? How does the model perform with, for instance, a 50%/50%, 90%/10%, 0%/100% or 100%/0% splits? What led to your decision of 75/25?\n\nIn table 2 there is a fairly consistent trend, with one outlier. Do you have any insight into why model performance degrades as the image patches get smaller in all cases except the 128-224 which is worse than 112-224?\n\nDo you think framing your model as competitive with the standard dinov2 performance is accurate? Or is your paper better framed as a slight trade off in accuracy but improvements in speed?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"FastDINOv2,\" a two-stage training curriculum for visual foundation models like DINOv2, which aims to improve training efficiency and model robustness simultaneously. The method begins with an initial training phase (75% of epochs) where the model is trained exclusively on down-sampled, low-resolution images to accelerate the learning of coarse, low-frequency features. In the second stage (the remaining 25% of epochs), the training transitions to full-resolution images and introduces a \"Gaussian Noise Patching\" augmentation. This augmentation is specifically designed to counteract the frequency bias induced in the first stage and to balance robustness across the entire frequency spectrum. Through extensive experiments, the authors demonstrate that their approach reduces training time by a factor of 1.6 and requires FLOPs by 2.25x, while maintaining competitive accuracy and robustness compared to the original DINOv2 model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper addresses a highly relevant problem of hugh computational costs in training foundation models. The achieved speedup are substantial.\n2. The combination of a frequency-based curriculum with a targeted augmentation (Gaussian Patching) to compensate the induced bias is novel. The paper convincingly argues how robustness can be deliberately \"built into\" the training process rather than being merely a by-product of extreme scale.\n3. The method is validated across various model and dataset sizes (ViT-S/B, ImageNet-100/1K). The evaluation is not limited to classification accuracy, but also includes robustness benchmarks (ImageNet-C), downstream tasks (semantic segmentation), and qualitative analyses (Grad-CAM, Fourier heatmaps), which makes the results very convincing.\n\n**Weaknesses:**\n\n1. The transition between the two training stages is set to a fixed value of 75% of the epochs. The authors themselves acknowledge that this heuristic approach may not be optimal. An adaptive mechanism would make the method more robust and generally applicable.\n2. The results, particularly in the tables, are presented without error bars or other measures of statistical significance. The authors justify this in the checklist by citing high computational costs, which is understandable, but remains a scientific limitation as the variance of the results is unknown.\n3. The term 'l40s' (e.g., in Abstract, Table 1) used to describe GPU resources is unclear and impairs readability. Clearly stating the GPU model (e.g., NVIDIA L40S) would enhance clarity. \n4. The rationale for using downsampling as a proxy for low-frequency extraction is somewhat unclear or oversimplified. A more explicit discussion or comparison with standard low-pass filtering methods (e.g., Gaussian blur) would significantly strengthen this argument.\n\nQUESTIONS:\n- The fixed 75% / 25% split for the curriculum is a key design choice. Have you investigated other splits or considered adaptive strategies where the transition point is determined dynamically based on convergence metrics? \n - In the introduction (line 89) and elsewhere, you describe the curriculum as inducing a \"high-frequency bias\". This is not very clear. Would it be more precise to state that the model develops higher robustness to low-frequency corruptions, which implies a more substantial reliance on high-frequency features? Could you further elaborate on this causality?\n - Bicubic interpolation was used for downsampling the images. Did you study the impact of other low-pass filtering methods (e.g., a Gaussian blur before downsampling) on the efficiency and robustness of the final model?\n - Given the lack of error bars, would it be feasible to run at least a few training runs with different random seeds for a smaller configuration (e.g., ViT-S on ImageNet-100) to provide at least a sense of the variance and stability of the results?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}

comments: SUMMARY:
The paper introduces FastDINOv2, a two-stage curriculum training approach for DINOv2, which aims to enhance both training efficiency and robustness to corruption. The framework considers training first on downsampled images for 75% of the epochs, then changing to full-resolution, together with Gaussian noise patching, for the remaining 25% of the epochs. The authors demonstrate a 1.5x reduction in training time while maintaining competitive performance on (mostly) ImageNet-based benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The approach achieves substantial acceleration without a major downgrade of results.
2. The frequency-based design of the curriculum is well-motivated and intuitive.
3. The authors provide insightful analysis, using Frequency-based analysis and Grad-CAM to support their claims.

Weaknesses:
1. Experiments are performed solely on DINOv2 (ViT). Will the curriculum show similar results for other methods? Is it also viable for CNNs?
2. Continuing the above, the experiments are performed mostly on an ImageNet subset (ImageNet-100). Raising concerns about how the curriculum performs on larger datasets (ImageNet-1K, ImageNet-22k, etc.)
3. The paper does not compare to competing frameworks.

QUESTIONS:
I would appreciate it if the authors could address my concerns in the weaknesses section. I would consider increasing my score if those are addressed properly.

LIMITATIONS:
The authors mention some limitations in the conclusion of the paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-06