Title: Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Supervised Learning

Curriculum Learning

Noise

Robustness

Representation Learning
Abstract: Self-Supervised Learning (SSL) has become a powerful solution to extract rich representations from unlabeled data. Yet, SSL research is mostly focused on clean, curated and high-quality datasets. As a result, applying SSL on noisy data remains a challenge, despite being crucial to applications such as astrophysics, medical imaging, geophysics or finance. In this work, we present a fully self-supervised framework that enables noise-robust representation learning without requiring a denoiser at inference or downstream fine-tuning. Our method first trains an SSL denoiser on noisy data, then uses it to construct a denoised-to-noisy data curriculum (i.e., training first on denoised, then noisy samples) for pretraining a SSL backbone (e.g., DINOv2), combined with a teacher-guided regularization that anchors noisy embeddings to their denoised counterparts. This process encourages the model to internalize noise robustness. Notably, the denoiser can be discarded after pretraining, simplifying deployment. On ImageNet-1k with ViT-B under extreme Gaussian noise ($\sigma=255$, SNR = 0.72 dB), our method improves linear probing accuracy by 4.8\% over DINOv2, demonstrating that denoiser-free robustness can emerge from noise-aware pretraining. The code is available at https://github.com/wenquanlu/noisy_dinov2.


Peer reviews:
comments: SUMMARY:
The paper offers a  method to have  off-the-shelf self-fupervised models like DINOv2 to robustly learn representations  on noisy images in a self-supervised way.  A denoiser is trained and then it generates noisy-denoised image pairs to build a curriculum for post-training DINO.  The latter parts of the curriculum have a teacher-based regularizer to force convergence. The denoiser is jettisoned after post-training. 

The idea  is not that different from the use of the decoder half of a diffusion model, or denoising-quantification as in syed & mirza.
The idea is built with visible increments, like showing DINO's weakness in  data scarce setting,  meidum improvements on data addition, and finally a new  paradigm for gradually building noise tolerance. Experiments are reflective.

STRENGTHS AND WEAKNESSES:
The strengths include having not to denoise during inference, and the tailorize use of a foundation model.

The paper does not use several fms, making the claim above a little weak. The  noise is random, and may vary across many runs, something that the experiment at larger scales (data size) avoids, raising a question over scaling.

QUESTIONS:
How important will the temperature of the curriculum be?

LIMITATIONS:
The noise added is ...added.  Multiplicative or mixed source often plagues data.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 2
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-supervised training approach aimed at enhancing DINOv2's performance in high-noise scenarios using curriculum learning. The authors provide thorough analysis and comparisons, demonstrating that the two-stage (Clean + noise) self-supervised training yields superior performance compared to single-stage training (using solely noisy data). The primary experimental setting involves training on synthetic noisy images and evaluating on synthetic noisy images. A notable advantage is that this method eliminates the need for a denoiser at test time, while the final results remain robust on both clean and noisy data.\n\nSTRENGTHS AND WEAKNESSES:\n1. Novel Setting:​​ SSL model training tailored for noisy images\n\n​​2. Extensive Experiments:​​ Evaluated across classification and instance-level recognition tasks, applied to multiple SSL methods beyond DINOv2\n\n​​3. Well-Written Paper:​​ Clear presentation and good organization\n\nQUESTIONS:\n1.  **Critical point:** Performance of DINOv2 trained on *clean images* evaluated on the *noisy test set*? The paper primarily reports DINOv2(clean) performance on clean images. However, the performance drop of DINOv2(clean) on the noisy test set is crucial for establishing this work's significance.  \n    **Suggestion:** Supplement Tables 2 and 3 with these results.\n\n2.  The inclusion of *denoised images* inherently enhances training set expressiveness.  \n    **Please show results** when training on:\n    -   Solely noisy images vs.\n    -   Mixed noisy + denoised images\n\n3.  Significant performance gap exists between the author-trained DINOv2-s (79.4% ImageNet accuracy) and the official DINOv2-small (81.1%). I understand computational constraints may limit training.  \n    **Key question:** Can the proposed method effectively *fine-tune* a **well-pretrained official DINOv2 model**? Enhancing the robustness of widely-used official weights against noisy data would be highly impactful.\n\n4.  **Urgently needed:** Evaluation on **real-world noisy datasets** (e.g., Hendrycks & Dietterich, \"Benchmarking Neural Network Robustness to Common Corruptions and Perturbations\", arXiv:1903.12261, 2019).\n\n5.  Can **PCA visualizations** on noisy images be shown for different models? (Similar to the style used in the original DINOv2 paper).\n\n6. **Releasing code would be beneficial to the community.**\n\nLIMITATIONS:\nThe authors have honestly acknowledged one limitation I previously noted: the lack of testing on **real-world noisy datasets**. However, one **critical limitation remains unaddressed**: The paper does not analyze whether the proposed method can effectively **fine-tune well-pretrained models** (e.g., off-the-shelf DINOv2 checkpoints) rather than training exclusively from scratch.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel, fully self-supervised framework to train noise-robust representation models without needing a denoiser during inference or fine-tuning. The central problem addressed is that state-of-the-art Self-Supervised Learning (SSL) methods, like DINOv2, perform poorly when pretrained on noisy data, a common issue in real-world applications such as medical imaging and astrophysics. The proposed solution involves a curriculum learning strategy where an SSL model is first trained on a denoised version of the dataset (created using an auxiliary SSL denoiser) and then continues training on the original noisy data. This \"denoised-to-noisy\" curriculum encourages the model to internalize noise-robust features, allowing the denoiser to be discarded after pretraining. Experiments on ImageNet under various synthetic noise conditions show that the proposed methods significantly improve performance over standard SSL training on noisy data and often match or exceed a pipeline that explicitly uses a denoiser.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The work addresses the critical and under-explored challenge of applying SSL to noisy, uncurated datasets, which is a frequent scenario in many scientific and industrial domains. \n2. The proposed noise curriculum (NC) is an intuitive and simple yet powerful strategy. It effectively builds noise robustness directly into the representation model without requiring complex architectural changes or a persistent denoiser module. \n3. The authors conduct extensive experiments across multiple datasets (ImageNet-100, ImageNet-1k) , noise types (Gaussian, Shot, Speckle), and severity levels. The method's effectiveness is demonstrated on different downstream tasks (classification, instance recognition) and its generalizability is tested on a wide range of SSL models (SimCLR, MoCo v3, iBOT, etc.). \n4. A key advantage is that the denoiser is only used during the pretraining phase and can be discarded for downstream tasks. This simplifies deployment and reduces computational overhead and latency during inference. \n\n\nWeaknesses:\n1. To my undestanding, the framework's success hinges on the initial step of training a self-supervised denoiser that can produce a reasonably clean version of the dataset.  The performance of the entire pipeline is therefore dependent on the effectiveness of the chosen denoiser, which might be a challenge for unusual or extremely high levels of noise.\n2. The timing of the switch from denoised to noisy training is a crucial hyperparameter that currently requires manual tuning.  While the paper shows the method is robust within a certain range, an automated or adaptive strategy for this transition would enhance the method's practicality.\n3. The proposed method introduces additional computational steps to the pretraining pipeline: training an SSL denoiser and then using it to process the entire dataset. While it saves computation at inference, this increases the upfront resource requirement for pretraining.\n4. The experiments are benchmarked using synthetic noise types.  While this allows for controlled and reproducible evaluation, real-world noise can be more complex. For example, in medical images, the CT images in sinogam can be viewed as poisson noise, however in the image domain, it is difficult to define what the noise is. The translation of these impressive results to real-world noisy datasets is not yet fully demonstrated.\n\nQUESTIONS:\n1. The performance of your framework depends on the initial SSL denoiser. How does the performance of DINOv2 w/ NC degrade if a less effective denoiser is used? Is there a performance threshold for the denoiser below which the curriculum provides diminishing or no returns?\n\n2. The restart epoch is a key hyperparameter. Have you explored any methods to automate the selection of this epoch? For instance, could the transition be triggered dynamically based on the convergence of the model on the denoised data?\n\n3. Do you have plans to evaluate your framework on such real-world datasets, and how do you anticipate its performance will translate from i.i.d. synthetic noise to more complex, structured real-world noise?\n\n4. The analysis on clean validation sets is very insightful. You suggest that DINOv2 w/ NC/NCT outperforms the N2N + DINOv2 baseline because explicit denoising leads to information loss.  Could an alternative or complementary explanation be that the two-stage curriculum acts as a powerful regularizer, forcing the model to learn more general features by being exposed to both clean (denoised) and noisy distributions?\n\n4. For the DINOv2 w/ NCT results on ImageNet-1k, the denoised teacher used for regularization is trained for a full 100 epochs, while the student model it guides only sees the denoised data for 30 epochs before switching to noisy training.  This seems to create a significant gap between the teacher and the student's initial state in the second training phase. Could you elaborate on why this design choice was more effective than using a teacher from the 30-epoch checkpoint, and how embedding alignment is preserved?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the major limitations of their work in Section 5. A constructive discussion of limitations is critical, and the authors handle this well.\n\nThe paper correctly identifies and discusses two key limitations:\nDependency on Denoiser Quality: The authors acknowledge that the framework assumes a reasonably effective self-supervised denoiser can be trained on the noisy data. They note that this is a fair assumption in many cases but remains an \"open dependency.\" \nManual Curriculum Scheduling: The paper points out that the current design requires manual tuning to determine the optimal point to switch from denoised to noisy pretraining. \n\nThe authors also implicitly acknowledge the use of synthetic noise as a necessary choice for controlled, reproducible assessment.  To further strengthen this section, I would suggest also adding a brief discussion on the increased upfront computational cost of the pretraining pipeline (training a denoiser and processing the dataset) as a trade-off for the gains in inference efficiency and robustness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}