Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
Authors: Wenquan Lu
Jiaqi Zhang
Hugues Van Assel
Randall Balestriero
Venue: NeurIPS 2025 poster ()
Keywords: Self-Supervised Learning
Curriculum Learning
Noise
Robustness
Representation Learning
Primary area: deep_learning
TL;DR: We show that self-supervised models like DINOv2 can develop strong noise robustness without any explicit denoiser at downstream fine-tuning or inference, by leveraging a data curriculum and a denoised regularization loss during pretraining.

Abstract: Self-Supervised Learning (SSL) has become a powerful solution to extract rich representations from unlabeled data. Yet, SSL research is mostly focused on clean, curated and high-quality datasets. As a result, applying SSL on noisy data remains a challenge, despite being crucial to applications such as astrophysics, medical imaging, geophysics or finance. In this work, we present a fully self-supervised framework that enables noise-robust representation learning without requiring a denoiser at inference or downstream fine-tuning. Our method first trains an SSL denoiser on noisy data, then uses it to construct a denoised-to-noisy data curriculum (i.e., training first on denoised, then noisy samples) for pretraining a SSL backbone (e.g., DINOv2), combined with a teacher-guided regularization that anchors noisy embeddings to their denoised counterparts. This process encourages the model to internalize noise robustness. Notably, the denoiser can be discarded after pretraining, simplifying deployment. On ImageNet-1k with ViT-B under extreme Gaussian noise ($\sigma=255$, SNR = 0.72 dB), our method improves linear probing accuracy by 4.8\% over DINOv2, demonstrating that denoiser-free robustness can emerge from noise-aware pretraining. The code is available at https://github.com/wenquanlu/noisy_dinov2.

Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
Wenquan Lu
Jiaqi Zhang'
Hugues Van Assel?
Randall Balestriero
Brown University Genentech _laqi_zhan6,, randall, hatpellaaaeesstterreeeeeoeoeeeee
{venquan_lu lu,
baleesrierro@@brom..eet van_assel hugee..ugeeeeeeeeee com
Abstract
Self-Supervised Learning (SSL) has become a powerful solution to extract rich representations from unlabeled data. Yet, SSL research is mostly focused on clean, curated and high-quality datasets. As result, applying SSL on noisy data remains challenge, despite being crucial to applications such astrophysics, medical imaging, geophysics or finance. In this work, we present fully self- supervised framework that enables noise-robust representation learning without requiring a denoiser at inference or downstream fine-tuning. Our method first trains an SSL denoiser on noisy data, then uses it construct a denoised-to- noisy data curriculum (i.e., training first on denoised, then noisy samples) for pretraining SSL backbone (e.g.. DINOv2), combined with teacher-guice regularization that anchors noisy embeddings to their denoised counterparts. This process encourages the model internalize noise robustness. Notably, the denoiser can be discarded after pretraining, simplifying deployment. On ImageNet- k with ViT-B under extreme Gaussian noise (a 255, SNR = 0.72 dB), our method improves linear probing accuracy by 4.8% over DINOv2, demonstrating that denoiser-free robustness can emerge from noise-aware pretraining. The code is available at
Introduction
16 9 have demonstrated remarkable success by leveraging unlabeled data to learn visual representations able to solve many tasks in zero or few shot manners (3]. However, the performances of those methods heavily rely on the availability of clean and high-quality datasets. As a result, some prior works have focused on building data curation pipelines for self-supervised learning (SSL) [1 But data curation disregards samples and introduces additional design questions and thus begs the following question:
Self-supervised methods like DINOv2 31
Can we enable off off-the-shelf SSL models like DINOv2 learn from highly corrupted data?
In real-world scenarios, datasets often contain noise that can severely degrade the performance of learned representations, and clean references required for supervised denoising are typically unavailable due sensor limitations, privacy constraints, or acquisition costs.
This is especially common in medical imaging 28
43 271, astrophysics [30], geological rock imaging (6]. remote sensing [17] and finance 29 For example, the earth radar images taken by ESA Sentinel-1 [4] comes with speckle noise, and the clean data for supervised denoising is difficult to acquire since it requires better sensor and atmospheric conditions. As another example, Sloan Digital Sky Survey (SDSS) imaging in the COSMOS region contains significant sky noise [30] which can obscure faint celestial objects. Despite advancements SSL, methods like DINOv2, as shown in Figure 5 still yield significant reduction in performance when trained directly on corrupted datasets.
To address the challenges outlined above, trivial baseline is aaaooiser-prprocesssee pipeline, which trains a self-supervised denoiser on the noisy data first, then trains the SSL models on the denoised
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Denoiser used in downstream tasks
Denoiser free downstream tasks
"Plano"
"Piano"
Denolsing
Pretrained
Ardiaacta
Pretrained
Head
Pretrained
Heed
Denoiser
SSL Model
SSL Model
Carry over denoiser's and artifacts model.
Fully salf-constreined. No external dependency during deployment.
Extra computation and memory overhead during inference and fine-
tuning.
Light-weight.
If downstream distribution differs from pretrain data, the denoiser needs be retrained separately from SSL
End-to rond-t--nndai fine -utiinnng simplicity.
model internalizes noise-robust features.
X SSL model operates on denoised data, with no built-in robustness,
Figure I: Comparison of downstream pipelines with (left) and without (right) denoisers. The denoiser- free pipeline shows numerous advantages in efficiency, simplicity and robustness.
images. In downstream tasks, the SSL denoiser always serves as preprocessor because the SSL model is only trained on denoised data thus cannot handle noisy inputs. While effective, this pipeline adds substantial inference latency, fine-tuning overhead, and deployment complexity (Figure I left). Instead, we explore training strategies that enable off-the-shelf SSL backbone (e.g.. DINOv2) to ingest noisy inputs directly, thereby ditching the denoiser during downstream use (Figure T right). To the best of our knowledge, this problem has little-to-no prior work in the existing literature. We take
a first step by corrupting ImageNet [12] with synthetic noise that mirrors real-world degradations, providing controlled yet challenging testbed for systematic evaluation and benchmarking.
In this paper, we propose novel, fully self superviee approach learn noise-robust representations. Our approach begins similarly as the denoising pipeline baseline: train a SSL denoiser to create denoised version of the dataset. We then employ curriculum [5] train DINOv2 on this denoised dataset for robust feature learning and subsequently restart training on the original noisy dataset to adapt the model to real-world noise. At inference time or downstream task finetuning, the denoiser can be discarded as the SSL model has learned to extract useful representations directly from noisy inputs-withour relying on external denoising. In addition, We further introduce regularization loss using denoised teacher to guide the student during noisy training, encouraging alignment between noisy and denoised embeddings and improving robustness under extreme noise. Extensive experiments show that method significantly improves classification and instance recognition performance over DINOv2 trained directly on noisy data. Remarkably, our framework achieves results comparable to, and sometimes exceeding, those obtained by the denoising pipeline baseline. In summary, we make the following contributions:
1. We propose a thorough sensitivity analysis of the latest SOTA SSL method (DINOv2) to noise being present its pretraining data, previously unexplored aspect in the literature. We identify that the method is sensitive to its pretraining data quality, especially in low-resource regimes (e.g., short training schedules or smaller datasets).
2. We find that, under mild noise levels, increasing training duration and data size can substantially mitigate performance degradation, but never fully recover the clean-data baseline. This offers actionable guidance for practitioners to deal with noisy data during pretraining.
3. We introduce tations directly from noisy pretraining data. By combining noise curriculum learning with a novel denoised regularization loss, our method achieves fully denoiser-free downstream fine-tuning and inference while maintaining strong performance across tasks. As shown in Figure 2 our methods, DINOv2 w/ NC and DINOv2 w/ NCT improve significantly over the DINOv2 baseline, and closely matches the denoiser- preprocessed pipeline N2N + DINOv2.
training paradigm for enabling models like DINOv2 to learn robust represen-
2 Related Work
Self-Supervised Learning for Images. A large body of work has focused on learning visual
representations using self-supervised methods [15 46 10]. Among them, joint-embedding methods have emerged as state-of-the-- tht-rr training models to produce similar embeddings for different views of the same image. These methods rely on architectural and loss regularizations to prevent representation collapse, such as enforcing cross-correlation constraints [4] or using momentum

---

N2N + DINO22
DINCV2 m/ NCT
DINOv2 w/ - DINO22
Lincer Probing Accuracy
Clean
Gaussian 50
Gaussian 100
Gaussian 255
80
10
SNR:
SNR:
SNR:
200
500
1000
200
50c
1000
200
503
1000
200
500
1000
Epochs
Epochs
Epochs
Epochs
Figure 2: Linear probing accuracies and noise examples on ImageNet- 100 dataset. The denoiser- preprocessed bascline N2N DINOv2 and our denoiser-free free methods DINOv2 w/ NC and NCT significantly improve over the DINOv2 baseline. Note that DINOv2 w/ NC and N2N + DINOv2 overlap at moderate noise levels, and DINOv2 w/ NCT is only evaluated at Gaussian o = 255.
encoders [6 7 47 [1. DINOv2 31. used in this work, is joint-embedding method that achieves state-of-the-ar performance on ImageNet classification and many evaluation benchmarks.
Self-Supervised Image Denoising learns to denoise noisy images without access to noisy-clean pairs. Noise2Noise 26 uses aligned noisy-noisy pairs derived from the same clean image to train
U-Net (35] model. However, obtaining such pairs is not always practical. Neighbor2Neighboo [20] as used in this paper, mitigates such issue by sampling two subimages from noisy image which serve as noisy-noisy pair [32]. It achieves competitive performance on range of denoising benchmarks [55] Another line of work uses blind-spot networks (BSN) g to learn denoise without clean-image supervision. BSN uses the same noisy image as its inputs to supervise its outputs. It either applies masks inputs 21
or masks in network structure 22 to prevent model collapse.
Self-Supervised Representation Learning for Noisy Data. There is very limited work on learning noise-robust features for image data. Most of the past works concern noisy time-series data, such as speech [25]
39 49 and EEG data 48]. 48 demonstrates applying time series SSL methods on
noisy datasets yields poor performance, and proposes denoiser-driven contrastive learning by using a conventional denoiser to create noised-conditione positive and negative pairs. This approach of leveraging denoiser to learn noise-robust representation shares similar motivations as our method.
3 Noise Robust Self-Supervieed Learning
In this section, we introduce our self-supervised method for robust representation learning on noisy image data. We start with a two-stage baseline, then moving towards fully self-supervised, denoiser- free noise robustness through curriculum learning and regularization.
3.1 Preliminary: Curriculum Learning Background and Denoiser-Preprocsse Baseline
Curriculum learning refers training scheme that trains a machine learning model from easier to harder data that imitates the learning order in human curricula. This often leads to faster convergence and better performance. The seminal work (5) formally defines curriculum as sequence of training distributions (QI...., Qt. Qr). Each distribution Q, is reweighting of the target training distribution P(z): Q+(z) x (())) Vz I (=)ddz
such that following conditions are satisfied: (1) the entropy of distributions increase over time, i.e., (QQ)) < H(Q++1). This implies that the model sees more diverse or complex examples as training progresses. rising entropy indicates reduced certainty and increased difficulty. (2) Unnormalized weighting for any example increases over time, i.e., (x(+) < W++ (z) Vz. (3) Final distribution equals the target, i.e., Qr(z) P(z). In practice, the first condition is most widely preserved, while the latter two are often relaxed to allow more flcxible curricula (38). In this work, we adopt this perspective and design training schedule to follow entropy- based curriculum from easy to hard examples.
N2N DINOv2: Joint Self-Supervised Denoising and Representation Learning. As a starting point, we consider denoiser- preprocessed baseline, an upper bound that combines self sepersueee denoiser with a joint-embedding SSL model. Given noisy dataset X, we first train denoiser fo on X and generate a denoised set Xdenoised fo (X). This is used to train self supervised learner go. such as DINOv2. While effective, this pipeline requires an explicit denoiser all downstream tasks. We use Neighbor2Neighhoor 220 as the denoiser for its robustness across noise levels, but emphasize that the pipeline denoiser-agnostic, a key benefit also shared by our later proposed methods. In practice, the denoiser should be selected based on the domain-specine noise characteristics.
3

---

SSL Denuising
SSL Represcnntionn Learing
Dowisticon
Noke
SSI. desnisa
aemoise
Rodket
DINOv2
DINDV2
Noicy
SSL desoller
EpiNow
DINOv
Perforn Red for the first k spocts, Sea switch to Blue.
Notey
Denoked
Epocts
Task Task 2 Task
Figure 3: Overview DINOV2 w/ NC which comprises SSL denoising, SSL representation learning and downstream tasks. A SSL denoiser is trained denoise the data (28). Then, DINOv2 is trained on the denoised data for early epochs followed by restarting training on noisy data. Lastly, after some fine-tuning, downstream tasks performed directly on noisy inputs with an added prediction head.
3.2 DINOv2 w/ NC (Noise Curriculum): Curriculum Learning for Self-Supervised Representation Learning
Ideally, instead of relying on separate denoiser, the representation model should develop robustness to noise on its own, enabling an end-to-end training setup in downstream tasks. Figure 3 outlines our curriculum learning approach achieve this. Following the definition in Section 3.1 design
a curriculum (Q1, Q2) that satisfies the key requirement H (Q:) < H (Q2). Here, Qi corresponds to the distribution over denoised images X deemiseed which lies in a more structured, lower-entropy subspace. Q2 corresponds to the distribution over the original noisy images X, which has higher entropy due to the injected stochastic corruption that expands the support of the data distribution.
Such curriculum is further motivated by the following toy example. We consider simple MLP with 1 hidden layer trained MNIST 2 using ReLU activation, and optimize the following self-supervised joint embedding loss Equation (T) that applies a soft identity constraint on empirical covariance matrix, which is canonical formulation that aligns embeddings of different augmented views while preventing representational collapse:
E B [mo(( - mo((2(2;))112 + S.o Ts
min
where Eg = - (2, 2) -E) mm)(r())) is the MLP and is the augmentation. Under Gaussian noise (a 0.4 255). we observed that: Training and linear probing on the clean train set (50 epochs each) yields 91.21 + 0.36% (I-sigma) accuracy on the clean test set. Training and linear probing on the noisy train set (50 epochs each) yields 64.55 + 3.47% on the noisy test set. Surprisingly. curriculum learning (training 30 epochs on clean, then 20 on noisy), followed by 50 epochs of linear probing on noisy, recovers performance to 83.05 + 0.45% on the noisy test set. This shows that the model can adapt the noisy data without forgetting what learned from the clean phase. The significant recovery in performance (see Appendix Figure 6). even with simple MLP and vector data, suggests that curriculum learning encourages the model to internalize noise-robust features. We now build on this intuition to scale our method to complex vision models like DINOv2:
Denoised Pretraining: We start with an initialized self-supervised model go (DINOv2) for rep- resentation learning. We train on the denoised set Xeenoised for k epochs to establish a stable initialization that captures the underlying structure of the data while reducing the impact of noise. Noise Curriculum Transition: Using the weights learned in the first epochs, we restart training on until convergence. Here 'restart' means re-initializing all training dynamics such as learning rate and weight decay scheduling. This enables the model to better adapt to the noisy dataset, learning noise-robust representations from the stable features learned on the denoised data. Denoiser-free Downstream Tasks and Inference: For downstream tasks involving noisy images, such as classification, the representations learned by go can be directly utilized with prediction head he for task-specific learning. Further fine-tuning can be performed directly on the noisy data without requiring denoising module, since the training strategy internalizes noise-robust features into the learned representations. This enables the entire model, both backbone go and head ha. to be optimized end-to-end on the downstream objective and deployed without any preprocessing at inference time as shown in Equation (2):
the original noisy

---

Finetuning: -= (s..)) [[hh) Inference: I == ho (g)
(2)
This highlights key advantage of the proposed DINOv2 w/ NC as it enables fully denoiser-free downstream fine-tuning and inference. contrast, the two-stage baseline N2N + DINOv2 requires a separate self-supervised denoiser trained and applied not only during pretraining but also downstream task fine-tuning and inference. This external dependency adds computational overhead and pipeline complexity, risks transferring the denoiser's bias to the SSL model, and deviates from our goal of learning roise-robust representations directly from corrupted data.
3.3 DINOv2 w/ NCT (Noise Curriculum Teacher): Anchored Curriculum Learning for Improved Self-Supervised Representation Learning
Under high or extreme noise levels, a good initialization from denoised training may not be sufficient, because strong noise may destabilize and disturb the prior learned representations during the noisy training. Thus, we propose to utilize the embeddings of the denoised images as anchors to regularize the training on noisy data. Let the teacher backbone of DINOv2 be T, and the student backbone be The original DINOv2 loss can be expressed as
Lainov Ldino&ibbo (T (T)r) ((T,<)))) + Lioton
(3)
Where Lkoleo is the Koleo Regularization, is the augmentation for teacher input, and Ta is the augmentation for student input. Ldinokibot is the combined DINO and iBOT loss that minimizes the cross-entropy of image-level and patch-level output scores between teacher {Pr img patch } and student {yime, ppasch }. Here the backbones T and S include 'heads' that output vector scores.
p
T(T(()) = {pt img patch3 S(Tx(x)) = {p-m iing, phatch3
(4)
Ldinokibot
Σ jmg logp!
1r patch log patch
(5)
Before restarting the noisy training, first extract the weights of the teacher backbone Tan that was trained on denoised data and freeze it. As shown in Figure the architecture of the model now contains three components: teacher T,, student S and frozen denoised teacher Tan- When restarting the training, given a noisy image x and its denoised version Idn = fa()), we apply exactly identical augmentations (i.e., same crops, blur, flip...) æ and Zdn, so their embeddings are aligned. In addition to the original DINOv2 loss, we encourage the output scores of the student to be close to that of the denoised teacher. This prevents the training from deviating significantly from its cleaner initialization. So the denoised denoised-regularized loss is given as:
L Ldiuov + A (Tau (Te ((T(()))
(6)
Where À is a parameter that controls the strength of the regularization. weights of I' and Tan are identical at the start of the training. Also, the technique presented is not restricted to only denoised settings. When clean images are available, such method can be used improve the overall noise-robustnes. of DINOv2 by simply replacing 'denoised' components with "clean" components.
is worth noting that the
Frozen Teacher
Denoised
Ldino&ibot
T
Student
EMA
Liioovz
It important emphasize that this regulariza- tion technique is specifically designed to comple- ment our curriculum training pipeline. For the regularization to be effective, the frozen teacher Tan and the trainable teacher T must have align- ment in their output embeddings at the beginning of the restart training. This is accomplished by assigning them identical weights. Using differ- ent frozen teacher, even if trained on the same denoised dataset, would not provide meaningful regularization due to misalignments, as shown Appendix Figure g Like NC, NCT is also denoiser- free at downstream fine-tuning and inference.
Noisy
Tt
Teacher
Figure 4: Structure of the denoised regulariza- tion loss. In addition to the original DINOy2 loss Ldinov2, a regularization loss term Ldinokibot is introduced by comparing the output scores be- tween the student and the frozen teacher. The frozen teacher takes in denoised inputs that un- dergo identical augmentations as the noisy inputs of the trainable teacher.
5

---

4 Experiments
4.1 Practical and Challenging Experimental Settings
Datasets. We conduct experiments both ImageNet-100 and ImageNet- 1k to evaluate the robustness of SSL methods under noise. ImageNet- 100 comprises 100 classes as defined in Mini- Mini-ImaeeNet [36.. The training set includes the first 500 images per class, totaling 50k training and 5k validation images. For large-scale evaluation, we experiment on ImageNet- [12] using the full training and validation
splits. For instance recognition evaluation, we use the Oxford and Paris dataset (34)
Noise Addition. We follow ImageNet-C [19] to introduce three types of noise: Gaussian, Shot
(Poisson), and Speckle noise, as these are among the most commonly encountered noise types in natural imaging. For the ImageNet- 100 experiments, we use Gaussian noise (a 50, 100, 255; SNR: 8.73 dB, 3.98 dB, 0.31 dB), Shot noise () = 10, 3, 1; SNR: 8.49 dB, 4.11 dB, 0.52 dB), and Speckle noise (a 102, 178.5, 255; SNR: 9.15 dB, 5.58 dB, 3.98 dB). For the ImageNet-1k l experiments, we tested Gaussian noise at 100 and = 255, with corresponding SNRs of 4.36 dB and 0.72 dB respectively. These SNRs are common values reported in real noisy medical imaging [4]. The example visualizations for each noise level are provided in Appendix (D2 The noises are added to raw images before preprocessing (e.g., resizing) to better reflect practical scenarios such as device or environmental noise. Consequently, while the initial noise level is fixed, the preprocessing steps introduce variations in actual noise levels that challenge the model's ability to adapt effectively. Notably, we generate the noise dataset once prior to training and do not apply noise on-the-fly fly during data loading, which closely simulate real-world scenarios where one starts with a fixed noisy dataset.
Implementation Details. In ImageNet- 100 experiments, we use ViT-S/16 as the base architecture for DINOv2, and unless otherwise mentioned, train it for 200 epochs with batch size of 40 using the W optimizer. The base learning rate is 7.9 x 10 (scaled by the square root of batch size). The denoiser is trained for 100 epochs with a batch size of using Adam. The linear probe uses a batch size of 128 and is trained for 12.5k steps. In ImageNet- Ik, we adopt ViT-B/16 as the backbone and train DINOv2 for 100 epochs with a batch size of 512 and drop path rate of 0.1. The learning rate is 2.8 x 10-3 The denoiser is trained on a 100k subset with batch size of 8 for 50 epochs (for Gaussian a= = 255) and 90 epochs (for Gaussian a = 100). The linear probe uses batch size 1024 for 25k steps. Each training is done on single RTX 4090 (-8h) or 4 GPUs (-65h). The strength parameters are determined via light parameter sweeps with the details provided in Appendix A.2
4.2 Linear Probing Reveals Superior Noise Adaptation
We begin by evaluating how different methods adapt to various levels and types of noise using a 200-epoch train- ing schedule on 1000 We train a linear classifier over a frozen DINOv2 backbone on the training set, and report classification performance over the validation set. We compare the classification performance of the DINOv2 backbone over four training strategies. (1) N2N + DINOv2: train the backbone on denoised images for
Clean N2N + DINOv2 DINOv2 w/ NCT DINOVv w/ NC - DINOv2
Gaussian 50
Gaussian 100
Gaussian 255
Accuracy
Accuracy
cccurayy
Accura
10
SNR: 8.73 dB
SNR 3.90
dB
SNR: 0.31
300
200
100
203
150
200
Epochs
Epachs
Epochs
Shot À 10
Shot 3
Shot À
Accurcyy
80
Accuracy
Accurads
60
80-
200 epochs and evaluate the model on denoised images (2) DINOv2 w/ NC: train the backbone on denoised im- ages for 140 epochs, then restart train- ing on noisy images for 60 epochs without regularization. Finally, eval- uate the model on noisy images. The rationale for such partitions of epochs is provided in Appendix B.3 (3) DI- NOv2 w/ NCT: same as the DINOv2 w/ NC but with the regularization loss applied in the 60 epochs of noisy training. We only evaluate the NCT
SNR: 8.49 dB
SNR: 4.11 dB
20
SNR: 0.52 dB
203
Epochs
Epochs
Speckle a 102.0
Speckle o - 178.5
Speckle a - 255.0
Jccuracy
30
80 Souuneg
30
SNR: 9.15 dB
SA/R: 5.58 m8
SNR: 3.98 dB
300
203
160
250
Epochs
Epochs
Epochs
Figure 5: Linear probing classification accuracies on DI- NOv2 backbone weights saved every 5 epochs during train- ing on ImageNet- 100 under various noise levels.

---

Table Performance comparison of different methods when scaling training duration and data size.
Noise Type
Method
200
Ep.
500 Ep.
1000 Ep.
Clean
DINOv2
74.1
80.3
81.4
N2N + DINOv2
71.8
76.9
78.2
Gaussian-50
Gaussian-100 Gaussian-255
DINOv2 w/ NC
72.8
76.9
78.1
Method
Clean
SNR: 8.73 dB
SNR: 4.36 dB
SNR: 0.72 dB
DINOv2
65.0
74,5
76.5
N2N DINOv2
73.1
57.2
Gaussian- 100
N2N + DINOv2
69.0
73.2
74.4
DINOV2 w/ NCT
72.1
55.8
DINOv2 w/ NC
68.1
72.7
73.6
SNR: 3.98 dB
DINOV2 w/ NC
70.9
53.5
DINOv2
55.4
68.4
71.6
DINO²2
79.0
70.7
$1.0
N2N + DINOv2
57.01
59.61
60.5
SimCLR
60.1
48.0
34,2
Gaussian-2S:
DINOV2 NCT
54.6
57.61
59.6"
SNR: 0.31
DINOV2 NC
47.9
54.7
59.0
(b) Linear probing classification accuracy on ImageNet- Ik under Gaussian noise using 100-epoch training schedule. DINOy2 w/ NCT and DINOv2 w/ NC show substantial improvements over the DINOv2 baseline, particularly at high noise levels (a = 255). SimCLR is included here as a reference baseline.
DINOv2
35.8
52.7
56.8
(a) Evolution of linear probing accuracies on ImageNet-100 100 under extended training durations. Rel- ative ranking among methods remain consistent across epochs. See visualization in Figure? 2
method the four most noisy scenarios (lowest SNRs), i.e., Gaussian noise (a=255, r=100), Shot noise (A=1), and Speckle noise (a=255). (4) DINOv2: train the backbone on noisy images for 200 epochs, and evaluate the model on noisy images. Additionally, we train and evaluate the model on original clean images, referred to as Clean. Figure 5 illustrates the classification accuracy curves.
As shown in Figure 5| our proposed method (green) significantly outperforms the backbone trained solely on noisy images (red) across all noise types and noise levels. At moderate noise levels, i.e., the first two columns, the accuracy of our method closely matches and even exceeds the backbone that is trained and evaluated on explicitly denoised data (shown in orange lines). Only at extreme noise levels, i.e., the last column, the gap between DINOv2 w/ NC and N2N + DINOv2 widens, primarily due to the significant loss of the original image signal. For instance, as shown in Appendix [D.2] with Gaussian noise at 255, many images become virtually unrecognizable the naked eye. Hence, regularization is introduced to guide the model in these extremely noisy scenarios. As illustrated in Figure 5 regularization DINOv2 w/ NCT improves the model's performance at all four scenarios. The improvement amplifies as the gap between N2N + DINOv2 and DINOv2 w/ NC widens, with a substantial improvement of 6.7% over NC achieved at Gaussian noise with o = 255. However, when N2N + DINOv2 is on par with DINOv2 w/ NC, the regularization provides diminishing benefit, as demonstrated with Gaussian noise at a 100. This expected since the guidance becomes redundant when its representation quality is comparable to or lower than that of the original model.
4.3 Scaling Improves Performance Across Datasets
Next, we scale both training and data size to evaluate performance in more resource-rich settings. Longer Training Duration. We further improve the performance of our methods by extending the training on ImageNet-100 to longer epochs. For 500-epoch configuration, we restart the training from the 200-epoch denoised checkpoint (i.e., N2N + DINOv2) as described in Section 4.2) for the 1000-epoch configuration, we restart the training from the 500-epoch denoised checkpoint. As shown in Table Ia and Figure 2 accuracies increase epoch increases and the relative ranking among methods remains consistent across training durations: N2N + DINOv2 achieves the highest accuracy, followed closely by DINOv2 w/ NC NCT, while the baseline DINOv2 remains the lowest. This demonstrates the scalability of the proposed methods. Interestingly, we observe that DINOv2 w/ NCT consistently converges to or outperforms its anchor's 's accuracy, as indicated by i and & in Table Tal demonstrating that the regularization objective successfully guides the model toward robust representations. Notably, the performance gap between DINOv2 and the other methods narrows as training progresses, suggesting that DINOv2 is sensitive to noise and benefits significantly from longer training. In contrast, our noise curriculum and regularization approach greatly accelerates convergence, achieving similar or better performance in roughly half the training time.
Larger Dataset. In our controlled ImageNet- 1k experiment, we constrain the DINOv2 backbone to undergo a total of 100 training epochs to enable fair comparison across methods. DINOv2 w/ NC: train the backbone on denoised images for 30 epochs followed by 70 epochs of training on noisy images. DINOv2 w/ NCT: While we find that restarting at 30 epochs yields the best performance for DINOv2 w/ NC, the denoised teacher at this early stage lacks sufficiently strong representations to serve as an effective regularizer. To address this, we continue training the denoised teacher to 100 epochs, then freeze it for regularizing the DINOv2 backbone for 70 epochs of noisy training. Since
7

---

Table 2:
Comparison of linear probing accura- cies on clean and noisy validation sets for models trained on noisy ImageNet-Il kk and ImageNet- 100 datasets. Surprisingly, our DINOv2 w/ NC or NCT method outperforms the two-stage N2N + DINOv2 clean validation sets, demonstrating more accurate and generalizable representations.
Table 3: Linear evaluation performance of ap- plying noise curriculum (NC) to other SSL models on ImageNet- 100. Performance is im- proved for all tested models, highlighting the broader applicability of our approach. The accuracy curve over epochs are visualized in Figure I2
in Appendix B.5
Dataset
Training Noise Method
Clean Noisy
SSL Model Architecture Method
Accuracy
N2N DINOv2
75.8
73.1
N2N + SimCLR
64.3
Gaussian- 100
DINOv2 w/ NCT
75.2
72.1
SimCLR
ResNet50
SimCLR w/ NC
61.1
SNR: 4.36 dB
DINOv2 w/ NC
74.3
70.9
SimCLR
59.0
ImageNet-1k (100 epochs)
DINOv2
74.1
70.7
N2N + MoCo v3
60.4
N2N DINOv2
64.1
37.2
MoCo v3
VIT-S
MoCo x3 w/ NC
55.3
Gaussian-255
DINOv2 w/ NCT
65.6
55.8
McCo v3
52.2
SNR: 0.72 dB
DINOV2 w/ NC
63.7
53.5
N2N + SimSiam
68.4
DINOv2
61.6
51.0
SimSlam
ResNet50
SimSiam w/ NC
65.7
Gaussian-50
N2N DINOv2
78.5
78.2
SimSiam
64.8
SNR: 8.73 dB
DINOv2 w/ NC
79.4
78.1
N2N + iBOT
61.9
DINOv2
76.5
iBOT
VIT-S
BOT w/ NC
62.7
Gaussian- 100
N2N DINOv2
76.0
74.4
BOT
56,9
ImageNet-
SNR: 3.98 dB
DINOv2 w/ NC
76.7
73.6
N2N + DINO
62.5
(1000 epochs)
DINOv2
74.6
71.6
DINO
VIT-S
DINO w/ NC
62.1
N2N + DINOV2
56.7
60.3
DINO
$7.9
Gaussian-255
DINOv2 w/ NCT
59.4
59.6
N2N DINOV2
69.0
SNR: 0.31 dB
DINOv2 w/ NC
57.9
59.0
DINOv2
VIT-S
DINOv2 NC
68.1
DINOv2
58.8
56.8
DINOv2
55.4
the frozen teacher and the DINOv2 backbone are derived from the same training run and pass through the same 30-epoch state, alignment is preserved. This allows us to meet the 100-epoch constraint while still leveraging stronger teacher. Table lb shows that DINOv2 w/ NCT improves over the DINOv2 baseline with substantial absolute gains of 1.4% and 4.8% at noise levels a = 100 and 255, respectively. DINOy2 w/ NC also achieves a strong improvement of 2.5% under the more challenging Ø = 255 setting. The relatively smaller gain of DINOv2 w/ NC at o = 100 is due to the fact that, in large-scale settings with moderate noise, longer training alone significantly improves DINOv2's robustness. Since DINOv2 w/ NC allocates a portion of the training budget to denoised pretraining, shortens the noisy training phase, slightly limiting its benefit. Overall, while scaling training on large datasets improves DINOv2's noise adaptation under moderate noise, denoised pretraining and regularization remain essential under high noise to maintain strong performance. As an additional baseline, we also evaluate DINOv2 trained exclusively on clean images and directly tested on noisy inputs. The results presented in Appendix C.T confirm that clean- peeat-peeta model suffers severe accuracy drop, which underscores the importance of our moise-aware training.
4.4 Probing on Clean Test Set Highlights Better Representation Quality
In previous sections, linear classification accuracies were measured on validation sets that had the same noise distribution as the training set. To further assess representation quality, we evaluate performance on the clean, original ImageNet validation set. Note that N2N + DINOv2 is evaluated without the denoiser because the clean images are already noise-free. Performance on this clean validation set serves as a strong indicator of how well a model captures representations of the "noise- free" real world. Surprisingly, as highlighted by bold numbers in Table DINOv2 w/ NC or NCT outperforms N2N + DINOv2 in most cases, strongly suggesting that our denoiser- free approach learns more robust and generalizable representations than the two-stage denoising pipeline. This can be attributed to the fact that explicit denoising in the two-stage approach inevitably loses some useful information, leading to a degradation in representation quality in the second stage. This finding has important real-world implications: in scenarios where large-scale noisy data is readily available for pretraining, but only a limited amount of clean data is available for specific downstream tasks, DINOv2 w/ NC and NCT offer more effective solutions for learning transferable representations.
4.5 Instance Recognition Validates Versatility
To further validate the versatility of our method, we extend the evaluation to another downstream task. Here, we examine the instance-level recognition task on noisy images using the embeddings output by the backbone. The approach is non-parametric as the images' embeddings are ranked based on their
8

---

Table Mean Average Precision (mAP) on instance-level recognition tasks using Oxford and Paris datasets. DINOv2 w/ NC consistently outperforms DINOv2 and closely matches N2N + DINOv2 across benchmarks, with notable gains extreme noise scenarios using regularization. This validates
the broader applicability of our approach to tasks beyond classification.
Noise
Method
Oxford Paris
Naise Method
Oxford Paris
Noise
Method
Oxford Paris
N2N DINOV2
21189
38.62
N2N + DINOv2
21.36
40.00
N2N + DINOv2
22.01
40.47
Gaussian
Shot
Speckle
DINOv2 w/ NC
21.30
40.94
DINOv2 w/ NC
21.94
42.20
DINOv2 w/ NC
22.40
41.28
o=50
X=10
d=102
DINOv2
19.15
40.04
DINOv2
21.13
39.73
DINOv2
20.33
39.04
Gaussian
N2N DINOV2
21.83
39.04
Shot
N2N + DINOV2
21.40
40.39
Speckle 1=178.5
N2N + DINOv2
21.31
39.31
DINOv2 w/ NC
20.13
40.14
DINOv2 w/ NC
22.11
39.12
DINOv2 w/ NC
21.21
38.74
g.100
A=3
DINOv2
16.39
32.54
DINOv2
35.65 18.59
DINOv2
18.63
37.06
N2N + DINOv2
15:67
33.68
N2N + DINOv2
18.96
37.22
N2N + DINOv2
20.11
39.19
Gaussian
DINOV2 w/ NCT
17.07
33.73
Shot DINOV2 w/ NCT
20.66
36.57
Speckle
DINOv2 w/ NCT
20.81
39.31
d=255
DINOv2 w/ NC
14.81
29.97
x=1
DINOv2 w/ NC
19.25
36.67
a=255
DINOv2 w/ NC
21.10
38.59
DINOv2
7.37
18.51
DINOV2
16.37
32.60
DINO+2
20.11 35.63
(x) Gaussian Noise
(b) Shot Noise
(c) Speckle Noise
cosine similarity with query image's embedding. The results are based on the medium difficulty evaluation setup in the Oxford and Paris datasets, where both easy and hard images are treated as positive images. We measure mean average precision (mAP) and report results Table 4 We see that the results closely align with Section 4.2 DINOv2 w/ NC greatly outperforms DINOv2 in all comparisons. The mAP of DINOv2 w/ NC is on par with N2N + DINOv2, even at extreme noise levels. For example, at Shot À = 1 and Speckle a 255, the mAp of DINOv2 w/ NC surpasses that of N2N + DINOv2 on the Oxford dataset. DINOv2 w/ NCT outperforms all other methods the majority of the time in high noise settings, again validating its effectiveness.
4.6 DINOv2 w/ NC is Applicable to Diverse SSL. Models
To test the applicability of DINOv2 w/ NC beyond DINOv2, we extend the DINOv2 w/ NC to various other SSL models on ImageNet- 100, including SimCLR [9], MoCo v3 [. SimSiam [[O]. iBOT 47, and DINO 77 These models encompass both contrastive and non-contrastive approaches, as well as ViT-based [3 and CNN-based architectures [88] All experiments are conducted with Gaussian (a=100) noise without denoised regularization. As shown in Table 3 DINOv2 w/ NC improves over the noisy baseline across all models. The improvement is modest or minor for the first three models, i.e., SimCLR, MoCo v3, and SimSiam. The variation in contrastivity and backbone architecture among these models suggests that neither factor is critical to the method's 's effectiveness. In contrast, iBOT and DINO (both are precursors to DINOv2) show substantial gains, similar to the improvements observed in DINOv2, where the SSL Model w/ NC closely matches N2N + SSL Model. This indicates that DINOv2 w/ NC has broad applicability. with more pronounced benefits for models sharing architectural and loss fusscfuntionn similarities with DINOv2. We also benchmark the performance of these models the instance recognition task in Table of Appendix [B.5] which shows exactly identical trends as the linear evaluation, further reaffirming the generality of our approach. Detailed training configurations are provided in the Appendix A.4
5 Limitations, Conclusion and Future Work
In this work, we proposed a fully self-supervised framework for learning noise-robust visual rep- resentations. Leveraging an SSL denoiser and the curriculum training strategy enables DINOv2 to improve significantly upon the noisy baseline. While our benchmark focuses on synthetic noise due to the scareity of large-scale labeled noisy datasets that allows for standard downstream evaluation, this choice enables controlled, reproducible assessment across noise types. One limitation is the assumption that self-supervised denoiser can denoise reasonably well; while it is a fair assumption as we show in Appendix B.6 that even very weak denoiser (1 training epoch) can still substantially improve accuracy, it remains an open dependency that could affect performance in more challenging settings. Another key direction for future research is to develop automated, adaptive strategies for curriculum schedule, as the current design requires tuning to determine when to shift from denoised to noisy samples during pretraining. More broadly. our methodology can be extended to other modalities like time series data, including noisy audio, EEG signals and financial data; we aim to establish a unified SSL approach for robust representation learning.

---

References
[1] Mahmoud Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning. In International Conference on Learning Representations (ICLR), 2023.
[2] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY,
USA, 2020. Curran Associates Inc. ISBN 9781713829557
[3] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari S. Morcos, Shashank Shekhar, Tom Gold- stein, Florian Bordes, Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learn- ing. ArXiv, abs/2304. 12210, 2023. URL https://api. semanticscholar owggCorrpsaaa..
258298825
[4] Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning, 2019.
[5] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48 2009.
[6] Florent Brondolo and Samuel Beaussant. Dinov2 rocks geological image analysis: Classification, segmentation, and interpretability, 07 2024.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 202 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021. doi: 10.109//CCV449922221.00999 100951.
[8] Shiyan Chen, Jiyuan Zhang, Zhaofei Yu, and Tiejun Huang. Exploring efficient asymmet- ric blind-spots for self-supervised denoising in real-world scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2814-2823, 2024,
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9640-9649. 2021.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEFE Conference Computer Vision and Patlern Recognition, pages 248-255. 2009. doi: 10. 10.1109/CVPR 200.. 0006000
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL http://rwwe..eeeeeoouuureeeoaao.
https:///operrevier net//forum?id--iicooda.
[14] European Space Agency (ESA).
Sentinel-1
satellite
data.
2014.
10

---

[15] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representations. 2018.
[16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond. Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Conference on Neural Information Processing Systems, 2020.
[17] Yue Guo, Shiqi Chen, Ronghui Zhan, Wei Wang, and Jun Zhang. Deformable feature fusion and accurate anchors prediction for lightweight sar ship detector based on dynamic hierarchical model pruning. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 1-11591115506., 2025. doi: 10.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[20] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighhor: Self- supervised denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14781-14790, June 2021.
[21] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2129-2137, 2019.
[22] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In Advances in Neural Information Processing Systems, 2019.
[23] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278 2324, 1998. URL https: //api semanticscholar. wwtpeschhhlarr.o//rrroeee orgg/orrpusD:115422255
[24] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-bsn: Self-superviser denoising for real-world images via asymmetric pd and blind-spot network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[25] Seong-Gyun Leem, Daniel Fulford, Jukka-Pekka Onnela, David Gard, and Carlos Busso. Adapting a self-supervised speech representation for noisy speech emotion recognition by using contrastive teacher-student learning. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023. doi: 10.1 10.1109/CAASP449337..00 202242235...0231099973333
[26] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference Machine Learning. 2018.
[27] Johannes Leuschner, Maximilian Schmidt, Daniel Otero Baguer, and Peter Maass. Lodopab-ct, a benchmark dataset for low-dose computed tomography reconstruction. Scientific Data 8(1): 109, 2021.
[28] Cynthia McCollough, Adam Bartley, Rickey Carter, Baiyu Chen, Tammy Drees, Phillip Ed- wards, David Holmes III, Alice Huang, Farhana Khan, Shuai Leng, Kyle McMillan, Gregory Michalak, Kristina Nunez, Lifeng Yu, and Joel Fletcher. Low-dose et for the detection and classification of metastatic liver lesions: Results of the 2016 low dose et grand challenge. Medical Physics, 44:c339- 10 2017. doi: 10. 20002/pp122453

---

[29] Omkar Nabar and Gautam Shroff. Conservative predictions on noisy financial data. In Proceed- ings of the Fourth ACM International Conference on Al in Finance, ICAIF '23, page 427-435, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702402.. doi: 10. 10.1145/3604237. 3666889.. URL httpp::tp//// http:::://doooo///////// org/10. 0141/3604423333 . 3626859
[30] Reiko Nakajima, R. Mandelbaum, U. Seljak, J. Cohn, Reinabelle Reyes, and Richard Cool. Photometric redshift requirements for lens galaxies in galaxy-gglaxy lensing analyses. Monthly Notices of the Royal Astronomical Society, 420, 07 2011. doi: 10.1 10.1111//j.166. 1/j. 2926.2011.222222 x.
[31] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[32] Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. peporruptee-t-rreorruppeee Unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2043-2052, June 2021.
[33] Deepak Pathak, Philipp Krahenbuhl. Jeff Donahue, Trevor Darrell, and Alexei A. Efros, Context encoders: Feature learning by inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[34] F. Radenović, A. Iscen, G. Tolias, Y. Avrithis, and O. Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In CVPR, 2018
[35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MIC -MCCAII 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[36] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 3637-3645, Red Hook, NY, USA,
2016. Curran Associates Inc ISBN 97815108388
[37] Huy V. Vo, Vasil Khalidov, Timothée Darcet, Théo Moutakanni, Nikita Smetanin, Marc Szafraniec, Hugo Touvron, Camille Couprie, Maxime Oquab, Armand Joulin, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Automatic data curation for self-supervised learning: A clustering- approach. Transactions on Machine Learning Research (TMLR), 2024.
[38] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, (4(9):455544577 2021.
[39] Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, and Yu Wu. Wav2vec-switch: Contrastive learning from original-noisy speech pairs for robust speech recognition. In /EEE International Conference on Acoustics, Speech, and Signal Processing, 2022.
[40] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. BlindZunblind: Self-supervised image denoising with visible blind spots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2027-2036, June 2022.
[41] Marijke Welvaert and Yves Rosseel. On the definition of signal-to- ratio and contrast-to- noise ratio for fmri data. PloS one, 8:,77089, 11 2013. doi: 10. 1/j.r7//joralllpoone 900.0700......0007779...........
[42] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In European Conference on Computer Vision (ECCV), 2020.
[43] Huidong Xie, Weijie Gan, Bo Zhou, Ming-Kai Chen, Michal Kulon, Annemarie Boustani, Xiongchao Chen, Qiong Liu, Xueqi Guo, Menghua Xia, Liang Guo, Hongyu An, Ulugbek Kamilov, Hanzhong Wang, Biao Li, Axel Rominger, Kuangyu Shi, Ge Wang, and Chi Liu. Dose-aware diffusion model for 3d low-dose pet denoising: multi-institutional validation with reader study and real low-dose data. Journal of Nuclear Medicine, 65, 2024.
12

---

[44] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, 2021.
[45] Dan Zhang, Fangfang Zhou, Felix Albu, Yuanzhou Wei, Xiao Yang, Yuan Gu, and Qiang Li. Unleashing the power of self-super image denoising: comprehensive review, 2024. URL
https://arxia org/ abs/23000 00247
[46] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[47]
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1xXv:2111.078933 2021.
[48] Shuang Zhou, Daochen Zha, Xiao Shen, Xiao Huang, Rui Zhang, and Fu-Lai Chung, Denoising- aware contrastive learning for noisy time series. In International Joint Conferences on Artificial Intelligence, 2024.
[49] Qiu-Shi Zhu, Jie Zhang, Zi- Zi-Qiang Zhang, Ming-Hui Wu, Xin Fang, and Li- Li-Rong Dai. A noise- robust self-supervised pre-training model based speech representation learning for automatic speech recognition. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3174-3178, IEEE, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the paper's main contribution: fully self-supervised framework that enables noise-robust representation learning without requiring denoiser during inference and downstream fine-tuning. The claim well sup- ported by the methodology and experimental results in Section 3 and Section 4 Additional contributions, including the sensitivity analysis and the impact of scaling training duration and data size, are demonstrated in Section 4.2 and Section 4.3
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations explicitly and thoroughly in Section 5 including the dependency on denoiser quality and the need for tuning the curriculum schedule.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present formal theoretical results or theorems. The contributions empirical and algorithmic, with claims requiring formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofss provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides comprehensive details of the models, datasets, training schedules, hyperparameters, and evaluation metrics used for all experiments, Implementation details, such as architectures (e.g., ViT-S/16 and ViT-B/16), training durations, batch sizes, optimizers, and noise levels, are thoroughly described in Section 4 The full source code to produce the experimental results is also included in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The full source code required reproduce all experiments is included in the supplementary material, along with documentation and instructions. The datasets used (ImageNet-Ik, 1k, Oxford, and Paris) are publicly available, and the paper describes preprocessing and noise generation protocols in detail (Section 4.1
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
oolpoolicyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips.
sonddeSubiissioPPooe httpc//gguide///eeuubbiiiisiee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings including data splits, data preprocessing, type of optimizer, learning rate, batch size and number of epochs are detailed Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

Justification: We do provide enror bars for the toy experiment in Section 3.2 However, error bars are not reported for large experiments because it would be too computationally expensive. However, the core findings are robust and extremely unlikely to be due to chance, as they are consistently observed across wide range of noise types, severity levels, model architectures, and dataset scales. In all tested scenarios, our methods demonstrate clear and repeatable improvements over the baseline.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed information about GPU types, number of GPUs and training time are provided Section 4.1 Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eeeeepecccecppe//////e
Answer: [Yes]
Justification: We have carefully reviewed the Code of Ethics and confirm that our research complies with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: All external assets used in this work, such as datasets and pretrained models, are properly cited in the main paper. For instance, DINOv2 is cited in Section (Introduction) and ImageNet- lk is cited Section 4.1 (Experimental Settings). We have respected the licenses and terms of use associated with these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new training strategy and regularization method for noise- robust SSL. Documentation and code for reproducing the experiments are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

Outline of the Appendix:
Appendix
A
Technical Details about Experiments
Appendix A.1
Details of Toy Experiment in Section 3.2
Appendix
A.2
Regularization Strength Details and Ablation
Appendix
A.3
Training Details for Main DINOv2 Experiments
Appendix
A.4
Training Details for Other SSL Models in Section 4.6
Appendix
A.5
Noise Addition Formula
Appendix A.6
Signal-to-Noise Ratio Details
Appendix
B
Ablation Studies
Appendix B.1
Ablation Unveils Synergy in Curriculum Learning
Appendix B.2
Critical Roles Aligmment and Initialization
Appendix B.3
Restart at Different Epochs
Appendix B
Enduring Effectiveness in Mixed Noisy-Clean Data
Appendix
B
3.5
Further Results for Other SSL Models
Appendix
B.6
Robustness to Denoiser Quality
Appendix Baseline Comparisons
-Appendix
Performance of -lean-Pretrained Model on Noisy Data
Appendix
Baseline Performance on Mixed Noisy dois--Deooised Data
Appendix D
Visualizations
Appendix
D.
Training Loss Visualization
Appendix
D.2
Noisy Images Visualization
Appendix
D.3
PCA Visualization
A Technical Details about Experiments
A.1 Details of Toy Experiment Section
In both pretraining and linear probing evaluation, we use Adam optimizer, batch size 256, and constant learning rate of 1.0 x 10-4 The augmentation includes random resized crop and random affine, shown in Listing I The strength for soft identity constraint is set to 0.5.
The results over runs are visualized in Figure We observe that adopting noise curriculum not only significantly improves accuracy, but also substantially stabilizes performance, as evidenced by the narrower green box compared to the wider orange box. This highlights the dual benefits of curriculum learning in enhancing both robustness and consistency.
trans transforms.o Compose (
transf RandommRes scale- (0.8, 1.0)),
transforme RandomAffine(
degrees=15, translate=(O. 1, 0.1),
scale= (0,95, 1.05), shear=10
)
1)
Listing 1: Data augmentation used in pretraining for the simple MLP with Midden-layer in the toy experiment.
A.2 Regularization Strength Details and Ablation
For results in Figure 5 we select optimal regularization strength based on light sweeps visualized in Figure 7 Specifically. y, is set to 1.1, 5.0, 2.0 and 4.0 for Gaussian 255, Shot 1, Speckle 255 and Gaussian 100 respectively. Notably, Figure shows that increasing the strength
21

---

95
90
85
MNIST Test Accuracy (%6
80
70
60
55
Clean
Noisy
Curriculum
Method
Figure 6: Box plots of toy experiments accuracies over 17 runs. "Clean" for MLP trained and tested on clean. "Noisy" for MLP trained and tested on noisy. "Curriculum" " for MLP w/ NC tested on noisy. Using noise curriculum not only significantly improves accuracy, but also stablizes performance, as shown by the narrower green box compared to the wider orange box.
results sharp increase in accuracy followed by fluctuatons around a higher plateau, demonstrating the effectiveness of NCT.
In scaling epoch experiment for ImageNet-100 (Table la
set to 1.1, 0.5 and 0.2 for 200, 500 and 1000 epochs respectively. We find longer training duration requires smaller regularization strength. This agrees with our main finding that scaling training can partially mitigate the performance degradation, thus requires less guidance from denoised teacher. In ImageNet- experiment (Table Tb). is set to 2.0 and 1.6 for Gaussian ø = 100 and Gaussian o = 255.
For evaluation clean set (Table 2 the strengths are set to the same value their noisy counterparts detailed the preceding paragraph.
In instance recognition experiments (Table 1. À = 0.2 for all settings we find it generally yields
better results. For mixed noise-clean data evaluation shown later in Appendix B.4 À is 1.1.
A.3 Training Details for Main DINOv2 Experiments
Training Details for Section 4.2 For curriculum-baat training, although the model is trained on
denoised images for only 140 epochs, the training dynamics the first stage, such as learning rate scheduling, are configured based on the total training duration of 200 epochs. In the second stage of training on noisy images, the training dynamics are configured for total duration of 60 epochs to ensure smooth convergence.
Training Details for Section In ImageNet- Ik experiments (Table lb), the training configuration for DINOv2 and N2N + DINOv2 are set as default, consistent with its original implementation. In the noisy-stage training of DINOy2 w/ NC and DINOv2 w/ NCT, reduce the linear learning rate warm-up epochs from 10 to 5, and the teacher temperature warm-up epochs from 30 to 15 to accommodate the reduced training duration for noisy adaptation under the constraint of 100 epochs. Intuitively, less warm-up is needed because denoised pre-training provides a good initialization.
Training Details for SimCLR Baseline in Table Ib. Consistent with DINOY2, set batch size to 512 and train for 100 epochs during pre-training. We use the source Py Tyrrch implementation from Andrew wtw.oo/ssoontnlleee..r.oo pytorhh Since SimCLR typically requires epochs during linear evaluation compared to DINOv2, we train the linear head for 90 epochs (112.59k steps) with a batch size of 1024 to ensure fair comparison.
Details Linear Probing. Consistent with the original DINOv2 paper, when performing linear probing, light grid search is conducted over hyperparameters, i.c., learning rate, output layer, and
22

---

Gaussian 255
Shot À
54.0
53.0
Accuracy
01.0
$2.0
09.5
50.0
19.0
SNR: 0.31 dB
59.5
SNR: 0:52 dB
Regularization Strength
Regularization Strength
Speckle = 255
Gaussian a 100
68.6
69.5
09.2
08.0
66.
67.6
68.0
SNR: 3.90 dB
07.4
SNR: 3.90 dB
Regularization Strength
Regularization Strength
Regularization strength ablation to determine optimal strength for DINOv2 w/ NCT in Figure Most plots show a rapid increase in linear probing accuracy followed by fluctuations around a higher plateau strength increases. Due computational constraints, the sweep was not performed with uniform resolution across all settings and may vary across noise types.
Figure 7:
whether concatenating average-pooled tokens. The highest accuracy value is reported which is common practice.
A.4 Training Details for Other SSL Models in Section 4.6
The linear evaluations of all models train the linear head for 50 epochs.
SimCLR. During training, we use batch size
and LARS optimizer. After square root scaling, the base learning rate set to 0.075 x v100 0.75. During linear evaluation, we use batch size of 200 and SGD optimizer. After square root scaling, the base learning rate is set to 0.35.
1.6 220004048
MoCo v3. During training, use batch size of 128 and AdamW optimizer. After square root scaling, the base learning rate is set to 10-4 x 1/28/ 1024 1.414 10-4 During linear evaluation, we use batch size of 200 and SGD optimizer. After linear scaling, the base learning rate to 0.2 (200/256) 0.15625.
SimSiam. During training, we use a batch size of 100 and SGD optimizer. After square root scaling, the base learning rate is set to 0.15 x /100/25 250 = 0.09875. During linear evaluation, we use batch size of 200 and LARS optimizer. We found the learning too small for effectively training the linear head. After some experimentation, we set the base learning rate 2.34375 after the linear scaling 3.0 (200/256).
provided in SimSiam's 's codebase
iBOT. During training, we use batch size of 80 and Adam W optimizer. After square root scaling, base learning rate set to 5 10 / /80/256 2. 795 x 10-4 During linear evaluation, we batch size of 128. We also adopt the sweeping hyperparameter implementation in iBOT to obtain the best linear evaluation value. This involves sweeping over a range of learning rates, and a set of candidate optimizers including LARS and SGD.
DINO. During training, we use batch size of 128 and AdamW optimizer. After square root scaling, the base learning rate set to 5 10-4 //28/22 = 3.54 x 10-4 During linear evaluation, we use batch size of 128 and SGD optimizer. After square root scaling, the base learning rate is set to
0.001 x
/ 128/256 256 7.1 1
23

---

A.5 Noise Addition Formula
As stated in Section we follow ImageNet-C 19 to introduce Gaussian, Shot, and Speckle noises to the images. Let original image be z, and normalized image be = x/255..
Gaussian noise:
Traisy 255 clip (a + (0.
(7)
where c 255
and a E {50, 100, 255}.
Shot (Poisson) noise:
Poisson(E: X)
Unoisy 255 clip
00,1
(8)
where À E [10,3,
Speckle noise:
T noisy 255 clip ( ( (N(00
(9)
where = 255 and E {102, 178.5, 255}.
Note that the clipping applied the [0, range in all noise types, which further increases the difficulty of denoising at high noise levels. A significant portion of the noise signal may get clipped and hence irreversibly distorted or lost.
A.6 Signal-to-Noise Ratio Details
All SNR values given are calculated from the raw noisy images without preprocessing, e.g., resizing, crop. We report the average SNR between the training and the validation set since they have significant difference. In the instance recognition, we report the average between the Oxford and Paris datasets. Detailed per-set SNR values are provided in Table 5 and Table
Table 5: SNR values of training and validation sets for ImageNet-100 100 and ImageNet- I datasets.
Table 6: SNR values for Oxford and Paris
datasets.
Dataset
Noise Type
(dB) SNR (dB)
Noise Type
Oxford (dB) Paris SNR (dB)
Gaussian
8.84
8.63
Gaussian
8.86
9.04
Gaussian 100
3.87
4,27
Gaussian 255
0.43
0.19
Gaussian 255
0.39
0.55
Shot À 10
8.55
8.44
Shot 10
8.66
8.79
ImageNet-100 Shot À 3
4.16
4.05
Shot À
4.24
4.39
Shot À 1
0,58
0.47
Shot À =
0.65
0.78
Speckle a = 102
9.17
9.14
Speckle = 255
4.09
4.17
Speckle a = 178.5
5.59
5.56
Speckle = 178.5
5.69
5.75
Speckle a = 255
4.00
3.96
Speckle
102
9.28
9.31
Gaussian 100
4,41
4.31
ImageNet-lk
Gaussian 255
0.78
0.67
24

---

B Ablation Studies
B.1 Ablation Unveils Synergy in Curriculum Learning
We conduct ablation studies on ImageNet- 100 to analyze the contribution the two key components
of the curriculum learning method, specifically in the context of linear probing classification on images with Gaussian noise (a 100).
Only Denoised Pretraining. We train DINOv2 on denoised images for 140 epochs, then continues to train it on noisy images for 60 epochs without resetting training dynamics.
Only Noisy Training with Restart. We train DINOv2 on noisy images for 140 epochs, then restart training on the same noisy set for 60 epochs.
Figure & iillustrates that denoised pretraining contributes nearly twice the accuracy improvement over the noisy baseline compared to restarting training. Notably, their effects are not merely additive, as our combined method outperforms the sum of their individual gains, highlighting a synergistic interaction between the two techniques. The synergy arises because, while denoised pretraining provides robust initial weights, the diminished learning rate in later stages limits adaptation to noisy images. Restarting training resets the learning rate, allowing the model to fully adapt to noise and leverage the learned representations effectively.
68
DINOVZ w/ NC
66
Only Denoised Pretraining
64
Accuracy
62
Only Noisy Training with Restart
60
58
DINOv2
54
120
140
160
180
200
Epochs
Figure Comparison of curriculum learning ablations showing the contributions of denoised pretraining and restarting noisy training linear evaluation accuracy. "Only Denoised Pretraining" (purple) provides nearly double the improvement of "Only Noisy Training with Restart" (brown) over the Noisy baseline (red).
B.2 Critical Roles of Alignment and Initialization
To evaluate the design choices of our regularization loss, we conduct ablation studies on ImageNet- 100 under extreme Gaussian noise (a 255), where the regularization loss has the most significant impact. These studies provide insights into the importance of alignment and initialization in our approach.
Unaligned Frozen Teacher. Instead of extracting the weights of the current teacher backbone at the end of the denoised training phase, we retrain separate backbone on the same denoised dataset to extract teacher weights. This alternative frozen teacher then used during the noisy training phase. Unlike the original method, this introduces misaligned embeddings between the frozen teacher, the trainable teacher, and the student. At regularization strength 0.2, as shown by the brown line in Figure 9 this setup results in a marginal performance improvement of just 1. % over the green baseline. significantly lower than the aligned case depicted by the purple line. At regularization strength 1.1, shown in Figure I0 the strong unaligned teacher instead destabilizes training, dragging accuracy downward as learning proceeds, ultimately underperforming even the DINOv2 w/ NC. These results highlight the seamless integration between the regularization loss and the curriculum training pipeline, where the first stage's outputs are essential for effectively regularizing the second stage.
Randomly Initialized Backbone. further investigate the role of denoised pretraining, we replaced the trainable teacher and student backbones with randomly initialized weights during the noisy training phase while keeping the frozen teacher unchanged. At regularization strength 0.2, this configuration yields extremely poor performance, with accuracy dropping below 20%, shown
25

---

Strength 2 0.2
DINOV2 NCT
50
Unaliged Frozen Teacher
40
Accuracy
DINOv2 w/ NC
DINOv2
20
Randomly Initialized Backbone
10 120
140
160
180
200
Epochs
Figure Comparison of regularization ablations at strength 0.2 showing the contribution of alignment and initialization to linear evaluation accuracy. Aligned regularization (purple) significantly outperforms unaligned regularization (brown) and random initialization (grey). This illustrates both the alignment and the denoised pretraining contributes substantially to the effectiveness of the regularization.
Strength A 1.1
DINOv2 w/ NCT
Unaliged Frozen Teacher
Accuracy
DINOv2 w/ NC
-00
DINOv2
Randomly Initialized Backbone (Diverged)
10
120
140
160
180
200
Epochs
Figure 10: Comparison of regularization ablations at strength 1.1 showing the contribution of alignment and initialization to linear evaluation accuracy. Aligned regularization (purple) signif- icantly outperforms unaligned regularization (brown) and random initialization (grey). Random initialization diverges after 30 epochs of training. Strong unaligned regularization drags the accuracy down training progresses.
by the gray line in Figure 9 Notably, at regularization strength 1.1, the training diverges after 30 epochs as the loss explodes, as shown by the discontinuous grey line in Figure I0 These results highlight the necessity of denoised pretraining, even in the presence of regularization term, as it provides a crucial foundation for the training process. Without it, the training fails leverage the frozen teacher's guidance.
B.3 Restart at Different Epochs
We investigated the impact of restarting training on noisy data at different stages: (1) training on denoised data for 140 epochs, followed by 60 epochs noisy data (a common setting in this paper); (2) 130 epochs on denoised data, then 70 epochs on noisy data; and (3) 120 epochs on denoised data, then 80 epochs on noisy data. As shown in Figure II the performances across these variations are similar, with no approach consistently outperforming the others. Thus, we decide to use 140 epochs as the restart point.
B.4 Enduring Effectiveness in Mixed Noisy-Clean Data
In real-world scenarios, datasets often contain images of varying quality, posing challenges for SSL methods. In this section, we investigate how different noise-clean distributions ImageNet- 100 affect the SSL model's performance and the effecttiveness of our method under these distributions. We replace x% (i.e., 0%, 2%, 10%) of noisy data with clean images. So in our noise curriculum, mix of (100- x²)% % denoised and x% clean images in the first stage, then
the model is trained on a mix of (100 - %%% noisy and 4% clean images in the second stage. The validation set is also adjusted to follow the same distribution the training set. From Table we see that generally
26

---

Clean
N2N + DINOV2
DINOV2
120 Epochs Restart
130 Epochs Restart
140 Epochs Restart
Gaussian 50
Gaussian 100
Gaussian 255
Accuracy
0.6
Accuracy
Accuracy
044
0.4
SNR: 8.73 dB
SNR: 3.98 d8
0.2
SNR: 0.31 dB
100
200
100
200
0
100
200
Epochs
Epochs
Epochs
Shot À 10
Shot À =
Shot
Accuracy
0.6
Accuracy
Accuracy
0.6
0.4
0.4
044
5NR: 8.49 dB
SNR: 4.11 dB
SNR: 0.52 dB
100
200
200
200
Epochs
Epochs
Epochs
Speckle 0 102.0
Speckle a 178.5
Speckle a = 255.0
Accuracy
0.6
Accuracy
0.6
Securacy
0.5
0.4
0.4
dB
100
Epochs
Epochs
Epochs
Figure 11: Linear Probing Classification accuracy for different restarting epochs. We observe that no approach consistently outperforms the others between restarts 120, 130 and 140 epochs.
the linear evaluation accuracy decreases across all methods when more clean data is introduced but noisy data remains the majority, which can be attributed to reduced overall representation alignment between noisy, denoised and actual clean data. On the other hand, through column- wise comparisons. DINOv2 w/ NC consistently outperforms DINOv2 and closely matches N2N DINOv2; regularized method DINOv2 w/ NCT brings substantial improvement over unregularized method. It is clear that curriculum learning framework and denoised regularization are still effective despite the varying training distributions.
Table 8: Mean Average Precision (mAP) on instance-level recognition tasks for other SSL models. The pattern consistent with Table 3 Noise curriculum (NC) improves performance for all models.
Table 7: Mixed Poise-Clean Data Evaluation: Impact of introducing varying percentages of clean data (0%, 2%, 10%) on linear evaluation accuracy. We observe that increasing the pro- portion of clean data generally reduces accuracy across all methods due to reduced representa- tion alignment. However, our DINOv2 w/ NC and NCT are still effective despite the varying training distributions.
SSL Model
Method
Oxford M
Paris M
N2N SimCLR
21.43
35.32
SimCLR
SimCLR w/ NC
19.08
33.39
SimCLR
18.04
31.65
N2N MoCo v3
15.34
29.52
MoCo v3
MoCo v3 w/ NC
13.79
30.05
MoCo
12.95
27.72
Type
Clean
Clean
N2N SimSiam
24.03
40.10
+ DINOv2
71.8
72.0
71.1
SimSiam
SimSiam
21.12
39.77
Gaussian-50
DINOv2 NC
72.8
72.3
71.6
SimSium
20.64
38.15
DINOv2
65.0
61.2
61.9
N2N iBOT
12,26
N2N DINOv2
69.0
67.8
67.7
IBOT
14.60)
Gaussian- 100
DINOv2 NC
68.1
66.6
67.0
IBOT
11.05
24.05
DINOv2
55.4
54.8
53.1
N2N DINO
15.04
29.16
N2N DINOv2
57.0
55.2
$3.5
DINO
DINO w/ NC
16.01
31.33
DINOv2 w/ NCT
54.6
53.8
51.7
DINO
12.73
27.70
Gaussian-255
DINOv2 w/ NC
47.9
47.3
47.3
N2N DINOv2
21.83
39.04
DINOv2 35.8
36.9
35.8
DINOy2
DINOv2 w/ NC
20.13
40.14
DINOv2
16.39
32.54
27

---

B.5 Further Results for Other SSL Models
Figure [2] shows the aceuracy over epochs for SSL models tested in Section 4.6 Table shows the performance of the SSL models on the instance recognition task using the Oxford and Paris dataset. Both results illustrate that the noise curriculum improves the performance of all SSL models on noisy data, demonstrating the wide applicability of the method.
N2N + SSL Model
SSL Model w/ NC
SSL Model
SimCLR
MoCo V3
SimSiam
(Contrastive ResNet50)
(Contrastive VIT
(Non Contrastive ResNet50)
60
Accuracy
Accuracy
50
Accuracy
60
50
40
40
40
30
100
200
100
200
100
200
Epochs
Epochs
Epochs
iBOT
DINO
DINOv2
(Non-Contrastive VIT-S)
(Non-Contrastive VIT-S)
(Non-Contrastive VIT-S)
60
60
Accuracy
Accuracy
Accuracy
60
40
40
40
100
200
100 Epochs
200
100
200
Epochs
Epochs
Figure 12: Linear evaluation performance over epochs of applying DINOv2 w/ NC to other SSL models on ImageNet-100. Noise curriculum (green) improves performance for all tested models, with substantial gains observed in iBOT and DINO variants, showing similar trend as DINOv2.
B.6 Robustness to Denoiser Quality
To investigate the level of tolerance DINOv2 w/ NC the denoiser quality, we conduct ablation studies on how the number of denoiser training epochs impacts the SSL model's final performance. We use the number of training epochs as quantifiable proxy for denoiser effectiveness. The evaluation follows our default setting: 200 epochs of DINOv2 ViT-S training with Gaussian- 100 noisy data. The results are shown in Table 9 We observe that even training the N2N denoiser for just epoch leads to substantial improvement of DINOv2 w/ NC over the DINOv2 baseline (i.e., 10.7 increase), while being only 2 percent less than that of well-trained denoiser (100 epochs). Although the 1-epoch denoiser still produces many undesirable artifacts like very weak denoiser, the curriculum still yields substantial improvements in downstream performance. This highlights the robustness of our curriculum, which has considerable tolerance to the denoiser's quality. A denoiser has to be unreasonably bad (e.g., training diverged or collapsed) to provide no return in performance.
Table 9: Performance of DINOv2 w/ NC on ImageNet- 100 with the N2N denoiser trained for varying epochs. Notably, even weak denoiser trained for only I epoch substantially improves DINOv2 w/ NC over DINOv2.
Method DINOv2
Denoiser Training Epochs
Accuracy
55.4
DINOv2 w/ NC
epoch
66.1
DINOV2 w/ NC
5 epochs
67.4
DINOv2 w/ NC
100 epochs
68.1
28

---

Table 10: Performance of clean- sean--petrained DINOv2 models under different noise conditions. Linear probes are tine-tune either on clean or noisy data while keeping the backbone frozen. The clean- prefrained DINOv2 suffers severe drop in performance when applied on the noisy test set.
Probe Tuned
Probe Tuned
Training Dataset
Architecture
Eval Dataset
Eval Noise
on Clean
on Noisy
Gaussian- 100
63.2
70.3
LVD-142M
ViT-B/14
ImageNet- lk
Gaussian-255
8.9
29.0
Clean
84.5
/
Gaussian-
49.2
61.0
LVD-142M
ViT-S/14
ImageNet-II
Gaussian-255
4.9
23.1
Clean
81.1
Gaussian-100
40.7
58.1
ImageNet- 1k (100 epochs)
ViT-B/16
ImageNet- 1k
Gaussian-233
2.2
20.5
Clean
79.0
/
Gaussian-3
60.0
74.3
ImageNet-100 100 (1000 epochs)
ImageNet- 100
Gaussian- 100
22.3
61.3
VIT-S/16
Gaussian- 255
3.1
34.5
Clean
81.4
/
Table 11: Linear probing accuracy on noisy ImageNet- 100 set when training on mixed noisytdenoised data. Mixing denoised images results in 7.4 point drop compared to training on noisy data alone.
Training Setting
Accuracy
100% Gaussian-100
55.4
50% Gaussian- (00+ + 50% Denoised
48.0
C Baseline Comparisons
C.1 Performance of Clean-Pretrained Model on Noisy Data
We evaluate the noise-robustness of DINOv2 model trained exclusively on clean images, order to reinforce the necessity of noisy training. Two strategies are employed to cover complementary baselines. (1) With the backbone frozen, we fine-tune a linear probe on noisy data drawn from the same distribution as the test set, and then evaluate the full model noisy data. (2) With the backbone again frozen, we fine-tune a linear probe on clean data and subsequently evaluate the model on noisy data. The first strategy simulates common downstream scenario where frozen feature extractor adapted to new domain via task-speeific heads, while the second serves as a more standard adversarial benchmark that tests the model's 's robustness noise without ever exposing it to noise during training. addition to our own trained models, we also benchmark the officially released DINOv2 weights, which were pretrained the LVD- 1--M dataset (142 million images). The results in Table 10 show substantial performance drops (15-70 points) when applying clean-pretrained DINOv2 to noisy test sets. The degradation is particularly severe Baussian-255 noise, where the accuracy of ViT-B/14 drops from 84.5 to 29.0 (a 55.5 -pontt decrease), even when the probe is tuned on Gaussian-255 noisy data.
Notably, despite the smaller model size and significantly reduced training data, our DINOv2 w/ NCT (Table (b) surpasses the official ViT-B/14 weights trained on I.VD-142M at both Gaussian- 100 (72.1 vs. 70.3) and Bausssinn255 (55,8 vs. 29.0). These results emphasize that standard DINOy2 is not robust input noise, underscoring the necessity of incorporating noisy images during pretraining to achieve strong downstream performance noisy settings.
C.2 Baseline Performance on Mixed Noisy-denoised Data
Another intuitive baseline is to investigate whether mixing denoised samples with noisy ones during training can improve representation learning by increasing diversity and expressiveness of inputs. Table II shows the linear probing accuracies of DINOv2 VIT-S the noisy ImageNet- 100 test set,
29

---

when trained for 200 epochs on 100% noisy data (from Table I(a)) versus on 50% noisy + 50% denoised data. We observe a significant drop 7.4 points in accuracy when noisy and denoised data are randomly mixed during training. As result, mixing clean (denoised) and noisy images during training leads to degraded performance, primarily due to reduced representation alignment between noisy and denoised samples, which hinders the model's ability to learn consistent features. This shows naive mixing is suboptimal, and reinforces the necessity using a staged curriculum learning approach that separates noisy and denoised images.
30

---

D Visualizations
D.1
Training Loss Visualization
18
DINOv2 Loss
16
DINOv2
Loss
14
L055
10
50000
100000
150000
200000
250000
Iterations
Figure 13: Comparison of training loss between DINOv2 and DINOv2 w/ NC. DINOv2 w/ NC can descend to lower loss due to its better initialization from the denoised pretraining.
Figure 13 shows the comparison between training loss of DINOv2 and DINOv2 w/ NC on the ImageNet- 100 noisy data with Gaussian noise (a 100). DINOv2 w/ NC loss (green) descends to lower value than DINOv2 (red). The two curves are similar in shape which reflects the restarting training technique in our noise curriculum learning.
D.2 Noisy Images Visualization
Gaussian Noise 50
Gaussian Noise o Denoised
Gaussian Noise a 100
Gaussian Noise = 100 Denoised
Gaussian Noise 255
Gaussian Noise 255 Denoised
31

---

Shot Noise À = 10
Shot Noise 10 Denoised
Shot Noise À 3
Shot Noise À = 3 Denoised
Shot Noise À
Shot Noise À Denoised
Speckle Noise 102
Speckle Noise 102 Denoised
Speckle Noise 178.5
Speckle Noise
178.5 Denoised
Speckle Noise a 255
Speckle Noise 255 Denoised
D.3 PCA Visualization
To visualize the dense features, we apply PCA to the feature space and map the first three principal components the RGB channels. We use the ViT-S 200-epoch Gaussian-255 checkpoints from Table Ia for visualization. As shown in the DINOv2 visualizations (middle of each triplet) fail to clearly separate salient objects from the background, and the resulting color patterns lack semantic consistency across images, suggesting weaker representation quality. In contrast, the DINOv2 w/ NCT visualizations (right each triplet) distinctly separate salient objects, and objects of the same class are consistently represented with similar color patterns across different images, indicating semantically aligned features. These results demonstrate that NCT yields substantial improvement in feature quality over the DINOv2 baseline.
32

---

Noisy Input
DINOv2
DINOv2 NCT
Noisy Input
DINOv2
DINOv2 NCT
Figure 23: PCA visualizations: visualize dense features of the models by performing PCA on the feature space and mapping the top three principal components to RGB. In each triplet, the left image is the noisy input with Gaussian noise 255. The middle and right images are produced by DINOv2 and DINOv2 w/ NCT (VIT-S, 200 epochs) respectively.
33


Reviews:
[{"review_id": "o0PW5LW9Tp", "reviewer": "Reviewer_BtzQ", "summary": "This paper presents a novel, fully self-supervised framework to train noise-robust representation models without needing a denoiser during inference or fine-tuning. The central problem addressed is that state-of-the-art Self-Supervised Learning (SSL) methods, like DINOv2, perform poorly when pretrained on noisy data, a common issue in real-world applications such as medical imaging and astrophysics. The proposed solution involves a curriculum learning strategy where an SSL model is first trained on a denoised version of the dataset (created using an auxiliary SSL denoiser) and then continues training on the original noisy data. This \"denoised-to-noisy\" curriculum encourages the model to internalize noise-robust features, allowing the denoiser to be discarded after pretraining. Experiments on ImageNet under various synthetic noise conditions show that the proposed methods significantly improve performance over standard SSL training on noisy data and often match or exceed a pipeline that explicitly uses a denoiser.", "questions": "1. The performance of your framework depends on the initial SSL denoiser. How does the performance of DINOv2 w/ NC degrade if a less effective denoiser is used? Is there a performance threshold for the denoiser below which the curriculum provides diminishing or no returns?\n\n2. The restart epoch is a key hyperparameter. Have you explored any methods to automate the selection of this epoch? For instance, could the transition be triggered dynamically based on the convergence of the model on the denoised data?\n\n3. Do you have plans to evaluate your framework on such real-world datasets, and how do you anticipate its performance will translate from i.i.d. synthetic noise to more complex, structured real-world noise?\n\n4. The analysis on clean validation sets is very insightful. You suggest that DINOv2 w/ NC/NCT outperforms the N2N + DINOv2 baseline because explicit denoising leads to information loss.  Could an alternative or complementary explanation be that the two-stage curriculum acts as a powerful regularizer, forcing the model to learn more general features by being exposed to both clean (denoised) and noisy distributions?\n\n4. For the DINOv2 w/ NCT results on ImageNet-1k, the denoised teacher used for regularization is trained for a full 100 epochs, while the student model it guides only sees the denoised data for 30 epochs before switching to noisy training.  This seems to create a significant gap between the teacher and the student's initial state in the second training phase. Could you elaborate on why this design choice was more effective than using a teacher from the 30-epoch checkpoint, and how embedding alignment is preserved?", "limitations": "Yes, the authors have adequately addressed the major limitations of their work in Section 5. A constructive discussion of limitations is critical, and the authors handle this well.\n\nThe paper correctly identifies and discusses two key limitations:\nDependency on Denoiser Quality: The authors acknowledge that the framework assumes a reasonably effective self-supervised denoiser can be trained on the noisy data. They note that this is a fair assumption in many cases but remains an \"open dependency.\" \nManual Curriculum Scheduling: The paper points out that the current design requires manual tuning to determine the optimal point to switch from denoised to noisy pretraining. \n\nThe authors also implicitly acknowledge the use of synthetic noise as a necessary choice for controlled, reproducible assessment.  To further strengthen this section, I would suggest also adding a brief discussion on the increased upfront computational cost of the pretraining pipeline (training a denoiser and processing the dataset) as a trade-off for the gains in inference efficiency and robustness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The work addresses the critical and under-explored challenge of applying SSL to noisy, uncurated datasets, which is a frequent scenario in many scientific and industrial domains. \n2. The proposed noise curriculum (NC) is an intuitive and simple yet powerful strategy. It effectively builds noise robustness directly into the representation model without requiring complex architectural changes or a persistent denoiser module. \n3. The authors conduct extensive experiments across multiple datasets (ImageNet-100, ImageNet-1k) , noise types (Gaussian, Shot, Speckle), and severity levels. The method's effectiveness is demonstrated on different downstream tasks (classification, instance recognition) and its generalizability is tested on a wide range of SSL models (SimCLR, MoCo v3, iBOT, etc.). \n4. A key advantage is that the denoiser is only used during the pretraining phase and can be discarded for downstream tasks. This simplifies deployment and reduces computational overhead and latency during inference. \n\n\nWeaknesses:\n1. To my undestanding, the framework's success hinges on the initial step of training a self-supervised denoiser that can produce a reasonably clean version of the dataset.  The performance of the entire pipeline is therefore dependent on the effectiveness of the chosen denoiser, which might be a challenge for unusual or extremely high levels of noise.\n2. The timing of the switch from denoised to noisy training is a crucial hyperparameter that currently requires manual tuning.  While the paper shows the method is robust within a certain range, an automated or adaptive strategy for this transition would enhance the method's practicality.\n3. The proposed method introduces additional computational steps to the pretraining pipeline: training an SSL denoiser and then using it to process the entire dataset. While it saves computation at inference, this increases the upfront resource requirement for pretraining.\n4. The experiments are benchmarked using synthetic noise types.  While this allows for controlled and reproducible evaluation, real-world noise can be more complex. For example, in medical images, the CT images in sinogam can be viewed as poisson noise, however in the image domain, it is difficult to define what the noise is. The translation of these impressive results to real-world noisy datasets is not yet fully demonstrated.", "was_revised": true, "final_justification": "The additional experimental evidence and theoretical insights the author provided will strengthen the revised manuscript significantly. I maintain my positive assessment of this work.", "rebuttal": "We sincerely thank Reviewer BtzQ for the thoughtful and highly positive review. Below, we would like to address each of your questions in detail:\n\n## Question 1: How does the performance of DINOv2 w/ NC degrade if a less effective denoiser is used?\n\nThanks for raising this practical question. To investigate it, we conducted additional studies on how the number of denoiser training epochs impacts the final SSL model’s performance. We use the number of training epochs as a quantifiable proxy for denoiser effectiveness. The evaluation follows our default setting: 200 epochs of DINOv2 ViT-S training with Gaussian-100 noisy data. The results are shown in the table below:\n\n**Performance of DINOv2 w/ NC on ImageNet-100 with the N2N denoiser trained for varying epochs**\n\n| DINOv2 Baseline | 1 epoch | 5 epochs | 100 epochs (from table 1 (a)) |\n|-|-|-|-|\n| 55.4 | 66.1 | 67.4 | 68.1 |\n\nWe observe that even training the N2N denoiser for just 1 epoch leads to a substantial improvement of DINOv2 w/ NC over the DINOv2 baseline (i.e., 10.7 increase), while being only 2 percent less than that of a well-trained denoiser (100 epochs). Although the 1-epoch denoiser still produces many undesirable artifacts like a very weak denoiser, the curriculum still yields substantial improvements in downstream performance. This highlights the robustness of our curriculum, which has considerable tolerance to the denoiser’s quality. A denoiser has to be unreasonably bad (e.g., training diverged or collapsed) to provide no return in performance. We will include this result as an additional ablation study in the Appendix B of our revised paper. Thank you again for the insightful suggestion; we believe this investigation further strengthens our contribution.\n&nbsp;\\\n&nbsp;\n## Question 2: Any method to automate the selection of the restart epoch?\n\nThank you for raising this important point. We agree that the restart epoch is a key hyperparameter. In our experiments, we fix the total training budget (e.g., 200 epochs), which makes automated selection of the restart point non-trivial: restarting too early leaves insufficient time to benefit from clean supervision, while restarting too late limits the model’s capacity to adapt to noisy data. The noise level, model size and learning rate schedule can also impact the convergence speed in both stages. In practice, we heuristically select the restart point by beginning from the midpoint and adjusting based on performance trends observed in preliminary runs.\n\nThat said, if the training budget were unconstrained, we believe a dynamic transition based on model convergence, such as monitoring linear probe accuracy on the denoised data, would be a highly promising heuristic. We view this as a valuable direction for future work and will mention it as a potential improvement in Section 5.\n&nbsp;\\\n&nbsp;\n## Question 3: Any plans to evaluate on real-world datasets? And how do you anticipate its performance will translate from synthetic to real noise?\n\nWe thank the reviewer for this important question. Indeed, as we acknowledged in Section 5, our current benchmark is limited to synthetic noise due to the scarcity of large-scale, labeled noisy datasets that support standard downstream evaluation. However, we fully agree that demonstrating robustness on real-world noise is critical for practical impact.\n\nYes, we do have plans to evaluate on real-world datasets. As a promising next step, we are actively exploring the curation of a large benchmark that contains real-world noisy images with accurate labels. Our goal is to include a diverse collection of noise types across domains such as biomedical imaging, astronomy, and remote sensing, where noise is often complex and structured. We believe this would provide a more rigorous testbed for evaluating our methods.\n\nFurthermore, we are confident that the performance gains observed on synthetic noise will translate to real-world settings. Firstly, Our method has been validated on a wide spectrum of synthetic noise types (Gaussian, Shot, Speckle), noise levels (SNR from 9.2 dB to 0.3 dB), and multiple architectures (e.g., SimCLR, MoCo v3, DINOv2, iBOT). The consistent improvements across these settings indicate that the curriculum-based robustness is not tightly coupled to any particular noise assumption. Secondly, as shown in our response to reviewer 5arj comment 2, our method continues to perform well under mixed noise (i.e., Poisson-Gaussian), which is a common real image degradation. This supports the hypothesis that robustness induced by our denoised-to-noisy training strategy can transfer to more complex noise distributions.\n&nbsp;\\\n&nbsp;\n## Question 4: Could the two-stage curriculum acts as a powerful regularizer, forcing the model to learn general features?\n\nThank you for the insightful observation.  We fully agree that, in addition to mitigating information loss from explicit denoising, the two-stage denoised-to-noisy curriculum can indeed act as a powerful regularizer. By exposing the model sequentially to denoised (low-entropy) and noisy (high-entropy) data distributions, the training process encourages the emergence of more general and robust features, as the model must reconcile representations across varying input conditions. This interpretation aligns well with our motivation for the curriculum design and supports our findings in Section 4.4, where DINOv2 w/ NC and NCT generalize better than N2N + DINOv2 on clean validation sets. We will incorporate this valuable perspective into Section 4.4 of our revised version to provide a more complete discussion.\n&nbsp;\\\n&nbsp;\n## Question 5: Why is the denoised teacher at 100 epochs a better regularizer than that at 30 epochs? And How is the embedding alignment preserved?\n\nThanks for pointing out this confusion. Here are our more detailed explanations. On lines 295-296, we explained that “the denoised teacher at this early stage lacks sufficiently strong representations to serve as an effective regularizer.”, this is because we empirically observed that the linear probing accuracy of the denoised teacher at 30 epochs is only 53.5 when tested on denoised dataset. This means that 53.5 would be an approximate upper bound of DINOv2 w/ NCT if using the 30-epoch denoised teacher, because DINOv2 w/ NCT consistently converges to or very slightly outperforms its anchor denoised teacher’s accuracy, as shown in table 1 (a) and line 285. To push for stronger performance and raise the ceiling, we thus employ denoised teacher at 100 epochs (i.e. accuracy = 57.2) to regularize the noisy training. And indeed, our experimental results show substantial improvement of NCT over both NC and DINOv2 baseline, and we expect its accuracy to reach 57.2 given long enough training epochs.\n\nAs for how the embedding alignment is preserved, on line 298, we explained that “the frozen teacher and the DINOv2 backbone are derived from the same training run and pass through the same 30-epoch state”. This means at 30 epoch, the frozen teacher and the trainable teacher share identical weights. Recent SSL literature [1][2][3] shows that self-supervised models evolve their representations in a coarse-to-fine manner: broad semantic clusters emerge early, while later training improves fine-grained separability. Consequently, although we continue to train the denoised teacher to full 100 epochs, its embeddings are generally still aligned with the backbone at 30 epoch state, as the coarse semantic structure (e.g., principal axes in embedding space) is preserved despite the continual training. Our empirical result also agrees with the theory as NCT solidly surpasses NC, demonstrating the 100-epoch denoised teacher can act as an effective regularizer during noisy training. We will make sure to add more discussion in section 4.3 in our revised manuscript.\n\n[1] Reverse Engineering Self-Supervised Learning, Ben-Shaul et al., NeurIPS 2023\n\n[2] On the Stepwise Nature of Self-Supervised Learning, Simon et al., ICML 2023\n\n[3] Understanding Learning Dynamics of Neural Representations via Feature Visualization at Scale, Kuntala et al., NeurIPS Workshop 2023\n&nbsp;\\\n&nbsp;\n## Comments on Computational Cost\n\nThank you for your suggestion. We agree that our pipeline introduces a slight increase in upfront computational cost due to training a denoiser and preprocessing the dataset. However, this additional cost is minimal compared to the full pretraining and downstream application stages. The denoising step only requires a single round of inference, and training the denoiser is highly efficient, as demonstrated by our ablation studies in Question 1. Moreover, in industry settings, inference cost increasingly becomes the primary bottleneck. A core contribution of our work is that it does not increase inference cost, which makes it particularly well-aligned with real-world deployment needs. Overall, we believe that the small upfront cost is a worthwhile trade-off for the substantial gains in inference efficiency and robustness. We will include this important discussion in Section 5 of our revised paper. Thank you again for pointing it out.\n&nbsp;\\\n&nbsp;\n\nWe sincerely thank you for your feedback. Your suggestions have significantly helped us strengthen our submission, and we will remain available during the remaining discussion period."}, {"review_id": "Yga61TwiFq", "reviewer": "Reviewer_1LLb", "summary": "This paper proposes a self-supervised training approach aimed at enhancing DINOv2's performance in high-noise scenarios using curriculum learning. The authors provide thorough analysis and comparisons, demonstrating that the two-stage (Clean + noise) self-supervised training yields superior performance compared to single-stage training (using solely noisy data). The primary experimental setting involves training on synthetic noisy images and evaluating on synthetic noisy images. A notable advantage is that this method eliminates the need for a denoiser at test time, while the final results remain robust on both clean and noisy data.", "questions": "1.  **Critical point:** Performance of DINOv2 trained on *clean images* evaluated on the *noisy test set*? The paper primarily reports DINOv2(clean) performance on clean images. However, the performance drop of DINOv2(clean) on the noisy test set is crucial for establishing this work's significance.  \n    **Suggestion:** Supplement Tables 2 and 3 with these results.\n\n2.  The inclusion of *denoised images* inherently enhances training set expressiveness.  \n    **Please show results** when training on:\n    -   Solely noisy images vs.\n    -   Mixed noisy + denoised images\n\n3.  Significant performance gap exists between the author-trained DINOv2-s (79.4% ImageNet accuracy) and the official DINOv2-small (81.1%). I understand computational constraints may limit training.  \n    **Key question:** Can the proposed method effectively *fine-tune* a **well-pretrained official DINOv2 model**? Enhancing the robustness of widely-used official weights against noisy data would be highly impactful.\n\n4.  **Urgently needed:** Evaluation on **real-world noisy datasets** (e.g., Hendrycks & Dietterich, \"Benchmarking Neural Network Robustness to Common Corruptions and Perturbations\", arXiv:1903.12261, 2019).\n\n5.  Can **PCA visualizations** on noisy images be shown for different models? (Similar to the style used in the original DINOv2 paper).\n\n6. **Releasing code would be beneficial to the community.**", "limitations": "The authors have honestly acknowledged one limitation I previously noted: the lack of testing on **real-world noisy datasets**. However, one **critical limitation remains unaddressed**: The paper does not analyze whether the proposed method can effectively **fine-tune well-pretrained models** (e.g., off-the-shelf DINOv2 checkpoints) rather than training exclusively from scratch.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "1. Novel Setting:​​ SSL model training tailored for noisy images\n\n​​2. Extensive Experiments:​​ Evaluated across classification and instance-level recognition tasks, applied to multiple SSL methods beyond DINOv2\n\n​​3. Well-Written Paper:​​ Clear presentation and good organization", "was_revised": true, "final_justification": "I am satisfied with the authors' response. Their revisions and clarifications have thoroughly resolved my questions.", "rebuttal": "We sincerely thank Reviewer 1LLb for the positive feedback, and for raising a series of very insightful questions. Below, we would like to give detailed responses to each of your suggestions and questions.\n\n## Question 1: Performance of DINOv2 trained on clean images evaluated on the noisy test set?\n\nThanks for raising the necessity of this baseline and we fully agree that it is crucial for establishing this work’s significance. To address this, we conduct further evaluation to supplement Table 2 with the results for both ImageNet-1k and ImageNet-100, as shown below. We observe that significant drops in accuracy occur (>20 points for Gaussian-100, >40 points for Gaussian-255) when evaluating the clean-pretrained DINOv2 on the noisy test set. This demonstrates SSL models trained exclusively with clean data cannot generalize to noisy images in downstream tasks, thus it is important to research training methods that produce noise-robust models, which is the main goal of our paper. We will update Table 2 and plan to include the additional results for Table 3 in our revised paper. \n\n| Dataset | Training Noise | Method | Clean | Noisy |\n|-|-|-|-|-|\n| ImageNet-1k (100 epochs)  | Gaussian-100 | DINOv2 Clean | 79.0 | 58.1 |\n| | Gaussian-255 | DINOv2 Clean | 79.0 | 20.5 |\n| ImageNet-100 (1000 epochs)| Gaussian-50 | DINOv2 Clean | 81.4 | 74.3 |\n| | Gaussian-100 | DINOv2 Clean | 81.4 | 61.3 |\n| | Gaussian-255 | DINOv2 Clean | 81.4  | 34.5  |\n\n&nbsp;\\\n&nbsp;\n## Question 2: Show results when training on mixed noisy + denoised images\n\nWe understand the reviewer’s valid suspicion that incorporating denoised images could enhance the training set’s expressiveness. In practice, we found mixing clean (denoised) and noisy images during training often leads to degraded performance. This is primarily due to reduced representation alignment between noisy and denoised samples, which hinders the model’s ability to learn consistent features. The table below shows the linear probing accuracies of DINOv2 ViT-S on the noisy ImageNet-100 test set, when trained for 200 epochs on 100% noisy data (from Table 1(a)) versus on 50% noisy + 50% denoised data. We observe a significant drop of 7.4 points in accuracy when noisy and denoised data are randomly mixed during training.\n\n| Training Setting | Accuracy |\n|-|-|\n| 100% Gaussian-100 | 55.4 |\n| 50% Gaussian-100 + 50% denoised | 48.0 |\n\nIn Appendix B.4, we comprehensively tested the performance of DINOv2 when trained and evaluated on mixed noisy + clean dataset, as seen in Table 7, introducing a small amount of clean images (e.g., 2%, 10%) during training destabilizes the process and slightly degrades performance. The above results shows naive mixing is suboptimal, and reinforces the necessity of using a staged curriculum learning approach that separates noisy and clean (denoised) images.\n&nbsp;\\\n&nbsp;\n## Question 3: Can the method fine-tune a well-pretrained official DINOv2 model?\n\nWe sincerely appreciate your understanding of the computational constraints. DINOv2-small in the original DINOv2 paper is based on the **closed-source** LVD-142M dataset, which contains 142 million images. In contrast, our model is trained on ImageNet-1k with only 1 million images, which helps explain the performance gap. Additionally, the official DINOv2 models use a patch size of 14, resulting in more parameters and higher input resolution compared to our model, which uses a patch size of 16.\n\nYes, the proposed method, in particular, NCT **can** effectively fine-tune a well-pretrained DINOv2 model against noisy data. While our method is primarily designed for pretraining from scratch, we demonstrate that our denoised-regularized loss can significantly improve upon the vanilla DINOv2 fine-tuning baseline. We use the official ViT-B/14 weights, and fine-tune them on ImageNet-1k with Gaussian-255 noise for 4 epochs due to the limited rebuttal time window. Two finetuning strategies are employed: 1. Finetune directly using the original DINOv2 loss, 2. Finetune using our NCT loss where a copy of the pretrained weights serves as the denoised teacher. The linear probing accuracies on both noisy and clean test sets are shown in the table below:\n\n| Test set | DINOv2 ViT-B (official) | Finetune w/ DINOv2 loss (baseline) | Finetune w/ NCT loss |\n|-|-|-|-|\n| Gaussian-255 | 29.1 | 39.2 | 47.2 |\n| Clean | 84.5 | 48.0 | 58.2 |\n\n\nWe observe that NCT loss achieves significantly higher accuracy in both noisy target set (+8.0) and clean set (+10.2) compared to original loss. This shows the regularization not only helps convergence on the noisy set, but also reduces distributional shift from the clean set. We acknowledge that tuning on very noisy data will inevitably reduce clean performance due to the significant distributional gap, but researching how to improve noise robustness of pretrained models without sacrificing the clean accuracy remains a promising direction for future work.\n\nWe would like to emphasize that due to rebuttal time constraints, the above results were produced out-of-box without any cross-validation or hyperparameter tuning. We believe that extending the number of training epochs would further improve performance. We would also like to highlight that fine-tuning the official DINOv2 weights is **non-trivial**, as they only release weights for the backbone, **but the iBOT and DINO heads are not released, which are crucial for fine-tuning**. Thus, we first train the heads on clean data for 75k steps with the backbone frozen, before fine-tuning on noisy data. These results likely represent a lower bound, and we believe that access to the official iBOT and DINO heads, which are trained on large-scale datasets, would substantially improve fine-tuning performance.\n&nbsp;\\\n&nbsp;\n## Question 4: Evaluation on real-world noisy datasets\n\nThanks for the valuable suggestion. The source “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations” (i.e., ImageNet-C) also utilizes **synthetic** noise injection processes. In our work, we actually employ the code base of ImageNet-C to create noisy datasets.\n\nAs acknowledged in Lines 354–356 of Section 5 (Limitations), a key challenge is the scarcity of large-scale real-world noisy datasets with reliable labels, which hinders standardized evaluation. As a promising next step, we are actively exploring the curation of such a benchmark to facilitate more realistic and comprehensive assessment. We fully agree that such a benchmark allows a fairer and more practical assessment of our method.\n&nbsp;\\\n&nbsp;\n## Question 5: PCA Visualizations\n\nThanks for the great suggestion. We agree that PCA visualizations provide an intuitive and vivid representation of a model’s learned features. However, due to the current NeurIPS rebuttal policy, which prohibits any links, we cannot post images here. Instead, we will describe the PCA visualizations verbally.\n\nWe strictly follow the style in the original DINOv2 paper to generate PCA visualizations. Under Gaussian-255 noise, we observe that the DINOv2 baseline generates very noisy visualizations, as it struggles to separate salient objects (i.e., birds) from the background. In contrast, NC and NCT generate visualizations that clearly separate the salient objects with sharp boundaries. Notably, objects of the same class are consistently represented using similar color patterns across different images, indicating semantically aligned features. This helps explain why NC and NCT substantially outperform the DINOv2 baseline in downstream tasks. Moreover, from the visualizations, we can infer that NC and NCT will outperform DINOv2 not only in classification tasks, but also in dense tasks like segmentation and depth estimation. We will ensure to include these visualizations in the Appendix of our revised paper.\n&nbsp;\\\n&nbsp;\n## Question 6: Releasing code\n\nWe have included all source code in the zipped supplementary materials, along with detailed instructions and links to download trained weights, which we will ensure are released to the community. We fully agree that open-sourcing the code and weights are beneficial to the scientific advancement of the community.\n&nbsp;\\\n&nbsp;\n\nWe truly appreciate the feedback and believe the changes have made our submission much stronger. We hope we've covered all your questions, and we will remain available for conversation until the discussion period ends."}, {"review_id": "Fv2CxZPt33", "reviewer": "Reviewer_5arj", "summary": "The paper offers a  method to have  off-the-shelf self-fupervised models like DINOv2 to robustly learn representations  on noisy images in a self-supervised way.  A denoiser is trained and then it generates noisy-denoised image pairs to build a curriculum for post-training DINO.  The latter parts of the curriculum have a teacher-based regularizer to force convergence. The denoiser is jettisoned after post-training. \n\nThe idea  is not that different from the use of the decoder half of a diffusion model, or denoising-quantification as in syed & mirza.\nThe idea is built with visible increments, like showing DINO's weakness in  data scarce setting,  meidum improvements on data addition, and finally a new  paradigm for gradually building noise tolerance. Experiments are reflective.", "questions": "How important will the temperature of the curriculum be?", "limitations": "The noise added is ...added.  Multiplicative or mixed source often plagues data.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "The strengths include having not to denoise during inference, and the tailorize use of a foundation model.\n\nThe paper does not use several fms, making the claim above a little weak. The  noise is random, and may vary across many runs, something that the experiment at larger scales (data size) avoids, raising a question over scaling.", "was_revised": true, "final_justification": "I was convinced of the usefulness of the idea for real world data, but had given a lower score  because of some impediments to the method really working well in the wild. The authors have clarified some, and admitted some, which  puts a better definition on the employment of the method.", "rebuttal": "We sincerely thank Reviewer 5arj for the positive review and helpful suggestions. Your comments have helped us better clarify important methodological details and highlight the robustness of our approach. Below, we would like to address each of your questions and concerns:\n\n## Question 1: How important will the temperature of the curriculum be?\n\nThank you for the thoughtful question. Our curriculum does not explicitly involve a temperature parameter. Instead, we adopt a discrete two-phase schedule, transitioning from denoised (easier) to noisy (harder) data. We experimented with gradually increasing the proportion of noisy samples during training but found that this neither accelerated convergence nor improved final performance.\n\nCurrently, the softmax temperatures used in the DINOv2 joint embedding loss are set to the default schedule for both training stages. The teacher temperature follows a warm-up phase and then remains constant. The student temperature is held constant. We are aware that, in the knowledge distillation literature [1], temperature can be dynamically learned to maximize the distillation loss, thereby increasing learning difficulty and forming an effective curriculum. While in our paper the learning difficulty is primarily controlled by the presence of noise, we think exploring the incorporation of dynamic temperature into our research is also highly worthwhile. We appreciate your insight that this is an interesting direction for future exploration.\n\n[1] Curriculum temperature for knowledge distillation Li et al., AAAI 2023\n&nbsp;\\\n&nbsp;\n## Comment 1 on the weakness: The noise is random, and may vary across many runs, something that the experiment at larger scales (data size) avoids, raising a question over scaling.\n\nThank you for raising this concern, and we apologize for the lack of clarity. We would like to clarify that the noise does not vary across runs in any of our experiments. We generate the noisy dataset once prior to training and do not apply noise on-the-fly during data loading. This setup is intended to closely simulate real-world scenarios where one starts with a fixed noisy dataset, aligning with realistic settings. We will make sure to explicitly clarify this detail in Section 4.1 (Noise Addition) of our revised paper.\n&nbsp;\\\n&nbsp;\n## Comment 2 on the limitation: The noise added is …added. Multiplicative or mixed source often plagues data.\n\nWe fully agree with your valid concern that multiplicative or mixed noises often plagues data. We would like to clarify that many noises tested in this paper, like shot noise and speckle noise, are not additive. In particular, speckle noise is a multiplicative noise, as shown in its formulation in Appendix A.5 equation (9). \n\nTo address the concern regarding mixed noise sources, we evaluated our method on Poisson-Gaussian noise, which is one of the most common mixed noises in imaging. The Poisson-Gaussian noise is formulated as $x = \\frac{\\operatorname{Poisson}(\\tilde{x} \\cdot \\lambda)}{\\lambda} + \\mathcal{N}(0, \\left(\\frac{\\sigma}{255}\\right)^2)$, where $\\tilde{x}$ is the clean image. We set $\\lambda = 3$ and $\\sigma = 100$, using the default 100 epochs to train the N2N denoiser, and 200 epochs to train DINOv2 ViT-S on ImageNet-100. The linear probing accuracies are shown in the table below:\n\n**Comparison of linear probing accuracies on Poisson-Gaussian mixed noise with 200 epochs fixed training budget**\n\n| Method | Accuracy |\n|-|-|\n| DINOv2 | 53.8 |\n| DINOv2 w/ NC | 63.2 |\n| DINOv2 w/ NCT | 65.1 |\n| N2N + DINOv2 | 65.8 |\n\nThe results are fully consistent with Figure 1 and Table 5 in the paper, where DINOv2 w/ NC significantly outperforms the DINOv2 baseline, and DINOv2 w/ NCT closely matches N2N + DINOv2. This demonstrates our methods are effective under mixed noise conditions and generalize to more complex scenarios. We will ensure to include these results in our revised paper to highlight the robustness of the methods under mixed noises.\n&nbsp;\\\n&nbsp;\n\nWe appreciate your thoughtful feedback and hope we have addressed all of your questions and concerns. We remain engaged and willing to provide further clarification during the ongoing discussion."}]

Author rebuttal:

Meta-review: This paper presents a self-supervised method for representation learning that doesn't need a denoiser. This addresses a common issue of poor performance when training with noisy data. The three reviewers of the paper were clearly in favor of accepting it. The authors also provided much additional information and experiments in the rebuttal that the reviewers found helpful in confirming their initial responses. The authors may wish to incorporate these post-discussion results and clarifications in the revision to improve the paper.
Decision: Accept (poster)