Demystifying How Self-Supervised Features Improve Training from Noisy Labels
Authors: Hao Cheng
Zhaowei Zhu
Xing Sun
Yang Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Learning with noisy labels
Self-Supervised Learning

Abstract: The advancement of self-supervised learning (SSL) motivates researchers to apply SSL on other tasks such as learning with noisy labels. Recent literature indicates that methods built on SSL features can substantially improve the performance of learning with noisy labels. Nonetheless, the deeper reasons why (and how) SSL features benefit the training from noisy labels are less understood. In this paper, we study why and how self-supervised features help networks resist label noise using both theoretical analyses and numerical experiments. Our result shows that, given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise. Further, we provide insights for how knowledge distilled from SSL features can alleviate the over-fitting problem. We hope our work provides a better understanding for learning with noisy labels from the perspective of self-supervised learning and can potentially serve as a guideline for further research.

Under review as a conference paper at ICLR 2022
DEMYSTIFYING How SELF-SUPERVISED FEATURES IMPROVE TRAINING FROM NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
The advancement of self-supervised learning (SSL) motivates researchers to apply SSL on other tasks such as learning with noisy labels. Recent literature indicates that methods built on SSL features can substantially improve the performance of learning with noisy labels. Nonetheless, the deeper reasons why (and how) SSL features benefit the training from noisy labels are less understood. In this paper, we study why and how self-supervised features help networks resist label noise using both theoretical analyses and numerical experiments. Our result shows that, given a quality encoder pre-trained from SSL, simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise. Further, we provide insights for how knowledge distilled from SSL. features can alleviate the over-fitting problem. We hope our work provides a better understanding for learning with noisy labels from the perspective of self-supervised learning and can potentially serve guideline for further research.
INTRODUCTION
Deep Neural Networks (DNNs) have achieved remarkable performance in many areas including speech recognition (Graves et al., 2013), computer vision (Krizhevsky et al. 2012 Lotter et al., 2016), natural language processing (Zhng & LeCun 2015) etc. The high-achieving performance often builds on the availability of auality-anootated datasets. In real world scenario, data annotation inevitably brings in label noise which degrades the performance of the network, primarily due to DNNs' capability in "memorizing" noisy labels (Zhang et al. 2016).
In the past few years, a number of methods have been proposed to tackle the problem of learning with label noise including robust loss design (Ghosh al. [017]] Zhang & Sabuneul 2018 Liu &
Guo 2020), sample selection (Han et al. 2018; Yu et al. 2019)] Cheng et al. 2021 and noise tran-
sition matrix estimation (Patrini al. 2017 Zhu al.; 202 Among all these methods, arguably the most efficient treatment is to adopt robust losses, since sample selection and noise transition matrix estimation always involve training multiple networks or need multi-stage training. Nonethe- less, though the designed losses are theoretically proven robust, they often suffer from significant performance drop when noise rate is high (Wang al. 2019: Ma et al. 2020 Cheng et al.) [202]] 2021a), hinting that the ability of converging to the optimal classifier is also important.
Zhu et al.
Very recent works (Zheltonozhski et al., 202T Nodet al. 2021 Ghosh & Lan 2021 Yao et al.) 2021
Tan et al. 2021) started applying self-supervised learning to solving the problem of learning from noisy labels. The experiments show that methods built on the self-supervised features can achieve exceptional performance even when the noise rate is high and largely outperform previously reported SOTA approaches. Despite the empirical observations, the reasons why self-supervised features lead significant performance improvement are not well understood. In this paper, provide theoretical insights to understand how self-supervised features improve classification with label noise and perform extensive experiments to support our theory. Our analysis provides new understanding on learning with noisy labels from the perspective of self-supervised learning. We summarize main contributions below;
We theoretically and experimentally show that by using self superviide features to fine-tune the network on noisy datasets, Cross Entropy itself is robust to label noise (Theorem ( The theory also answers the question of whether or not fix the encoder when performing fine-tuning.

---

Under review as a conference paper at ICLR 2022
We theoretically and experimentally show that by using self-supervised features, regularizer commonly used in knowledge distillation (Hinton et al. 2013)) (where the dataset does not contain label noise) can greatly alleviate ver-fitting problem of DNN noisy datasets (Theorem 3
1.1 RELATED WORKS
Learning with Noisy Labels: Due to the over-fitting problem of DNN, many works design robust loss to improve the robustness of neural networks. (Ghosh et al. 2017) proves MAE is inherently robust label noise. However, MAF has a severe under-fitiing problem. (Zhang & Sabuncul 2018) propose a loss which can combine both the advantage of MAE and CE, exhibiting good performance on noisy datasets. (Liu & Guo 2020) introduces peer loss, which is proven statistically robust to label noise without knowing noise rate. The extension of peer loss also shows good performance on
instance-de dependentt label noise (Cheng et al.) 202T Zhu al. 2021a). Another efficient approach to combat label noise is by sample selection (Jiang et al. 2018 Han et al.) 2018 Yu et al. 2019] Northcutt et al., 202T Yao et al., 2020; Wei et al. |020 Zhang et al. 2020). These methods regard "small loss" examples as clean ones and always involve training multiple networks to select clean samples. Semi-supervised learning is also popular and effective on learning with noisy labels in recent years. Some works (Li ct al., 2020; Nguyen et al.) 2020) first perform clustering the sample loss and divide the samples into clean ones and noisy ones. Then drop the labels of the "noisy samples" and perform superviee learning all the samples.
The goal of self-supervised learning (SSL) is to learn good presentation without using the information of the labels. Generally, the methods of SSL can be divided into two categories: designing pretext tasks or designing loss functions. The designed tasks or losses do not involve any labels. Some popular tasks include patch orderings (Doersch et al.. 2015 Noroozi & Favaro 2016), tracking (Wang& & Gupta 2015) or clustering features (Caron et al. 2018: 2019). However, the SSL performance of pretext tasks is limited. Recent SOTA methods for SSL is by designing contrastive loss functions. The representative works include Moco (He et al. 2020 and SimCLR (Chen al.) 2020) which train neural networks based on InfoNCE loss (Oord al.. 2018). In our paper, we also adopt InfoNCE for performing self-supervised training to get SSL pre-trained features. The first part of our paper relates to the works that apply SSL features to perform fine- tuning on noisy dataset (Nodet et al. 2021 Ghosh G02sh & Lan 2021 and our goal is to build theoretical understanding on this aspect.
Self-Supervised Learning:
The second part of our paper is very related to the research field of knowl- edge distillation (KD). The original idea of KD can be traced back to model compression (Buciluă et al. 2006), where authors demonstrate the knowledge acquired by large ensemble of models can be transferred to a single small model. (Hinton et al. 2015) generalize this idea to neural networks and show small, shallow network can be improved through teacher-student framework. Due to its great applicability, KD has gained more and more attention in recent years and numerous meth- ods have been proposed to perform efficient distillation (Mirzadeh et al., 2020 Zhang et al., 2018 2019). However, the dataset used in KD is assumed to be clean. Thus it is hard to connect KD with learning with noisy labels. In this paper. we theoretically and experimentally show that a regularizer generally used in KD (Park et al. (219)) can alleviate the over- over-fitting problem on noisy data by using SSL features which offers new alternative for dealing with label noise.
Knowledge Distillation:
2 PRELIMINARY
We introduce preliminaries and notations including definitions and problem formulation.
Problem Formulation: Consider classification problem on set of N training examples denoted by D = {(F,,, nn[]) where [N] == 21, ... ,,N is the set of example indices. Examples (In, yn) are drawn according to random variables (X. Y) from a joint distribution D. The classi- fication task aims to identify classifier C that maps X to Y accurately. Our theoretical analyses focus on binary classifications thus Y E {0, 1]. In real-world applications, the learner can only observe noisy labels. For instance, human annotators may wrongly label some images contain- ing cats as ones that contain dogs accidentally or irresponsibly The label noise of each instance is assumed to be class-dependen (Liu & Tao 2015). i.e., P(Y = | Thus the er-
rates defined e+ PP(Y Q/
1),e- P(Y = 1 0). The corresponding
2

---

Under review as a conference paper at ICLR 2022
Random Initialization
Path-1: Traditional learning
Path-2 2: Unfixed encoder
Path-3: Fixed encoder
Converged Classifier
Encoder f
Figure 1: Illustration of different learning paths.
noisy dataset and distribution are denoted by D == {Ins (n)) neeN] and D. Define the expected risk of classifier C' R(C) Ep [X(C(X) + Y). The goal is learn classifier C' from the noisy distribution which also minimizes R(C). i..., learn the Bayes optimal classifier such that C'aayes (z) arg 4a (0.1) P(Y i|X z). For better presentation, define the following
notations: X = X = 1, = X = 0, and X = - = Y, X X + Y.
Evaluation of SSL (Self-Supervised Learning): SSL usually evaluated by two steps: First, use SSL to train an encoder with only unlabeled data X, then add a linear classifier g following the pre-trained encoder f and only fine-tune 9 on (X,Y) with a fixed f. The high-level intuition is that, if the encoder f is well learned by SSL, only fine-tuning linear classifier g is often sufficient to achieve good performance on test data. If the test performance is comparable to SL (Supervised Learning), call the gap between SSL and SL is small (Chen al. (2220). Denote by the space of linear classifier g. Fine-tuning linear layer on () ~ D can be represented as:
min (p((E(((((X)), gEG
where CE denotes the Cross-Entropy loss. Note the dimension of s(X) is determined by the network structure, e.g.. 512 for ResNet34. In binary classifications, g(f(X)) [0, where g(f(X)) < 0.5 indicates predicting class-O ) and g(f(X)) >> 0.5 corresponds to class-1. g(f(X)) is supposed predict the same label as (ar).
ROBUSTNESS OF
WITH SSL FEATURES
We will analyze the robustness of Cross-Entropy with SSL features by comparing three different learning paths as illustrated in Figure [ Path-1 is the traditional learning path that learns both encoder and linear classifier g at the same time. Path-2 is the strategy applied (Ghosh Lan 2021) that firstly pre-trains encoder f with SSL, then treats the pre-trained model as a network initialization and jointly fine-tunes and g. Path-3 an alternate SSL-based path that first learns the encoder f then only fine-tunes the linear classifier y with fixed ʃ.
3.1 THEORETICAL TOOLS
We prepare some theoretical tools for our analyses. Our first theorem focuses on demonstrating the effectiveness of only fine-tuning linear classifier g as in Path-3. We present Theorem | below.
Theorem Let arg mingeg Ep (x)))(((((X)) Y),, g2 = arg mingeg E,s(CE((f(X)), Y)]. Then e = e_ < 0.5, we have:
Round( (f())) Round((s(f(X)))
(1)
where f is fixed encoder and g is the linear classifier, g(((-)) denotes the output whose value ranges from 0 to 1. Round(p) (p) is predictor function that outputs 1 if p >> 0.5 and outputs O otherwise.
Theorem I shows with balanced error rates, simply fine-tuning a linear classifier g the noisy data distribution D can achieve the same decision boundary the optimal linear classifier obtained from the corresponding clean distribution D. i.e. g1 (f) and (2(.) have the same predictions for all the samples. Theorem can be generalized the case with arbitrary classifier beyond linear.
3

---

Under review as a conference paper at ICLR 2022
[a] Features
Features CIFAR10
(c) Features AR100
50
40
30
40
20
20
10
10-
-20
Figure 2:
Tsne visualization of the self-supervised encoder outputs f(X) on DogCat (Kaggle CIFARIO and CIFARI00 (Krizhevsky et al.). Different colors stand for different classes. SimCLR Chen et al.
2020 is selected for SSL training. Quality of encoders: (a) > (b) > (c).
However, admittedly with limited data, training complicated classifiers is hard to converge to the optimal decision boundary. We defer more details to the next subsection.
We then evaluate the former part of Path-3, i.e., the performance of SSL., Recall in Section 2 SSL is usually evaluated by performance gap between f Bayes and Cayys where y Bayes is the optimal linear classifier trained on D, fo Bayes denotes the joint model given by Buyes (f(X)). We consider tractable case in Assumption
Assumption The encoder outputs f(X+) and [(X-) follow Gaussian distribution with parame- ters (HI, E) and (122 E), where Σ = -² is the identity matrix.
follow
Istates states that the self-supervised features for each class follow simple Gaussian distribu- tions. We check the effectiveness of this assumption by Figure It can observed that the features of each class may have overlaps. but good SSL method is supposed to return features with good separations (small overlaps). In Assumption we use || u22l and a to capture the overlapping area of two classes. If || #all is large and is small, then there exists small overlapping, Based on this assumption, we show the performance SSL in Theorem
Assumption
Theorem P/Y = 1) = P(Y = 0), the risk (error rate) of Bayes optimal classifier f Bayes follows as:
R(f g Bayes) = 11 + (IIm H2ll
2
where o is the cumulative distribution function (CDF) of the standard Gaussian distribution.
Wrap-up With Theorem [ we know CE is robust, the performance of which is subject to f. Theorem 2 implies that if SSL features learned by f exhibit good property, i.e.. when || μ2 is large and a is small, only fine-tuning g can approach the Bayes optimal classifier Bayes Therefore, good SSL features induce high performance. In summary, Theorem and Theorem connect SSL features with robustness and generalization ability of CE loss, providing an insight on why SSL features improve classification with label noise.
3.2
CAN WE FIX THE ENCODER?
We compare the performance of Path-2 with Path-3 in this subsection.
3.2.1 THEORETICAL ANALYSES
Denote by VC(g) the VC-dimension of G.
Always fix good encoders When we fine-tune g on fixed f, the learning errors will come from two parts: the encoder error and the classification сттоr. The encoder error is bounded in Theorem 2 Noting that g
a linear classifier with relatively low VC-dimension and CE is calibrated and convex function, the optimization can achieve global optimum with enough samples (Bartlett et al. 2006), e.g., w.p. 1 6, the learning error will be with O VC i..dd. instances (Vapnik
4

---

Under review as a conference paper at ICLR 2022
DogCat
(6) CIFARLO
OFAR100
(4) CIFARIO
100
200
200
100
80
50
60
02
40-
40-
40
40-
(unfined encoded
lunflaed erenter)
uuttaed encoded
onfieedd ennsted
0.0
6.6
150
rpiring
Figure 3: (a) (b) (c): Performance of CE on DogCat, CIFARIO and CIFARIO0 under symmetric noise rate. For each noise rate, the best epoch test accuracy is recorded. The blue line represents training with fixed encoder and the red line represents training with unfixed encoder; (d): test ac- euracy of CIFARIO
each training epoch under symmetric 0.6 noise rate. We use ResNet50 for DogCat and ResNer34 for CIFARIO and CIFARI00. SimCLR is used to deploy SSL pre-training. Detailed experimental setting are reported in the Appendix.
2013). Recall Theorem 1 also guarantees the optimality of g learned under the noisy distribution D. Therefore, given a good encoder with large enough | we should keep it fixed since the simple 0,
linear classifier trained under label noise only requires N = e (C((())) 1/6 i.i.d. instances in D to achieve around errors with high probability, which performs well enough to approximate Bayess
Updating f is not stable If f is unfixed, due to the non-convexity and over-fitting capability of network trained on vanilla D will likely and ultimately overfit to noisy labels (Zhang et al. 2016), particularly when the number of instances is insufficient. For example, w.p. ò. jointly learning and g and achieving error E requires O VC((Fo))++ i.i.d. instances, which is far more than the number required in only tuning due to the huge capacity of DNNs. Note the effective i.i.d. instances would decrease with the increase of noise rates. Although good initialization contributes to better convergence, f will be misled by label noise in extreme cases. Therefore it is unreasonable to assume that CE will stably induce the network to approximate the Bayes optimal classifier under label noise. In other words, if f is not fixed, due to the overfitting capability of f. the outputs of ((X) would be contaminated by the label noise before g. In contrast, the original SSL features are independent from the label noise if f is fixed. We therefore conclude that CE on vanilla D is not robust. Note Nodet et al. (202T) claimed the encoder should not be fixed during fine-tuning based on some experiments. We include more discussions in Appendix.
3.2.2 EMPIRICAL EVIDENCE
We conduct experiments in Figure 3 to support our theorems and analyses above.
Overall performance From Figure 3 [c),, when encoder f is fixed, the performance of CE (fixed encoder) exhibits great robustness as noise rate increases and only has little drop when the symmetric noise rate approximates the maximum theoretical value for each dataset (0.5 for DogCat, 0.9 for CIFARI0) which verifies Theorem [ and Theorem 2 Note that the drop is because, to reach the condition specified in Equation (I). a substantial number of training samples are needed. However, for example, each class only has 500 samples in CIFARIOS 100. As the noise rate increases, the number of clean samples will further decrease which makes the classifier hard to learn from the distribution.
Fixed encoder benefits extreme label noise Next, we observe the performance gap between SSL and SL when datasets are clean. This gap can be visualized in Figure (a) (a) (b) (c) at noise rate = 0. For DogCat, the gap very small, thus CE with fixed encoder always exhibits better performance. For CIFARIO, the gap is moderate, the performances of CE (fixed encoder) and CE (unfixed en- coder) are close when noise rate is small. However, when noise rate is high, CE with fixed encoder still exhibits better performance. For CIFAR100 the gap is large, thus CE with fixed encoder shows better performance only when noise rate is over 0.7.
Fixed encoder has better convergence Note that the performance in Figure (a) (b) (c) are selected from best epoch accuracy. If we take the last epoch accuracy into consideration. CE with fixed encoder has great advantage over unfixed encoder. Figure 3 (d) depicts the performance of CE in
5

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of test accuracies of each method under instance-based label noise. CE (un- fixed f with random imit.) is common baseline adopted in the literature which trains a random initialized DNN using CE loss on noisy dataset.
Inst. CIFARIO
Inst. CIFARIOO
Method
c 0.2
= 0.4
= 0.6
=
0.2
8
0.4
= 0.6
CORES (Cheng et al. 2021)
89.50
82.84
79.66
61.25
47.81
37.85
CE (unfixed J with random init.)
87.16
75.16
44.64
58.72
41.14
25,29
CE (fixed f. sampling)
88.74
75.71
25.7
59.38
46.13
24.75
CE (fixed f, down-sampling)
90.12
84.19
82.06
62.88
61.1
58.93
terms of training epochs under 0.6 symmetric noise rate on CIFARIO. We find that even though these two approaches have almost same best epoch accuracy, CE with fixed encoder is more robust as training proceeds. For CE with unfixed encoder, since the self-supervised pre-trained network's weights serve as better initialization, it shows good performance in the beginning. However, because of the memorizing effect DNN, it will still over-fit to noisy labels in the end.
Generalize to class- and instance-dependeni label noise: Theorem shows that CE is robust to label noise when e = e+. However, for a more general label noise such as asymmetric class- dependent label noise or cestance-dependee
label noise, CE with fixed encoder on vanilla (f()), Y) hard to be guaranteed to be robust because for instance noise, noise rate varies for each class. But it does not mean SSL features can not help. We may still fix the encoder and perform simple down- sampling to balance the noisy dataset before fine-tuning the linear classifier. The down-sampling is conducted to make PPYY = i) P(Ý = j) in the noisy dataset which can well approximate the condition in Theorem (lllusraaionn is deferred Appendix). Note that with wown-sampling, the target distribution of clean labels also changes, However, Theorem is proved without specifying the distribution of clean labels, suggesting that classifier is consistent even for imbalanced dataset under symmetric label noise. Secondly, with SSL features, classifier is robust to imbalanced imbal- anced dataset (Yang Xul (020). Thus down-sampling strategy reduce error rate imbalances is helpful for other noise types beyond symmetric. We perform an experiment in Table I to verify the effectiveness of the down-sampling in the setting of instance-dependenn label noise (generation of label noise is followed by (Cheng et al.) (02T]). From the experiments, if no sampling strategy conducted, CE with fixed encoder does not show many benefits. In some cases, it even performs worse than CE baseline (unfixed f with random init). However, with down-sampling, CE with fixed encoder even outperforms CORES (Cheng et al. (021)) by a large margin. Even though our main purpose in the paper is not to develop new methods for achieving SOTA, think this down- sampling strategy motivated by Theorem | can inspire further research on instance-dependent label noise by using SSL. features.
3.3 TAKEAWAYS
Our both theoretical and empirical results demonstrate that: We need to simply fix the encoder when 1) the encoder is sufficiently good, 2) the high noise rate decreases the number of effective i.i.d. instances to a certain level, 3) the over- over- parameterizze model makes it easy to overfit label noise. However, there are
concerns: 1) If the gap between SL and SSL is relatively large, CE with unfixed encoder has more advantage on small noise rate (e.g., Figure (c)); 2) Consider an online classification system which always needs new training data update itself. If we only fine-tune the linear classifier for keeping robustness to label noise, then due to the limited Icarnability of affine functions, this online classification system may learn very limited knowledge from unseen data. Therefore, it is necessary to update the encoder while keeping it stable and robust to label noise, which is the focus in the next section.
4 REGULARIZER FROM KD IMPROVES NETWORK ROBUSTNESS
In this section, show that regularizer from knowledge distillation (KD) PPark et al. 2019) can well alleviate the over fiter-fittig problem by using the structure of SSL features. Different from classical KD settings where the dataset is clean, analyses aim regularizing classifiers learned with noisy

---

Under review as a conference paper at ICLR 2022
SSL features
projection head
SSL training
encoder
X
regularize
f
linear clssilter
training
Y
features
Figure 4: In this training framework, we adopt CE for SL training and InfoNCE for SSL training. During training, SSL features utilized to perform regularization on SL features.
supervisions by SSL features. Understanding the working mechanism is important since it may contribute the community new perspective learn with label noise by distilling information from SSL features. In the rest of this section, we will first provide theoretical understanding towards the regularizer under label noise, then perform experiments on different settings test our analyses.
4.1 TRAINING FRAMEWORK
The training framework is shown in Figure where learning path (SSL training) added to be parallel to Path-2 f - (SL-training) in Figure [ The newly added projection head h is one-hidden-layer MIP (Multi Layer Perceptron) whose output represents SSL features (after dimension reduction). Its output is employed to regularize the output of linear classifier g as Reg(h( ((X) 9(())))(())))) InfoNCE from SimCLR (Chen al. (0200) is adopted for SSL training and CE is for SL training. InfoNCE and CE share a common encoder, inspired by the design of self distillation (Zhang et al. 2019). The loss function defined as:
L ==E0 E (CE(g((())) Y) + In()NCE(((((X))) + Reg(MM((())))X()))
Intuitively, SL features is supposed to be benefited from the structure information from SSL features, e.g., clusterability (Zhu et al. (b22b) that instances with similar SSL features should have the same true label and instance with different SSL features should have different true labels. Mathematically. let f; = hzf();),, = (x:)) and X to be the set of N tuples of data samples. The distance
between 4 and 1, can be represented as oi (1,, (2) - 1| 1 where w (1,2} and m is normalization term:
m
||, -1,,, 11
(4)
X2
(xx?
Then the expectation of Reg (eg(h(X).99X)) in Equation
can be estimated by:
Ex (eg(h((()))
d(d (te,tt)), (s).
(5)
(x
where d(-) is distance measure for two inputs. Popular choices are 4, la or square l2 distance.
4.2 ANALYTICAL FRAMEWORK
Under the label noise setting, denote by dis decomposition:
8,)), we have the following
di
Σ
dig +
dis
Σ
2. 2.dis) (6)
2
X2
(
(#,,2,) EE2
(E sarr
ErE E
Term-1
Term
Term-3
where X' = Xetean U Xpoiay. Xavisy denotes the set samples whose labels are flipped. Note the regularizer mainly works when SSL features "disagree" with SL features, i.e. Term-3. For further analyses, we write Term-3 the form of expectation with chosen as square la distance, i.e., MSE loss:

---

Under review as a conference paper at ICLR 2022
The y Xaai (I((((Xka))) 9 (X ))
1(ss((XXX)) [{(
1)
m2
where m and mg are normalization terms in Equation Note in Le, we use w for SL features and w = for SSL features Denote the variance by var(-). Define notations:
x - 0),
=
X == x 0, Y. = 1).
To theoretically measure and quantify how feature correction relates to network robustness, we make three assumptions follows:
Assumption 2 (Memorize clean instances) Vn E {nz Un = yn CE(g((((nn) yn)) = 0.
Assumption (Same overfitting) (f (X)())) and var(g(((X""
Assumption (Gaussinndissrrbuted SSL features) The SSL features follow Gaussian distribu- lions, ie., h([(x+1) ~ (P1, Σ) and h((((-1)) ~ (22,, E).
Assumption implies that DNN has confident predictions clean samples. Assumption B im- plies that a DNN has the same degree of overfitting for each noisy sample. For example, an over- parameterized DNN can memorize all the noisy labels (Zhang al.) [0066] Liu 2021). Thus these two assumptions are reasonable. Assumption follows Assumption Note in Figure SSL fea-
tures from h(f()) rather than f(X).
4.3 THEORETICAL UNDERSTANDING
Based on Assumptions 4 we present Theorem 3 to analyze the effect of Le. Recall
moisy
XI(Y L,Y X == XI 0, 1).
Theorem When e_ = and P(Y = 1) = P(Y 0), minimizing Le respect to g (X on DNN results in the following solutions:
E Exyd (x })] = 2 2+ 2+ 11, P2) E [o(S(X )) = 2 (1,11),
where A M1, (2) = (r(E)/ =22l tr(.) denotes the mairix trace,
Theorem reveals a clean relationship between the quality of SSL features (given by {(f(x))) and the network robustness noisy samples. Note the true label for X is 0 and the true label for Xmaisy is 1. Theorem shows the regularizer is leading the classifier to predict the feature with corrupted label to its corresponding true label, and the guidance is particular strong when A (E,, 0, e.g., (r(E) - 0 or [M #2l oc, such that (X ))] 0 and moisy
E x [x( + 1. Beyond this ideal case, we discuss more interesting findings as:
Worst case: The regularizer can always have a positive effect to avoid memorizing wrong la- bels long the model ʃ h is better than randomly initialized models. This is because A(E, Pl, A2) > 0 when and #2 are different, and the regularizer is Icading the classifier to the true direction when (22, 11, U2) O.
Practical case: The model f o h is always better than randomly initialized models since SL training can guarantee the performance of encoder f on clean instances, which may generalize to noisy instances, and SSL training can further improves the quality of ʃ o h. Thus in practical cases. using SSL features to regularize learning with noisy labels is always beneficial.
More possibilities: Note the proof of Theorem 3 does not rely on any SSL training process. From our first finding, we know any encoder helps if it is better than random models. This makes it possible to use some pre-trained encoders from other tasks.
High-level understanding on structure regularization: Even though we have built Theorem 3 to show SL features can benefit from the structure of SSL features by performing regularization, there
Practically, different choices make negligible effects performance, See more details in Appendix.
8

---

Under review as a conference paper at ICLR 2022
(<) 20% symm dinit: SimCLi)
10) 40% symm (init SimCUR)
[c) 40% symm (inth: Random)
(d1 60% symm Unit: Random)
200
80-
tot 2008323
40-
40-
40-
40
20
20-
Nogu
80
50 200
50
30
Figure 5: Experiments with respect to the regularizer on CIFARIO. ResNet34 is deployed for all the experiments. (a) (b): Encoder is pre-trained by SimCLR. Symmetric noise rate is 20% and 40%, respectively; (c) (d): Encoder is randomly initialized. Symmetric noise rate is 40% and 60%, re- spectively. The value of hyper-parameters and other detailed setting in the experiments are reported in the Appendix.
still lacks high-level understanding of what the regularization is exactly doing. Here we provide an insight Theorem 4 which shows the regularization implicitly maximizing mutual information between SL features and SSL features.
Theorem 4 Suppose there always exists mapping & to map h(f(X)) to ((f())) Ie., g(f(X)) ((K()(X)) Then minimizing Equation (5) is implicitly maximizing Mutual Information be- tween (x)( and g(f())) Lee, when Equation (5) achieves 0, the mutual information (( achieves maximum.
The above results facilitate better understanding on what the regularizer is exactly doing. Note that Mutual Information itself has several popular estimators (Belghazi et al. 2018 HHjelm et al. 2011)). is a very interesting future direction develop regularizes based on MI to perform regularization by utilizing SSL features.
4.4 EXPERIMENTS
We provide experiments on CIFARIO to verify our theory and analysis. The overall experiments shown in Figure 3 In the experiments, Regularizer is added at the very beginning since recent studies show that for random initialized network, the model tends to fit clean labels first (Arpit et al., 2017) and we hope the regularizer can improve the network robustness when DNN begins fit noisy labels. From Figure (c) (d), for CE training, the performance first increases then decreases since the network over-fits noisy labels as training proceeds. However, for CE with regularizer, the performance is more stable after it reaches the peak. For 60% noise rate, the peak point is also much higher than vanilla CE training. For Figure (a) (b), since the network not randomly initialized, it over-fits noisy labels at the very beginning and the performance gradually decreases. However, for CE with regularizer, it can help the network gradually increase the performance as the network reaches the lowest point (over-fitting state). This observation supports Theorem that the regularizer can prevent DNN from over-fitting to noisy labels. More experiments and ablation studies can be found in the Appendix.
5 CONCLUSIONS AND DISCUSSIONS
In this paper, we have provided a theoretical understanding on why and how self-supervised features improve training from noisy labels and perform experiments to verify our analysis.
We show that with SSL pre-trained encoder, CE itself is theoretically robust to symmetric label noise. With the down-sampling strategy. it can also exhibit great power on class- and instance- dependent label noise. This part answers why SSL features help (Theorem (
We show that by utilizing SSL features, the regularizer in KD can help DNN alleviate the over- fitting problem on the noisy dataset and build theoretical insights on how SSL features relate to
the network robustness. This part answers how SSL features help (Theorem 3
Future directions can be done to develop more efficient sampling strategies for solving instance- based label noise with theoretical guarantees and develop efficient regularizers to combat label noise. The key for these future works is to leverage the nice structural property of SSL features.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volumm 70, pp. 233-242. JMLR. org, 2017.
Peter I. Bartlett, Michael Jordan, and Jon D McAuliffe.
Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-155 2006.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04000 2018.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535-541, 2006.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-trate pre-training of image features non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2959-2968, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Rep- resentations, 2021.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acousties, speech and signal processing, pp. 6645-6649, IEEE, 2013.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arxiv preprint arXiv: 1808 06670, 2018.
Karan Grewal, Phil Bachman, Adam
10

---

Under review as a conference paper at ICLR 2022
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp.
PMLR, 2018.
Kaggle. Kaggle dogcat dataset. URL https: // /www http://wwwwwkwlwe...eeee waggle...g........... wtt.c/ccccccooo c/dgss vs cats
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- (canadian institute for advanced re- search). URL http: /twp://wwwwcccc toronto. wwwp/eeri/eeeeeeeeeee ctrii//cifar.ttm
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020. URL
http::::///peeeeeeeeee net wtt.eewweetoeeeeeeeeeeeeeoee d=
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance rewcighting. IEEE
Transactions on pattern analysis and machine intelligence, 38(3):447-44) 2015.
Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In Interna- tional Conference on Machine Learning, pp. 6725-6735. PMLR, 2021.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226-6236. PMLR, 2020.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning. arXiv preprint arXiv: 1605.08804 2016.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Nor- malized loss functions for deep learning with noisy labels. In International Conference on Ma- chine Learning, pp. 4543-6555 PMLR, 2020.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5191-5198, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204 2013.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling. In International Conference on Learning Representations, 2020. URL https:
//openzeview.
Pierre Nodet, Vincent Lemaire, Alexis Bondu, and Antoine Coruuéjoss. Contrastive representations for label noise require fine-tuning, arXiv preprini arXiv:2 100.09144, 2021.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pp. 69 84. Springer, 2016.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70: 70:1373-1411, 2021.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1807 1807,00778 2018
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1954-1952, 2017.

---

Under review as a conference paper at ICLR 2022
Cheng Tan, Jun Xia, Lirong Wu, and Stan Z Li. Co-leaming: Learning from noisy labels with self-supervision. arXiv preprint arXiv:2108. 0.000000003.... 2021.
Vladimir Vapnik. The nature of statistical learning theory. Springer Science & Business Media, 2013.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pp. 2794-2802, 2015.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross en- tropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 322-330, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13726 13735, 2020.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learn- ing with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:21 10. 12088, 2021.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. arXiv preprint arXiv:2006.07529, 2020.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James T Kwok. Searching to exploit memo- rization effect in learning with noisy labels. In Proceedings of the 37th International Conference on Machine Learning, ICML '20, 2020.
Yazhou Yao, Zeren Sun, Chuanyi Zhang, Fumin Shen, Qi Wu, Jian Zhang, and Zhenmin Tang. Jo-src: A contrastive approach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5192-5201, 2021.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint arXiv:1901.042:0 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.00
2016
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3713-3722, 2019.
Xiang Zhang and Yann LeCun. Text understanding from scratch. arXiv preprint arXiv: 150200/700, 2015.
Xuchao Zhang, Xian Wu, Fanglan Chen, Liang Zhao, and Chang- Tien Lu. Self-paced robust learn- ing for leveraging clean labels in noisy data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6853-6860) 2020.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778-8788, 2018.
Eygenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Con- trast to divide: Self-supervised pre-training for learning with noisy labels. arXiv preprint arXiv:2/03. 13646, 2021.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10113-10123, 2021a.
12

---

Under review as a conference paper at ICLR 2022
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability alternative to anchor points when learning with noisy labels. arXiv preprint arXiv:2102.05291, 2021b.
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
The Appendix is arranged as follows: Section A proves Theorem 1 and Theorem 2 in the main paper. Section proves Theorem B and Theorem in the main Paper. Section [ illustaates why down-sampling can decrease the gap of noise rates. Section D Provvides additional discussions and experiments including further discussion of whether or not to fix the encoder when fine-tuning DNN on noisy dataset; analyses of why robust losses (Wang et al., 2019 Ma et al. 2020) may suffer severe performance degradation under high noise rates; the effect of distance measure in Equation 4 (w 1 2); ablation study in Section 4 visualization of features trained by vanilla CE loss; the effect of different SSL. pre-trained methods; Experiments with human-annotater label noise. Section elaboratts the detailed experimental setting of all the experiments in the paper. Section generalizes Theorem[]c multi-class classification. Section provides theoretical and experimental proof on why directly fine-tuning SSL features cannot be robust on asymmetric and instance label
noise.
A PROOF FOR THEOREM 1,2
A.1 PROOF FOR THEOREM I
Define 91 ming (C((E((((()))) and 92 ming E& (CECE(((()))) Suppose 91 Bayes are Bayes optimal classifiers under P(f(X), and ((f( Y). respectively. Since CE is calibrated and convex loss, with enough samples, 91 and 92 will converge (approximate to)
Bayes
and 92
Bayes and 92
Bayes Bartlett et al. 2006). Thus prove gi and g2 have the same prediction for f(X),
I1
it is identical to prove 91 Bayes and 92 Hayes have the same predictions for fX)).
Let p+ (x) be the distribution of f(X+) and p-(x) be the distribution of f(X-), according to the bayes rules:
P(Y 1) (x)
P(Y = 1/x)
(8)
P(Y = 1) -2 (z) + =0) (x)
P(Y =0 0) (xx)
P(Y 0/z))
P(Y = 1) P+ (x) P(Y 0) (x)
Define (xx) = PY 1) 11- (x) P(Y = 0) . (a:). Thus vb(x) = 0 is the decision boundary for optimal Bayes classifier 91
Bayes on clean dataset. Since when (xx)) > P(Y = 1/x) > IP(Y and vice versa. Since the generation of label noise independent to f(X). the samples for each class whose labels are flipped also follows original distribution. Denote pf (ar) to be the distribution of samples whose observed labels are 1 under label noise. Thus p (xx) and pf (ar) can be represented as:
(zz)) from Equation (8) (9). the classifier will output the prediction
(x) mi (P(Y =1) (1 (1- P4 (z) + PPV =0) --.0))---p00 (x)) P+ (x) m2 (P(YY =) e4 (z) IPYY 0) (1-e-1) p- (x))
where mi and m2 are normalization term make distribution valid. Specifically, mi
PV= and m2 -(= + + (1=e)- Under label noise, P(Y = 1)
P(Y = 1) (1 = 0) -))e. e_ and P(Y = 0) = P(Y = =) -) + +P = 0) (1 -e-).
According bayes rules, we have:
P(Ý = 1) p4 (x)
P( 1|x)
(10)
P(Ý = 1) p$, (x) P(Ý = 0) (x)
P(Ý = 0) p² (x:)
P(Ý
0|z)
(11)
F( = 1) pf (x) + IP( 0) p" (x)
14

---

Under review as a conference paper at ICLR 2022
Without loss of generality, assume there exits Zy in the feature space which satisfy (x > 0.
Thus for clean dataset, P(Y = 1) p+(x) > P(Y = 0) p- p-(x)) and P(Y = = > P(Y = 0 z0)
We aim to show that for noisy dataset, P(Y = 1 m > P(Y 0/ which means (b(x)) = 0 is also the decision boundary that gives Bayes optimal classifier 92
Bayes under label noise.
According to Equation (10) and Equation
P(Y = 1/z0) P-Ý- 0 xc))
(1 - c.4) IP(Y = 1) p+ (2o) - (1 - 2 .P(Y = 0) pp-pppppp (2
(12)
normalization
where normalization = IP( 1)
P-Y = 0) (x) > 0. Since IP(Y = 1) .p p+)))))) >
P(Y 0) p-p (20), if e.+ e- 0.5, we have P(Y 1 PPY 0/ 0. Thus vb(x)
also the decision boundary of 92 91 samples.
Bayes Bayes and g2
Bayes will output the same prediction for all the
Proof done.
A.2 PROOF FOR THEOREM
Let d = || 12ll- Without loss of generality, we assume μ1 = (-2. and μ2 = (5 .........0 0) (This can be done by applying affine transformation on original coordinates without changing actual positions of f(X+) and f fX-)). Under the distribution of Gaussian assumption, we write the distribution of f f((+) as follows:
1
p+ (x) ot expl
(ar - 11) m41
(13)
20
where (I11,.... F&k ) RR. 3e denotes the variable of in the k-th dimension. Similarly, the pdf of f(X-) can be represented as:
expl 1 (= (2
(x (2) 22)
(14)
20
Following notations from A.1 solving (bxx) = 0 under the conditions of P(Y = P 0)
gives 31 (rr is the variable of the first dimension). Thus, when 21 < 0, the Bayes optimal Bayes
classifier f Bayes outputs prediction as 1. lee. for samples whose labels are 1, f g gives right prediction when x and wrong prediction when æj 0. The risk of of g Bayes be written as:
R(f Bayes)
= Ex o Bayes (X) + Y)
= Ey ExEx Bayes (X) + Y)
P(Y = 1) Ex Expr=1111 1(f Rayes (X) # + 0) Exiy== 10 og Bayes Baye((X) # 0)
EExr== o Bayes (X) * + Exly=0 Bayes # 0)
2
(0)
Ex
Bayes (X) #1)
p+ (+()dd } .0+ [ p+ (x))dx
<0
d
(a) is satisfied because of the symmetry of p+ (x)) and p- (a:). We derive (b) as follows:
15

---

Under review as a conference paper at ICLR 2022
P+ p+(x: (x:2))
a//a <0
/
exp(
202 (x - 11)²((x (x - pi)jdæ
(a-)))
alliic (2
-/ Jala: <0 (2
(2r + 4)² x3 + + xx
exp(
dæ
20²
12
(x) + =
exp(-
|d
expl
|dzk
exp(
}dr
22 V2TO
202
12m
20
2150
22Ta
20²
(1)
d/2
=1.1
expl
]d
ai <0
VTT
20
Since San V2ma exp( ())+
20 | is one dimensional gaussian with mean - and deviation Ø, it
is easy to calculate that Jzy <0 V250 gaussian distribution.
exp( (x)
Jdzı
where & is the cdf of standard 2-
d
20
Thus S21>0 V2na exp( (x) 212
(21 2)2
}dl = 1 -S2,<0 V2n0
exp
}dn = ( 20 and
20
20
Equation (b) is satisfied.
Proof done.
B PROOF FOR THEOREM 3,4
Lemma X and Y are independent and follow gaussian distribution: X N(xx, Ex) and Y ~ N(px, ), Then: Exy([IX Y11²) = Ilux uy tr(Ex + Ex).
B. PROOF FOR THEOREM 3
Before the derivation, define some notations for better presentation. Define the labels of Xclea
as Y ylean and the labels of X as muiisy Under the label noise, it is easy to verify P(Y clean = 1) =
P(Y=1)-(1-e]) PI (1--- and P(ynoisy 1)
P(Y=0)-e- +YY==eee Let p1 P(yckea 1),
F22 PP = 1), g((()) and h(f(X)) to be simplified as g and hx(X).
In the case of binary classification, (x) is one dimensional value which denotes the network prediction on x belonging to Y 1. Le can be written as:
Exdion x [X gf (X (a |||
(X hf ((niniy))) 3
m
m2
denoted as (e X
(a) E xdeas yreken ( clean noiss) (endy yaido
P1 PP- E Xddon u V (Xetean x -1-1100 (1- P2 E Xcen x V/ (Xeeen X
+ Pi (1- p2) E "xaee y V (eeea muisy + (1 - pi) (1 p2) Exaan, x v clean mobsy
(a) is satisfied because V (Kckan Xnoisy) is irrelevnt the labels. We derive V (Xelaa X follows:
where mi and m2 normalization terms from Equation
16

---

Under review as a conference paper at ICLR 2022
E yda x Vi V (X )
gf (x
| 11s(X )13
(b) E Exten x
m1
12
of (X
Ihf(Xdem
( )12
(c) Xxdon Xpily
)2
mi
72
(d)
gf(
1
1X X
- (x Doois
)|1 ))²
X X
m1
m1
m2
(b) is satisfied because from Assumption 2 DNN has confident prediction on clean samples. (c) is satisfied because g( (x(x)) is one dimensional value which ranges from 0 to 1. From Assumption (x) and h((x-) follows gaussian distribution with parameter (F1, Σ) and (12, E). Thus
maisy )11 = || 12ll + tr().. according to Lemmal we have IE Xyten x || ( fxx(f((4
Similarly, one can calculate E Exttin X I - hf(X )12 tr(E). can be seen that
(d) is function with respect g [ (x ). Similarly, V (Xeclem x is also function with respect to g g (X4 while V (x x and (XX (X x are functions with respect to gx(x Denote d(+,+)= E "x f hx (x+ )|² After organizing 1/r (X (X X X and
/ (Xccaa x x we have:
min P1 P2 E xdo x 10 (X xx x + (1 -P1) P2 Ex x ( (Xrelean moiky gf(X
min (E (Exain gf (X maisy gf(
(15)
- -(2-pi mi . d( d(+++)) +)
mi d)(, E Ex (x + 2 (1-p1))
me
m2
+ constant with respect to g (X
Note in Equation 15) use (IE xenio gf g (x )² to approximate Ex gf (X (X.mi since from
Assumption
var (orggg((() (( + O. Now we calculate mi and m2 from Equation 4
mi = P1 P2 (1 E x gf (X + (1 p1) P2 E Exsio g.
(16)
P1 (1 - (1))) (1
E3 m g f(X moisy + (1 -P): (1 p2) E Ex gf ( moisy)
p2) m2 = P1 P2 d( +, +) + (1 - PP) (n)).p P2 d( d(-++) +) + -p) --)).d(( -(+,-)+(( p12)1-p2) . -((--))
Under the condition of P(Y = 1) = P(Y = 0),
c+. we have p1 P2
m2 +1 mi 2' which is constant with respect to E gx gf (X and
E Exmir gf ( in Equation 16.. Thus Equation [15 is a quadratic equation with respect to
E Exma gf (X Then when Equation 15 achieves global minimum, we have:
E yan gf (X
mi (p) d( +) - (1 pI) d(-,+))
m2
(17)
2 2
I 1 12
B.tr(3)
Similarly, organizing V (Xelean x and V (Xeelen x gives the solution of Exuan gf (f( (X
17

---

Under review as a conference paper at ICLR 2022
E Xanin of (X (x
pp
m1
(p1 d(-,--)(11- -1 .ppppp).. d(++-))
m2
(18)
2
2 + 1 -2
8.tr(2))
Proof Done.
B.2 PROOF FOR THEOREM 4
We first refer to property of Mutual Information:
I(X;Y) I(+(();
(19)
where th and & are any invertible functions. This property shows that mutual information is invariant invertible transformations (Cover) 1999). Thus prove the theorem, we only need to prove that
& in Theorem must be an invertible function when Equation (5) is minimized to O. Since when invertible, (((((()) = (MM((()))
b(((()))
We prove this by contradiction.
Let 4 h(f(x))) and 8; = ((f()))). Suppose & is not invertible, then there must exists 8; and 8j where * Sj which satisfy t, = {(s;) = fi. However, under this condition, f; tj 0 and 8i - 8, * Equation (5) can not minimized to 0. Thus when Equation (5 minimized 0, must be an invertible function.
Proof done.
B.3 PROOF FOR LEMMA I
By the independence condition, Z = also follows gaussian distribution with parameter (μx Hy, Σx + ).).
Write Z as 2 = μ + LU where is a standard gaussian and μ = μx - py, LL Ex + Ey. Thus
1 zT 1 j -U
(20)
Since is standard gaussian, BE()) 0. We have
[[| 211)) = n. L
n² P E E ((T
hid
( 1TT 4 E(I L)RK
(21)
=T + L)
-|ıx 112 + tr Ey)
(a) is satisfied because U standard gaussian, thus E(U2)) = and (UU,U) (b: +1.
Proof Done.
C ILLUSTRATING DOWN-SAMPLING STRATEGY
We illustrate in the case of binary classification with e4 te_ 1. Suppose the dataset is balanced, at the initial sate, e, > e... After down-sampling, the noise rate becomes et, and et.. We aim to prove two propositions:
18

---

Under review as a conference paper at ICLR 2022
Proposition 1 If e+ and are known, the optimal down-sampling rate can be calculated by e+ and e. to make e = =
Proposition 2 ef and e- are not known. When down-sampling strategy is to make P(Ý = 1)
P(Y 0), then 0 < et - < E+
Proof for Proposition / Since dataset is balanced with initial e+ > e-, have PP(Y 1)
P(Y 0). Thus sown-snm is conducted at samples whose observed label are 0. Suppose the
e- ret and E=_ -((---)te_" If e we random down-sampling rate is r, then e = tree
have:
r.c.
(22)
e + r +r.04 84
r (1-e_)+ee
(1-c1) Thus the optimal down-sampiln rate V V (1-e-)) which can be calculated if e_ and e+ known.
Proof for Proposition 2 If down sampling strategy is to make P(Ý 1) P( = 0), then 1-e+te- 6 + c_, we have 1-c Thus e can be calculated as:
(ct + 1 ---
4 1 ct +r
(1-e+ .e4
(1 (1- e1 + e+ (1 - e, +e-)
Denote a te- tec
1-e+te-
Since e+ > e-, - e_ + > - -4 + te.,
1-e-te-
1-4+te-
=1
te- +c-
Similarly, et can be calculated as:
e_ +r (1-
(1- +e4
e_ (1 -1--- +e+) + ([---))
+e_)
Denote 8 = (1-6 to ]+(l-e >(1-e| Since e_ > >e,, 1 - -e- + > 1 - et ee,,
1e4
B = -1-e te 1+(1-E )(--e++) e (1-e +el)+(l-e )(1- tel) = 1. Since e+ < <6
l-e_te+
1-E_te+
and B- > e-, have e_= _"= a..ee.4 e+ B.e_ < e4 -
Next, we prove ex > e", following the derivation below:
4 >
r.ee
6_
+r.
+r. e_.
e_ (1 e+
e e_)
(23)
1 64 =e_
e_ (1 - e+)
1 e_ +e+
c . e_)
e+ (1 - e4) c4) + e_ .e
e_ e
c_ c_. - --)
et
e_
c Since we have assumed e_ < 6, and I-c+ -e_ (1 - -e--)- - T-c-
Let f(++) = =4 (1 (1- - e+) +
e_ +, < 1. Thus proving et > e" is identical to prove f(,)) > 0 when e_ e4 1 c..
Firstly, it is easy to verify when e, = e_ or or., 1- c.. f(e+) = 0. From Mean Value Theory, there must exists a point en which satisfy ('((o) = where e+ <ep < 1 1 Next, we differentiate
19

---

Under review as a conference paper at ICLR 2022
decreased gap by down-sampling
0.035
0.2
e_ 0.3
0.030
6.020
0.015
0.010
0.000
0.6
0.7
e+
Figure 6: Visualizing decreased gap by down-sampling strategy.
f(e+) as follows:
2 22 e1+)) f(e+) (1 (4- (1--)) e (1- (1-e_) -2
(24)
(1- (1 - e__
can be verified that f'(e-) (1-e 11-e)))((-e) and f'(1 e-) -- -1((1--)) = 0.
Further differentiate ((()), get when e 1 ((1 --)- e ("((e) < and when e4 > - ((1 - --l- e f( > 0. Since e_ < 6+ and e_ + < 1, we have e and i.., e- (1- c2) < c-, i.e. 1 ((1 e_) ea ) locates in the point between
and e.. Thus, when e__ <e < 1 (01-(((ee)))))
((+)) is a strictly concave function and <44 < e_, [(++ is strictly convex function.
when 1 - (e) 2
Since f'(e_) and (1 e.) 0, eo must locates in the point between e and ((1 (1- e²)$ which satisfy ('e) = 0. Thus when e... < e+ < en. f(4)) monotonically increases and when eg e+ <1 e-, [(e.. monotonically decreases. Since ((e_) f(1 -e_) 0. We have f(e4) > when 6_ < 6 <
Proof done.
We depict figure in Figure better show the effect of down-sampling strategy. It can be seen the curves in the figure well support our proposition and proof. When e e large, down-sampling strategy to make IP(Y = 1) P-Y 0) can well decrease the gap even we do not know the true value of e_ and e4. Thus down-sampling strategy can approximate the condition in Theorem [ which explains why CE with fixed encoder can also work well on instance-dependent label noise in Table
D MORE DISCUSSIONS AND EXPERIMENTS
D.1 WHETHER OR NOT TO FIX THE ENCODER
Nodet et al. (202 experimentally test many methods and losses based on the pre-trained SSL features and draw the conclusion that the encoder should not be fixed during fine-tuning. We argue that this conclusion is not complete and neglects the specialty each loss and the gap between SSL and SL varies for each dataset. First, the robustness of CE in Equation (T) is based the fact that CE is a calibrated and convex loss respect to the linear classifier. However, the losses tested by Nodet al. (202T) do not all satisfy this property when performing fine-tuning. For example, GCE
20

---

Under review as a conference paper at ICLR 2022
Table 2: Comparing CE (fixed encoder) with robust losses CIFARIO
Symm
Method
0.2
0.4
0.6
0.8
SCE
87.63
85.34
80.07
53.81
APL
89.22
86.05
79.78
55.06
Peer 90.70
88.29
82.10
33.03
CE (fixed encoder)
91.06
90.73
90.2
88.24
Zhang & Sabuncu 2018 is not convex with respect linear classifier. Second, if the gap between SSL and SL large, then the encoder should not fixed since supervised features can exhibit better performance. However, if the gap between SSL and SL is very small, it is guaranteed by Theorem and Theorem 2 that CE on noisy dataset can approximates SL on clean dataset when encoder is be seen in Figure (a). Thus, the above argument suggests that whether
fixed. The observation to fix the encoder or not is determined by the gap between SSL and SL for given dataset.
D.2 PROBLEM OF OTHER ROBUST LOSSES
Many robust losses have been proposed in recent years. For example, some losses (Natarajan al. Liu & Guo 2020) are also proven statistically robust which satisfy:
2013
min Ex,y (CE(g(f(X)) Y)] min x(lggx((()((( P)]
(25)
9.
where l is the robust loss and f. g in Equation (25) are simultaneously updated. However, Equation 25) not only needs sufficient training samples achieve small learning error c, but also does not consider the optimization difficulty of deep neural networks. Because of the great non-convexity of f. it is very hard guarantee that f and g will converge global optimal when noise rate varies. Thus the performance of É in Equation (25) may suffer severe performance degradation under high noise rate. However, for Equation (T). when f is fixed, CE loss with respect g strictly convex which is very friendly for optimization.
To further verify our analysis, we compare CE with SSL pre-trained encoder with some robust losses including SCE (Wang et al. 2019), APL (Ma et al. (020) and Peer loss (Liu & Guo 2020) on CIFARIO with symmetric label noise. The results are shown Table 2 It can be observed that when noise rate is very high, the performance of these robust losses will drop significantly. However, when SSL pre-trained encoder is fixed, loss with respect to linear classifer strictly convex. Thus the performance is more stable when noise rate increases.
D.3 THE EFFECT OF DISTANCE MEASURE IN EQUATION
4
In this paper and experiment, we use le norm to calculate the feature distance between SL features and square la norm to calculate the distance between SSL features. This choice can lead good performance from Theory 3 and Figure 5 Practically, since structure regularization mainly cap- tures the relations, different choice does not make a big effect on the performance. We perform an experiment in Figure 7 which shows that the performance of both types are quite close.
D.4 ABLATION STUDY
In Figure 4 SSL training to provide SSL features to regularize the output of linear classifier g- However, SSI. training itself may have positive effect on DNN. To show the robustness mainly comes from the regularizer rather than SSL training, we perform an ablation study in Figure From the experiments, it is the regularizer that alleviates over-fitting problem of DNN.
D.5 VISUAL.IZING FEATURES OF DNN TRAINED BY CE WHEN ENCODER IS NOT FIXED
We visualize the features DNN trained by CE when encoder is not fixed. Figure 9 shows the Tsne visualization of features before linear classifier under different noise rate. It can be seen, analyzed
21

---

Under review as a conference paper at ICLR 2022
(e) 20% symm unit: SimCUR)
10) 40% symm (init SimCUR)
tc 40% symm (int:: Random)
(d1 60% symm Unit: Random)
40-
60
60
80
40
40
40
60
20
20
Figure 7: Comparing difference choices of distance measure in Equation (4). Type 1 denotes using norm calculate distance between SL features and square la norm to calculate distance between SSL features, which is adopted in our paper. Type 2 denotes using l2 norm to calculate distance for both SL and SSL features.
symm Random)
symm Randem)
100
100
60
test
20
20
InfoNCE
CE + InfoNCE
CE InfoNCE Regullarizrr
CE InfoNCE Regularizer
20
40
60
80
100
120
140
20
40
60
100
120
140
training
tra
Figure 8: Ablation study of using the regularizer to train DNN on noisy dataset.
CIFAR10 Festures under 20% relse
(0) CIFARIO Features under 60% noise
(c) CIFAR1O Features under 80% naise
40
40
10
30-
20
20
10
10
-10
10
-40
-10
-20
-30
Figure 9: Tsne visualization of supervised features before linear classifier on CIFARI0 under 20%, 60% and 80% symmetric label noise, respectively.
in the paper, due to non-convexity and great over-fitting capability of f (encoder), the features of f(X) have already been contaminated by the label noise before linear classifier, thus CE on vanilla (X, Y) is not robust.
D.6 THE EFFECT DIFFERENT SSL-PRETRAING METHODS
Our analyses and experiments not restricted to any specific SSL method. Experimentally, other SSL methods are also adoptable to pre-train SSL encoders. In Figure SimCLR (Chen et al. 2020 adopted pre-train SSL encoder. For comparison, pre-train encoder with Moco (He al. 2020) on CIFAR 10 and fine-tune linear classifier on noisy labels in Table
22

---

Under review as a conference paper at ICLR 2022
Table 3: Comparing different SSL methods CIFARIO with symmetric label noise
Symm label noise ratio
Method
0.2
0.4
0.6
0.8
CE (fixed encoder with SimCLR init)
91.06
90.73
90.2
88.24
CE (fixed encoder with MoCo init)
91.55
91.12
90.45
88.51
can be observed that different SSL methods have very similar results, which further verifies our Theorems in the paper.
D.7 EXPERIMENTS ON HUMAN-ANNOTATED NOISY DATASET
We perform experiments CIFARI00-N (A challenging
noisy dataset) (Wei
et al. 202T) with structure regularization. From Table 4 it can be observed that by using SSL Teatures, Structure Regularization can outperform many benchmark methods real-world human annotated noisy dataset.
Table 4: The best epoch (clean) test accuracy for each method on CIFAR100-N.
Method
CE
Forward 7
GCE JoCoR
Peer Loss
ELR
Structure Regularization
Acc.
55.50
57.01
56.73
59.97
57.59
58.94
61.12
E DETAILED SETTING OF EXPERIMENTS
Datasets: We use DogCat, CIFARIO and CIFAR100 for experiments. DogCat has 25000 images. We randomly choose 24000 images for training and 1000 images for testing. For CIFARIO and CIFARIOO, we follow standard setting that use 50000 images for training and 10000 images for testing.
Setting Section 3 SimCLR is deployed for SSL pre- training with ResNet50 for DogCat and ResNet34 for CIFAR10 and CIFAR100. Each model is pre-trained by 1000 epochs with Adam optimizer (Ir = le-3) and batch-size is set to be 512. During fine-tuning, we fix the encoder and only fine-tune the linear classifier noisy dataset with Adam (Ir 1e-3) for 100 epochs and batch-size set to be 256.
identical Section 3, except when train from scratch (random initialization). learning rate is set to be 0.1 at initial state and is decayed by 0.1 50 epochs. In Equation (7). we use MSE loss for measuring the relations between SL features and SSL features. However, since MSE loss may cause gradient exploration when prediction is far from ground-trutte we use smooth I loss instead. Smooth I loss an enhanced version of MSE loss. When prediction is not very far from ground-truth, smooth I loss is MSE, and MAE when prediction is far.
Setting in Section 4 The basic
The code with running guideline has already been attached in the supplementary material.
F GENERALIZE THEOREM I TO MULTI-CLASS CLASSIFICATION
Theorem For K-class classification, let gi aigninges E=o(E(((((())) Y)}, g2
arg mingeg (x)) Then if symmetric noise ratio c < K-1 for each sample Zi. have
K
arg max 9i (xxz)) arg max (x:()):))
(26)
where f is fixed encoder and is the linear classifier, g(f(-)) denotes the network output after softmax layer.
23

---

Under review as a conference paper at ICLR 2022
proof: Following the proof for Theorem suppose 9i
Bayes and 92 are Bayes optimal classifiers
Bayes
under (Xf()), Y) and (X), Y), respectively. We aim to show 91 Lee. the decision boundaries are the same.
Bayes and 92
Bayes
are consistent.
Denote X; x| i. Define pz()) to be the distribution of f(X)). According to the bayes rules:
P(Y = =p))
P(Y = i|x)
(27)
Σラ=1 P(Y = j) P)(
Let X' be the space of f(X):
X = 1 U X/2 ... UUX
(28)
where X = fæ E f(X)PP(Y = =|/> > P(Y | , i + j}.
Let (x) be the distribution of samples whose observed labels i under symmetric label noise rate e. We have:
(x(x) my P(Y (1--)) (z)(z) -P(Y= =)) K 1 pz)x)
(29)
where my is the normalization term to make distribution valid.
Specifically, mi
P[ Exz (((=))xンー) Under label noise, IPY =) P(Y i) (1 E) + Σe P(Y
i) - = 1. According to the bayes rules, we have:
m.
P( = i) -1) (2(
P( = =1bz)
(30)
Σ/= P(Y = 1)(())
Let X' = X U. ... U.. ..K where X = fe E f(X)FF( = i1æt) > PPÝ = j + j}. To
Rayes and 92 prove 91
Rayes are consistent, it is equivalent to show X, XS when K-1
K
Without loss of generality, we prove X; X1 when < K-1
K
When j # 1, we have:
P( = 1/x) P(Ý- j|æ)
(1 (1--- Fー P(Y =) 1) pi (x) - (1 -1-6- K-T
P(Y = (z(
(31)
normalization
Since X1 = fx E P( = 1/z) >
- if c K-1 > 0, i.e.
K-1 K
Proof Done.
G ILLUSTRATING BAYES CLASSIFIER IS NOT CONSISTENT UNDER ASYMMETRIC AND INSTANCE LABEL NOISE
Asymmetric label noise:
We give an example K-class classification (K=3), Following the def- inition and notation in Section suppose the asymmetric label noise ratio is e, which is generated between two adjacent label 2.2 3.3 1), we have:
P(Ý = 1|z) - P(Y = 2|x²
(1-c) PPY 1) (1 (2.c- 1) :)) 22)-(())) - P(Y =3) pp(2))
(32)
normalization
24

---

Under review as a conference paper at ICLR 2022
P(Y = 1|x) - -P(- = =3/z)
(1 (1-2. 2.) = 1) p)(()) P. = 2) (2(2)22)))))) (1 -) c) P(Y = =3) p2(
(33)
normalization
Since Xi = f E ((X))P(X = 1/x) > P(Y = j(=)., #1 and X = (a E [(X)P(( = 1| >
P(Ý [[==j + 1} it is easy to verify X = Af only when = 0. Thus asymmetric label noise does not induce consistent Bayes classifier. We also provide an experiment on CIFARIO further verify our analysis in Table 5 It can be seen that for asymmetric label noise, there is only marginal improvement compared to symmetric label noise in Figure
Table 5: Comparing CE (fixed encoder) with baseline (CE with random initialized unfixed encoder) on CIFARIO with asymmetric label noise
Asymm label noise ratio
Method
0.1
0.2
0.3
0,4
CE (baseline, unfixed encoder with random init)
90.69
88.59
86. 36.14
80.11
CE (fixed SSL pre-trained encoder)
90.85
89.92
88.25
82.46
Instance- deeendeed label noise behaves like symmetric label noise, i.e. the noisy labels may exist in all the classes. However, the noise ratio for each class not the same and dependent on the features. From Section F when noise ratio for each class not the same, the Bayes classifier is not consistent. However, from Table [ and Section C we the- oretically and experimentally show that simple down-sampling strategy can approximate condition in Theorem I and Theorem 5 which is helpful for instance level label noise.
Instante-dependent label noise:
25


Reviews:
[{"review_id": "z2YOhZV2D7d", "reviewer": "Reviewer_GRMb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros: \nProviding a theoretical understanding of how self-supervised pre-trained features help to improve network resistance against noisy labels.\nThe paper conducted experiments that supported their theoretical claims\n\nCons:\nThe experiments were conducted under limited noisy ratio levels from 20 to 60 percent. No extreme noise levels were checked, i.e., >=80%. In addition, only symmetric synthetic noise scenario is consider, while more complicated asymmetric and real-life noise cases are not considered. \n", "summary_of_the_paper": "This paper studies how self-supervised pre-training impacts the resistance of the neural network to noisy labels. The paper provides both theoretical analyses and numerical experiments of their study.  The main contribution of the provided study is (i) given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise (ii) providing insights for how knowledge distilled from SSL features can alleviate the over-fitting problem.", "summary_of_the_review": "This paper aims to provide a theoretical understanding of why self-supervised pre-trained features help neural networks improve resistance under noisy labels. While the theoretical claims look solid, the important, more general settings outside of symmetric synthetic noise are missing (please see \"Cons\"). \nHence, in the pre-rebuttal phase, my score is 5 and it can be re-evaluated once the authors will address my concerns \n\nPost rebuttal decision:\nI am grateful to the authors for their responses to the reviewers' concerns during the rebuttal phase. Nevertheless, the paper presents an important study the experimental setup is not comprehensive enough to support the paper's concussions, Hence, I vote to reject the paper in its current form and maintain my current score, I encourage the authors to expand the experimental study by my and other reviews comments. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "yBRuk2I_Vem", "reviewer": "Reviewer_PASS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. Using SSL representation can help learning with noisy labels is an interesting discovery. Using theirs theories, the authors also motivate why fixed encoders are important\n2. The authors motivate a regularizer from knowledge distillation to further improve the training performance.\n\nWeakness:\n\nTheoretically, (1-2); Experimentally (3-6); Clarity(7)\n1. The novelty of the theorem 1 and 2 is limited. It is trivially true that when you have majority labels that are not noisy, you can learn an optimal classifier. \n2. The authors need to explain or cite when SSL give you representations that satisfy assumption 1.\n3. It's not clear whether the authors use a validation set. Validation set is important for model selection in machine learning. The authors seem to choose the best test performance directly.\n4. The motivation of downsampling is unclear. The authors mentions that downsampling can help satisfy the condition in Theorem 1. However, with downsampling, the target distribution may change and the Bayes classifier may be different. This may change the conclusion in theorem 1. Does downsampling also help for the baseline CORES?\n5. Though the authors claim that they do not aim to get SOTA results, more datasets should be introduced. At least, I would like to see the results on cifar 100.\n6. Somehow the authors mention the regularizer in section 4. but do not use it in Table 1. Why is that?\n7. The paper can be re-organized. The authors describe the use of SSL first and then describe some experiments. After that, the authors describe the regularizer and some other experiments. They look like two separate parts. I suggest the authors put all the experiments and discussions in one section. ", "summary_of_the_paper": "The authors illustrate why self-supervised learning can help learning in training with noisy labels.\nThe main contributions are:\n1. The authors illustrate theoretically why learning good representation can help learning with noisy labels\n2. The authors describe why fixed encoders are important.\n3. The authors motivate a regularizer between the SL and SSL.", "summary_of_the_review": "The authors propose an interesting idea that SSL can help learning with noisy-labels but due to the theoretical, experimental and clarity concerns above, I think the paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "roxpOVufe_Z", "reviewer": "Reviewer_GhTK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: (1) Theoretical analysis on one kind of noisy-label learning method.\nWeakness: (1) The empirical analysis could be improved (see below).", "summary_of_the_paper": "The paper performs a theoretical analysis on one kind of noisy label learning method. The paper also presents an empirical analysis.\n", "summary_of_the_review": "The experiment needs to be improved. The paper performs empirical analysis only on one setup. It could make the paper better if the authors could perform analysis on other settings (e.g., asymmetric noise, etc). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SYPTdTq3O7G", "reviewer": "Reviewer_WCJg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Understanding how self-supervised learning can be used, and particular with regards to the impact of fine tuning representations is an important question, as is dealing with label noise.\n- The paper has quite a large scope of topics that it tries to connect.\n\nWeaknesses:\n- I find the paper somewhat fragmented, jumping between different ideas and topics without going too deep enough into any single one. It's not really clear to me that section 4 is related to section 2/3, they seem to studying quite distinct subjects?\n- I find the lack of analysis regarding the specific type of SSL algorithm you use, and the effects that that may have, a weakness. The authors sidestep this by assumptions 1 and 4, but as seen in Figure 2c) on standard small vision datasets these assumptions do not seem to hold (never mind imagenet). What effect does using different SSL methods have on the results, even empirically?\n- Many of the results do not seem particularly surprisingly to me and I'm not sure if the application to noisy labels is sufficiently novel. For example, it seems intuitive that one should not fine tune a pretrained NN if the fine tuning dataset is small, i.e. that there isn't much 'signal'. Likewise, it seems intuitive that one should do the same when faced with noisy data.\n- It is not clear to me how you train h, and with what objective, in eq 3? If h is the regulariser surely it cant be trained towards g? Please can you explain this to me.\n- I think the setting for fig 3 is a bit strange. Surely you should not stop training at the best test accuracy but rather for a fixed number of epochs to show the 'damage' of training with different noise levels for a fixed amount of training. It seems like for higher noise rates that the you are taking very few epochs by stopping based on test accuracy. The results would look more in your favour too I imagine and reflect well on fixed encoder, given fig 3d.\n\nMinor comment:\n- Is X_+ for label 1 or 0, it is defined as label 1 but in Theorem 3 it is 0?", "summary_of_the_paper": "This paper studies the usefulness of self-supervised features when encountering data with noisy labels. It presents an array of ideas on this topic: including the question of fine-tuning pre-trained representations, using ideas from distillation as regularisation to improve generalisation. Some theoretical and empirical results are relating to the authors arguments.  ", "summary_of_the_review": "I do not recommend the paper for acceptance currently as I do not think it presents a coherent picture (of a very interesting topic) and also have some confusions about the presentation of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an analysis of the robustness of self-supervised learning (SSL) features to noisy labels in downstream supervised learning, and provides empirical verification of the results (mostly in the symmetric noise setup); a SSL regularization scheme is also analyzed (section 4). While the paper contains plausible insights, the reviews share similar concerns that the analysis is mainly based on the noise being symmetric, and that the SSL features already have good class separation and Gaussian clusters, which are strong assumptions. Given that the assumptions are not theoretically verified, and that there is not sufficient empirical results in heavy non-symmetric noise scenario on large benchmark datasets, the reviewers think the paper does not provide practical guidance for noise label learning in its current form.
Decision: Reject