Mitigating Memorization of Noisy Labels via Regularization between Representations
Authors: Hao Cheng
Zhaowei Zhu
Xing Sun
Yang Liu
Venue: ICLR 2023 poster (2023)
Keywords: learning with noisy labels
representation learning

Abstract: Designing robust loss functions is popular in learning with noisy labels while existing designs did not explicitly consider the overfitting property of deep neural networks (DNNs). As a result, applying these losses may still suffer from overfitting/memorizing noisy labels as training proceeds. In this paper, we first theoretically analyze the memorization effect and show that a lower-capacity model may perform better on noisy datasets. However, it is non-trivial to design a neural network with the best capacity given an arbitrary task. To circumvent this dilemma, instead of changing the model architecture, we decouple DNNs into an encoder followed by a linear classifier and propose to restrict the function space of a DNN by a representation regularizer. Particularly, we require the distance between two self-supervised features to be positively related to the distance between the corresponding two supervised model outputs.  
Our proposed framework is easily extendable and can incorporate many other robust loss functions to further improve performance. Extensive experiments and theoretical analyses support our claims. Code is available at https://github.com/UCSC-REAL/SelfSup_NoisyLabel.

Published as a conference paper at ICLR 2023
MITIGATING MEMORIZATION OF NOISY LABELS VIA REGULARIZATION BETWEEN REPRESENTATIONS
Hao Cheng" Zhaowei Zhu"i Xing Sun and Yang Liu University of California, Santa Cruz, Tencent YouTu Lab (haocheng, zwzhu, yangl in @ucsc. edu, (winfred. sun) @gma il....... com
ABSTRACT
Designing robust loss functions is popular in learning with noisy labels while existing designs did not explicitly consider the overfitting property of deep neu- ral networks (DNNs). As result, applying these losses may still suffer from overfiting/memoriiingg noisy labels as training proceeds. In this paper, we first theoretically analyze the memorization effect and show that lower-capacity model may perform better on noisy datasets. However, it is non-trivial to de- sign neural network with the best capacity given an arbitrary task. To cir- cumvent this dilemma, instead of changing the model architecture, we decou- ple DNNs into an encoder followed by a linear classifier and propose to re- strict the function space of DNN by representation regularizer. Particu- larly, we require the distance between two self-supervised features to be posi- tively related to the distance between the corresponding two supervised model outputs. Our proposed framework is easily extendable and can incorporate many other robust loss functions to further improve performance. Extensive experiments and theoretical analyses support our claims. Code is available at https://githuu:
wttp://wwww...e////SEEEEEEEA/// Sttm///UOOORAAA/effuuuu. NoisyLabel Noiiyayaaal
INTRODUCTION
Deep Neural Networks (DNNs) have achieved remarkable performance in many areas including speech recognition (Graves et al., 2013), computer vision (Krizhevsky al. 2012: Lotter al.) 2016), natural language processing (Zhang & LeCun] 2015), etc. The high-achieving performance often builds on the availability of quality- aanotat- datasets, In real-world scenario, data annotation inevitably brings in label noise (Wei et al. 2022d which degrades the performance of the network, primarily due to DNNs' capability in "memorizing' noisy labels (Zhang al. 2016).
In the past few years, a number of methods have been proposed to tackle the problem of learning with noisy labels. Notable achievements include robust loss design (Ghosh al. 2017 Zhang & Sabuncu 2018: Liu & Guo 2020 Wang et al., 2021), sample selection (Han et al. 2018 Yu et al.
2019 2019/Cheng et al. [021] [iia et al. (021b) transition matrix estimation (Patrini et al.
|
2017 /Zhu
al. | 22221b] Xia et al. | 2020b and loss correction/reweighinnn based on noise transition matrix
(Natarajan et al. 2013 Liu & Tao. 2015; Patrini et al.) 2017 Jiang et al.) 2021 Zhu et al. 2021b
Wei et al. 2022a Zhu et al. (222cc). However, these methods still suffer from limitations because
they are agnostic to the model complexity and do not explicitly take the over-fitting property of
DNN into consideration when designing these methods (Wei al. 1021 Liu et al. 2022). In the
context of representation learning, DNN is prone to fit/memorize noisy labels as training proceeds (Wei et al. 2022d Zhang al. 2016), i.e., the memorization effect. Thus when the noise rate is high, even though the robust losses have some theoretical guarantees in expectation, they are still unstable during training (Cheng et al. 2021). has been shown that early stopping helps mitigate memorizing noisy labels (Rolnick et al. 2017 Xia et al.) 2020a). But intuitively, early stopping will handle overfitting wrong labels the cost of underfitting clean samples not tuned properly. An alternative approach is using regularizer to punish/avoid overfitting (Liu & Guo 2020 Cheng et al.
Equaal contributions.
Corresponding author; Yang Liu yangl u@ucac.

---

Published as a conference paper at ICLR 2023
2021: Liu et al. 2020), which mainly build regularizers by editing labels. In this paper, we study the effectiveness of a representation regularizer.
To fully understand the memorization effect learning with noisy labels, decouple the gen- cralization error into estimation error and approximation error. By analyzing these two errors, we find that DNN behaves differently on various label noise types and the key to prevent over-fitting is to control model complexity. However, specifically designing the model structure for learning with noisy labels is hard. One tractable solution is to use representation regularizers cut off some redundant function space without hurting the optima. Therefore, we propose unified framework by utilizing representation to mitigate the memorization effect. We list main contributions below:
We first theoretically analyze the memorization effect by decomposing the generalization error into estimation error and approximation error in the context of learning with noisy labels and show that a lower-capacity model may perform better on noisy datasets.
Due to the fact that designing neural network with the best capacity given arbitrary task requires formidable effort, instead of changing the model architecture, we decouple DNNs into an encoder followed by linear classifier and propose to restrict the function space of DNNs by the structural information between representations. Particularly, we require the distance between two sepe-supe features to be positively related to the distance between the corresponding two supervised model outputs.
The effectiveness of the proposed regularizer is demonstrated by both theoretical analyses and numerical experiments. Our framework can incorporate many current robust losses and help them further improve performance.
1.1 RELATED WORKS
Learning with Noisy Labels Many works design robust loss to improve the robustness of neural
networks when learning with noisy labels (Ghosh et al. 2017 Zhang & Sabuncu 2018 Liu & Guo
2020 Xu et al. 2019) Feng et al.] 2021 Yong et [ et al. 2022 Wei et al., 2022c b). GGhosh
al. 2017) proves MAE is inherently robust to label noise. However, MAE has severe under-fitiigg
problem. (Zhang& & Sabuncu 2018) proposes GCE loss which can combine the advantage of MAE and CE, exhibiting good performance on noisy datasets. (Liu & Guo 2020) introduces peer loss, which is statistically robust to label noise without knowing noise rates. The extension of peer loss also shows good performance on instance-dependent label noise (Cheng et al.) 2021; Zhu al. (021a). Another efficient approach to combat label noise is by sample selection (Jiang et al. 2018: Han et al.
2018. Yu et al.) 2019: /Nortccutt et al. 2021 Yao et al., 2020 Wei et al., [0200] Zhang et al. 2020 Xia al. 202 These methods regard "small loss" examples clean ones and train multiple networks to select clean samples. Semi-supervised learning is also popular and effective on learning with noisy
labels in recent years. Some works (Li et al. 2020; Nguyen et al. 2020) perform clustering on the sample loss and divide the samples into clean ones and noisy ones. Then drop the labels of the "noisy samples" and perform semi-supervised learning all the samples. However, the semi-supervised pseudo labels can cause disparate impact on different groups of data /Zhu et (2.226) (20226). Recently,
some works apply self-supervised learning handle noisy labels (Ghosh & Lan 2021 Li et al.
2022a
Wei et al. 2023). Our work can also explain some findings from (Ghosh & Lan 2021).
Knowledge Distillation Our proposed learning framework is related knowledge distillation (KD). (Hinton et al. 2015) shows that a small, shallow network can be improved through teacher-student
framework. Due to its great applicability. KD has gained more and more attention recent years and numerous methods have been proposed to perform efficient distillation (Mizadeh et al. 2020 Zhang et al. 201 2019). However, the dataset used in KD assumed be clean. Thus it is hard to connect KD with learning with noisy labels. In this paper, we theoretically and experimentally show that a regularizer generally used in KD (Park et al. (2019) can alleviate the over-fitting problem on noisy data by using DNN features which offers new alternative for dealing with label noise.
2 PRELIMINARY
We introduce preliminaries and notations including definitions and problem formulation.
Problem formulation Consider a K-class classification problem on a set of N training examples denoted by D (()nn))))nn)))) where [N] == (1,2, NN is the set of example indices.
2

---

Published as a conference paper at ICLR 2023
Examples (In, Vn) are drawn according to random variables (X from joint distribution D. The classification task aims to identify a classifier C' that maps X to Y accurately. In real-world applications, the learner can only observe noisy labels ÿ drawn from Y|X (Wei et al. 2022d). e.g., human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly. The corresponding noisy dataset and distribution are denoted by D = {(I,. (m) I ne[N]] and D. Define the expected risk of a classifier C as R(C) = Ep [1(C(X) + Y)]. The goal is learn classifier C from the noisy distribution D which also minimizes R(C), i.e., learn the Bayes optimal classifier such that C* (2) = arg [ P(Y = i|Xx = x))
Noise transition matrix The label noise of each instance is characterized by Ti, P(Ý y Y i). where T(X) is called the (instance-dependenn)) noise transition matrix Zhu al. 20216 Li et al. 2022b). There are two special noise regimes (Han et al. 2018) for the simplicity of theoretical analyses: symmetric noise and asymmetric noise. In symmetric noise, each clean label is randomly flipped to the other labels uniformly w.p. c, where is the noise rate. Therefore, T = 1 --6 and Ty K-I, i F j. i E [K]. In asymmetric noise, each clean label is randomly flipped to its adjacent label, i... T.
e, Ta + T(,(+1)< I, where (i + 1)x := mod K
Empirical risk minimization The empirical risk on a noisy dataset with classifier C' writes as 1 EnENNy (Z,)) (n). where is usually the cross-entropy (CE) loss. Existing works adapt to make it robust to label noise, c.g., loss correction NNtarajan et al., 2013 Patrini et al., 2017). loss reweighting (Liu & Tao 2015), generalized cross-entropy (GCE) (Zhang & Sabuncu 2018), peer loss (Liu & Guo 2020). f divergeence (Wei & Liu 202T]. To distinguish their optimization from the vanilla empirical risk minimization (ERM), we call them the adapted ERM.
Memorization effects of DNNs Without special treatments, minimizing the empirical risk on noisy distributions make the model overfit the noisy labels. As result, the corrupted labels will be memorized (Wei et al. 2022d Han et al. 2020 Xia et al.) 2020a) and the test accuracy clean data will drop in the late stage of training even though the training accu- racy is consistently increasing. See Figure [ for an illustration. Therefore, it is important to study robust methods mitigate memorizing noisy labels.
Accuracy with training spachs
300
Truin Acc, bym 2.A
That Sym
Test Acc, Sym
The rest of the paper is organized as follows. In Section we theoretically understand the memo- rization effect by analyzing the relationship among noise rates, sample size, and model capacity, which motivate us to design a regularizer to alleviate the memorization effect in Soction/4byy restricting model capacity. Section 5 empirrcalyy validates our analyses and proposal.
Outline
Training and test accuracies on CIFAR-10 with symmetric noise with noise rates 0.4 (blue curves) and 0.6 (red curves). We use ResNet 34 for conducting the experi- ments (See detailed setting in Appendix E)
Figure 1:
3 IMPACTS OF MISLABELED DATA ON DNN PERFORMANCE
We quantify the harmfulness of memorizing noisy labels by analyzing the generalization errors on clean data when learning on a noisy dataset D and optimizing over function space C.
3.1 THRORETICAL TOOLS
Denote by the optimal clean classifier Co = arg mincec [D)((((X), Y)). the optimal noisy classifier Cp arg mincec (C)),(((X), F)], and the learned classifier on the noisy dataset C arg mincee EnE[n] Fn)). The expected risk w.r. the Bayes optimal classifier C* can be decomposed into two parts: Ble(C, (f()), Y)] - [X)(C"(X),, Y) = Errorg(Cp, Cp) + E C*), where the estimation error Errorg and the approximation error Error can be written as Error Cp) E (X), Y)) - [E(C (2),Y) We analyze each part respectively.
(II(C;(X),,)) Y)] E (((Co(X), Error, C*)
Estimation error We first study the noise consistency from the aspect of expected loss.
3

---

Published as a conference paper at ICLR 2023
Definition 1 (Noise consisteney). One label noise regime satisfies the noise consistency under loss
i if the following affine relationship holds: E.g({(((x)))) (1 72, where
71 > 0 and are constants in a fixed noise setting.
Following the probabilistic decomposition of label noise Natarajan et al. (2013); Ghosh et al. (0017); Cheng et al. (202I), we have Lemmas I and 2.
Lemma general noise regime with noise transitions Tiş(X): : P(Ý = j|Y X) can be decoupled to the following form:
E [(((())))) -.. ED)((C(()))
Σ Σ P(Y =
(E[]]
where Uz₂(X) = =T,}) - T
minx, T
Lemma D shows the general instance-dependent label noise hard to consistent since the second term is not constant unless we add more restrictions to T(X). Specially, in Lemma we consider two typical noise regimes for multi-class classifications: symmetric noise and asymmetric noise.
Lemma 2. The symmetric noise consistent with 0-1 loss: E.g [e(C(X), 8) Ex(((((()) VY]
72. where n
ck কं. The asymmetric noise is not consistent: BB [(((())))) K c) Ep [x(((((X), Y)] E P(Y =)) (i 1)x)].
With Lemma 2 we can upper bound the estimation errors in Theorem 1
Theorem 1. With probability at least 1 8, learning with symmetriicasymmetticc noise and 0-1 loss has the following estimation error:
[C] log(N e/(CI) + 1og(8/))
Errorg (CD.CD)= ( 16
Bias Cp)
(N(1-E)
where e 2.718 is the base of the natural logarithmms, |C] is the VC-dimension of function class eK K-1 for
(Bousquet al. 2003 Devroye et al. 2013). The noise rate parameter satisfies
symmetric noise and ffrr asymmetric noise. The bias satisfies Bias Cp, CT) 0. symmetric noise and Bias(CD, Cp) (p)== = I EiE[K] IP(Y =))pp==(((C(()) 1)k) e((p(x),
1)x) for asymmetric noise.
Approximation error Analyzing the approximation error for an arbitrary DNN is an open-problem and beyond our scope. Generally, according the trade-off between approximation error and estimation eтror (a.k.a. bias-complexity trade-off (Shalev-Shward & Ben- Ben-David, (0144). a large function space C reduces the approximation error the cost of increasing the estimation error. Note an alternative of the generalization bound is using the Rademacher complexity of C. Both bounds help reveal the tradeoff between approximation error and estimation error w.r.t C. We use VC-dimension since it shows clearer relationship between model capacity C (numerator) and the effective number of samples ((1 - e)² (denominator).
Trade-off From Theorem | and the above analyses, the bias complexity trade-off is more severe in the presence of label noise. In the case of symmetric label noise, a larger [C] will lead to smaller approximation error but at the cost of larger estimation error given large N. Although the trade-off may not be remarkable in traditional learning with clean data (e = 0), it is critical when label noise exists since the existence of will significantly reduce the effective datasize from N N -)? In practice, it is non-trivial to find the best function space or design the best neural network given an arbitrary task for reducing the total generalization error. We will introduce a tractable solution the following sections.
3.2 DECOUPLED CLASSIFIERS: FROM FUNCTION SPACES TO REPRESENTATIONS
One tractable way to restrict the function space is fixing some layers a given DNN model. Particu- larly, can decouple C into parts: C / g. where the encoder extract representations from
4

---

Published as a conference paper at ICLR 2023
Random Initialization
Path-1: Traditional learning
Path-2: encoder
Path-3: Fixed encoder
Converged Classifier
Encoder
Figure 2: Illustration of different learning paths (distinguished by colors). The curve with arrow between two green dots indicates the effort (e.g., number of training instances) of training a model from one state to another state.
raw features and the linear classifier maps representations to label classes, i.e., ((x))g(((X))) Clearly, the function space can be reduced significantly if we only optimize the linear classifier g- But the performance of the classifier depends heavily on the encoder By this decomposition, we transform the problem of finding good function spaces to finding good representations.
Now analyze the effectiivneess of such decomposition. Figure iillustrates three learning paths. Path-I is the traditional learning path that learns both encoder f and linear classifier g at the same time (Patrini al.) 2017). In Path-2, pre-trained encoder f is adopted as initialization of DNNs and both and g are fine-tuned on noisy data distributions D (Ghosh & Lan 2021). The pre-trained encoder f is also adopted in Path-3. But the encoder f is fixed/frozen throughout the later training procedures and only the linear classifier is updated with D. We compare the generalization errors of different paths to provide insights for the effects of representations on learning with noisy labels.
Now we instantiate function spaces C₁ and C2 with different representations. With traditional training or an unfixed encoder (Path-1 or Path-2), classifier C optimized over function space C₁ 9 with raw data. With fixed encoder (Path-3), classifier C' is optimized over function space G given representations fX)..
Symmetric noise Let Ci = Go C2 = Gl Denote the optimal classifier learned within the above two functions spaces by Chee and C respectively. Then the approximation errors of both cases can be denoted by Error, (Cg.F ,C*) and Error (C&) CC*). Assume x ,C*) Error (C C6 C*). Note the assumption holds generally for analyzing the bias-complexity trade-off; otherwise we should always prefer fixed encoder.
With the eτror proxy (E 6) in Theorem [ error, model capacity and noise rate in Corollary
we reveal the relationship among total generalization
Corollary When Error (CSO C*)) ECrorA(Oee C.)) the error proxy of the expected gener- alization error for the fixed encoder (9lf) is not greater than that for the unfixed one (G F) when
o e/
v log(4N c/19])
EK
16
1
K S $B(90F.91)
=
V2N
Errora CC*) - E ,C*)
Recall in above for hypothesis space denotes its VC-dimension. RHS is greater than 0 and LHS is decreasing with the increase of e. Corollary limplies that, for symmetric noise, a fixed encoder is likely better in high-noise settings. Section provides empirical results validate this claim. also provide empirical evidence in the Appendix D.4 that a shallow network (low capacity model, similar fixing the encoder) performs better than deeper network for high-noise noise regimes.
Other noise Based on Theorem I for asymmetric label noise, the noise consistency is broken and the bias term makes the learning error hard to be bounded. As result, the relationship between noise rate and generalization error is not clear and simply fixing the encoder may induce larger generalization error. For the general instance- -estadeedde label noise, the bias term is more complicated thus the benefit of fixing the encoder is less clear.
Insights and Takeaways With the above analyses, we know learning with an unfixed encoder is not stable, which is easier to be affected by noisy patterns and yields a worse result than a properly selected fixed encoder when the noise rate is high. Restricting the search space makes the convergence stable (reducing estimation error) with the cost of increasing approximation emors. This motivates us to find a way to compromise between a fixed and unfixed encoder. We explore towards this direction in the next section.
5

---

Published as a conference paper at ICLR 2023
SSL features
SSL training
teat
X
regularize
training
Y
festures
Figure 3: The training framework using representations (SSL features) to regularize learning with noisy labels (SL features).
4 COMBATING MEMORIZATION BY REPRESENTATION REGULARIZATION
Our understandings in Section 3 motivate us to use the information from representations to regularize the model predictions. Intuitively, as long the encoder is not fixed, the approximation error could be low enough. If the ERM is properly regularized, the search space and the corresponding estimation could be reduced.
4.1 TRAINING FRAMEWORK
The training framework is shown in Figure 3 where new learning path (Self-supervised learning, SSL) f - is added to be parallel to Path-2 f + y (SL-training) in Figure 2 The newly added projection head h is one- wew-hhdddddaaaye -laeer MLP (Multi Layer Perceptron) whose output represents SSL features (after dimension reduction). Its output is employed to regularize the output of linear classifier g- Given example (In,0e.) and random batch of features B (I, E B), the loss is defined as:
[(( f.9.f.9.)) h) = SL Training
Un)
intto (n(o)) +A (hef (Ixx)), (z()) B), (1)
SSL Training
Representation Regularizer
where À controls the scale of regularizer. The loss for SL training could be either the traditional CE loss or recent robust loss such loss correction/rweeighing (Patrini et al.) (2017: [iu & Tao 2013). GCE (Zhang & Sabuncu] 2018). peer loss (Liu & Guo 2020). The SSL features are learned
by InfoNCE (Van den Oord et al. 2018):
exp(sim(t(/(tz.))
Into (b((())))
log Σn Ex EB, exp(sim(h[ (xx)), (x))))
Note InfoNCE and CE share common encoder, inspired by the design of self distillation ZZang et al. 2019). The regularization loss EReg writes as:
EReg (he)(
Σ
(lo (nr), 6t (8ns )n)
B
En,
where d(-) is a distance measure for two inputs, e.g., square lz distance, e, h(f((n)) 8n = ((((())) (In, (n)) Ien t,2 ||" where {1,2) represents / norm and squared l2
metric, m normalizes the distance over a batch:
m
Σ Fn EB,atn'
14, - 1"
(2)
BI B
The design of (Reg follows the idea of clusterability (Zhu et al. 2021b 2022a) and inspired by relational knowledge distillation (Park et al., 2019), i.c., instances with similar SSL features should have the same true label and instance with different SSL features should have different true labels, which is our motivation to design Rey.. Due to the fact that SSI. features are learned from raw feature X and independent of noisy label Y, then using SSL features regularize SL features is supposed to mitigate memorizing noisy labels. We provide more theoretical understandings in the following subsection to show the effectiveness this design.
4.2 THEORETICAL UNDERSTANDING
We theoretically analyze how Leg mitigates memorizing noisy labels in this subsection. As discussed previously, SSL features are supposed to pull the model away from memorizing wrong labels due clusterability (Zhu et al. 2021bb. However, since the SL training is performed on the

---

Published as a conference paper at ICLR 2023
noisy data, when it achieves zero loss, the minimizer should be either memorizing each instance (for CE loss) or their claimed optimum (for other robust loss functions). Therefore, the global optimum should be at least affected by both SL training and representation regularization, where the scale controlled by A. For a clear presentation, we focus on analyzing the effect of breg in a binary classification, whose minimizer is approximate to the global minimizer when À is sufficiently large.
Consider a randomly sampled batch B. Denote by X² := ((1,;, [x) E B, E B. i ji the set of data pairs, and ر d( (14, (2), (si; 8,)). The regularization loss of batch B is decomposed as:
(Rog(h(((5.)):: o(
Σ
d
Σ d +
Σ
2d
(3)
B
|
(
Term-1
Term-2
Term-3
where X' Xr Kp. KT/Aip denotes the set of instances whose labels are true/false. Note the regularizer mainly works when SSL. features "disagree" with SL. features, i.e., Term-3. Denote by
~ 1), X_ ~ P(X = 0). X ~ P(X Y), xF ~ P(X * Y).
For further analyses, write Term-3 in the form of expectation with d chosen as square distance, i.e, MSE loss:
Le = EXT, [lo(f(x")) -(((())) [I((((xT) - (((((x))) 2
(4)
mi
m2
where my and m2 are normalization terms in Eqn 2 Note in Le. we use w = for SL features and = 2 for SSL features | Denote the variance by var (.). In the setting of binary classification, define notations: X P(X = 1,Y 0), x P(X 0, = 1).
To find a tractable way to analytically measure and quantify how feature correction relates to network robustness, we make three assumptions as follows:
Assumption (Memorize clean instances). Vn E {mlin = yn (xx(((x)))) 0. Assumption 2 (Same overfitting). var(g(f(x{))) = 0 and var(g(((XE)) = 0. Assumption 3 i.e., h(f(x+)) ~ NN(H,, Σ) and h(f(X-)) ~ (N2, L). where Σ is the covariance matrix.
SSL features). The SSL features follow Gaussian distributions,
Assumption implies that DNN has confident predictions clean samples. Assumption iimplies that a DNN has the same degree of overfitting for different classes of noisy samples. For example, an wver-parameterized DNN can memorize all the noisy labels (Zhang et al. (016)), which is the focus of this paper. Thus these two assumptions are reasonable (we also provide empirical evidence for Assumption Appendix D.5 D.5 Assumption assumes that SSL features follow Gaussian distribution when we add the regularize. Intuitively, to provide useful regularization, the SSL features should not be arbitrarily bad. In our experiments, we find that SSL features of CIFARIO are relatively good, thus regularizer can be added in the very beginning while for CIFAR100, we need certain warmup epochs before adding the regularizer. Next, we present Theorem 2 to analyze the effect of L,, Consider case that the model is over-parameterizze and traditional training can memorize all samples. Let
c+ P(V OΙY = 1), = P(Ý 1 0), we have:
Theorem Based on Assumptions 3 when e = e+ = e, P( 1) P-Y 0), assuming that the Bayes classifier achieves zero error, N and B are sufficiently large, then minimizing The over g, h, and f on DNN results in:
Ep 11 (f)) (
1
2
where A(E, pl, p2) = 8 trt)/l1 #2l13 tr(-) denotes the matrix trace, and g", f denote the optimal model.
Theorem reveals clean relationship between the quality of SSL features (given by hx)())) and the network robustness on noisy samples. When tr(E) -> or || - /22|| -> oo, the expected risk of the model Ep [II (X), Y)] will approximate to 0. L.e., for any sample x, the model will predict to its clean label. Note the proof of (icorrm@@ooe not rely on any SSL training process. This makes it possible to use some pre-trained encoders from other tasks. In the Appendix, we also provide an theoretical understanding on the regularizer from the perspective of information theory.
Practically, different choices make negligible effects performance. See more details in Appendix.
7

---

Published as a conference paper at ICLR 2023
CHFARIO
CIFARIOO
CIFA16
40-
Start
(Eace mosate
0.3
0.5
0.0
0.2
0.6
40 60
300
Figure 5: (a) (b) (c): Performance of CE on DogCat, CIFARIO and CIFAR 100 under symmetric noise rate. For each noise rate, the best epoch test accuracy is recorded. The blue line represents training with fixed encoder and the red line represents training with unfixed encoder, (d): test accuracy of CIFARIO on each training epoch under symmetric 0.6 noise rate. We use ResNet50 (He al. 2016) for DogCat and ResNet34 for CIFAR1O and CIFARION 100. Encoder is pre-trained by SimCLR et al. 2020). Detailed settings are reported in the Appendix.
CChnn
EMPIRICAL EVIDENCES
5.1 THE EFFECT OF REPRESENTATIONS
We perform experiments to study the effect of representations on learning with noisy labels. Figure shows the learning dynamics on symmetric label noise while Figure shows the learning dynamics on asymmetric and instance-dependen label noise. From these two figures, given a good representation, we have some key observations:
Observation-1: Fix encoders for high symmetric label noise
Observation-2: Do not fix encoders for low symmetric label noise
Observation-3: Do not fix encoder when bias exists
Observation-4: fixed encoder more stable during learning
Observation-1, Observation-2 are verified by Fig- ure 5 (a) (b) (c) and Observation-3 is verified by Figure a) (b). Observation-4 is verified by Figure 5 (d). These four observations are consistent with
C. aeymmente lane
1b) Q5AR30 initace abel
our analyses in Section 3 We also find an interest-
ing phenomenon in Figure (b) that down-sampling (making PPY i) IP j) in the noisy dataset) is very helpful for instance-dependent label noise since down-sampling can reduce noise rate imbal- ance (we provide an illustration on binary case in the Appendix) which could lower down the estimation error. Ideally, if down-sampling could make noise- rate pattern be symmetric, we could achieve noise consistency (Definition [ which results in O Bias from Theorem T. Another interesting phenomenon is that from Figure 5 (a) (b) (c), the crossing point is different for each dataset. This phenomenon
End wroder
Foed
asymmetriic noise rale
C4 instance noise rale
Figure 4: (a) performance of CE on asym- metric label noise. (b) performance of CE on dustance-dependee
label noise. The genera- tion of instance-dependern label noise follows from CORES (Cheng al. (02I).
can be explained by Corollary [1 Corollary T
iimplis that the encoder learned very well, i.e.,
Error (CfoF C*) Error (C CC), fixing the encoder has benefits over unfixed encoder even when noise rate is small. Since for DogCat, CIFARIO and CIFARI00 dataset, each class have 12500 samples, 5000 samples and 500 samples, respectively. When applying the self-supervised learning on these datasets, the encoder quality is DogCat > CIFARIO CIFAR100. Thus the crossing point is small for DogCat and large for CIFAR100.
5.2 THE PERFORMANCE OF USING REPRESENTATIONS AS REGULARIZER
Experiments on synthetic label noise We first show that Regularizer can alleviate the over-fitting problem when in Equation (1) is simply chosen as Cross-Entropy loss. The experiments are shown in Figure 6 Regularizer is added at the very beginning since recent studies show that for randomly initialized network, the model tends to fit clean labels first (Arpit et al. (017) and we hope the regularizer can improve the network robustness when DNN begins to fit noisy labels. From Figure 6 (c) (d), for CE training, the performance first increases then decreases since the network over-fits noisy labels training proceeds. However, for CE with the regularizer, the performance is more stable after it reaches the peak. For 60% noise rate, the peak point is also much higher than vanilla CE training. For Figure (a) (b), since the network is randomly initialized, it over-fits noisy labels
8

---

Published as a conference paper at ICLR 2023
syerm
10
00-
40
40
169
60 100
150
50 3010
350
Figure 6: Experiments w.t. regularizer (A = 1) on CIFARIO. ResNet34 is deployed for the experiments. (a) (b): Encoder is pre-trained by SimCLR. Symmetric noise rate is 20% and 40%, respectively; (c) (d): Encoder is randomly initialized with noise rate 40% and 60%, respectively. Table 1: Comparing each method on CIFARIO. The model is learned from scratch without SSL pretraining for all methods with = 1. Best and last epoch test accuracies are reported: best/last.
Symm.
CIFARIO
Asynm. CIFARIO 8 = 0.4
Method
8 0
5 == 0.8
CE
61.29/32.83
38.46/15.05
67.28/56.6
CE + Regularizer
69.02/65.13
61.94/56.78
73.38/58.51
GCE (Zhang & Sabuncu, 2018
72.56/62.84
50.71/20.53
69.19/53.22 24
GCE + Kegulanzer
72.61/68.38
63.63/63.05
69.79.61.32
FW (Patrini et al. 2017
63.95/60.13
40,08/26.7
68.62/58.01
FW + Kegularizer
68.73/65.90
00.94/59.88
75.64/67.66 7.66
HOC (7hu et al. 202Th
62.53/46.17
39.61/16.90
05.88/78.89
HOC+ + Regularizer
70.07/66.94 94
60.9/34.90
83.53/82.56
Peer Loss (Liu & Guo 2020
71.52/76.07
15.60/10.00
84,47/68.95 93
Peer Loss Regularzerr
77.61/73.26
61.64/53.52
81.58/75.38
ELR (Liu et al. 2020
72.56/71.58
42.76/23.57
86.65/85.27
EI.R + Kegulanzer
76.95/75.98
63.4/57.45
88.62/87.92
Table 2: Test accuracy for each method on CIFAR ION and CIFAR I0ON.
GCE
Co- Co-Teaching+
Peer Loss
JoCoR
ELR
CE+ Regularizer
CIFAR ION (Worst)
77.69
80.66
N3.26
82.53
83.37
83.58
88.74
CIFARIOON
55.50
56.73
57.88
57,59
59.97
58.94
60.81
Table 3: Test accuracy for each method on Clothing IM dataset. All the methods use ResNet50
backbones. DS: Down wwwn-Sampling. Reg: With structural regularizer.
Foward-T
Co-teaching
CORES+DS ImageNet 73.24
ELR+DS
CE
CE
CE DS RRg
Initializer
ImageNet
ImageNet 69.21
ImageNet
SiMCLR
SimCLR 72.95
SiMCLK 73.48
Accuracy
70.83
72.87
70,90
the very beginning and the performance gradually decreases. However, for CE with the regularizer, it helps the network gradually increase the performance as the network reaches the lowest point (over-fitting state). This observation supports Theorem that the regularizer can prevent over-fitting
Next, we show the regularizer can complement any other loss functions to further improve perfor- mance learning with noisy labels. Le., we choose E in Equation ( to be other robust losses. The overall experiments are shown in Table [ It can be observed that our regularizer can complement other loss functions or methods and improve their performance, especially for the last epoch accuracy. Note that we do not apply any tricks when incorporating other losses since we mainly want to observe the effect of the regularizer. It is possible to use other techniques to further improve performance such as multi-model training (Li et al. 2020) mixup (Zhang et al. 2018a).
Experiments on weal-woldd label noise We also test our regularizer on the datasets with real- world label noise: CIFAR1 ON, CIFAR 100N (Wei et al. (022d) and Clothing (Xiao et al.) 2013). The results are shown in Table 2 and Table 3 we can find that our regularizer is also effective on the datasets with real-world label noise even when E in Equation (1) is simply chosen to be Cross Entropy. More experiments, analyses, and ablation studies can be found in the Appendix.
6 CONCLUSIONS
In this paper, we theoretically analyze the memorization effect by showing the relationship among noise rates, sample size, and model capacity. By decoupling DNNs into encoder followed by linear classifier, our analyses help reveal the trade-off between fixing or unfixing the encoder during training, which inspires us new solution restrict overfitting via representation regularization. Our observations and experiments can serve as a guidance for further research to utilize DNN representations to solve noisy label problems.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is partially supported by the National Science Foundation (NSF) under grants IIS-2007951, IIS-2143895, CCF-2023495, and the Office of Naval Research under grant N00014-20-1-4
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks, In Proceedings of the 34th International Conference on Machine Learning
Learning-Volume Volume 70, pp. 233-242. JMLR. org, 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair. Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint MeXivv/801.04000 2018.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. In Summer school on machine learning, pp. 169-207. Springer, 2003.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. International Conference on Learning Representations, 2021.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Luc Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 2206-2212, 2021.
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2703-2708, 2021.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing. pp. 6645-6649 IEEE, 2013.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:2011.04406, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.
10

---

Published as a conference paper at ICLR 2023
R Devon Hjelm, Alex Fedorov, Samuel Maroii-Marrhilonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 100.06000000007, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp. 2304-2313. PMLR, 2018.
Zhimeng Jiang, Kaixiong Zhou, Zirui Liu, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. An information fusion approach to learning with instance-dependent label noise. In International Conference Learning Representations, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 1105, 2012.
Imagenet classification with deep convolu-
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels semi- supervised learning. In International Conference on Learning Representations, 2020. URL
net Ho http:///eaa.aeeeeeoooooooooooaar
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-superiseed contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 316-325, 2022a.
Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge. and Tongliang Liu. Estimating noise transition matrix with label correlations for noisy multi-label learning. In Advances in Neural Information Processing Systems, 2022b.
Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez-Grandee Granda. Early-learning regularization prevents memorization of noisy labels. Advances neural information processing systems, 33:20331-20344 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. arXiv preprint arXiv:2202. 14026, 2022.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE
Transactions pattern analysis and machine intelligence, 38(3)4447-461 2015.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226-6236. PMLR, 2020.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint 1rXiv:1605.08/11 2016.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5191-5198, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204 2013.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling In International Conference on Learning Representations, 2020. URL httl:tttss///oeereviiie net/ wtt.rummiiiHHgshhhhhh.
Curtis Northeutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411 1411, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
11

---

Published as a conference paper at ICLR 2023
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952. 2017.
Scott Reed, Honglak Lee. Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich, Training deep neural networks noisy labels with bootstrapping. arXiv preprini arXiv:1412.6596, 2014.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint 1rXiv:1705.106%3 2017.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Jingkang Wang, Hongyi Guo, Zhaowei Zhu, and Yang Liu. Policy learning using weak supervision. Advances in Neural Information Processing Systems, 19:199900-9977 2021.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEFE International Conference on Computer Vision, pp. 322-330, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13726-13735, 2020.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems, 34, 2021.
Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An. Deep learning from multiple noisy annotators union. IEEE Transactions Neural Networks and Learning Systems, 2022a.
Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Logit clipping for robust learning against label noise. arXiv preprint arXiv:2212.04005, 2022b.
Jiaheng Wei and Yang Liu. When optimizing SiS-diveree robust with label noise. In Interna- tional Conference on Learning Representations, 2021. URL URLptttss//ggenneeeee...ee
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, and Yang Liu. To smooth not? when label smoothing meets noisy labels. In ICML, 2022c.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-worke human annotations. In International Confer- ence on Learning Representations, 2022d. URL htt.tt:::/ppeeeee/e//e net forum?id=
TBWA6PLJZ.Q
Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu. To aggregate or not? learning with separate noisy labels. arXiv preprint arXiv:2206.07188. 2022e.
Qi Wei, Lei Feng, Haoliang Sun, Chenhui Guo, Ren Wang, and Yilong Yin. Fine-grained classification with noisy labels. CVPR, 2023.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems, 32, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust carly-learning: Hindering the memorization of noisy labels. In International conference on learning representations, 2020a.
12

---

Published as a conference paper at ICLR 2023
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance dependend label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020b.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Instance correction for learning with open-set noisy labels. arXiv preprint 1rXiv:2106.00453 2021a.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106. 004404 2021b.
Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended-T: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2691-2699) 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi: novel informationntheoreee loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems, volume 32, 2019.
Quamming Yao, Hansi Yang. Bo Han, Gang Niu, and James T Kwok. Searching to exploit memoriza- tion effect in learning with noisy labels. In Proceedings of the 37th International Conference on Machine Learning, ICML '00, 2020.
LIN Yong, Renjie Pi, WEIZHONG ZHANG, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. A holistic view of noise transition matrix in deep learning and beyond. In International Conference on Learning Representations.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint arXiv:1901.0422 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 11116611113300 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018a. Sa. URL
htttps://openrevviee net. worumridirioorro 1
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 3713-3722, 2019.
Xiang Zhang and Yann LeCun. Text understanding from scratch. arXiv preprint arXiv: 2015.
Xuchao Zhang, Xian Wu, Fanglan Chen, Liang Zhao, and Chang-Tien Lu. Self-paced robust learning for leveraging clean labels in noisy data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6853 6860, 2020.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4320-4328. 2018b.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778-8788 2018.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10113-10123, 2021a.
13

---

Published as a conference paper at ICLR 2023
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability alternative anchor points when learning with noisy labels. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 22912-12923, PMLR, 18 24 Jul 2021b.
Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training model to predict. In International Conference on Machine Learning, pp. 27412-27427. PMLR, 2022a.
Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi-supervised learning. In International Conference Learning Representations, 2022b. URL https://
openreview net/forum wattooooommmidd----o-
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. In International Conference on Machine Learning, pp- 27633-27655 PMLR, 2022c.
14

---

Published as a conference paper at ICLR 2023
APPENDIX
The Appendix is arranged as follows: Section A proves Lemmas and Theorems in Section Section B proves Theorem 2 in Section 4 and provides an high level understanding on the Cillutttttlsst why down- wown-sampling can decrease the gap of noise rates. Section|Dprovidde the effect of distance measure Eqn ( (w = or 2): ablation study in Section I the effect of different SSL pre-trained methods; the performance of shallow network and deeper network on high noise settings; empirical validation of Assumption
Outline
regularizer from the perspective of Information Theory. Section illustrates
2} Experiments towards the regularizer on CIFARII 100 dataset. Section E elaborates the detailed experimental setting of all the experiments in the paper.
A PROOF FOR LEMMAS AND THEOREMS IN SECTION 3
A.1 PROOF FOR LEMMA J
Let T := minx,; (x)
Considering general instance-dependent label noise where Tx(() -P(Ý =jlY X), we have (Cheng et al. 2021
Ex((C(())
= E P( =,,x= dr JE(K]
Σ
Σ
/ P(Ý =j,Y=i,-- 2))(C((X))) dr
(E[K] JE[K]
Σ
Σ
P(Y =1) / |P(Ý- =ily = i, = =)) = x|Y== = -)e(c(x),)) dx
(E(K]
Σ Σ P(Y- = =i)Bp|== [P( = i|l =i x ==)((((())
=
(E(K]
Σ Σ P(Y- -1
RE[K] SE[K]
Σ P(Y Y (x + Σ Σ
P(Y = 1,,
[E[K]
iE[K] JE
=T Σ
PPV = )()=)))) + Σ
P(V = D) -
*E[K]
iE[K]
Σ
Σ
P(Y (2,
]E[K] k
Σ
Σ P(Y = -YY-
JE(K) (E[
where Uzj(X) = +, T.
15

---

Published as a conference paper at ICLR 2023
A.2 PROOF FOR LEMMA 2
Consider the symmetric label noise. Let T(X) = T,XX, where T. = 1 1 1- = R-I Vi + The general form in Lemma
1 can be simplified as
B ((C(())) 8Y
- =(1-c) (DI()))))) K-
Σ
E
P(Y (
(E]
=(1- c1-c- K K- ED] K-1
Σ
Σ
P(Y = 1) ((
jE K] KE[K]
When is the 0-1 loss, we have
E JE[K] [E[K]
E
P(Y = -(C))))
and
eK
[)((((((), (1 (1-
K (((() K-1
Consider the asymmetric label noise. Let T(X) T, VX, where Tú = -c, T.((+t))) = c. The general form in Lemma can be simplified as
Ep ((C(( V) =1-x)))[((((()))
Σ KE[K]
P(Y B) |(((())) (1+ 1)x)]
A.3
PROOF FOR THEOREM 1
For symmetric noise, we have:
EB [(()(x))))
ED [((B(X),n) - eKK
e/(K 11
1 - eKeK(KK -
Thus the learning error is
Ep [(()(x))))] - EED (XCx(X))))
1
(E,5 [(()((), 8)] -Ep [((Co(x),)))) 8)))
1 (K/(K
Let
EB [[(((()) 8)| = 1
Σ ((C(an). Un).
N
nE[N]
Noting BB [((Cb(x),)) 13 - EB ((()(x), 53] 2 0, we have the following upper bound: Ep [(C((x),))) EB [(C)((x))))
< [((Op(x),)) E (((x(x),))
+ BB [((Co(x), - EE [((o((),,T]
E [((}(()))))
8) - LB ((()(()),)) |+ [ (((DD())))))) 15 (((Co(X),
Recall C E C. Denote the VC-dimension of by |C| (Bousquet al. 2003 Devroye al. 2013). By Hoeffdiig inequality with function space C, with probability at least 6, we have
1E, [((x((), 8)] BB [((C;(x), 8) + | 1 [(CC(()) 8)] 15 ((C)
< arg Ex (e(C(x),)) - EB [(((()))))
CEC
CC log(NN e/(cl) + log(8/5)
/16
2N
16

---

Published as a conference paper at ICLR 2023
Thus
ED [((Ō(())))] ED [I(Co(X), / 16,
CC Io e/(Cl) + 18g(8/0)
2N
ck
)²
K-1
Similarly, for asymmetric noise, we have:
ED [((Ōp(x))n] Es [(()
Bias (pp)
1-c
where
Bias (aas
Σ KE|K]
P(Y [0
Thus the learning error is
Ep [(C [(C)(x))))) ED ((CD((X))))
I-c D e( ((Cp(x),))]
Bx ((Co(()))8)) + (Bias(Cp) Bias Cp)
By repeating the derivation for the symmetric noise, we have
Ep
ED (((C=(X), [[C])X))))]]
-DD
16V
CC log((g (N. e/ICl) + 1og(8/8)
(Bias(CD) Bias(Cp))
2N
A.4 PROOF FOR COROLLARY
Symmetric noise Let Ci Go gl Denote the optimal classifier learned within the above two functions spaces by Cha. and C respectively. Then the approximation errors of both ,C*). Assume a ,C*) Error (Cg / C*). Note the assumption holds generally and the bias-complexity trade-off does not exist if the assumption does not hold.
cases can be denoted by A ,C**) and
From Lemma A.4 in Shalev-Shwartz & Ben-David 2014) and our Theorem I
we know
log(NN e/ e/Cl) +2
E|Error (B), 16
V2N
Therefore, by requiring the difference between two total generalization errors large than O, we have:
Es E 5)1 + (C1) - Eg A (C2, A (C2) 2
11 G o F log AN e/l o
lo e/19]) +2
Error (C9 ,C*) Error (COoT 2
16
22N(1 xK))
/2N(1 - ck K-1
}
(Vig FI Fog((N 9 lo (lo((4--e/)) e/
cK
16
+1
1-1
V2N
Error @l C**) Error (C9oF, x ,C*)
B PROOF FOR THEOREMS IN SECTION 4
Lemma If and Y are independent and follow gaussian distribution: N(xx,, Ex) and Y ~ N(y,, Ey), Then: (x YI12) = ||xx pyll2 + tr + Ex).
B. PROOF FOR THEOREM 2
Before the derivation, we define some notations for better presentation. Following the notation in Section define the labels of XT YT and the labels of X as YF, Under the label noise, it is easy
17

---

Published as a conference paper at ICLR 2023
to verify P(Y =) - P = 1). p2 PPPPP (f())) and h((())) to be simplified as g gf and (X(.
P(Y=1)(LL-))
and P(Y =] -- +P(Y=1)-er Let
P(Y=0)-e
In the case of binary classification, g. (x) is one dimensional value which denotes the network prediction on x belonging to Y = 1. Le can be written as:
[9S(XT) 95(X5)) [ - 11(
X
m1
7722
denoted ((XT,X))
@ y (XX .xF) (x",))
= P1 P2 E ExT x 4 (XT, x + (1 p1) Ext x ( xx+
+ P1 (1 (1-p2) Ex x V (X. x + (1 p1) p2) Exi V/ V(X..
where my and m2 are normalization terms from Equation
Specifically,
mi = lim
Σ
- 99f
8
B (
m2 == lim
Σ
8
[
(a) is satisfied because v7 (XT, XF) is irrelevant to the labels. We derive VX xF, as follows:
Xx xt (X (.)
9f(xE)) IIf(xT)) -f((X)I1 ,2
E ExI.
mi
m2
9f(xt) (-9f(xT)
1
[If((X])
1x((E))1 )²
Exp,x Xx
m1
m2
gf(xF)
[ - hf (x((111 ))²
E EXT.xt X
m1
m2
(b) is satisfied because from Assumption DNN has confident prediction clean samples. (c)
satisfied because gf(X) is one dimensional value which ranges from 0 to 1. From Assumption
3 hf(+)) and hf(X follows gaussian distribution with parameter (P1, Σ) and (p2, Σ). Thus according to Lemma we have ExI .xe [|ff(X]) - hf(x5))l
= [| 22ll + tr(2). Similarly, one can calculate EXX,Xx [[xs(XT) hs(x = 2 tr(E). It can be seen that (d) is function with respect to g Similarly, ((X xF) is also a function with respect to xf(E while (XX,, XE) and V (XX,, XE) are functions with respect gf(X).. Denote d(+, +) xt [ - hs(x After organizing (XT,, XE) and V (X xt), have:
min Pt P2 Ext,x V VX x.) + pı) .2 Exi,x x V(X xt. sf(x))
min (E (Ex.gf(X))) xf(E)) -(2 P1 d(+, +)
(5)
m1 m1. )(-,,t))) (xx)
+ 2. (1 p1)(
m2
m2
+ constant with respect to (
Note in Equation (5) we use
to approximate Exi gf since from Assumption
var()(((XE)))
0. Now we calculate my and m2 from Equation 2
18

---

Published as a conference paper at ICLR 2023
m1 = Pi P2 (1 Ext Exxgf(Xx))
(6)
P1 (1 - (p) (1
xx gf gf (1 (1- p1) p2) Ext gf
m2 = P1 P2 d(+, +) + (1 p1) .2 dl - +p pr (2-p2) -)+ pı) -p2) -=)
Under the condition of PP(Y- 1) P(Y 0), 6_ e+,, have pt p2 2. m2
4ar(E)+ -aal my 3. which is constant with respect Exp (x and E (x in
Equation (6. Thus Equation (5) is quadratic equation with respect to Ex; of (x Then when Equation achieves global minimum, we have:
Ext ( Pr
(pi d(+, +) (1 pı) d(-,+))
m2
1
2
2 +
(3)r(3) Iar -
Similarly, organizing VXTT. X and vi (XT, XE) gives the solution of xxx gf (XE):
Ex Exzos(X=)
p1 +
(p1 dl d(-,-) p1 -()),+(+++---
m2
1
(8)
2
2 +
8.tr(E)
|| 1
Denote A(E,11.12) = 8-.rr())/ 12112 Now we can write the expected risk as:
Ep 11 = e) EXT, [1 (o(J(x)))))] te Exe, [1 (o([(x*), Y)]
(a) e Exy [1 (o(s(x*), y]]
Exp = [I (o(F(x)).)) 1 Exy,y= [1 (o(f(xt)),)
2
1
1 A ((E,,1,12)
(a) is satisfied because of Assumption that model can perfectly memorize clean samples. (b) is satisfied because of balanced label and error rate assumption. (c) is satisfied by taking the results from Equation and Equation (8)
Proof Done.
B.2
HIGH LEVEL UNDERSTANDING ON THE REGULARIZER
Even though we have built Theorem 2 to show SL features can benefit from the structure of SSL features by performing regularization, there still lacks high-level understanding of what the regular- ization is exactly doing. Here we provide an insight in Theorem 3 which shows the regularization is implicitly maximizing mutual information between SL features and SSL features.
Theorem 3. Suppose there exists a function such that C(X) [((((X))), The mutual informa- tion (X)()())), C(X)) achieves its maximum when L. in Equ. (4)
The above results facilitate a better understanding on what the regularizer is exactly doing. Note that Mutual Information itself has several popular estimators (Belghazi et al. 2018 Hjelm et al. 2008). It a very interesting future direction to develop regularizes based on MI to perform regularization by utilizing SSL features.
19

---

Published as a conference paper at ICLR 2023
Proof for Theorem 3 We first refer to property of Mutual Information:
I(::)) = (()))
(10)
where ab and & are any invertible functions. This property shows that mutual information is invariant
invertible transformations (Cover) 1999). Thus prove the theorem, we only need prove that
in Theorem 3 must be an invertible function when Equation (4) is minimized to O. Since when & is
invertible, I (h(f(X)), C(X))
= ((((f())),(((())))
We prove this by contradiction.
Let 4 bx(())) and s; g(f(x:)). Suppose & is not invertible, then there must exists 8; and 8j where si + 8j which satisfy 1 = {(s:) = 4;- However, under this condition, 4 4 Lj = 0 and
8 8, * O, Equation (4) can not be minimized O. Thus when Equation 44 minimized 0, must be an invertible function.
Proof done.
B.3 PROOF FOR LEMMA
By the independence condition, Z = X - Y also follows gaussian distribution with parameter (ux ,y, Xx + ).).
Write Z as Z μ + LU where U is standard gaussian and pxx py. LLT 2x E.y Thus
1 1 =f= = 1 u+ + +U OF
(11)
Since U is standard gaussian, E(U)) 0. We have
E(1 (11) = nT L'
- 4 E
(L
k.l
(a)
12 #+ [(LF
(12)
n² p + tr(
= ||xx Hy 112 + tr + Ex)
(a) is satisfied because U is standard gaussian, thus B(U?) and E(URUt) 0 (k (1.
Proof Done.
C ILLUSTRATING DOWN-SAMPLING STRATEGY
We illustrate in the case of binary classification with e4 + e_ < 1. Suppose the dataset is balanced, the initial state, e_. After down-sampling, the noise rate becomes et, and e=. We aim to prove two propositions:
Proposition 1. If c4 and e_ are known, the optimal down-sampling rate can be calculated by e+ and e.. to make e4 =
e+ and e_ are not known. When down-sampling strategy is to make P(Ý = 1) Proposition 2. e
P(Ý 0), then 0 < e4 e c+
Prooffor Proposition 1 Since dataset is balanced with initial e4 > e_,, we have P(ý 1) < P( 0). Thus down-sampling conducted at samples whose observed label are Suppose the random down-sampling rate is r, then e 1-eltr-el
r-e.
and e" r-(1-e He
e- If e e", we have:
r r.
e_
(13)
1- e+ +r r. --
20

---

Published as a conference paper at ICLR 2023
Thus the optimal down-sampling rate r v Ve (1-0)_)) which can be calculated if e and ex are (1-ee_)))
known.
Proof for Proposition 2 If down sampling strategy is to make PP( = 1) = P(V = 0), then
r (e+ + 1 - --) = c + e-, we have r 1-e+te- Thus e7 can be calculated as:
1-e-te_
c 1- e +7
+e- (1 (1- (1 -e_ +c4 + c+ (1 c +e_)
Denote a = [1-c4) (1-c-)(1-c---0) +e4 04 Since Et > e-, - e- + 1 - Et te..
a == (1-c))(1-c_ (1e (1-4tt)) (1-c+)(1-e 1-e, tt. =1
Similarly, el can be calculated as:
-_ +r. (1---
(1- + (+)
- e_ +e+) (1---) e-).(1 e +6-)
Denote B = tc- (1-ette_)
1-0-t4
Since et > e-, 1 - e +e+ - 6 + e-.
3 =
tCl
tc₁
= 1. Since a e+ < e4
and B. > e_, we have e__ E 64 B. E+ c..
Next, we prove e e following the derivation below:
e >E E
r-6+
e_
1 +r
e- tr. (1
e_ (1 e4)
e4 (1 (--
(14)
1 +
e e- (1-4)
1 e_ +e
4 (1--e_
e (1 - c+) + e_ .e_ 2
e_
1
1 1-c_
Let f(e+) = 6 (1 e,) +
Since we have assumed e_ < e. and I- e_ - -e-)-- I-c-
e- +4 < 1. Thus proving e4 > c is identical to prove f(e+) > 0 when e 64 1 6..
Firstly, it is easy to verify when e+ = e_ or 6 = 1- e_, f(+)) = 0. From Mean Value Theory, there must exists a point eu which satisfy (co) = 0 where e+ cu < e_e_. Next, differentiate f(e,) as follows:
(((+)) (1 (1-e4)2 (1 e-) + cL. (1 e-_- - 2. (1 - =+)
(15)
(1- --_
1-e It can be verified that f'(e-) (I-e-)((1-e)) and f'(1 e_) -1-11--)) = 0.
0
Further differentiate f((4), we get when et < ((1 e-) e² 2 f"( < and when e+ > 1 ((1 --) (*()+) > 0. Since e_ < e_ and e_ + < 1, we have e < and
e_ ((1 ((l-e))-ed e2) < e_, ie,,, 1- ((1 (e-e_)- o22 locates in the point between e and - e_. Thus, when e <e < 1- - ((1-e_) } f(e+) is a strictly concave function and when 1 ((1 - e-) ) < < 1 e_, f(e+) is strictly convex function.
Since fee > and f(1 (1(1-_)) --)== 0, en locates in the point between e.. and 1 1-((1-e_)))) (e)
which satisfy ('(co) 0. Thus when e_ < ee co. f(++) monotonically increases and when
21

---

Published as a conference paper at ICLR 2023
decmasnt dove doan--orrgti
0.00
080
Figure 7: Visualizing decreased gap by down-sampling strategy.
(a) 20% symm (int: SimCUR)
13) 40% symm (init: SimCUR)
(c) 40% symm (Int:: Random)
(e1 60% symm Onit: Random)
10
50
00-
03
Lol 200830e
60-
00
40
60
20
20
20
20
Type
Type
Tyre
Type Type
Typa
Type
Type
0
20
60
00
200
0
20
50
20
200
0
50
100
155
50
1.00
250
2raning spoctrs
braining spocha
training spocta
craining spacta
Figure 8: Comparing difference choices of distance measure in Equation
Type denotes using l2 norm to calculate distance between SL features and square l2 norm to calculate distance between SSL features, which is adopted in our paper. Type 2 denotes using l2 norm to calculate distance for both SL and SSL features.
60 <e -1 f(e+) monotonically decreases. Since ((e-) f(1 e-) 0. We have f(e+) > when 6_ < et < -_..
Proof done.
We depict figure Figure to better show the effect of down-sampling strategy. It can be seen the curves in the figure well support our proposition and proof. When e4 e is large, down-sampling strategy to make PPYY = 1) PY 0) can well decrease the gap even we do know the true value of e_. and e4.
D MORE DISCUSSIONS AND EXPERIMENTS
D.1 THE EFFECT OF DISTANCE MEASURE IN EQN
In this paper and experiment, we use le norm to calculate the feature distance between SL features and square le norm to calculate the distance between SSL features. This choice can lead to good performance from Theory 2 and Figure 6 Practically, since structure regularization mainly captures the relations, different choice does not make big effect the performance. We perform experiment in Figure g which shows that the performance of both types are quite close.
D.2 ABLATION STUDY
In Figure SSL training is to provide SSL features to regularize the output of linear classifier g. However, SSL training itself may have positive effect on DNN. To show the robustness mainly comes from the regularizer rather than SSL. training, we perform an ablation study in Figure 9 From the experiments, it is the regularizer that alleviates over-fitting problem of DNN.
D.3 THE EFFECT OF DIFFERENT SSL-
METHODS
Our experiments are not restricted to any specific SSL method. Experimentally, other SSL methods also adoptable pre-train SSL encoders. In Figure SimCLR (Chen al. 2020) is adopted
22

---

Published as a conference paper at ICLR 2023
(n) 40% symm (init: Random)
(b) 60% symm finit: Random)
CE
Ct
InfoNCE:
CE InflNNCE
Regularizer
InfoNCE Repularizer
20
60
6o
100
120
340
20
6o
60
1.00
120
144
training epachs
tning epocha
Figure 9: Ablation study of using the regularizer to train DNN noisy dataset.
to pre-train SSL encoder. For a comparison, we pre-train encoder with Moco on CIFAR10 and fine-tune linear classifier on noisy labels Table 4
Table 4: Comparing different SSL methods CIFAR1O with symmetric label noise
Symm label noise ratio
Method
0.2
0.4
0.6
0.8
CE (fixed encoder with SimCLR init)
91.06
90.73
90.2
88.24
CE (fixed encoder with MoCo init)
91.55
91.12
90.45
88.51
It can be observed that different SSL methods have very similar results.
D.4 PERFORMANCE OF SHALLOW NETWORK AND DEEPER NETWORK HIGH NOISE SETTINGS.
Apart from empirical results in Section we also provide empirical evidence that a shallow network performs better than deeper network on nigh-symmetric noise settings to further validate Corollary 1. Since similar fixing the encoder, a shallow network also has lower capacity than deeper network. The experimental settings are as follows: network structure: ResNet18 vs ResNet50, dataset: CIFAR- 10, loss: Cross entropy; number of epochs (100), batch size (64), learning rate (0.1 at first 50 epochs and 0.01 for last 50 epochs), optimizer (SGD). We report the best epoch test accuracy in 5
Table 5: Comparing the performance of different network structures on CIFAR-10 with symmetric label noise
Symm label noise ratio
Models
0.8
0.85
ResNet18
33.06 38.74
45.57
ResNet50
30.54
can be observed that a shallow network behaves better for high symmetric noise ratio which supports our claim in the paper.
D.5 VALIDATION OF ASSUMPTION
The zero variance assumption (Assumption
2 to proceed the proof in Theorem 2 is backed up by (Zhang al. 2016) showing that DNN will memorize all the noisy samples when DNN converges,
23

---

noisy data for training and clean data for Published as a conference paper at ICLR 2023
resulting to near 0 loss. We perform experiments CIFAR-100 with symmetric label noise to
validate this assumption. The results are reported in Table It can be seen that the variance noisy
Table Variance of each noisy id in CIFAR-100 with training epochs
Labels id
Epoch 50
Epoch 100
Epoch 150
Epoch 200
3.829
0.038
0.009
0.001
1
4.153
0.019
0.006
0.005
3.388
0.055
0.015
0.002
3.572
0.018
0.0002
0.0002
4
3,952
0.112
0.002
0.0007
samples in each label id is tending to when training converges.
D.6 VALIDATION OF THE EFFECT OF BATCH SIZE
We perform experiments CIFAR100 under symmetric label noise ratio 0.6 with regularizer for different batch size. Table 7 shows that increasing batch size has slight perfomance gain.
D.7 EXPERIMENTS TOWARDS REGULARIZER ON CIFAR100
In this section, we examine our regularizer on CIFAR100 dataset with certain SOTA methods from (Liu et al. 2020). Results are reported in Table from which we can see that our proposed regularizer can also improve performance on CIFAR100 dataset.
E DETAILED SETTING OF EXPERIMENTS
We use DogCat, CIFAR1O, CIFARI00, CIFARION and CIFAR10ON and Cl for experiments. DogCat has 25000 images. We randomly choose 24000 images for training and 1000 images for testing. For CIFARIO and CIFARIO0, we follow standard setting that use 50000 images for training and 10000 images for testing, CIFARION and CIFAR F1AR have the same images of CIFARIO and CIFAR except the labels are annotated by real human via Amazon Mturk which contains real-world huamn noise. For Clothing IM, testing.
Datasets:
We use ResNet34 for conducting the experiments. All the experiments in Figure are trained from scratch with papper-prramters below: learning rate (0.1 at first 50 epochs and 0.01 for last 50 epochs), batchsize (256), optimizer (SGD).
Setting for Figure 1:
Setting in Section 5.1 (Figure 5 and Figure 4) SimCLR is deployed for SSL pre-training with ResNet50 for DogCat and ResNet34 for CIFAR10 10 and CIFAR100. Each model is pre-trained by 1000 epochs with Adam optimizer (Ir = Ic-3) and batch-size is set to be 512. During fine-tuning, we fine-tune the classifier on noisy dataset with Adam (Ir 1e-3) for 100 epochs and batch-size to be 256.
Setting in Section 5.2 For Table all the methods are trained from scratch with learning rate set to be 0.1 at the initial state and decayed by 0.1 at 50 epochs. For Table and Table 3 the encoder pre-trained by SimCLR and we finetune the encoder on the noisy dataset with CE Regularier. The optimizer is Adam with learning rate 1e-3 and batch-size 256. Note that in Eqn ( we use MSE loss for measuring the relations between SL. features and SSL. features. However, since MSE loss may cause gradient exploration when prediction far from ground-trutte instead. Smooth 4 loss is an enhanced version of MSE loss. When prediction is not very far from ground-truth, smooth 1 loss is MSE, and MAE when prediction is far.
use smooth 4 loss
Setting for Table g All the methods are trained from seratch with learning rate 0.001. The optimizer is Adam and the training epochs is 100. Note when applying regularizer with each method, for example, Regularizer + CE, we first use CE to warmup DNN for certain epochs, then apply regularizer prevent overfitting.
24

---

Published as a conference paper at ICLR 2023
Table 7: Comparison of test accuracy with different batch size on CIAFR100 under symmetric noise
ratio 0.6.
Methods
batch size 100)
batch size 200
batch size 300
CE Regularizer
39.08
40.07
40.1 14
Table 8: Comparison test accuracies with each method on CIFAR
The model is learned from
scratch for all methods with 1. Best and last epoch accuracies reported: best/last.
Symm. CIFARTOO
Asymm. CIFARIO0
Method
= 0.6
== 0.8
== 0.4
ELR (Liu et al.
2020
34.36/14.21
18.95/5.54
55.36/33.5
ELR
Regularizer
37.17/23.45
20.92/13.16
56.55/36.49
Bootstrap Reed et al.
2014)
34.21/13.8
18.45/6.68
39.28/30.93
Bootstrap + Regul larizer
35.63/27.88
19.92/16.3
40.21/34.56
FW (Patrini et al.
2017)
38.72/22.26
17.29/8.15
42.35/40.46
FW + Regulau lanzur
39.01/29.39
26.92/21.02
43.35/40.02
SL (Wag al. 2019)
34.16/14.12
19.17/6.68
40.76/32.07
SL Regula lanzerr
34.20/15.3
19.65/6.02
40.81/34.05
GCE (Liu al.
2020)
38.08/26.79
23.24/16.9
41.26/31.4
GCE + Regulanzeer
44.27/43.48
35.01/34.55
42.35/35.74
25


Reviews:
[{"review_id": "_cwRLzz40kk", "reviewer": "Reviewer_wiWY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses the memorization effect of deep neural networks and proposes a framework to mitigate the negative effects of memorizing wrong information when learning with noisy labels. The analyses focus on the tradeoff between estimation error and approximation error due to the model capacity. By two case studies, learning with a fixed encoder and learning with an unfixed encoder, the authors conclude that restricting the search space by fixing the encoder reduces the estimation error but possibly increases approximation errors. Based on the takeaways from the tradeoff analyses, the authors further propose a learning framework that tries to combine the beneficial parts of fixing the encoder (low estimation error) and not fixing the encoder (low approximation error). Specifically, they exploit the power of self-supervision that regularizes supervised learning with self-supervised features. The advantage of the proposed framework is also theoretically analyzed in a simplified case. They also run experiments on both the synthetic label noise and real-world label noise to demonstrate the effectiveness of their framework.", "summary_of_the_review": "This paper presents work in learning with noisy labels. Both theoretical analysis and experimental results can provide insights into the community. The reviewer affirms its worth. The responses to the mentioned concerns are expected. \n\n####Post rebuttal#### \n\nThe response is detailed and addresses my concerns. I hence raise my score. ", "strength_and_weaknesses": "**Strengths**\n1. The paper theoretically reveals how the noise rate affects traditional learning with noisy labels. Besides, the theoretical bounds are connected to the model capacity, which motivates the design of their framework.  \n\n2. The proposed framework is demonstrated to be effective from both the theoretical and experimental aspects. Specifically, the theoretical analyses show why and how the self-supervised features can help regularize the learning with noisy labels (weak supervision), which provides insights for understanding the benefit of self-supervised learning in making the model robust to noisy labels.\n\n**Weaknesses and Questions**\n1. The paper claims that \"early stopping will handle overfitting wrong labels at the cost of underfitting clean samples if not tuned properly\". However, the issue may be also not addressed by this paper. Perhaps, holistic regularization cannot avoid the underfitting issue well. \n\n2. It is not very new to exploit contrastive learning to handle noisy labels, e.g., [1] and [2]. More discussions about related work can be added. \n\n3. It seems that Figure 2 is not highly related to the proposed method of this paper. Besides, the power of reducing model capacities of three different paths is different. Could they always converge the same classifier?\n\n4. The analysis of the approximation error is not very solid. Besides, the trade-off analysis may contradict the double descent phenomenon in deep learning. How to address this concern?\n\n5. It is a bit hard to follow \"a larger $\\mathcal{C}$ will lead to smaller approximation error but at the cost of larger estimation error given large $N$\". It seems that the paper provides an error bound, but not the exact values. \n\n6. Assumption 2 is a bit to understand since different classes always have different memorization rates in training [3]. \n\n7. Theorem 2 is not highly related to the proposed method/framework. It seems that the theorem tells us the power of the Gaussian assumption in tackling noisy labels, which is similar to [4]. \n\n8. From Figure 5, subfigures (a)--(c) show the crossing points between fixed and unfixed encoders are different, i.e., the noise rates are 0.0, 0.4, and 0.7 for different datasets. The authors should provide some insights into this observation.\n\n9. The authors claim the performance of a fixed encoder depends on the pre-trained model. However, current experiments only focus on SimCLR. It is interesting to see whether the proposed framework works well with other self-supervised learning methods.\n\n10. It is interesting to see whether the disentangled representations would be beneficial.\n\n11. Minor comments: \n- The name of Section 3 could be revised. The memorization effect mainly claims that the deep network would first fit training data with clean labels and incorrect labels. The analysis and discussion are more related to the memorization and impacts of mislabeled data.\n- The citation format needs to be revised. Some ''\\cite{}'' need to be changed to ''\\citep{}''.  \n- It is interesting to show the effect of batch size on the effectiveness of the regularizer since the regularizer is based on self-supervised learning.\n----\n[1] Shikun Li, et al. Selective-Supervised Contrastive Learning with Noisy Labels. CVPR 2022.   \n[2] Diego Ortego, et al. Multi-objective Interpolation Training for Robustness to Label Noise. CVPR 2021.  \n[3] Yiwen Wang, et al. Symmetric Cross Entropy for Robust Learning with Noisy Labels. ICCV 2019.  \n[4] Kimin Lee, et al. Robust Inference via Generative Classifiers for Handling Noisy Labels. ICML 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YY8K5GqjoBv", "reviewer": "Reviewer_L66U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to use regularization between learned representations to study the problem of learning with noise features. The paper offers some theoretical discussions, and proposes a new methods, however, I do not see how the theory is connected with the proposed method other than a generic lesson learned on \"restricting the search space with regularization is a reasonable approach\", which is true, but probably we do not need an entire theoretical discussion for this lesson. ", "summary_of_the_review": "It's a nice and interesting work, but there are too many issues in the theoretical discussions, I think the work needs a deep and rigorous pass on the theoretical discussions before being published. ", "strength_and_weaknesses": "- strength\n    - the paper studies an interesting problem on learning with noise labels\n    - the paper is condensed with theoretical discussions, might offer interesting results in a broader scope\n- weakness\n    - the discussion of the theoretical result does not seem rigorous enough (see below)\n    - the paper contains a lot of contents, some contents might be better to be moved to appendix. It's quite hard to parse all the theoretical information within the main manuscript. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7aFJo5FDh4G", "reviewer": "Reviewer_DbNo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to tackle the problem of learning with noisy labels from the perspective of representation regularization. In detail, the self-supervised learning (SSL) head and supervised noisy labels head (SL) after the feature extractor (encoder) are utilized to extract SSL features and SL features respectively, and then a regularization loss is proposed to minimize the distance between the SSL features and SL features.", "summary_of_the_review": "With the findings above, I currently give the paper a reject score.\n\nUpdate: I raised my score to 6 after discussing with AC and other reviewers.", "strength_and_weaknesses": "Strength:\nThe proposed framework is extendable and can be plugged into other robust loss functions to further improve performance. \n\nExtensive experiments validate the effectiveness of the proposed method, and theoretical analyses are provided to support the claim.\n\nWeakness: \nThe motivation of the method is hard to follow and not convincing. Although the authors hope to use representation regularizers to cut off some redundant function space without hurting the optima, the actual method proposed in the paper seems to be not related to what the author claims.  Also, what’s the high-level motivation of the proposed method? How does the self-supervised branch work in the method?\n\nIn table 2 and table 3, the improvement of the proposed regularization loss is limited compared with previous methods.\n\nAre there any large real-world datasets with label noise other than toy datasets, such as CIAFR variants? If yes, what’s the empirical performance of those datasets?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6FwvzRP-qU", "reviewer": "Reviewer_Y66C", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work discusses how regularizing the representations of deep neural networks can help improve model generalization when the dataset is noisy. In particular, the authors first discuss how limiting the capacity of a neural network can help improve generalization in noisy data settings. This is achieved via the regularization of the representation space of the model. Regularization is performed by penalizing the distance between the representations from the projection head of SSL and SL features. Experimental results are demonstrated on various noise types and noise severities to suggest that the additional regularization objective can help improve the generalization performance of various noisy data training algorithms.", "summary_of_the_review": "The paper shows strong empirical gains in mislabeled data settings, but misses comparisons with state-of-the-art methods. I am unsure about the significance of the theoretical results.\n\nRaising score to 8 after author rebuttal", "strength_and_weaknesses": "## Strengths\n1. The key strength of this work is that it utilizes the advances in contrastive learning to regularize the representations in a supervised learning framework.\n2. The experimental observations complement some of the theoretical implications. (I was not able to appreciate the theoretical results and found them less intuitive, but I would like to see what other reviewers feel about it)\n3. The regularization method can be added on top of any existing method for training under label noise and results in consistent gains on the CIFAR10 dataset.\n4. I would have liked to see a larger portion of the paper devoted to the empirical analysis and ablations which is the strength of the work in my view.\n\n\n## Weaknesses and Questions:\n1. Theoretical Analysis: The theoretical analysis and assumptions seem to be very distant from the empirical approach. \n   (1) The notations are very confusing and non-succinct. \n   (2) The assumptions such as 0 variance, and infinite training data size are unrealistic. \n   (3) Connection between the theoretical results and the empirical approach is weak -- Gaussian features and discussion on the relevance of the theoretical analysis on the empirical observations.\n\n2. Figure 2 is very confusing till we read the first paragraph of page 5. Similarly, in Figure 1, what is the training setup? these things should ideally be self-contained in the captions.\n\n3. What is the problem setup in Figures 4 and 5? What is the noise level for asymmetric and instance-level noise? How many samples get down-sampled?\n\n4. Experimental Configurations: How was the value of $\\lambda$ selected? What was the pertaining dataset for the encoder? Which augmentations were used?\n\n4. Comparisons with state of art defenses such as ELR and SOP are missing. Refer to this paper for a list of more competitive baselines: https://proceedings.mlr.press/v162/liu22w.html\n\n5. It is suggested to fix the encoder for high symmetric label noise based on Figure 5. Are you doing this for the table numbers?\n\n6. How is observation 3 explained? What is its relevance? Can you provide some more information about the same?\n\n7. Results in Table 1 are only provided for the CIFAR10 dataset. This is not comprehensive enough.\n\n## Other Comments\n1. \\citet{} and \\citep{} have been used incorrectly throughout the paper. Please fix this.\n2. Figure 4 appears after Figure 5\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5plqp0246h", "reviewer": "Reviewer_kNp2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes how the representation learned by a DNN and its downstream generalization is affected by noise in the training labels. The key insight in this paper is that the authors suggest to decouple the classifier into an encoder and a linear model. The encoder is fixed for a downstream task, while the 'linear' model component is fine-tuned. This insight comes from the fact that 'smaller' (in terms of VC dimension) hypothesis classes have more difficulty learning label noise. Consequently this can serve as a regularizer on the task. Given noisy label scenario, the paper gives an upper bound on the estimation error for a particular hypothesis class that depends on the amount of label noise (through a $(1 - \\epsilon^2)$ term where $\\epsilon$ is the noise parameter), and VC dimension of the model class. A version of this theorem for a fixed decoupled encoder+linear model setting  suggests that the fixed encoder setting leads to a reduction in the ability of the class to fit noisy labels at the expense of higher approximation errors. Overall, this  insight is instantiated on an SSL setting where a precise relationship is revealed between the quality of the representations and the error on noisy examples. The paper concludes with experiments to back up this claim.", "summary_of_the_review": "This paper is well-written and clear. Overall, the paper provides an important insight about how to stem a model's reliance on  noisy labels using a fixed encoder and regularizing a model's representations.", "strength_and_weaknesses": "### Strengths\n- The key insight of this work is that constraining the representation learning portion of a classifier, to be fixed, helps regularize the ability of the classifier to memorize noisy labels. This insight is demonstrated effectively through the theorems and empirical analyses. \n\n\n### Weaknesses\n\n- While the theorems are helpful, generally, it is a bit hard for me to read too much into whether they should be meaningful. In the sense that we now know that the VC dimension term, for neural networks, might not encode as much information as we thought about the generalization behavior of the neural networks hypothesis classes.\n\n- Can the authors comment on how other performance properties of these trained models are affected by noise in labels? For example, should these insights also translate to something like the model's calibration?\n\n- The insights from section 5.1 are interesting and it might make sense to discuss these in the introduction. However, the results seem to suggest that one needs to one which noise regime that one is in. For example, for a task, in practice, I would have to know when the data has high symmetric label noise or how much bias to expect. These seem like challenging things to know ahead of time.\n\n- Some papers that might be useful to consider: Feldman et. al. (Does learning require memorization? a short tale about a long tail), and Li et. al. (How does a Neural Network's Architecture Impact its Robustness to Noisy Labels?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster