Robust early-learning: Hindering the memorization of noisy labels
Authors: Xiaobo Xia
Tongliang Liu
Bo Han
Chen Gong
Nannan Wang
Zongyuan Ge
Yi Chang
Venue: ICLR 2021 Poster (2021)

Abstract: The \textit{memorization effects} of deep networks show that they will first memorize training data with clean labels and then those with noisy labels. The \textit{early stopping} method therefore can be exploited for learning with noisy labels. However, the side effect brought by noisy labels will influence the memorization of clean labels before early stopping. In this paper, motivated by the \textit{lottery ticket hypothesis} which shows that only partial parameters are important for generalization, we find that only partial parameters are important for fitting clean labels and generalize well, which we term as \textit{critical parameters}; while the other parameters tend to fit noisy labels and cannot generalize well, which we term as \textit{non-critical parameters}. Based on this, we propose \textit{robust early-learning} to reduce the side effect of noisy labels before early stopping and thus enhance the memorization of clean labels. Specifically, in each iteration, we divide all parameters into the critical and non-critical ones, and then perform different update rules for different types of parameters. Extensive experiments on benchmark-simulated and real-world label-noise datasets demonstrate the superiority of the proposed method over the state-of-the-art label-noise learning methods.

Published as a conference paper at ICLR 2021
ROBUST EARLY-LEARNING, HINDERING THE MEMORIZATION OF NOISY LABELS
Xiaobo Xia' Tongliang Liu' Bo Han
Chen Gong Nannan Wang' Zongyuan Ge Yi Chang'
'Trustworthy Machine Learning Lab, School of Computer Science, The University of Sydney 2 Department of Computer Science, Hong Kong Baptist University
3 Schooo of Computer Science and Engineering, Nanjing University of Science and Technology
"ISN State Key Laboratory, School of Telecommunication Engineering, Xidian University Medical AI Group, Faculty of Engineering, Monash University
6 "iirdoc Research, Monash University
7 School of Artificial Intelligence, Jilin University
ABSTRACT
The memorization effects of deep networks show that they will first memorize training data with clean labels and then those with noisy labels. The early stopping method therefore can be exploited for learning with noisy labels. However, the side effect brought by noisy labels will influence the memorization of clean labels before early stopping. In this paper, motivated by the lottery ticket hypothesis which shows that only partial parameters are important for generalization, that only partial parameters are important for fitting clean labels and generalize well, which we term as critical parameters; while the other parameters tend to fit noisy labels and cannot generalize well, which we term as non-critical parameters. Based on this, we propose robust early-learning to reduce the side effect of noisy labels before early stopping and thus enhance the memorization of clean labels. Specifically, in each iteration, we divide all parameters into the critical and non-critical ones, and then perform different update rules for different types parameters. Extensive experiments on poachaar--simllatd and real-world label-noise datasets demonstrate the superiority of the proposed method over the state-of-te ofte--f-theerrr art label-noise learning methods.
find
INTRODUCTION
Deep neural networks have achieved remarkable success in various tasks, such as image clas- sification (He et al., 2015), object detection (Ren et al., 2015), speech recognition (Graves et al., 2013), and machine translation (Wu et al., 2016). However, the success is largely attributed to large amounts of data with high-quality annotations, which is expensive or even infeasible in practice (Han al., 2018a; Li al., 2020a; Wu et al., 2020). On the other hand, many large-scale datasets are collected from image search engines or web crawlers, which inevitably involves noisy labels (Xiao al., 2015; Li al., 2017a: 7a; Zhu et al., 2021). As deep networks have large learning capacities and strong memorization power, they will ultimately overfit noisy labels, leading to poor generalization performance (Jiang et al., 2018; Nguyen et al., 2020). General regularization techniques such as dropout and weight decay cannot address this issue well (Zhang et al., 2017).
Fortunately, even though deep networks will fit all the labels eventually, they first fit data with clean labels, which helps generalization (Arpit et al., 2017; Han al., 201 8018b; Yu et al., 2019; Liu et al., 2020). Thus, the early stopping method can be used to reduce overfitting to the noisy labels (Rolnick et al., 2017; Li et al., 2020b; Hu et al., 2020). However, the existence of noisy labels will still adversely affect the memorization of clean labels even in the early training stage. This will hurt generalization (Han et al., 2020). Intuitively, if we can reduce the side effect of noisy labels before early stopping, the generalization and robustness of the networks can be improved.
'Correspondence Tongliang Liu (tongliang. liu

---

Published as a conference paper at ICLR 2021
Note that over-parameetrrization of deep networks is one of the main reasons for overfitting noisy labels (Zhang et al., 2017; Yao et al., 2020a). The lottery ticket hypothesis (Frankle & Carbin, 2018) shows that only partial parameters are important for generalization. The deep networks with these important parameters can generalize well, or even hetter by avoid overfitting. Motivated by this, for learning with noisy labels, it remains a question if we divide the parameters into two parts to reduce the side effect brought by noisy labels, which enhances the memorization of clean labels and further improves the generalization performance of the deep networks.
In this paper, we present a novel and effective method to find which parameters are important for fitting data with clean labels, and which parameters tend to fit data with noisy labels. We term the former as critical parameters, and the latter as non non-critical parameters. Then on this basis, we proposed robust early-learning to reduce the side effect of noisy labels before early stopping. Specifically, in each iteration during training, we first categorize all parameters into two parts, i.e., the critical parameters and the non-critical parameters. Then we designed different update rules for different types of parameters. For the critical ones, we perform robust positive update. This part of the parameters are updated using the gradients derived from the objective function and weight decay. For the non-critical ones, we perform negative update. Their values penalized with the weight decay, and without the gradients derived from the objective function. Note that the gradients for updating are based on the loss between the prediction of deep networks and given labels. For the critical ones, they tend to fit data with clean (correct) labels to help generalization. Their gradients can therefore be exploited to update parameters. However, for the non-critical ones, they tend fit data with noisy (incorrect) labels, which hurts generalization. Their gradients will misguide the deep networks overfit data with noisy labels. Thus, we only use a regularization item, i.c., the weight decay, to update them. The weight decay will penalize their values to be zero, which means that they are penalized to be deactivated, and not to contribute to the generalization of deep networks. In this way, we can reduce the side effect of noisy labels and enhance the memorization of clean labels. In summary, the main contributions of this work are as follows:
We propose a novel and effective method which can categorize the parameters into two parts according to whether they are important to fit data with clean labels.
Different update rules have been designed for different types of the parameters to reduce the side ffect of noisy labels before early stopping.
We experimentally validate the proposed method on both synthetic noisy datasets and real- world noisy datasets, on which it achieves superior robustness compared with the state-of-the-ar methods for learning with noisy labels.
Related Work, Farly stopping is quite simple but effective in practice. It was used in supervised learning early (Prechelt, 1998; Caruana et al., 2001: Zhang et al., 2005; Yao et al., 2007). With the help of a validation set, training is then stopped before convergence to avoid the overfitting. While learning with noisy labels, the networks fit the data with clean labels before starting overfit the data with noisy labels (Arpit et al., 2017). Early stopping is then formally proved to be valid for relieving overfitting to noisy labels (Rolnick al., 2017: Li et al., 2020b). has also been widely used in existing methods to improve robustness and generalization (Yu et al., 2018;; Xu et al., 2019; Yao et al., 2020b; Cheng et al., 2021).
The lottery ticket hypothesis (Frankle & Carbin, 2018) shows that deep networks are likely to be over-parameterized and only partial parameters are important for generalization. With this part of the parameters, the small and sparsified networks can be trained to generalize well. While this work is motivated by the lottery ticket hypotheis, this work is fundamentally different from it. The lottery ticket hypothesis focuses network compression. It aims to find sparsified sub-network which has competitive generalization performance compared with the original network. This paper focuses on learning with noisy labels. We want to find the critical/non-critical parameters to reduce the side effect of noisy labels, which greatly improves the generalization performance.
Lots of work proposed various methods for training with noisy labels, such exploiting a noise transition matrix (Liu & Tao, 2016; Hendrycks et al., 2018; Xia et al., 2020a; Li et al., 2021), using graph models (Xiao et al., 2015; Li al., 2017b), using surrogate loss functions (Zhang & Sabuncu, 2018; Wang et al., 2019; Ma et al., 2020), meta-learning (Ren et al., 2018; Shu et al., 2020), and employing the small loss trick (Jiang et al., 2018; Han et al., 2018b; Yu al., 2019). Some methods among them employ early stopping explicitly or implicitly (Patrini et al., 2017; Xia et al., 2019). We
2

---

Published as a conference paper at ICLR 2021
also early stopping in this paper. We are the first to hinder the memorization of noisy labels with analyzing the criticality of parameters.
Organization. The rest of the paper is organized as follows. In Section 2, setup the problem and introduce the neural network optimization method. In Section 3, we discuss how to find the critical parameters and perform different update rules. In Section 4, we provide empirical evaluations of the proposed learning algorithm. Finally, Section concludes the paper.
2 PRELIMINARIES
Notation. Vectors and matrices are denoted by bold-faced letters. The standard inner product between two vectors is denoted by ) We use || llp as the E, norm of vectors matrices. For function f. we use f to denote its gradient. Let [n] = (1,
classes. Let X' and y be the feature and Problem Setup. Consider a classification task, there label spaces respectively, where X' E I with d being the dimensionality, and y - [c]. The joint probability distribution over X' y is denoted by D. Let S = [x)],,,)]]]= be an i.i.d. sample drawn from D, where n denotes the sample size. In traditional supervised learning, by employing S, the aim to learn a classifier that can assign labels precisely for given instances. While learning with noisy labels, we are given a sample with noisy labels S = F.)) which drawn from a corrupted joint probability distribution D rather than D. Here, y is the possibly corrupted label of the underlying clean label y. The aim is changed learn robust classifier that could assign clean labels to test data by only exploiting training sample with noisy labels.
2.1 NEURAL NETWORK OPTIMIZATION METHOD
The optimization method is essential for training neural networks. Stochastic gradient descent (SGD) is the most popular one nowadays among the optimization methods (Allen-Zhu et al., 2019; Cao & Gu, 2019; Zou et al., 2020). Our proposed method is directly related to SGD. We analyze the optimization problem of typical supervised learning with clean labels as knowledge background. Consider a classifier to be trained, let W E I be all the parameters, where m is the total number of the parameters. Let L TRC y + be the surrogate loss function, e.g., cross entropy loss. With regularization item, e.g., & regularizer, optimization method would involve minimizing an objective function as:
min L(W; = -inn Σ L(W; (x)) + A|III1
where À RR, is regularization parameter. The update rules of the parameters W can be represented by the following formula:
W(k + +(K+ (L(W()); + (Wg)))()))) OW(k)
(2)
where 17 > 0 is the learning rate, W(k) is the set of the parameters at the k-th iteration, sgn(.) is the standard sgn function in mathematics, and S* is subset randomly sampled from S. With SGD, the regularization parameter X is equivalent to the weight decay coefficient in the training process (Loshchilov & Hutter, 2019).
3 METHODOLOGY
In this section, first introduce an alternative interpretation for the optimality criterion (Section 3.1). Then, we present how to determine the pnitica//oon-criten criticai parameters by exploiting this interpretation during training and the memorization effect of deep neural networks (Section 3.2). Finally, different update rules are proposed for different types of parameters to cope with noisy labels (Section 3.3).
3.1 OPTIMALITY CRITERION
For the optimization of the objective function L(W; S), the optimality will achieved at W when VL(W; = 0 (Boyd et al., 2004; Bubeck, 2014). However, modern neural networks are complex and preepparamtteiized which makes VL(W: S) extremely nigh-dimensional. It is unintuitive for us to analyze digenssonnii vectors. The optimality is hard to be effectively judged. To address this issue, we will use a more intuitive interpretation for optimality criterion in this paper, which can associate the optimization with scalar. Specifically, if we let G(l) L(IW;S)
3

---

Published as a conference paper at ICLR 2021
G'(t) S)( T. Let = 1, then ( W). We know that the optimality can be reached when VL(W = 0, then G1(1) O. In this way, the optimality can be checked by exploiting the scalar G' (1).. Note that the new optimality criterion is sufficient, but is not necessary. In this paper, we focus on learning with noisy labels, and the necessity of the new optimality criterion does not affect the effectiveness of the proposed method. We will discuss this carefully in the next subsection.
3.2 JUDGING THE IMPORTANCE OF NETWORK PARAMETERS
We have shown that the optimization of the objective function can related to a scalar G1()). Its value is equal to the inner product between the value of the parameters and the gradient w.r.t. the parameters. To achieve the optimality, we need to push the value of (1) to be zero. Thus, we can judge the importance of each parameter through its influence on the value of (1). Note that the memorization effects of deep networks show they first memorize the data with clean labels. The
parameters that contribute to the optmiality the early stage therefore important for clean labels. Consider a parameter, denoted by w; E W, its gradient is VL(w;; S). The judgement criteria is denoted by gi. i.e.
9: VL(w;() wal. E [m].
If the value of g, is large, w, is viewed as a critical parameter, as gi has a great influence on the value of G' On the contrary, if the value of 9i is small, e.g., zero or very close to zero, W; is regarded to be a non-critical parameter. is not important for fitting clean labels. If we update it, it will tend to fit noisy labels.
The underlying issue of directly using the gradient of L(w,; 5) as a criterion for the criticality can be identified. When we only exploit gradient information, we ignore the value of the parameter w,. However, if the value is zero or close to zero, the parameter is inactivated. It is also non-critical for optimality (Han et al., 2015; Frankle & Carbin, 2018; Lee et al., 2019). Note that we use early stopping in this paper. The deep networks mainly fit clean labels in the early training. Thus, even with the existence of noisy labels, we can use the criterion to analyze the criticality of the parameters.
should be noted when 9 = O, there will be three possible scenarios: (1) only VL(w;; = 0; (2) only w; 0; (3) both V VL(w;; = 0 and w; 0. In all three cases, can judge the importance of parameter Wy using 9i as analyzed. In other words, when f = O, we allow that the value of VL(w,; S) is not zero, i.e., the new optimality criterion is not necessary, which does not influence the effectiveness of the proposed method.
3.3 COMBATING NOISY LABELS WITH DIFFERENT UPDATE RULES
We have presented how to judge the importance of the parameters, and then divide them into the critical ones and non-critical ones. We exploit the label noise rate to help divide the parameters into the critical/non-
ones. Intuitively, if the noise rate is high, the number of clean labels is small. The number of required critical parameters for memorizing clean labels is then small. The number of the critical parameters has a negative correlation with noise rate. We therefore use the noise rate to help identify the critical parameters. We use T to denote the noise rate. If T is not known in advanced, it can be easily inferred (Liu & Tao, 2016; Yu et al., 2018a). We show that the proposed method is insensitive to the estimation result of the noise rate in Section 4.4. Then, the number of the critical parameters can be defined as:
me (1 -1-11 T)m.
In each iteration, for each parameter Wa, i E [m]. The critical and non-critical parameters are determined according to the result of numerical sorting of g., which has been explained before. The critical and non-critical parameters are denoted by W. and W., respectively. Two different update strategies are performed for two types of the parameters.
Robust positive update. For the critical ones We, we use the gradients derived from the objective function and weight decay. We clip the gradients to perform gradient decay in this paper. The update rule is:
W.(k + 1 - W.(k) - (a1 71) (L))) + Asga(WW()))). (W.(k)
(5)
where St is a subset randomly sampled from 3. Note that we directly use 5* here rather than S* in Eq. (2). It is because the proposed method exploits the memorization effects of deep networks.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1 CDR algorithm.
Input: initialization parameters W. noisy training set Dr, noisy validation set D., learning rate n). weight decay coefficient A, fixed т, epoch Z' and Tmax: iteration N maxi
for T 1, 1.2....... Tmax do 2: Shuffle training set D::
for N 1,...,... N max do
3: Fetch mini-batch Dr from Dr;
Divide W into W. and W. with Eq. (3) and Eq. (4); //define the types of the parameters; //update We using the robust positive update; //update Wn using the negative update;
5: Update We with Eq. (5); Update W2. with Eq. (6);
end end
//Early stopping criterion: if the minimum classification error is achieved with W on D.
8: Output: parameters W after update.
Though we have only noisy training data, deep networks will first memorize training data with clean labels. As can be seen in Eq. (5), the gradient decay coefficient is set to 1 - T, which can prevent over-confident descent steps in the training process.
Negative update. For the non-critical parameters Wn. we only use the weight decay to update them. The update rule is:
Wr, + + (,) (W)(k)))ssgn(WW..... (ks)
(6)
The gradients of the objective function exploit loss between the prediction of deep networks and given labels. Robust positive update uses the gradients to update the critical ones, which helps deep networks memorize clean labels. For the non-critical ones, they tend to overfit noisy labels, their gradients misleading for generalization. Thus, we only use the weight decay, to update them. The weight decay will penalize their values to be zero and help generalization (Arora et al., 2018). As they are deactivated, they will not contribute to the memorization or generalization. The use of two update rules makes us achieve the goal, i.e., reducing the side effect of noisy labels and thus enhance the memorization of clean labels. The overall procedure of combating noisy labels with different update rules (CDR) is summarized in Algorithm 1.
4 EXPERIMENTS
In this section, we first introduce the datasets used, and implementation details the experiments (Section 4.1). We next introduce the methods used for comparison in this paper (Section 4.2). The ablation study is conducted to show that the proposed method is not sensitive to the estimation result of the noise rate (Section 4.3). Finally, we present the experimental results on synthetic and real-world noisy datasets to show the effectiveness of the proposed method (Section 4.4).
4.1 DATASETS AND IMPLEMENTATION DETAILS
To verify the effectiveness of the proposed method, we run experiments on the manually corrupted version of four datasets, i.e., MNIST (LeCun et al., 1998), F-MNIST (Xiao al., 2017), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009), and two real-world noisy datasets, i.e., Food-101 (Bossard et al., 2014) and WebVision (Li al., 2017a). MNIST and F-MNIST both have 28x28
gayscale images of 10 classes including 60,000 training images and 10,000 test images. CIFAR-10
and CIFAR-100 both have 32> 32 color images including 50,000 training images and 10,000
test images. CIFAR-10 10 has 10 classes while CIFAR-100 has 100 classes. Food-101 consists of 101 food categories, with 101, 100,00 images. For each class, 250 manually reviewed clean test images are provided as well as 750 training images with real-world label noise. Web Vision contains 2.4 million images crawled from the websites using the 1,000 concepts ImageNet ILSVRC12 (Deng et al., 2009). Following the "Mini" setting in (Jiang al., 2018; Chen et al., 2019; Ma et al., 2020), we take the first 50 classes of the Google resized image subset, and evaluate the trained networks on the same 50 classes of the ILSVRCI2 validation set, which is exploited test set. For all datasets, following prior works (Patrini et al., 2017; Wang et al., 2021b), we leave out 10% training data as a validation set, which is for early stopping.
5

---

Published as a conference paper at ICLR 2021
We consider four types of synthetic label noise in this paper, i.e., symmetric noise, asymmetric noise, pairflip noise and instance- eestacc--dependent noise (abbreviated as instance noise). These settings are widely used in existing works (Ma et al., 2018; Thulasidasan et al., 2019; Pleiss et al., 2020; Wang et al., 2021a). The noise rates T are set to 20% and 40%. The details of the noise setting are described as follows:
this kind of label noise is generated by flipping labels in each class uniformly to incorrect labels of other classes.
Symmetric noise:
Asymmetric noise: this kind of label noise is generated by flipping labels within a set of sim- ilar classes. In this paper, for MNIST, flipping 2 - 7, + 8. 5 + 6. For F-MNIST, flipping T- SHIRT + SHIRT, PULLOVER - COAT, SANDALS + SNEAKER. For CIFAR-1O, TRUCK + AU- TOMOBILE, BIRD - AIRPLANE, DEER -> HORSE, DAT+DOG. For C/FAR- 100, the 100 classes are grouped into 20 super-classes, and each has 5 sub-classes. Each class is then flipped into the next within the same super-class.
Pairflip noise: the noise flips each class to its adjacent class. More explanation about this noise setting can be found in (Yu et al., 2019; Zheng et al., 2020; Lyu & Tsang, 2020).
Instance noise: the noise is quite realistic, where the probability that instance is mislabeled depends on its features. Following (Xia et al., 2020b), we generate this type of label noise to validate the effectiveness of the proposed method.
For MNIST, we train a LeNet (LeCun et al., 1998) with batch size 32. For F-MNIST, we train a ResNet-50 (He et al., 2015) with batch size 32. For CIFAR-10 and CIFAR-100, train ResNet-50 with batch size 64, and typical data augmentations including random crop and horizontal flip are applied. For all the training, we use SGD optimizer with momentum 0.9 and weight decay is set to 10-3 The initial learning rate is set to 10-2 For Food-101, we use a ResNet-50 pre-trained on ImageNet with batch size 32. The initial learning rate is changed to 3, For WebVision, we use an Inception-Resse ResNeet v2 (Szegedy et al., 2016) with batch size 128. The initial learning rate is set to 10-1 We set 100 epochs in total for all the experiments. For fair comparison, all the codes are implemented in PyTorch 1.2.0 with CUDA 10.0, and run on NVIDIA Tesla V100 GPUs. Our implementation is available at https:/// http://gttuu.coom. coo/xiaaooxii/CCDRRRR
4.2 COMPARISON METHODS
We compare the proposed method with the following methods: (1) CE, which trains the deep neural networks with the cross entropy loss on noisy datasets. (2) GCE (Zhang Sabuncu, 2018), which unites the mean absolute error loss and the cross entropy loss to handle noisy labels. The hyperparameter q in this work is set to 0.7. (3) DMI (Xu al., 2019), which copes with noisy labels from the perspective of information theory. (4) APL (Ma et al., 2020), which combines two mutually reinforcing robust loss functions. For this baseline, we employ
combination of NCE and RCE for a comparison. (5) MentorNet (Jiang et al., 2018) 8), which learns a curriculum to filter out noisy data. (6) Co-teaching (Han et al., 2018b), which maintains two networks and cross-trains on the instances with small loss. (7) Co-tcaching+ (Yu et al., 2019), which maintains two networks and finds small loss instances among the prediction disagreement data for training. (8) S2E (Yao et al., 2020a), which exploits automated machine learning to handle noisy labels. (9) Forward (Patrini et al., 2017), which estimates the noise transition matrix to correct the training loss. (10) T-Revision (Xia et al., 2019), which employs importance reweighting technique and introduces slack variable revise the noise transition matrix. (11) Joint (Tanaka et al., 2018), which jointly optimizes the network parameters and the sample labels. The hyperparameters a and are set 1.2 and 0.8 respectively. Note that we do not compare with some statc-of-the art methods like SELF (Nguyen et al., 2020) and DivideMix (Li al., 2020a) as baselines because of the following reasons. (1) Their proposed methods are aggregations of multiple techniques while this paper only focuses on one, therefore the comparison is not fair. (2) We are focusing on proving the concept, i.e., how to reduce the side effect of noisy labels before early stopping, but not on boosting the classification performance.
4.3 ABLATION STUDY
We need the noise rate to determine the number different types of the parameters and the gradient decay coefficient mentioned above. Compared with symmetric noise, asymmetric noise and pairflip noise, the noise rate of instance noise is hard to be estimated (Cheng et al., 2020; Xia et al., 2020b). We present that our proposed method is insensitive to the estimation result of noise rate. The experiments are conducted on CIFAR-IO and CIFAR-1O0 datasets with instance noise. The noise rates set to 20% and 40%, respectively. In Figure 1, we show that how the elassificaition performance

---

Published as a conference paper at ICLR 2021
Table 1: Mean and standard deviations of classification accuracy (percentage) on synthetic noisy datasets with different noise levels. The experimental results are reported over five trials. The best mean results are bolded.
Dataset
Method
20%
Symmetric
40%
20%
Asymmeetric
40%
20%
Puirlii
40%
20%
Instance
40%
98.60.1.007
96.18.0..16
99.00.00.08
98.2
08.74.1..17
94,0810,95
98.14:10.02
92.76.10.21
GCF
98.84:40.11
18.12.00333
98.92:10.09
00.21
98.9411.055
97.39-40.62
98.17:10008
94.97-10.32
DMI
98.94::0.02
98.62=0.17
90.07
08.61=0.26
98.96:-012
97.27=0.39
98.
98.3440.17
95.34+:0.28
APL.
98.74:10.09
97.0411.335
98.90:10.07
97.23.10.33
98.24.10.39
95,2411.49
97.03:10.29
90.04.1..93
MentorNet
97.21+0.13
93.96=0..76
98.51
93.47=0..0
97.25::0:2
93,27-40.33
95.17:00.26
90.05
MNIST
Co-teaching
97,22++0.18
94.64+0.33
98.63+0.12
93.62=1.277
97.44:+0:26
94.81=0445
97.32=+0.15
92.45+00.59
Co-teaching+
98.11:10.07
95.87
98.83:10008
9.76
98.81:10011
95,42:00333
98.07:10.122
94.37:00.48
S2E
98.93.10.39
93.2112.33
99.23.10.07
0..3
99.10:10:04
30,15
98.42.10.47
03.38.10.4
Forward
98.10:::.11
96.83-0.28
98.79:40.8
97.94=0.77
98.62:0.011
16
95
97
18
T-Revision
98.93=+0.07
08.40=0.553
99.05=00.16
08.23=0.54
98.82:40007
97.43=0.19
98.33=00155
95.644-0.34
Joint
98.54:4-0113
08.30=00.28
98.96--00005
98.40=0.11
98.70=20.08
9..
98.11-20.13
93.15+0.43
CDR
99.00-00004
9.180=20.22
99.30:50.04
98.80=0.17
99.17=0008
98.12=0.23
98,4900099
94.45:1.044
CE
90.3620.21
87.61
91.31
86,4322011
91.6520122
76,4224.13
88.1==0677
28.0242.22
GCE
91.77:+0.13
90.02=0.37
91.45:00229
73.62+2.92
91.99:00.66
84.21+22.05
91.06=10.55
74.82:00.94
DMI
91.87:1:0.66
92.33:10.11
89.62.10.48
91.33-10.37
83,93:10.22
90.87:10115
80.51110.6
API
87:23:1:0119
73.62.10.88
86.21110.24
81.03
84.52.10.73
26.39-12.85
84.38.10.70
60.38.18.1 17
F-MNIST
MentorNet
88.12:40.12
86.05+0.27
89.7640.18
68.93+3.20
87.39-10.57
76.90-5.72
86.00+0.26 26
78.37+
Co-teaching
89.03++0.32
87.04=0.69
92.03-+0.16
72.23+4.38
89.63+00.78
14=1022
89.27=0.86
13,49+1.27
Co-teaching+
91.34:1:0117
90,23+00211
83.98111055
66.27
91.08:1:0.25
02.65:00.49
83.78:10.73
38.79-19.93
S2E
90.89.10.27
33.68.13.73
91.20.10.11
87.0610.500
91.52.10.19
72.09
89.1711.322
72.62.12.73
Forward
90,72:t:.11
0.73
92.05:10.21
03.42=00,74
90.02:10.87
03.06-20.79
87.95:
15.341119
T-Revision
91.95.10.20
92.07.10.11
0..5
91.06.10.19
85.67.10.88
91.05510.28
Joint
82.01:40.77
72.36-2..84
85.92-4-0.83
73.09-00.91
86.04-10.99
30.87+3.95
82.07-40.94
50.62-44.77
CDR
92.24::0.11
9.29
93.01-50.14
0.33
93.66:0019
87.55=1007
91.52=0.17
85.04+1.02
CE
89.14:0.41
1.325
88.21=00199
86.3
1.03
09.6200.72
86:53=0.77
86.73 36 75.33-22.72
QCE
89,48:+0.28
86.07=0.41
89.03=+0.21
84.12 1.24
88.58:10.34
83,23+3.98
88.02:00.44
04
76.89--0.96
DMI
89.29:10.30
1.07
89.37-10.82
86.32
18.4111.01
8.702
88.93:10.29
79.35
API
88.2111..32
1.30
89.03:10075
85. 2,
87.34.1:44
14
10.12.13365
76.31:12.24
40.73.1..89
MentorNer
83,26::0.72
1.73
84.07
60,22:3.47
89
09,37+3.28
83.06:10.92
19.40:2..1
CIFAR-10
Co Co-tennhing
88.20.1:0.27
84.45.10.68
87.4210.38
64.03
62.66.10.32
73.6810.62
86.71
1..14
Co-teaching+
86.47:400.2
78.9340.77
45
1743344
14.01
1.3
85.92.40.00
57.9543317
S2E
90,26-+-0.24
25.20=2..05
90.73++0.32
07
89.92=0.37
1,61
90.32:0.21
1.
Forward
88.36:10.34
86.47
89.30+0.71
85.33
87.62:10.24
1.32
85 03.39:1..23
1.68
T-Revision
89,43.10.62
86.98.10.77
89.9410.744
88.11.1.22
91.01110.29
1.3
90.43.10.38
85.46.1
Joint
89.94.10.25
87.17
90.83:10.18
08.2410.79
91.31
15.6221..75
90.13-10.34 34 85.23.10.44
CDR
90.6:10.31
87.19:0.43
92.00:0.227
88.68=0667
921135023
0.
91.4=:023
86.2510.57
CE
63.93:072 36.8220.82
64.12:50:54
$28620092
64.10:20:6
$2.7110.19
63.33:0:299
508410899
GCE
65.62++0.82
57.97=1.21
65.34:10.4
14.35=1.28
6..32+1004
55.03=1.225
66.67:00.0
1571441.77
DMI
62.77:+:0:2
57.42=0.53
64,30+:0.4
51.31+2.73
58.77=0064
42,89=00.7
59.04=20355
0.699-+0.62
APL
59.37:40.82
51.03=1.04
54.31+0.84
18.2241.35
59.77-10.74
53.25+0092
49.1742.72
38.18:4..04
CIFAR-100
MentorNet
57.27:::1.32
49.011:2.09
54,1010.92
1.821=1.82
54.73:::226
45.31
50.02:1:0.33
1.627
Co-teaching
61.47.10.41
53,44110.40
57.35
37.62
58.11:1047
48.4610.64
57.73.10.37
43.28.10.55
Co-teaching+
64.13::0.32
55.9220.811
58.9741119
16.2..7
56.31:041
38.03:0.55
55
11.11
S2E
64.21+00.72
43.1242..77
63.92::0.66
42.45=1.733
58.21:0.43
41.74+2.09
61.08:0.59
47.06
Forward
54.88:10.92
1.764:11.77
53.84
58.37-10.56
19.82-0.733
58.55:10.31
46.42-40.95
T-Revision
64,67+0.38
1.01
68.02+0.53
54.93=1.09
62.69+0.73
12,31=146
60.22-+0.68
10,23+
Joint
66.12::0.42
59.45200.88
68.2914025
55.53=0.47
67.35-10.31
12.22+1885
65.91-10.3
55.09110.95
CDR
68.6810.33
62.72
0.38
70.64 55.58
71.93 0.37
36.9411.00 69.82
0.42
6.77
of the proposed method varies with the change of the estimated noise rate. We can clearly see that the proposed method robust to the estimation result of the noise rate.
In this paper, we set the proportion of non-critical parameters to the noise rate. Also, it is interesting to investigate that how the proposed method works we set the proportion of non-critical parameters as constant number. Note that if we set the constant number too randomly, the performance of the proposed method may be hurt. Therefore, we use noisy validation set to locate and compare the difference between the located constant and the noise rate. The search of the constant is within the range (0.10, 0.20, 0.903. The experiments are conducted on MNIST, F-MNIST, and CIFAR- 10. The experimental results provided in Table 2. As we can see, in many cases, the located constant and the label noise rate are numerically equal. However, it is complicated to locate a suitable constant with noisy validation set, there is huge search range. On the contrary, the noise rate always can be estimated effectively (Liu & Tao, 2016; Yu et al., 2018a). Therefore, it is reasonable and feasible to assume that the proportion of non-critical parameters is the same as the noise rate.
7

---

Published as a conference paper at ICLR 2021
NNSORCO-4EN)
Tol
044
(a)
rate
(c)
Illustration of robustness to the estimation result of noise rate. For each estimated noise rate, we report experimental results over five trials. The blue dots represent the result of each experiment. The orange dots represent the mean of five experimental results in each case.
Figure 1:
Table 2: The located constant on synthetic noisy datasets with different noise levels. The result with underline means that the located constant and the noise numerically equal.
Symmetric
Asymmetric
Pairflip
Instance
Dataset
20%
40%
20%
40%
20%
40%
20%
40%
MNIST
0.20
0.50
0.20
0.40
0.20
0.50
0.30
0.60
F-MNIST
0.20
0.40
0.20
0.50
0.30
0.50
0.20
0.60
CI
0.30
0.50
0.30
0.30
0.30
0.50
0.20
0.40
Fiice Swwenss AWM
CV44 3N3 40M
JCAM 200, FARE)-
(a)
(b)
(c)
(d)
Figure 2: Illustration of the experimental results on noisy CIFAR- We can elearly see that the proposed method (CDR) can reduce the side effect of noisy labels the early training stage, which improves generalization (red line vs. green line).
4.4 CLASSIFICATION PERFORMANCE ON NOISY DATASETS
Results on synthetic noisy datasets. Table 1 shows the experimental results on four synthetic noisy datasets with various types noisy labels. For MNIST, as can be seen, our proposed method produce the best results in the vast majority of cases. When the noise is instance-dependent. the proposed method achieves competitive results. Note that T-Revision achieves the best classification performance in this case. Compared with the other synthetic noisy datasets, MNIST is less challenging. In this case, T-Revision thus can exploit the noise transition matrix and the slack variable to well model label noise, which leads to the best performance. However, for the instance- deeenndddddedeeet label noise the other datasets, i.c., F-MNIST, CIFAR- 10, and CIFAR- 100, estimating the transition matrices does not work well and the proposed robust eary eary-learning method achieves the best performance.
8

---

Published as a conference paper at ICLR 2021
Table 3: Classification accuracy (percentage) on foo dataset. The best result is in bold.
CE
GCE
DMI
APL
MentorNet
Co-teaching
84.03
84.96
85.52
82.17
81.25
83.73
Co-teaching+
S2E
Forward
T-Revision
Joint
CDR
76.89
84.97
85.52
85.97
83.10
86.36
Table 4: Top-1 validation accuracies (percentage) on clean ILSVRCI2 validation set of Inception- ResNet v2 models trained on WibVision dataset, under the "Mini" setting in (Jiang et al., 2018; Chen et al., 2019; Ma et al., 2020). The best result in bold.
CE
GCE
DMI
APL
MentorNet
Co-teaching
57.34
55.62
56.93
61.27
57.66
61.22
Co-teaching+
S2E
Forward
T-Revision
Joint
CDR
33.26
54.33
56.39
60.58
47.60
61.85
For F-MNIST and CIFAR-1O, 10, proposed method is consistently superior other state-of-the-ar methods across all the settings, Note that S2E achieves impressive performance in the case of Symmetric-20% on CIFAR-IO. However, fails to generalize well compared with the proposed method, especially in the cases of 40% label noise rate. In contrast, CDR achieves a clear lead over S2E in these cases, which verifies the effectiveness of the proposed method. Lastly, for the more challenging dataset, i.e., CIFAR-100, the proposed method once again outperforms all the baseline methods. particular, in the very challenging case of Instance-40%, the proposed method takes a nearly 6% lead compared to the second best method GCE.
Results on real-world noisy datasets. The experimental results on Food-101 and WebVision datasets are reported in Table 3 and 4. Again, classification accuracy, our method surpasses all other baselines. This verifies the effectiveness of our proposed method against real-world label noise.
Note that the CE method exploits early stopping in all experiments. Compared the classification performance of CE with the classification performance CDR, i.e., carly stopping vs. robust early stopping, can clearly see that the proposed method can achieve better performance. We also present the illustration of the experimental results on CIFAR-100 with 40% noise. As shown in Figure 2, CDR can effectively reduce the side effect of noisy labels at the early training stage. The illustrations of the experimental results on the other settings can be found in Appendix A.
5 CONCLUSION
In this paper, motivated by the lottery ticket hypothesis, we provide novel method to distinguish the critical parameters and non-critical parameters for fitting clean labels. Then we propose different update rules for different types of parameters to reduce the side effect before early stopping. The proposed method very effective for learning with noisy labels, which supported by experiments on synthetic datasets with various types of label noise as well as real- real-world datasets. Our method is simple and orthogonal to other methods. We believe that this opens up new possibilities in the topics of learning with noisy labels. It would be interesting explore the potential characteristics of the updated parameters such as the distance to initial parameters (Li et al., 2020b; Hu et al., 2020) and the mutual information with the vector of all training labels given inputs (Harutyunyan al., 2020).
ACKNOWLEDGMENTS
TLL. was supported by Australian Research Couneil Project DE- 19010143 and 10-1001434 BH was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202, HKBU Tier-1 Start-up Grant, and HKBU CSD Departmental Incentive Scheme. CG was supported by NSF of China (No. 61973163 162) and CCF-Tencent Open Fund (No: RAGR20200101, NNW was supported by National Key Research and Development Program of China under Grant 2018AAA0103202. ZYG supported by the Airdoc-Monate research centre fellowship. YC was supported by the NSF of China (No.6197610,, No.U19A) We thank anonymous reviewers for giving constructive comments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In NewrIPS, pp. 6158-6169, 2019.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Zhang. Stronger generalization bounds for deep nets via a compression approach. In ICML, pp. 254-263, 2018.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. arXiv preprint
2017.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative compo- nents with random forests. In ECCV, 2014.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. university press, 2004.
Cambridge
Sébastien Bubeck. Convex optimization: Algorithms and complexity. arXiv preprint arXiv: 1405,4980, 2014.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. NeurlPS, pp. 10836-10846, 2019.
Rich Caruana, Steve Lawrence, and C Lee Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In NeurlPS, pp. 402-408, 2001.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. arXiv preprint arXiv: MXXiv:905..5500 00000000000 2019.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: sample sieve approach. In ICLR, 2021.
Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance-and label-dependen label noise. In /CML, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. leee, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint /803.03633 2018.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649. IEEF, 2013.
Bo Han, Jiangehao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In NeurIPS, pp. 5836-5846, 201 8a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, pp. 8527-8537. 2018b.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor W Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In ICML, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NewrlPS, pp. 1135-1143 2015.
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information in neural network weights. In ICML, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint 1rXiv:15/2.03388 2015.
10

---

Published as a conference paper at ICLR 2021
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NewrIPS, pp. 10456-10465. 2018.
Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In ICLR, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pp. 2309-2318, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The MNIST database of handwritten 1998.
digits. http://aam..ecuu..coomeedebeaaae
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based connection sensitivity. In ICLR, 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels semi- supervised learning. In /CLR, 2020a.
Mingehen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. AISTATS, 2020b.
Wen Li, Limin Wang. Li Wei, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 1708iv:700.0088800 02862, 2017a.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. arXiv preprint arXiv:2102.024000 2021.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In /CCV, pp. 1910-1918 1918, 2017b.
Sheng Liu, Jonathan Niles Niles-Weed, Narges Razavian, and Carlos Fernandez-Grrane Farly-learning regularization prevents memorization of noisy labels. arXiv preprint arXiv:2007.0015// 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3):447-461, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2020.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-diien learning with noisy labels. JCML, pp. 3361-3370, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICML, 2020.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning filter noisy labels with self-ensembling. In ICLR, 2020.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: loss correction approach. In CVPR, pp. 1944-1955 2017.
Geoff Pleiss, Tianyi Zhang. Ethan R Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. arXiv preprint arXiv:2001.10528, 2020.
Lutz Prechelt. Early stopping- but when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 1998.
11

---

Published as a conference paper at ICLR 2021
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331-4340, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS. pp. 91-99, 2015.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: 10694, 2017.
Jun Shu, Qian Zhao, Zengben Xu, and Deyu Meng. Meta transition adaptation for robust deep learning with noisy labels. arXiv preprint arXiv:2006. 206900006.077, 2020.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alex Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.0726. 2016.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, 2018.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Combating label noise in deep learning using abstention. In ICML, pp. 2234-6243 2019.
Qizhou Wang, Bo Han, Tongliang Liu, Gang Niu, Jian Yang, and Chen Gong. Tackling instance- dependent label noise via a universal probabilistic model. In AAAI, 2021a.
Qizhou Wang, Jiangchao Yao, Chen Gong, Tongliang Liu, Mingming Gong, Hongxia Yang, and Bo Han. Learning with group noise. In AAAI, 2021b.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019.
Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A new perspective on learning with label noise. arXiv preprint arXiv:2006.000 07831, 2020.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao,, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 100000000111000 144 2016.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. 2019.
Are anchor points really indispensable in label-noise learning? In NeurlPS, pp.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, and Yinian Mao. Extended T: Learning with mixed closed-set and open-set noisy labels. arXiv preprint arXiv:2012.000932, 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurlPS, 2020b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:1708.0772: 2017.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi: A novel information-theoreti loss function for training deep nets robust to label noise. In NewrIPS, pp. 6222-6233, 2019.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James T Kwok. Searching to exploit memoriza- tion effect in learning with noisy labels. In ICML. 2020a.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: Reducing estimation emor for transition matrix in label-noise learning. In NewrIPS, 2020b.
12

---

Published as a conference paper at ICLR 2021
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289 315, 2007.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? In ICML, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In CVPR, pp. 4489. 2018a
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV, pp. 68 83, 2018b.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Tong Zhang, Bin Yu, et al. Boosting with early stopping: Convergence and consistency. The Annals of Statistics, (3(44):5388153) 2005.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, pp.
2018.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, 2020.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In CVPR, 2021.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109(3):467-492 2020.
13

---

Published as a conference paper at ICLR 2021
A DETAILED EXPERIMENTAL RESULTS
In section 4.4, provide the illustrations of the experimental results. However, because of limited space, we only show the illustrations of the experimental results on noisy CIFAR-1O0. The noise rate is set to 40%. In this supplementary material, we provide the illustrations of the experimental results on the other employed datasets and settings as follows.
C24
(a)
(b)
(c)
(d)
Figure 3: Illustration of the experimental results on noisy MNIST. The noise rate is set to 20%.
(N6.T. www.S..nnacccc.co 4010
Seme m
ESA
COR
(a)
(b)
(c)
(d)
Figure 4: Illustration of the experimental results on noisy MNIST. The noise rate is set to 40%.
maverr: Spme
(TANST ApTmertao00
(TAMSTT FanEEs
(TANSST POM
CTR
CTR
CTR
(a)
(b)
(c)
(d)
Figure 5: Illustration of the experimental results on noisy F-MNIST. The noise rate is set to 20%
14

---

Published as a conference paper at ICLR 2021
C24
(a)
(C)
(d)
Figure 6: Illustration of the experimental results noisy F-MNIST. The noise rate is set to 40%.
Opech
(a)
(d)
Figure 7: Illustration of the experimental results noisy CIFAR-IO. The noise rate is set 20%.
(a)
Figure 8: Illustration of the experimental results noisy CIFAR-1O. The noise rate is set to 40%.
(a)
(d)
Figure 9: Illustration of the experimental results on noisy CIFAR-108 The noise rate is set to 20%.
15


Reviews:
[{"review_id": "wtHgi5ZzzX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to exploit the early stopping method to solve the problem of learning with noisy labels. Specifically, this paper finds that only partial parameters (critical parameters) are important for fitting clean labels and generalize well; while the other parameters (non-critical parameters) tend to fit noisy labels and cannot generalize well. Based on this observation, this paper proposes to divide all parameters into the critical parameters and non-critical ones, and perform different update rules for the two types of parameters, in each iteration. Extensive experiments on benchmark-simulated and real-world label-noise datasets demonstrate the effectiveness of the proposed method.\n\nThis paper has the following advantages:\n1.\tThis paper is well-written and the motivation is very clear. This paper has clearly explained the two types of parameters, i.e., critical parameters and non-critical parameters.\n2.\tThis paper proposes a novel method with an interesting idea. Unlike many existing methods that are aggregations of multiple techniques, this paper only focuses on one concept. It is simple but effective. The view of updating different parameters by different rules is quite novel for learning with noisy labels. I think this view may bring some new insights to the area of learning with noisy labels.\n3.\tExperiments are quite thorough and results on both synthetic and real-world datasets validate the effectiveness of the proposed method.\n\nThis paper also has some minor issues:\n1.\tI would suggest the authors to carefully check the notations used in the paper. For example, $\\mathbf{w}$ denotes all the learnable parameters of the model in the paper, while $\\mathbf{w}$ usually means a vector. So I would suggest the authors to use another symbol to denote the set of all the learnable parameters of the model, e.g., $\\mathcal{W}$.\n2.\tIt is not well justified why this paper chooses the $\\ell_1$ regularizer. Is there any consideration to use the $\\ell_1$ regularizer except that could be associated with weight decay?\n3.\tThe hyper-parameter $\\tau$ (noise rate) needs to be known. It may not be a big issue as the ablation study in this paper demonstrates that the proposed method is insensitive to this hyper-parameter.\n\nOverall, I think this is a good paper with an interesting idea and convincing experimental performance. So I prefer to accept it.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "i_fZV9ttgbw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------Overall------\nThis paper utilizes the memorization effects of deep models and aims to improve their robustness to noisy labels before early stopping. As the deep models fit training data with clean labels in the early stage of training, the authors propose a novel method to identify those more important parameters for fitting clean labels. They then deactivate the unimportant parameters to reduce side effects brought by noisy labels, which enhances the fitting to clean labels implicitly. I think this paper is interesting and makes sense. The major comments and issues are as follows: \n\n------Major comments------\n1. Different from other complex methods for learning with noisy labels , this work discusses that standard cross entropy loss can achieve competitive performance with early stopping. We can therefore focus the training stage before early stopping to handle noisy labels. The authors skillfully allow the optimality to be checked by a scalar, and then judge the importance of the parameters by analyzing the influence of the parameters on this scalar. The idea of this paper is novelty and meaningful. \n\n2. The paper is very well-written. The description of its motivation and technical details is clear and flows smoothly, which makes it easy for readers to understand the core idea of this paper and follow its implementation details.\n\n3. The experimental results are convincing. The authors provide a very detailed description of experimental settings. Besides, this paper exploits multiple methods for comparison and considers various noise settings to verify the effectiveness of the proposed method. The experimental results on synthetic and real-world datasets are convincing. The authors also perform an ablation study to present the proposed method is insensitive to the estimation of noise rate. \n\n------Issues------\n1. I only find the illustration of comparison between CE and CDR in the case of noisy CIFAR-100. This paper aims to reduce the side effect of noisy labels before early stopping, thus CE is an importance baseline in this paper. Can the authors add illustrations of the experimental results in other cases like Figure.2? \n\n2. The baselines and experimental results are sufficient. Could the authors add some introduction for the baselines and more detailed discussion for experimental results. \n\n3. The authors may need add some explanation for Eq.(2) and Eq.(5). The proposed method makes use of the memorization effects of deep models. The authors directly write \\tilde{S} rather than S in Eq.(5). However, this may be easy to misunderstand. I suggest that the authors can emphasize it or change it. \n\n4. Some typos need to be corrected. (1) “The underlying issue of directly using the gradient of......”; (2) “Robust positive update uses the gradients to update the critical ones......”.\n\n5. Some minor comments. (1) The experimental results in Table 1 are too dense. (2) The figures are not readable, especially the title is small for me. This makes it a little hard to match the figures with specific cases. (3) The parameter (noise rate) $\\tau$ still needs to be estimated, which may be challenging. It will be promising to automatically set this parameter during training. Thus, a more in-depth analysis is worthy of further learning. \n\nI hope the authors can address these issues carefully to improve this work. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "VIojxqDxeMz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for deep learning with noisy labels, which distinguishes the critical parameters and non-critical parameters for fitting clean labels and updates them by different rules. The method is easy to implement and the empirical results are promising. Experiments on both simulated and real-world datasets show it reaches new state-of-the-arts results.\n\nQuestions:\n- I don’t think the ablation is convincing – why does the proportion of non-critical parameters is assumed to be the same as the noise rate? The ablation only shows the method is insensitive when the estimation of the noise rate is not precise. However, why do we need to estimate the noise rate? What if the proportion of non-critical parameters is assumed to be a constant number? In other words, what will the performance be like if the estimation is largely different from the real noise rate? \n- Since the validation set is also noisy, why does the early stopping criterion adopts the minimum classification error on it? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5OuwXvLJRUD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of learning with noisy labels and proposes a novel method CDR which is inspired by the lottery ticket hypothesis. In particular, the proposed method categorizes the parameters into two parts, including critical parameters and non-critical parameters, and applies different update rules to these parameters. Using comprehensive experiments on synthetic datasets and real-world datasets, the authors verify that the proposed method can improve the robustness of the classifiers against noisy labels.\n\nPros.\n1. The proposed method is interesting in its design. The authors provide an alternative interpretation for the optimality criterion, which reveals that the value of the parameter can also be used to check the optimality. It is novel and of significance. Meanwhile, the proposed method is easy to implement. This method can also be applied to existing algorithms to further improve their robustness.\n2. Overall, this paper is very well written and well organized. The technical details are easy to follow, and Algorithm 1 helps understand the procedures. \n3. Extensive experiments are performed to verify that the proposed method indeed helps over the baselines at fighting noisy labels. I like the details of experimental settings, which really can help reproduce these experimental results.\n\nCons.\n1. The new interpretation for the optimality criterion is important, but lacks detailed explanation. I find that there is only simple analysis. Perhaps the authors could add some details or citations for better understanding.\n2. Though the baseline S2E uses AutoML, it seems that S2E is an improvement on co-teaching or co-teaching+? It is not suitable to place it at the end. \n3. For some baselines such GCE and Joint, there are hyperparameters to consider. The authors should explain how to set their values for a fair comparison. For APL, there are multiple combinations of loss functions. In this experiment, which one did you choose? I suggest that the authors add such explanation in Section 4.2, which will make the results more convincing. \n4. The proposed method implicitly exploits the memorization effects of deep models, and can reduce the side effect of noisy labels before early stopping. After early stopping, noisy labels still affect the performance. How to reduce the side effect during the whole training by using this idea? I personally think this is an interesting and meaningful direction. The authors can regard this as future work to improve this paper. \n\nOverall, I think this paper makes sense in learning with noisy labels. The proposed method is novel and effective. I recommend to accept this paper, and hope that the authors can address the above issues carefully.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel method for learning with noisy labels based on an interesting insight into the learning dynamics of deep neural networks. 

Reviewers unanimously vote for acceptance. I agree with their assessment, and it is my pleasure to recommend the paper for acceptance. 

If I can draw attention to one comment, I strongly agree with R1 that the criterion in Eq. (3) is somewhat poorly motivated. I believe the paper would benefit from a clearer exposition of this part. 

Please make sure to address all reviewers' remarks in the camera-ready version. Thank you for submitting your work to ICLR.
Decision: Accept (Poster)