Label Cleaning with Likelihood Ratio Test
Authors: Songzhu Zheng
Pengxiang Wu
Aman Goswami
Mayank Goswami
Dimitris Metaxas
Chao Chen
Venue:  (2020)
Keywords: Deep Learning

Abstract: To collect large scale annotated data, it is inevitable to introduce label noise, i.e., incorrect class labels. A major challenge is to develop robust deep learning models that achieve high test performance despite training set label noise.  We introduce a novel approach that directly cleans labels in order to train a high quality model. Our method leverages statistical principles to correct data labels and has a theoretical guarantee of the correctness.  In particular, we use a likelihood ratio test(LRT) to flip the labels of training data.  We prove that our LRT label correction algorithm is guaranteed to flip the label so it is consistent with the true Bayesian optimal decision rule with high probability.  We incorporate our label correction algorithm into the training of deep neural networks and train models that achieve superior testing performance on multiple public datasets.

Under review as a conference paper at ICLR 2020
LABEL CLEANING WITH A LIKELIHOOD RATIO TEST
Anonymous authors
Paper under double-blind review
ABSTRACT
To collect large scale annotated data, it is inevitable to introduce label noise, i.e., incorrect class labels. A major challenge is to develop robust deep learning models that achieve high test performance despite training set label noise. We introduce a novel approach that directly cleans labels in order to train a high quality model. Our method leverages statistical principles to correct data labels and has theo- retical guarantee of the correctness. In particular, we use likelihood ratio test (LRT) to flip the labels of training data. We prove that our LRT label correction algorithm is guaranteed to flip the label so it is consistent with the true Bayesian optimal classifier with high probability. We incorporate our label correction al- gorithm into the training of deep neural networks and train models that achieve superior testing performance on multiple public datasets.
INTRODUCTION
Label noise is ubiquitous in real world data. It may be caused by unintentional mistakes of manual or automatic annotators (Yan et al. 2014; Veit et al., 2017). It may also be introduced by malicious attackers (Steinhardt et al. 2017). Noisy labels impair the performance of a model (Smyth et al.) 1994 Brodley & Friedl| 1999), ) especially deep neural network, which tends to have strong memo- rization power (Frmay& & Verleysen 2014, Zhhng et al. 2017). Improving the robustness of a model to label noise is crucial yet challenging task in many applications (Mnih & Hinton 2012 Wu al.) 2018). Existing methods mainly follow two directions, probabilistic reasoning and data selecting.
Probabilistic methods explicitly model noise transition matrix, namely, the probability of one label being corrupted into another (Goldberger & Ben--Reueen 2017 Patrini et al. 2017). The transition matrix is often estimated from the data, and is used to re-calibrate the training loss or to correct the prediction. Explicit estimation of the transition matrix can be problematic due to the large variation of noise patterns, c.g., uniform noise, asymmetric noise, or mixtures. Furthermore, the transition matrix size is quadratic to the number of classes, making the estimation task prohibitive when the data has hundreds or even thousands of classes.
Data-selecting methods are agnostic of the underlying noise pattern. These methods gradually col- lect clean data whose labels are trustworthy (Malach & Shalev-Shwarz) [017]] Jiang et al.] [0188] Han et al. 2018). As more clean data are collected, the quality of the trained models improves. The major issue of these methods is the lack of a quantitative control of the quality of the collected clean data. Without principled guideline, it hard to find the correct data collection pace. An aggressive selection can unknowingly accumulate irreversible errors. On the other hand, an overiy-conservative strategy can be very slow in training, or stops with insufficient clean data and mediocre models.
We propose novel method with the benefit from both the probabilistic and the selecceee ap- proaches. Similar to data-selecting methods, our method continuously improves the purity of the data labels by correcting the noise-corrupted ones. Meanwhile, we improve the classifier using the updated labels. Our label correction algorithm is based on statistical principles and theoretically guaranteed to deliver high quality label set. Instead of explicitly estimating the transition matrix, the correction algorithm only depends on the prediction of the current model, denoted as f. Using an f-based likelihood ratio test, we determine whether the current label of each data should be cor- rected. Our main theorem proves that the label correction algorithm will clean a majority of noisy labels with high probability.
In practice, we incorporate the label correction algorithm into the training of deep neural networks. Our method iteratively updates the labels of the data while continuously training a deep neural

---

Under review as a conference paper at ICLR 2020
network. To ensure the deep neural network does not overfit with noise labels that are yet to be cor- rected, we introduce a new retroactive loss term that regulates the model by enforcing its consistency with models in previous epochs. The rationale is that the model in an earlier training stage tends to fit the true signal rather than noise, although its overall performance is sub-optimal. Through experiments on various datasets with various noise patterns and levels, we show that method produces robust neural network models with superior performance.
To the best of our knowledge, our method is the first to correct labels with theoretical guarantees. It is has advantages over both probabilistic methods and data-selecting methods. Compared with other data-selecting methods, it has a better quantitative control of the label quality and thus is less brittle when generalizing to different datasets and different noise patterns. Also note that we are not selecting clean data. Instead, we correct labels and always use the whole training set to train. This brings an additional advantage of fully leveraging the data. Compared with other probabilistic methods, our correction algorithm assumes rather general family of underlying noise patterns and avoids an explicit estimation of the transition matrix.
1.1 RELATED WORK
One representative strategy for handling label noise is to model and employ noise transition matrix to correct the loss. For example, Patrini et al. (2017) propose to correct the loss function with esti- mated noise pattern. The resulting loss is an unbiased estimator of the ground truth loss, and enables the trained model to achieve better performance. However, such an estimator relies on strong as- sumptions and could be inaccurate in certain scenarios. Reed et al. (2014) consider modeling the noise pattern with a hidden layer. The learning of this hidden layer is regularized with a feature re- construction loss, yet without guarantee that the true label distribution is learned. Another method mentioned in their work is to minimize the entropy of neural network output: however, this method tends predict a single class. To address this weakness, Hendrycks al. (2019) propose utilize a small number of kosher data to pre-train a network and estimate the noise pattern. However, such clean data may not always be available in practice.
Alternatively, another direction proposes to design models that are intrinsically robust to noisy data. Crammer et al. (2009) introduce regularized confidence weighting learning algorithm (AROW), where parameters of classifiers are assumed normally distributed and the mean and covariance of this distribution is updated during training. The idea here is to preserve the weight distribution as much as possible while requiring the model to maintain predictive ability. The follow-up work (Crammer & 2e10 proposes to improve this algorithm by herding the updating direction via specific velocity hield (NHERD), achieving better performance. Both of these works impose parametric constraint on parameters, which could prevent classifiers from adapting to complex dataset. |Apit et al. (2017) show that deep neural networks tend to learn meaningful patterns before they over-fit to noisy ones. Based on this observation, they propose to add Gaussian or adversarial noise to input when training with noisy labels, and empirically show that such data perturbation is able to make the resulting model more robust. Other commonly adopted techniques, such as weight decay and dropout, are also shown to be effective in increasing the robustness of trained classifier (Arpit et al. [207]] /Zhang et al. 2017). However, the intrinsic reasons for this phenomenon still remains unclear and overfitting to noisy label is still inevitable.
Apart from the above mentioned strategies, one recent work proposes to correct the corrupted labels during training. In particular, Tanaka et al.] (2018) propose to jointly train the deep network and estimate the underlying true labels. While achieving improved performance, their method largely relies on the prior distribution and is difficult deploy under cases where there is large number of classes. In addition, there are a few other works attempting to select clean data while eliminating noisy ones during training (Malach Shalev-Shwartz/ 2017 Jiang et al. 2018: Han et al., 2018). These methods demonstrate promising results, but lack quantitative control of the quality of the collected clean data.
Finally, beyond deep learning framework, there are several theoretic works that demonstrate the robustness of a variety of losses to label noise (Long & Servedio 20100 Nataaaaaaaaann Natarajan et al. 2013 Ghosh et al. 2015 van Rooyen al. 2015).. Following the work of WWang & Chaudhuri 2018). Gao et al. (2016) propose an algorithm that can converge to the Bayesian optimal classifier under different noisy settings. Moreover, they also provide in-depth discussion regarding the performance of k-
2

---

Under review as a conference paper at ICLR 2020
nearest neighbor (KNNN classifiers. However, the problem with KNN is that it is computationally intensive and difficult to be incorporated into a learning context. Within the framework of deep learning, there more efforts that need to be made to bridge theory and practice.
2
METHOD
Our method has two synchronized modules, the training module and the label correction module. The training module continues to Iearn a classifier based on the current labels. Meanwhile, the label correction module uses the prediction of the classifier to correct labels.
We start with some preliminaries necessary for the expo- sition (Section 2.1). In Section 2.2 explain rection algorithm.
uses the prediction of the classifier (trained on noisy labels) for likelihood ratio test. Based on the test result, it decides whether to correct the label of data. Theoretically, sumptions of the prediction, f. the algorithm will change the labels the correct ones, i.e., the ones consistent with the Bayes optimal classifier (Theorem D.
Updated labels y
prove that under certain as-
Neural
Label Correction
Network Training
In Section 2.3 we present the overall training method. We incrementally train a deep neural network based on the corrected labels, y. Meanwhile, the network's prediction is used for label correction. To improve the quality of the prediction, we introduce new loss term, called the retroactive loss. The goal is to regulate the model using models trained in earlier epoches, they may be less overfitting with corrupted labels.
Updated model
Figure 1: The overview of our method.
For the label correction module, we focus binary classifier. But the algorithm and the theoretical results can easily be generalized to the multiclass setting (Corollary) Tp.
2.1 PRELIMINARIES
- {0, 1} be the label space, and be an unknown distribution on X' x y. The joint probability can be factored as D( y) = Pr(y Pr(w)) We denote by 1/x>) the true conditional probability. The Bayes risk of a classifier h : X - y is R(D, h) = Pr(m.))-((h(a) # y). A Bayes optimal classifier is the minimizer of the Bayes risk, ie., h"" = arg min, R(D, h). It can be calculated using the true conditional probability, n.
Let X' be the feature space, y y
n(æ> Pr(y
1 if 77x) >= 1
h (a) 1 (1(m)2))
otherwise
We assume the true conditional probability, 77. satisfies the Tsybakov condition (also called the TNC condition) (Tsybakov al., 2004). This condition, also called margin assumption, stipulates that the uncertainty of 17 (region close to decision boundary at 1/2) and thus the Bayes optimal classifier bounded.
Definition 1 (Tsybakov Condition). There exist C > 0, O, and to E (0, } such that for all t < lo.
Pr [((x) [ <1 < CCA.
The noisy label setting. Instead of samples from D, we are given a sample set with noisy labels S {(x 7}} where is the possibly corrupted label based the true label y. We assume transition probability Tı-s = Pr(ū = jly = i). i.e., the chance a ground truth label y is flipped from class i to class j. For simplicity, independent of the true
denote Tij = Ti--j- The transition probabilities Toi and Tı0 true joint distribution D and the feature æ. We denote the conditional probability of the noisy labels 7xx) Pr(j 1|2=). In short, call the noisy conditional probability. is related linearly to the true conditional probability,
7xx) (1 (10))(())))) + Tai ()))) 701 -To))(2F) 701-
3

---

Under review as a conference paper at ICLR 2020
1.00
class
class
0.50
025
025
0.0
0.00
X00X X0E
20000 XX
-0.25
(a) Noisy labels and f.
(b) Corrected labels and 17.
LRix)
155
LRtx
d=1
A=1
1.0
class 0
class
XX0X
0.0 X30X
43.4%
300000
-2
2
(c) LR for y 1.
(d) LR for y 0.
Figure 2: An illustration of the label correction algorithm. A is 1. (a): corrupted sample and its corresponding classifier prediction f. (b): after correction, the labels are consistent with the true conditional probability, πj. (c): likelihood ratio for = Data with x 0 are corrected to Nnew = 0 as I (.) are below A = 1. (d): likelihood ratio for 0. Data with O are corrected to Tnew as LR((z) are below A 1.
2.2 THE LABEL CORRECTION ALGORITHM AND ITS THEORETICAL GUARANTEES
Our label correction algorithm takes in current neural network prediction X (0, (i.e., estimation of 1) based on the noisy labels). For all training data and their current noisy label (æ, 7). the correction algorithm uses f to run likelihood ratio test and to decide whether to flip the label according to the result. The goal of the likelihood test is to decide whether the null hypothesis, Hu : : y, is true. If yes. ÿ accepted as it is. Otherwise, we flip y. so that hopefully becomes Formally, the likelihood ratio is defined as
LR(f 2, f(x) f(x))
(3)
f(z))²
We compare this likelihood ratio with a predetermined value Lf x. y) < A, we reject the null hypothesis and flip the label Fnew = 1 - y. Otherwise, the label remains unchanged, Fnew y.
See Figure for for an illustration of the algorithm.
Note that the constant A depends on the underlying noise pattern and f. Below we show that if we choose A carefully, the label correction algorithm is guaranteed to make proper correction and clean most of the corrupted labels. However, in practice, A is unknown and needs to be tuned.
Intuition. In the likelihood ratio (Eq. (3 3 the numerator the likelihood that the prediction f is consistent with the noisy label y. The denominator is the likelihood of the opposite case. When this ratio is smaller than 1, we know that the prediction whether ff agrees with ÿ is not the hypothesis to test. To test the intended null hypothesis (ÿ v), we need to check whether is consistent with the true conditional distribution 17, namely, the Bayes optimal classifier prediction 1 (x:). To this end, assume is close enough approximate of 7 as is trained on the noisy labels. This way, testing whether f agrees with ÿ is close to testing whether ñ agrees with y. except that the threshold A needs to be carefully chosen. Another issue need to consider is that the A is unknown. Our main theorem will bound the chance of failed correction by how close f approximates 7j and how close we can A the perfect one.
f is more likely to be inconsistent with y. But
2.2.1 FORMAL STATEMENT OF THE ALGORITHM AND THE THEOREM
Assume the knowledge of some prediction function f (perhaps from neural network). The label correction algorithm is given in Procedure I. It checks the likelihood ratio with regard to a parameter ). If T.R is greater than A, we flip the label. Otherwise, Bnew is 14710 To1 1+ -1g
A min( 14:7 Tia
4

---

Under review as a conference paper at ICLR 2020
the same V. Note that one only needs to know the difference Toi T10 in order know A. In practice, may be unknown and needs to be decided empirically.
Procedure LRT Correction
Procedure 2 LRT MoltiCorrection
Input: (x. f()). A.
Input: (x, 3). f(x) A.
Output: Unew
Output: Vnew
1: if T.R(J.a) x. 17) S A then
:- arg max, (x)] LR(J, x) = [[(m]]
[mm)
2: Inew = -1-3
if LR x. 17) S A then
else
2: Inew
4: Unew = y
else
end if
4: Unew
5:
end if
Our main theorem states that suppose the classifier prediction, f. close approximation of the noisy conditional probability, 7j. And suppose we can find a good enough A that close enough to the ideal A. Then there is very good chance that algorithm corrects most labels to the correct ones, i.c., the same as the Bayes optimal classifier prediction. Please note that here "proper correction" means that the new label, Vnew, is the same the Bayes optimal classifier prediction, h" (x)),, instead y. This is well justified as means that the correction will give us a classifier as good as the Bayes optimal one.
Theorem 1. Assume satisfies the Tsybakov condition with constants C > 0 and X >> 0. Let h²
denote the Bayes optimal classifier, and îj the noisy conditional probability 1j(x) (1 Toi
Tia) T01. Let be any classifier such that II Āx S for some
0. Let A
min -Im T+ T10 [=m=a=Tm) and let A 0 be constant such that A {A e, + E). If Inew
denotes the outpul of the L.T-Correction with V, J, and the given A, then
(Nnew # +h)sc SC (2.g)=
Pr
T10 T01
2
2(1 TO0 To1
If Toi = -i0, then PE * = (2e)
Intuition of the proof. We will prove lemmas. The first lemma shows that when f is linear in i.e., f(x)= an) + 6, with known coefficients a, b, we can find a A such that then the correction algorithm can be correct everywhere. The second lemma proves more relaxed version. If is exactly 7j and if we only know the difference between the transition probabilities, To1 Ti0, then we can bound the chance of mistakes of the correction algorithm. Finally, based on these two lemmas, careful case analysis, and the Tsybakov condition of the true conditional probability, we can prove the theorem. The complete proof can be found Appendix A
Multiclass setting. So far, all the description and theoretical results are based on binary classifi- cation setting. However, the results can be generalized to a multiclass setting without any technical difficulties. Let be the observed (possibly) corrupted label, f(x) be a classifier vector, where [[]]]; (the ith coordinate of f(x)) specifies the confidence of assigning æ to class i. The algorithm presented in Procedure 2 Informally,
present the theoretical guarantee for multiclass LRT as Corollary ! The proof is left for a journal version.
Corollary Theorem can be generalized the multiclass setting, when using algorithm Tt--aaliiCorreett
2.3 TRAINING DEEP NEURAL NETWORKS WITH LLT-LLBBLLCCRRECCTIEE
Our training algorithm continuously trains a deep neural network while correcting the noisy labels. Procedure 3 is the pseudocode of the training method, called AdaCorr. It trains neural network model iteratively. Each iteration includes both label correction and model training steps. In label correction step, the prediction of the current neural network, f. is used to run LRT test on all training data, and to correct their labels according to the test result. Since f is used to approximate the conditional probability 7/. we use the softmax layer output of the neural network f. After the
5

---

Under review as a conference paper at ICLR 2020
labels of all training data are updated, we use them to train the neural network incrementally. We continue this iterative procedure until the whole training converges.
We also have burm-in stage in which we train the network using the original noisy labels for epochs. During the burn-in stage, we use the original cross-entropy loss, Les Afterwards, we add addi- tional retroactive loss which will be ex- plained below.
Procedure 3 AdaCorr
Input: S {2r V3. A, m, T
1: for epoch=l m do
2: Train neural network with LCE
3: end for
Training with retroactive loss. After the burn-in stage, we want avoid overfit- ting of the neural network, so that its out- put better approximates 7j. To achieve this goal, we introduce refrouctive loss term [feero(((ar)) 9) The idea to enforce the consistency between ʃ and the prediction 10:
4: f" current model prediction
5: for epoch=m + 1 to do
6:
if epoch 2 + 10 then
7:
f f= = current model prediction
8:
for all (z, 0) S do
9:
Unew LRT [Rt.-oorttrtttioffll.... V),A)
y = Inew
of the model at a previous epoch, f'. It has been observed that a neural network 12:
11:
end for
end if
at earlier training stage tends to learn the true pattern rather than to overfit the noise (Arpit et al. 2017). Formally, the loss can be written
13: Train using Listro + LOE, with f' and
14:
end for
Eee fo((x) log fa( in which Ne is the number of possible label classes. The training loss is the sum of the retroac- tive loss and the cross-entropy loss
N.
((((),,)) (()))) + LCE(f(E)
Σ fa(x) log fo(x)
Σ To log fo(x))
In practice, we set I' to be the prediction of the model at the m--h epoch. In other words, once the burn-in stage is finished, the training switches from LCE to ICE + Tiretro- And the model at the end of the burn-in stage is used for the retroactive loss. We also set the label correction to start slightly after the burn-in stage, say m + 10. The key hyperparameter is the starting epoch m. Another hyperparameter is A. We select both and A empirically. Ablation study in Section&shows that our method is robust to these hyperparameters.
3 EXPERIMENTS
In this section we empirically evaluate our proposed method with several datasets, where noisy labels are injected according to specified noise transition matrices.
We use the following datasets: MNIST (LeCun & Cortes 2010), CIFARIO (rizhevsky et a). CIFAR100 (Krizhevsky b). and ModelNet40 (Z. Wu & Xiao 2015). MNIST consists of 28 x 28 grayscale images with 10 categories. contains 60,000 images, and we use 45,000 for training. 5,000 for validation and 10,000 for testing. CIFARIO and CIFAR 100 share the same 60,000 32 32 3 image data, with CIFAR 10 having 10 categories while CIFAR IIFARI0O having 100 categories. Similar MNIST, we split 90% and 10% data from the official training set for the training and validation respectively, and use the official test set for testing. ModelNet40 contains 12,31 CAD models from 40 categories, where 8,859 are used for training. 984 for validation and the remaining 2,468 for testing. We follow the protocol of (Qi et al. (2017)) to convert the CAD models into point clouds by uniformly sampling 1, 1,024 points from the triangular mesh and normalizing them within a unit ball. In all experiments, report the performance on test set.
Datasets.
early stopping on validation set to tune hyperparameters and to
Noise patterns. Following (Reed et al. 2014 Patrini et al. 2017), ,, corrupt our labels artificially using a noise transition matrix T, where Tú = Tı = Pr(y jly = 1) is the probability that category i is flipped to category the true label i is flipped to other classes with equal probabilities, i.e., Tíş = p/ (N,, - 1) for i +
In our work we focus two types of T: (1) uniform, where
and Ti = 1 P, where p is the noise level and N, is the class number; (2) pair flipping, where the

---

Under review as a conference paper at ICLR 2020
true label i is flipped with Ti transition matrices are as follows:
p for i # j and Ti =1 p. Examples of these two types of
0.7
0.1
0.1
0.1
0.7
0.3
0.0
0.0
0.1
0.7
0.1
0.1
0.0
0.7
0.3
0.0
T =
T2 =
0.1
0.7
0.0
0.0
0.7
0.3
0.1
0.1
0.1
0.7
0.3
0.0
0.0
0.7
in which Ti for uniform noise with level 0.3, and 72 is for pair flipping with noise level 0.3,
Baselines. We compare the proposed method with the following methods: (1) Standard, which trains the network in a standard manner, without any label resistance technique; (2) Forward correction 2017)) which explicitly estimates the noise transition matrix to correct the training loss; PPatrini et et
(3) Decoupling (Malach & Shalev-Shwarzz 2017). which trains two networks simultaneously and updates the parameters selected data whose labels are possibly clean; (4) Coteaching (Han et al. 2018), which also trains two networks but exchanges their error information for network update; (5) MentorNet (Jiang et al. 2018), which learns curriculum filter noisy data; (6) Forgetting Arpit et al. 2017). which uses dropout to help deep models resist label noise.
Experimental Setup. For the classification of MNIST, CIFARIO 10 and CIFARI00, we use preactive resnet34 (He et al. 2016) as the backbone for all the methods. On ModelNer40, we use PointNet (Qi ct al. 2017). We train the models for 180 epochs to ensure that all the methods have converged. We utilize RAdam (Liu et al. 2019) for the network optimization, and adopt batch size 128 for all the datasets. We use an initial learning rate of 0.001, which decayed by 0.5 very 60 epochs. The experimental results are listed in Table As is shown, our method outperforms the competing methods across the datasets under different noise settings.
Table 1: The classification accuracies of different methods.
Noise Level of Uniform Flipping
Noise Level of Pair Flipping
Data Set
Method
0.2
0.4
0.6
0.8
0.2
0.4
Standard
99.0
98.7 0.4
98.1
0.3
91.3 0.9
99.3
0.1
99.2 1 0.1
98.8 0.1
Forget
99.0
98.8
+ 0.1
97.7
- 0.2
62.6
99.3 +
96.5
+.5++200 2.0
89.7
Forward
99.1
98.7
98.0=04
89.6 19.6:488
99.4 0.0
9..2
96.5
4.4
Decouple
9.1
9..0
9.
9.26
99.4 9.00
9.13
99.1
MentorNet
99.2
98.7 0.1
98.1+0.1
5.25
98.6
99.1
9.19
MNIST
Coteach
9.2
0..3
0.3
0.7
9.1
99.0
98.9
AdaCorr
0.00
9..4
99.1 0.0
9..7
99.5 0.0
99.6 0.0
9..4
Standard
8.2
83.1
764 0.4
2..6
8.2
88.4
0,3
Forget
87.1
83.4
76.5-0.7
33.0
8.
83.7 0.1
0.54
Forward
8..4
8.8
74.7 1..7
3.0
0.5
87.4
0.5
Decouple
8.46
0.5
7..6=0.1
0.9
0.3
0.3
0.5
MentorNet
9.3
8.52
75.5
34.1
9.24
88.9
83.3 1.0
CIFARIO
Coteach
0.5
80.9
0.5
2.60
91.8 + 0.1
89.9
0.2
80.1 + 0.7
AdaCorr
91.0 + 0.3
88.7
0.5
81.2
0.4
49.2
2.4
92.2
0.1
91.3 + 0.3
89,2 0.4
Standard
58.9
1.0
20.8
1.0
$9.5
0.4
$2.9
44.7
Forget
59.3
5.20
40.9
7.7
1.111.1
61.4 0.9
0.6
37.7 4.6
Forward
0.5
0..2
41.1 0.5
2066
58.3
0..2
44.4
Decouple
59.0
0.7
40.2
0.4
+8.5
60.8 + 0.7
56.1 0.7
48.4
MentorNet
0.5
1..4
38.7
0.8
1.9
0.2
1.8
47.4
CIFARI00
Coteach
66.1 -
6.60
48.3 18.3-0.1
16.1
0.94
57.6 0.3
49.2 0.3
AdaCorr
67.8
60.2
46.5 12
24.6
0.2 68.3
61.1
49.8 1 0.7
Standard
75.3 2.6
70.0 3.0
2.39
84.4 1.2
1.3
78.9
1 0.7
Forget
80.1 1.8
7.69
69.0 0.7
26.2
83,3
62.0
59.5 2.9
Forward
52.3
1.88
43.5 1
28.2 1.5
48.1
48.0 3.7
49.1
4.4
Decouple
82.5
1..7
1
2.7
85.7
84.3
80.5
ModelNet40 MentorNet
86.5
1.
70.9
199
5..7
83.7
8.0
79.3
2.1
Coteach
0.9
0.88
81.8
1.1
68.9 2.8
85.7 1 0.8
79.1 1 3.0
69.1
2.4
AdaCorr
86.9
85.1 15.1
78.6
72.1
87.6 - 0.4
84.6 + 0.5
83.7
0.5
Clothing 1M. We also evaluate our method on a large scale Clothing 1M dataset (Xiao et al. 2015
We pretrained resnet-50 and trained the model using SGD for 20 epochs. Our method has 71.47 accuracy. outperforms Standard (68.94) and Forward (69.84). Other baselines (Forget, Decouple, MentorNet, Coteach) did not report result this dataset.

---

Under review as a conference paper at ICLR 2020
4 ABLATION STUDY AND DISCUSSION
We conduct ablation study to see the significance of our contributions. We compare our method (LRT +Iyetrr)) with two baselines: our method without the retroactive loss (LRT+I and using cross-entropy loss only without LRT-Correction (Lee Only). We report the test accuracy on CIFARIO in Table We observe that adding LRT Label Correction to Lee alone helps improve the performance significantly. The numbers inside the parenthesis are the percentage of correct labels after the training and label correction. We observe that the LRT Correction corrected large portion of noise labels. Also observe adding the retroactive loss improves the method further in both test accuracy and label correction rates.
Table 2: Effect of LRT Correction and Lyetro- The experiments are performed on CIFARIO (aceu- racy in %). The number in parenthesis denotes the rate of correct labels after training.
Method
uniform 0.2
uniform 0.4
uniform 0.6
uniform 0.8
pair 0.4
Lee Only
87.5(80.0)
83.1(60.0)
76.4(40.0)
47.6(20.0)
84.5(60.0)
LRT+Lee
91.1(95.7)
87.9(91.2)
80.7(81.7)
47.3(47.1)
87.5(91.9)
Lo
91.0(95.7)
88.7(90.5)
81.2(82.5)
49.2(49.0)
89.2(91.3)
We evaluate how different hyperparameters affect the performance our method. We compare our method with different m, the length of the burn-in stage. We start introducing the retroactive loss after m epochs, and start label corrections after m + 10 epochs. The final testing accuracies are shown in Table 3 We observe the performance of our method is rather robust w.r.t. different m's. We choose m 20 this data set (CIFARIO) and similarly in other datasets.
Table Effect of different m's. The experiments are performed on CIFARIO (accuracy in %). The number in parenthesis denotes the rate of correct labels after training.
Noisy Type
Epoch 15
Epoch 20
Epoch 25
Epoch 30
Epoch 35
Epoch 40
uniform 0.4
87.6(90.1)
88.7(90.5)
87.4(90.7)
86. 86.7(90.6)
84.8(88.7)
84.1(87.2) 78.1(81.7)
uniform 0.6
79.4(81.0)
81.2(82.5)
80.9(81.9)
79.3(81.9)
79.1(81.8)
pair 0.4
75.8(80.0)
89.2(80.1)
90.8(87.0)
89.2(89.3)
88.2(90.1)
86.7(90.1)
We also evaluate the performance on different A's. A is the unknown value for our likelihood ratio test. controls how aggressive we would like to correct the labels. From Table 4 we observe bigger A tends to give better results (as it is less aggressive in correcting labels). We observe 1/1.2 the best one for CIFARIO. Similar values of optimal A are found other data sets.
Table 4: Effect of A. The experiments are performed CIFARIO, and the number in parenthesis denotes the rate of correct labels after flipping.
Noisy Type
1/1.0
1/1.2
1/1.5
1/2.0
1/2.5
1/3.0
uniform 0.4
88.3(91.7)
88.7(90.5)
83.0(89.3)
77.1(86.7)
75.2(85.1)
75.5(84.0)
uniform 0.6
79.9(81.0)
81.2(82.5)
80.9(81.9)
79.3(81.9)
79.1(81.8)
78.1
pair 0.4
88.2(92.2)
89.2(80.1)
84.4(89.8)
77.0(86.8)
76.4(85.1)
77.0(84.1)
In general, we observe our hyperparameters are rather consistent across different datasets. This reveals better generalization power of our method over other datasets and noise patterns. We believe this is due the principled approach we take in label cleaning.
In Table we show the performance of different methods trained on clean dataset. We observe that when without label noise, all the methods achieve very similar performance. However, note that on CI CIFAR 100 and ModelNet40, the performances of MentorNet and Forward are slightly inferior to the standard approach. The reason could be that MentorNet is designed specifically for handling noisy data, and would behave too conservatively under clean data setting; for Forward, al- though the data is clean, the estimated noise transition matrix is imperfect and still indicates a certain level of noise, leading to inferior performance. AdaCorr remains competitive in these experiments.
8

---

Under review as a conference paper at ICLR 2020
Table 5: The performance of different methods trained clean dataset.
Dataset
Standard
Forget
Forward
Decouple
MentorNet
Coteach
AdaCorr
MNIST
99.5 + 0.1
99.5 0.1
99.5 0.1
99.5 = 0.0
99.4 + 0.2
99.5 + 0.0
99.4 = 0.3
CIFARIO
93.6 1 0.3
93.5
91.2 # 0.4
93.7 = 0.4
91.5 - 0.4
93.6 0.2
93.9 = 0.2
CIFARI00
72.0
0.3
70.1 0.3
72.0 0.3
71.4 0.3
72.2 = 0.5
74.1 = 0.0
ModelNet40
87.6 0.5
87.7
60.3 + 0.6
88.4 = 0.6
85.0 0.5
85.9 + 0.4
88.4 0.3
In Figure we draw convergence curves on CIFAR10 with Uniform 0.4 noise level. On the left, show the curves of our proposed AdaCorr method. The model continues to fit with the corrected labels (Unew and the test accuracy on clean labels does not drop. This shows that the model and the label correction curves of the Standard model. Without label correction, the model overfits with the noisy label and loses on test data catastrophically.
improving in harmonic fashion and do not collapse. On the right, show the
AdaCorr
Standard
100
Accurrty (5)
Accurscy (%)
50-
40-
30
training accuracy against y new
30-
training accuracy against y bide testing accuracy
testing accuracy
20
47
68
111
132
153
175
47
68
90 Epochs
111
132
153
175
Epochs
Figure 3: Convergence curves for CIFARIO with uniform 0.4 noisy level. Left: AdaCorr training accuracy evaluated against the corrected label (Unew), testing accuracy against clean label. Right: Standard training accuracy against noisy label (g) and testing accuracy against clean label.
CONCLUSION
We propose label correction algorithm to combat label noise. The correction algorithm performs a likelihood ratio test for each input label decide whether to correct the label. Theoretically, we prove that our method corrects noisy labels with high probability. By combining with deep neural network training with a novel retroactive loss, our method can produce models robust to different noise patterns. Experiments various datasets show that our method outperforms state-of-the -htt--rr-t-- and is robust to hyperparameters.
REFERENCES
Devansh Arpit, Stanislaw K. Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look memorization in deep networks. In ICML, pp. 233-242, 2017.
Carla E. Brodley and Mark A. Friedl. Identifying mislabeled training data. J. Artif. Int. Res., 11(1): 131-167, July 1999. ISSN 1076-9757.
Koby Crammer and Daniel D. Lee. Learning via gaussian herding. In NIPS, pp. 451-459. 2010.
Koby Crammer, Alex Kulesza, and Mark Dredze. Adaptive regularization of weight vectors. In NIPS, 2009.

---

Under review as a conference paper at ICLR 2020
Benot Frnay and Michel Verleysen. Classification in the presence of label noise: A survey. Neural Networks and Learning Systems, IEEE Transactions on, 25:845-80000 05 2014.
Wei Gao, Bin bin Yang, and Zuo cheng Zhou, On the resistance of nearest neighbor to random noisy labels. 2016.
Aritra Ghosh, Naresh Manwani, and P.S. Sastry. Making risk minimization tolerant to label noise. Neurocomput. 160:93-107, July 2015.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 8536-8546, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645, 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, pp. 2712-2721, 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentorett: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). a. Utt wwwp://wwww///////////////.. cs www.ttoooooooooo wwt.:etttotooooooooooooooooottte
http://www......oooooooooooooaaa htmll
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). URLpttt:: tttp://w.wwcc.troooooooooo www/
html
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. BRLhttp://yann
lecun. henn//xxb/mniit..
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint /90Xiv:::8.... 03266,, 2019.
Philip M. Long and Rocco A. Servedio. Random classification noise defeats all convex potential boosters. Machine Learning, 78(3)2287-304 Mar 2010.
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In NeurIPS, 2017.
Volodymyr Mnih and Geoffrey E. Hinton. Learning to label aerial images from noisy data. In ICMI., 2012.
Nagarajan Natarajan, Ambuj Tewari, Inderjit S. Dhillon, and Pradeep Ravikumar. Learning with noisy labels. In NeurlPS, 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach. In CVPR, pp. 2233-2241, 2017.
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pp. 77-85, 2017.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and An- drew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. CoRR, abs/1412.6596. 2014.
Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. Inferring ground truth from subjective labelling of venus images. pp. 1085-1092, 1994.
10

---

Under review as a conference paper at ICLR 2020
Jacob Steinhardt, Pang Wei Koh, and Percy S. Liang. Certified defenses for data poisoning attacks. In NIPS. pp. 3520-3532, 2017.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, 2018.
Alexander B Tsybakov al. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics, 32(1):135-166, 2004.
Brendan van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In NeurIPS, pp. 10-18. 2015.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J. Belongie. Learn- ing from noisy large-scale datasets with minimal supervision. In CVPR, pp.
2017.
Jha Wang and Chaudhuri. Analyzing the robustness of nearest neighbors to adversarial examples. In ICML, pp. 5133-5142, 2018.
Xiang Wu, Ran Hc, Zhenan Sun, and Tieniu Tan. A light CNN for deep face representation with noisy labels. IEEE Trans. Information Forensics and Security, 13(11):2888) 2896, 2018
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699 2015.
Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learning, 95(3):991-327, 2014.
A. Khosla F. Yu L. Zhang X. Tang Z. Wu, S. Song and J. Xiao. 3d shapenets: A deep representation for volumetric shape modeling. In CVPR, pp. 1912-1920, 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
A PROOF OF THEOREM
We restate the theorem below for completeness.
Theorem Assume 1) satisfies the Tsybakov condition with constants C >
and X > 0. Let h*
denote the Bayes optimal classifier, and 7 the noisy conditional probability 7)xx) (1
()0) + 701- Let f be any classifier such that || Āl S for some 6
Let A
min( 1+110-101 1+101-710
TT 1+ ), and let A' > 0 be a constant such that A' E [A c, + c]. If Inew
denotes the output of the LRT-Correction with y. x, f. and the given A', then
Pr (onew h²) S =c
T10 T01
+
(z.g)~D
2(1 T10 To1)
If T01 = T10. then PT( (unew # =
To prove this theorem, we first prove two lemmas. In the following we will state and prove the lemmas, and then continue to finish the proof of Theorem [
Lemma 2. Assume y satisfies the Tsybakov condition with constants > 0 and > 0. Let
denote the Bayes classifier. If f depends linearly n, i.e., f(x) an(x) + b, a, b 0, and a, b and u are known, and A is chosen as in Table then
If LLIICCorretion(
(f.oofffff.. (xx M. b.) flips the label o. then Inew h" (xx),, and (Unea (x. ū) # h²t (x:)) O.
Lemma 3. Assume y) satisfies the Tsybakov condition with constants C > 0 and X > O. Let ht denote the Bayes classifier. Lei 7]xx)) (1 To1 T10))((;)) + Toi denote the corrupted Bayesian classifier. If A min( 1+120 14 T10
10100T01) then
11

---

Under review as a conference paper at ICLR 2020
Table 6: Values of A.
b+ aa2
a/2
2 2-
2-
a+t26
If LRT-Correction(ff.er Ø), A) flips the label of then Inew = h² (x),
PT(a.))~D( (z. 0) + (x)) S C (1 (110-10--100 and To1 T10, Pr(ma)) (Wnww (z, 7) + (x))) O.
(Tnew
Proof of Lemma
We begin with some observations:
Observation 1: ÿ 1( > b+2/2), the algorithm never flips. This is because A = O in all such cases, and since the likelihood ratios f/(1 f) or f)/ are always positive, y is not flipped.
Observation 2: fa) < b ++ a/2 if, and only if, ny(x) < 1/2. This follows by a straightforward calculation using fxx = an)x)) + b.
In the remainder of this proof, we assume that Tio > Toi- The case Tio < Toi follows by symmetry.
Observation 3: If ((x)) and ((x) + a/2 (so non-trivial test (unlike ones in Observation 1) is performed), then f(x)
(x) < Á [ = ((x) < b + a/2) if, and only if, nn(x) < 1/2.
Similarly if f(x)
and f() > b+ a/2 then (1 (()))
A j 0, f(x) > 6+ a/2) if,
and only if, nTx) > 1/2.
Proof of Observation 3: First notice from the table that A 1, 1/2) is the reciprocal of A 0, (x) > 1/2). Since the likelihood test ratio is also the reciprocal in the two situations, proving one statement suffices. We prove the first statement. For convenience we denote A (x) < 1/2) by A.
f(x)
< A
(4)
f())
f(x) < A/(1+
(5)
4+
f(x)
2-0-26
(6)
1
a++b
2-a-26
a a+ 2h
f(x)=
(7)
a b
(8)
a
n (z) 1/22.
We can now prove the first assertion of Lemma 2 (under the assumption that T10 > Toi). That is, if LRT-Correctionff.e 7). a, b) flips the label of J, then Vnew h²" hx))). Let first consider the case ((x)) = 1. If f(x) >> b+ a/2, then it implies that v7(x) > 1/2 (Observation 2). The corresponding A in the table is meaning that
test is performed, and Inew (x. 9)
7(xx = h" (x). On the other hand, if f(x) < b + a/2, we perform a likelihood ratio test on f/(1 f). which less than the corresponding value of A iff m(x)) 1/2 (Observation 3), so again Inew (ar, .) ((x)) ht (a:). The case ((x) = 0 can be analyzed analogously.
Thus the only points x where we may not match h² (a:) are ones where the flipping algorithms does not flip, and the result mismatches with h", By the above observations, the following is where this happens.
Observation 4: Assume T10 > 701- Then
{a: Vnew (zr 7) * h² (x=) {a: ((x) (()) < a/2,) 1/2).
12

---

Under review as a conference paper at ICLR 2020
However, according to Observation 2, f(x) < b + a/2 implies ()x))
1/2. Thus this case cannot
happen. Hence Pr(a.y)>> (Inew (Cnew # h" (a)) = 0, and Lemma is proved.
Proof of Lemma
This proof is an extension of the proof of Lemma tion 4, get the following
All the steps are analogous, except in Observa-
Observation 5: Assume T1o > Tu1 Then
{x: Unew((; v) + l1" (x)) = fx: 7(x) = 0, f]()) < m()) > 1/23.
This measure is non zero. However, it is bounded above by the measure of the set (7(x) < (xx) 1/2]. Since 7j(x) (1 Toi Tia)n(a) + Tul this the same as the measure of the set {ar 1/2 < n)()) < 1/2- 701_ By the Tsybakov condition, this measure at most 1-710
This was for the case 710 ToL, and when TI0 < 7oi we can check by analogous calculation that the numerator is reversed. Hence we have proved
T10
3(1-r10-701
(neew (2, 17) + (x)) SC (1
Toi - T10 2(1 T10 - -r10) 1)
Pr (
The last assertion of Lemma follows by just substituting T01 T10 in the above formula. Hence
Lemma 3 is proved.
Proof of Theorem
1
< and B denote the event ( (+/= In the following we describe the cases when our algorithm results (flips or does not flip) label inconsistent with It".
Let A denote the event
T f < 1/2,n > 1/2, LRT flips). According to the LRT scheme, this happens with probability at most Pr(An (12))))) 1/2)).
( = 1, 1/2, < 1/2, LRT does not flip). According to the LRT scheme, this happens with probability at most Pr(A n (1 < 1/2)). Here A denotes the complement of the event A.
: 1. >
1/2, > 1/2,LRT LRT flips). According to the LRT scheme, this happens with probability at most Pr(An (n>> ( 1/2)).
11 1/2, < 1/2, LRT does not Accordinn to the LRT scheme, this happens with probabiiiiyy most Pr(A* (1 < 1/2)).
00 1/2, 100.f(1/22,
LRT does not flip). According to the LRT scheme, this happens
with probability most Pr(B" n (1) > 1/2)).
(ū < < 1/2, LRT flips). According to the LRT scheme, this happens with
probability at most Pr[B (n < 1/2)).
1/2, LRT Accps).Aco to the LRT scheme, this happens with probability at most Pr(Bn(r 11/2).
( 0. >
@ 0, 1/2, 1/2, LRT does not flip). According to the LRT scheme, this happens with probability at most Pr(B" n (n > 1/2)).
In all of these cases, observes (after using the fact that |lf 70000 I that the probability of these events is either of the form
Pr 1/2 < 7)x)= <
Tea Tor
+ or
T
Pr
I1.7 Tai
E < < 1/2 or
T-
2/
Pr
1/2 < (Tx)
<
1+47 To
+ or
1- 710
A/'
Pr
T+ T01
E < < 1/2
T-for
Next, we observe that
A-A'
Also,
1''
A Sc
1+
1+A
[++))++1]]
1+
13

---

Under review as a conference paper at ICLR 2020
Using these observations, the four expressions be bounded by the analogous four expressions found by replacing A by A.
Pr 1/2 2 < ()))
701
+ 2e or
1-701
Pr
113 T01
2e <
< 1/2 or
1-ror 710
N/
Pr
1/2 < 11()))
<
11 T01
+ 2e or
1-
A/' 1+a
Pr
T01
2e < 1/2
--fo
Assuming
bound each of these probabilities by (1211- (1 + 2c Notice that although there are 8 cases, they are mutually disjoint, so the final failure probability is bounded by the maximum of these cases, also of the same form. Hence we have proved
Tsybakov condition,
Nnew +1²) SC (2:2))~D) D
Pr
T10 To1
+ 2e
2((1 T10 - To1
which finishes the proof of Theorem
14


Reviews:
[{"review_id": "ryxITI7jFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Label noise widely exists in the large-scale data sets. This paper proposes a novel approach that directly cleans labels in order to train a high quality model. The proposed method leverages statistical principles to correct data labels and has a theoretical guarantee of the correctness. In particular, a likelihood ratio test (LRT) to flip the labels of training data is used, and it can prove that the LRT label correction algorithm is guaranteed to flip the label so it is consistent with the true Bayesian optimal classifier with high probability. The experimental results on several benchmark data sets show that the proposed method is promising. Overall, this paper could be a significant algorithmic contribution. But I also have some minor concerns:\n[1] The theoretical analysis and the experimental results are both well organized in the paper. How about the time complexity of the proposed method. If the authors can show the time cost in the paper, I will much more agree with the paper.\n[2] In the experimental parts, the convergence curve of the proposed method during the training epochs may be better to prove the theoretical analysis.\n[3]The details of the compared methods should be given, and it will be better to give the results without any noise labels. In this way, the confidence of the paper will be further improved.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rkxMYnMaKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for correcting training label noise, using a likelihood ratio test on the predicted probability of the classifier trained on the noisy labels, and then proposes an algorithm for iteratively cleaning training labels and retraining a model.  The paper also provides a theoretical guarantee for the probability of correctly re-labeling the training set, and provides empirical results showing that the proposed method significantly outperforms existing approaches for handling noisy training labels.\n\nOverall, I think the empirical results appear very strong, but think this paper is below the acceptance threshold due to three factors, in ranked order:\n- (1) The theoretical guarantee, which is positioned as a core contribution of the paper (and in fact claims it as \"the first to correct labels with theoretical guarantees\", which is not true), is based on assumptions that seem overly strong; these are somewhat relaxed in a \"Remark\", but this seems unproven and is a confusing presentation regardless.\n- (2) The theoretical bound itself is somewhat vacuous as it contains several constant factors that seem very material to the bound, but totally opaque to the reader.\n- (3) The experiments are very strong overall, which is a major plus for the paper; however, there are some questions about the hyperparameter tuning and some other points where more clarity could improve the strength of the empirical results\n\nRegarding (1):\nAs a reader, my first natural reaction was to worry about circularity/degeneracy in the proposed method: basically, we are using the confidence (as a ratio of predicted cond. probabilities) of the model trained on the corrupted labels to correct those labels... if the labels are so corrupted that the model is also way off, then intuitively, this method should not work.  I was wondering about how this situation would be bounded / handled.\n\nIt turns out in Thm 1 that an incredibly strong assumption is made, namely that the model trained on the corrupted labels, f, is a linear function of the true model, with constants a, b known to some small degree of error epsilon (note that the theorem statement says that these constants are unknown- but it then assumes that \\Delta, which is set based on a and b, is known up to \\epsilon error).  This seems like an incredibly strong assumption- and no context / motivation is given about why it should be taken as reasonable.\n\nThen, immediately after the Theorem, \"Remark 2\" states that this condition is not actually needed at all- but (a) then why not just strike it from Thm 1 statement, and (b) there does not seem to be any proof of this Remark in the appendix (where the proof of the main theorem itself is closer to a sketch than a standard proof...).\n\nRegarding (2):\nThe bound produced in Thm 1 seems somewhat vacuous: letting \\tau_{01} = \\tau_{10}, then the probability of the label correction being erroneous is bounded by 8C(O(\\epsilon))^\\lambda.  This quantity is presumably in (0,1/2], so it's a small range to start... but it seems hard to get anything from this bound without some idea of what the constant factors (C, and those hidden in O(\\epsilon)) and \\lambda are.  In particular, as presented, it seems implausible that \\epsilon- the error in specifying the \\Delta threshold- gets that small, in which case these constants become very important to know!  Another way of phrasing this remark: many theoretical bounds have lots of unknown constant factors, but are ultimately just trying to expose some scaling with respect to one parameter, e.g. number of data points, and therefore the constants don't need to be known that well for the statement to have some value.  This doesn't exactly seem to be the case here- therefore it seems hard to extract something from this statement (even ignoring the strong assumptions it is predicated on).\n\nRegarding (3):\nOverall, I think the empirical performance reported in Table 2, and overall thoroughness of the ablation in Section 4, are major strong points for the paper- the performance is very impressive!  However I have a few questions, clarification of which would be very helpful in my mind:\n- (i) A major issue that seems to be raised in the earlier sections is that there is a hyperparameter \\Delta--the threshold for the likelihood ratio test--that everything depends on, and must be chosen empirically.  Table 5 shows that the effect of choosing it is not crazy, but also clearly not insignificant.  My question is: how is it chosen?  On the validation dataset?  And is this validation dataset also corrupted in the same way as the training dataset?  If not, that seems like a major whole in the setup.\n- (ii) I also have a high level question for understanding: how is it possible for the various approaches to do so well with 0.6 and 0.8 noise level of uniform flipping?  In the p=0.8 noise model, for example, the probability of a data point getting flipped to *any individual wrong label* is *greater than that of it being the correct label*.  How is it possible to learn a model based on such a dataset?  Was there some kind of pre-training?  Was the validation set not corrupted?  I don't conceptually understand how the results shown are possible...?\n\nOverall, I think points in (1) and (3) could be helped with additional clarification and contextualization, and possibly (2) as well.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xGBLK0tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a label correction approach based on a likelihood ratio test, for robust training of deep neural networks against label noise. First, this paper introduces the LRT-Correction procedure, which is the main component of the proposed label correction approach. LRT-Correction uses the current model prediction to run a likelihood ratio test and flip labels when they are rejected. The decision is made by comparing the likelihood test results with a predefined value \\Delta. Then they introduce the full algorithm, AdaCorr, where the LRT-Correction procedure serves as an inner loop for the label correction. Lastly, there are experiments done on four datasets to conclude the superior performance of the proposed AdaCorr in contrast to several existing methods. \n\nOverall, this paper proposes a new label correction approach based on a likelihood ratio test. Standard experiments show that the proposed AdaCorr is superior to several existing methods. \n\nThe following questions are expected to be addressed during rebuttal:\n1.\tThe LRT-Correction procedure introduces additional computation costs. What is the difference in computation costs between Standard and the proposed label correction approach? Is the extra computation cost significant? \n\n2.\tThe LRT-Correction procedure is introduced based on a binary setting. The main theorem (Theorem 1) also only supports the binary setting. There is a statement in Corollary 1 that “LRT-Correction can be generalized to multiclass classification tasks, by flipping \\tilde{y} to be the best prediction of f when the null hypothesis is rejected. Theorem 1 can be generalized to multiclass classification tasks, by considering all pairs of class values.” How exactly did you do for that? Please provide more details.\n\n3.\tThis paper uses the ablation study to demonstrate that the proposed AdaCorr is robust to several important hyper-parameters, e.g. the number of epochs m for the burn-in stage, the predefined value \\Delta for likelihood ratio test. How did you exactly choose the optimal value for these hyper-parameters? These is a statement “We choose m=20 in this data set (CIFAR10) and “similarly” in other datasets.” Did you use the same m(=20) for all datasets? Does this also hold for the hyper-parameter \\Delta ?\n\n4.\tThe experiments are too standard. Any results on real-world datasets, e.g. Clothing1M [1]？\n\nMinor comments:\n1.\tThe summarization of the existing related work is not consistent throughout the paper. In Introduction section, this paper believes that the existing methods mainly follow two directions, i.e. probabilistic reasoning and data selecting; while in Related Work section, they are classified into three categories. Please clarify your arguments. \n\n2.\tPage 5: In Corollary 1, “LRT-Correctioncan” -> “LRT-Correction can”\n\n3.\tPage 7: In Table 2, MINIST -> MNIST. \n\n4.\tPage 7: In Experiment Setup, please provide more training details, e.g. learning rate. \n\n\n[1] Xiao, Tong, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. \"Learning from massive noisy labeled data for image classification.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699. 2015. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a very interesting topic, and the authors clarified various issues raised by the reviewers.
However, given the high competition of ICLR2020, this paper is unfortunately still below the bar.
We hope that the detailed comments from the reviewers help you improve the paper for potential future submission.
Decision: Reject