A Simple Approach to the Noisy Label Problem Through the Gambler's Loss
Authors: Liu Ziyin
Ru Wang
Paul Pu Liang
Ruslan Salakhutdinov
Louis-Philippe Morency
Masahito Ueda
Venue:  (2020)
Keywords: noisy labels
robust learning
early stopping
generalization

Abstract: Learning in the presence of label noise is a challenging yet important task. It is crucial to design models that are robust to noisy labels. In this paper, we discover that a new class of loss functions called the gambler's loss provides strong robustness to label noise across various levels of corruption. Training with this modified loss function reduces memorization of data points with noisy labels and is a simple yet effective method to improve robustness and generalization. Moreover, using this loss function allows us to derive an analytical early stopping criterion that accurately estimates when memorization of noisy labels begins to occur. Our overall approach achieves strong results and outperforming existing baselines.

Under review as a conference paper at ICLR 2020
A SIMPLE APPROACH TO THE NOISY LABEL PROBLEM THROUGH THE GAMBLER'S 's Loss
Anonymous authors Paper under double-blind review
ABSTRACT
Learning in the presence of label noise is a challenging yet important task, and it is crucial to design models that are robust when part of the dataset are mislabeled. In this paper, we discover that a new class of loss functions called the gambler's loss provides strong robustness to label noise across various levels of corruption. Training with this modified loss function reduces memorization of data points and is a simple yet effective method to improve robustness and generalization. Moreover, using this loss function allows us to derive an analytical early stopping criterion that accurately estimates when over-memorization begins to occur. Our overall approach achieves strong results and outperforming existing baselines.
LEARNING FROM NOISY LABELS
As a first step of supervised learning tasks, the user often collects a large amount of data in the form of (x,y) y) pairs where z represents the input data and the desired labels. However, parts of real world data can often be mislabeled due to 1) annotator mistakes as a natural consequence of large-scale crowdsoureing procedures (Howe) 2008), 2) the difficultly in fine-grained labeling across a wide range of possible labels (Russakovsky et al. 2015). 3) subjective differences when annotating emotional content (Busso et al.) (2088), and 4) the use of large-scale weak supervision (Dehghani ct al. 2017). Learning in the presence of noisy labels using neural networks is challenging since overparametrized neural networks regularization (Zhang et al. 2017). When the model memorizes the noisy labels, its generalization deteriorates (c.g., see Figure and Section I.I] In fact, this agrees with the theoretical finding in (Nakkiran et al., 2019) that neural networks trained with SGD often learns functions stage by stage from simpler functions to more complex ones, and functions whose output is uniformly random are the functions with highest complexity, and so memorization of random labels often come late in the training. Based on these, we may devise special methods to extract signal from noisy labels.
known to be able to memorize all labels even with strong
Similar to Patrini et al. (2017); | Yu et al. (2019); Han et al. (2018), we deal with an idealized noisy label setting, where a noise transition matrix is invoked to define the problem, and each label has the probability defined by the transition matrix to be flipped to another label (see appendix for formal
mathematical definition).
Organization: In this section, we first hypothesize and then empirically verify the existence of three stages during training in the presence of noisy labels; in Section 2 we propose that the gambler's loss function is noise robust loss function that can be used for noisy label problems, and we further propose an analytical early stopping criterion using the gambler's loss. We then verify our methods through a series of experiments and comparisons with existing approaches.
1.1 TRAINING TRAJECTORY OF NOISY LABELS
In practice, one often faces the problem of learning from dataset = {z, v whose labels are partially corrupted, i.c. D = Detean U D Deorrupt- We consider a setting where the labels are symmetrically corrupted (i.e. any label has equal probability to be flipped to any other class). Let 1 - denote the probability that data point (x., y) is clean, being the probability of mislabeling. classification problem can be denoted as learning a function f(-): X + Y. where x E X = R² are the data points, and y Y IR are the targets. In practice, we parametrize f with parameters 0 (i.e. fe) and learn 9 by minimizing the empirical risk E [[((f(()). y) for some loss function f. The standard loss function for classification is cross entropy (or the nlll loss). However, naive training on D using n/l loss often results in a "peaked" learning curve as shown in Figure lb We observe that such learning curves often consist of three stages (See Figure T):

---

Under review as a conference paper at ICLR 2020
100
100
23
train Fuea
fooal
Irain Laow
train Eoenat
80
30
60
115
1002
40
fast learning stage
1.0
mnanor mnaoorbattin
2012 stage
0.0
60
30
100
120
0
20
80
60 opoch
80
100
120
epoch
(a)
(b)
Figure I: Different stages in the presence of label noise. (a) We plot fiotal (total loss), lelean (loss Delean), and feorrupt (loss D Deorrupt). we see that during the gap stage, there clear "gap" between the Eelean leorrups where training on clean lahels has completed Hypothesized qualitative division of the three stages: fast learning stage, stage, and the memorization stage. Experiment done MNIST with corruption rate 0.5.
training on noisy labels has barely started; (b)
1) Fast Learning Stage: The model quickly learns the underlying mapping from data clean labels; one observes rapid decreases in training loss and increases in test accuracy.
2) Gap Stage: From Figure la we notice learning on the clean labels is almost complete (Eatean 0.5) but training on noisy labels has not started yet (fcorrupt ~ 2.5), and large gap in training loss exists between Detean and D Deorrupe. Both the train loss and the test accuracy reach a plateau, and this is the time at which the generalization performance is the best.
3) Memorization: When there are only noisy labels left to learn from, the model will memorize these noisy labels and the train loss decreases slowly to O. However, memorizing noisy labels hurts generalization and the test accuracy decreases as well.
These stages seem to be present across datasets and model architectures when label noise is present. This is problematic because the testing accuracy reaches peak early in training and then decreases steadily because the noisy gradient from D corrupt starts to dominate that from Deleans leading to memorization of noisy labels and affecting generalization. Therefore, tackling the problem of learning from noisy labels involves reducing the negative effect from D Doorrupt. This is hard in general, since one does not know a priori which points are mislabeled. We mainly deal with symmetric noise, and we denote the corruption rate (the probability that data point has label that is not true label) as == E, where is the clean rate. Various approaches have been proposed (Patrini et al. [017 Han al., 2018 Yu al. 2019), but, unfortunately, the best current methods can do only to slow down the drop in test accuracy instead of preventing it completely (Han et al. 2018).
Our method will be based on the above observation; summarize the key point, we note:
Observation. When training on label-corrupted dataset, there is stage when the average loss on the correct points is much lower that of the corrupted points.
Intuitively, since features easier to learn than random noises average. This phenomenon also motivates a label noise method sometimes called the "small loss method" (Han et al. 2018 Yu et al.) 2019), in which the data points with loss higher than threshold is screened to prevent the network from learning wrong information from such point. The observation also holds theoretically. As shown in (Nakkiran et al.) 2019), neural networks trained with SGD often learns functions stage by stage from simpler functions to more complex ones, and functions whose output uniformly random the functions with highest complexity. From this observation, we make the following idealized assumption that our method is based on:
Assumption. (Idealized gap stage) During the gap stage, the model has not learned anything about the data points in D porrupto and predicts strictly uniform score across all the labels, and it achieves perfect accuracy on D Detean-
We find that this assumption holds well for simple datasets such as MNIST and on datasets with complicated
very high corruption rate, where our method achieves best results, and less
2

---

Under review as a conference paper at ICLR 2020
datasets such CIFARIO (for example, on gap stage, the training accuracy the clean dataset do not reach close to 100%).
2 THE GAMBLER'S S LOSS FUNCTION AND HOw TO EARLY STOP
We propose an approach that uses the gambler's loss function (Ziyin et al. 2019). Intuitively speaking, the gambler's loss builds on the analogy that making classification given data point is the same as gambler making a bet on horse race given the horses, and that minimizing the nll loss in the classification problem is the same as maximizing the doubling rate of wealth. The gambler's 's loss may be generalized to include a reservation option in which the gambler can play "safe" and reserve part of his money in pocket. We refer to this generalized version as the gambler's loss. Mathematically. the gambler's loss augments the target space by an "reservation" dimension: Y - Y' = IR*+1, where the (m + 1) dimension defined as the reservation score. In the case of a neural network, using this loss involves adding one more output neuron followed by softmar layer to normalize the output. The gambler's 's loss function for a single data point (x. y) with y being a 0 1 label, and with neural network fo): X Y' is
e(f(x),)) log + m1) ((0))m))
o is hyperparameter for the loss function, and reasonable range of o is e (1, m] (also see ). Note that we omit 0 when unnecessary, but f always denote a neural network in this work, and should always come with subscript 6, denoting its parameters. When training with gambler's 's loss, we notice that usually the following intuition hold. Larger o encourages learning while smaller discourages learning. When o is appropriate, lowering makes the neural network "cautious" about learning noises present in the dataset, and so lowering o makes the model more robust to such noise. However, when o is below critical value, the minimum becomes to refrain from learning completely, and so achieving best performance using gambler's 's loss often involves approaching this critical value from above.
2.1 EARLY STOPPING CRITERION FOR THE GAMBLER'S Loss
Now we propose criterion accurately estimate the location of the gap and perform early stopping at the peak in test accuracy. It is shown to be theoretically true that early stopping is effective against label noise in Li et al. (2019) for overparametrized networks trained with gradient descent. However, it is surprising that the existing literature only focuses on delaying or relieving the negative effect of
learning the noisy labels, but not on finding the best time to do early-stop stop (Patrini et al. 2017 Han
et al. 2018 Yu et al. 2019). As in the previous works Han et al. (2018); Yu et al. (2019). we assume that the is known (to perform early stopping); otherwise, it may be estimated.
Let be the gambler's hyperparameter. Let á be the predicted probability on the true label y, and let k denote the prediction made on all the wrong classes added altogether, be the predicted confidence score by the gambler's ' loss. By definition of a probability distribution, have ş k i= 1. Based on our observations, we make the following idcalized assumption: during the the gap stage, the model has not learned anything about the data points with wrong labels, i.e., it makes strictly uniformly random guess across the all labels, the predicted probabilities are the same and are all equal to & This allows to write down the general loss function during the gap stage:
1: 1- 1. (1 (11-))log) log
1
1-
k
E B[- e log p+
m
m
where the expectation is taken over the dataset, and we will omit it since it does not affect clarity. The following theorem uses the above assumptions and predict the loss of the gap stage.
Theorem 1. During the idealized gap stage, the optimal solution has ie = O, and so the loss function takes the form:
- = p+ 1-6) )+ (1 +1-c)) log
1-
(3)
which exhibits an optimal solution at
1 1-c
-8 (E, -min- m(in(((p) = e logg e + (1- e) log
(4)
0
3

---

Under review as a conference paper at ICLR 2020
Algorithm Early Stopping on the Gambler's Loss
1: Given: untrained classifier parameters 6, noise rate e, gambler's hyperparamter o.
2: Change classifier output from m--ay to (m + 1)-way softmax.
3: for (I, in each batch do
4: Compute gambler's loss Lg from equation
5: If Lg < < oe, end training.
6: Update parameters 0 using VoLg.
7: end for
which only depends on and and we have p" co-1
The proof given in the appendix. This theorem tells that a network trained with the gambler's loss function with hyperparameter
symmetrically corrupted dataset with corruption rate 1 - should have training loss around (e, o) when entering the gap stage. This motivates using -f" (c, as early stopping criterion. Notice that we expect our loss to have the following limits:
1. As > 1, we recover the no-noise setting, and the optimal loss is O.
2. As e - 0, we go to the full noise case, and the dis-confidence score would dominate, and the gambler's loss converge log (when is large) ZZyinn al. 2019). We summarize our proposed method in Algorithm
2.2 ROBUSTNESS To LABEL NOISE WITHOUT EARLY STOPPING
130
One common approach (among others) to label noise involves intro- ducing a surrogate loss function that is specialized for the corrupted dataset hand and not very useful when one is not aware of the existence of label noise. Therefore, one surprising finding in this work is that simply using the gambler's loss improves the perfor- mance of the model at convergence, thus provides robustness to label noise automatically, For example, in Figure 4a we see that, without knowing e, the gambler's s improves over the baseline by over 40% testing accuracy at convergence.
We hypothesize that the reason is that the gambler's loss function makes learning on corrupted points slower than the clean points. As a result, this widens the test accuracy plateau by slowing down the mem- orization phenomenon. Let a data points be v, whose "observable" label is y, and true label is y'. Intuitively speaking, the mechanism is that it is easier to learn to reserve (by increasing f(x), m++1 than to memorize the noisy points (by increasing f fx)yy, and so f(x) m+1 saturates the softmax layer, making the gradient vanish. Also, cannot be too small, since that will also make the gradient vanish on the clean data points.
40077
(a) Training with Adam
6-65
To be slightly more formal. Notice that the gambler's loss function has derivative (defining g(x), == (x) + (z)))
0-5
320
(b) Training with SGD
Ve (o) f(z), f2)))) Vg(x),
(5)
Figure 2: Training trajecto-
and by Theorem 1 (in Section 2.1 during the gap stage we have:
ries on the gambler's loss us- ing different optimizers, with
yf==
(6) r 0.5.
f(x), + ffx) m+1
y y
and the probability we update a clean data point is c, while that of corrupted data is 1-c therefore, in expectation (sloppy notation), the gradient of order
m-11 and,
E [[Vlogg()))) Vg(l) m
(7)
4

---

Under review as a conference paper at ICLR 2020
137
20
14
08
(a) testing accuracy at o (b) testing accuracy at o= (c) critical behavior around (d) critical behavior around 8.25
8.20
0= 8.2 at epoch = 20 0= 8.2 at epoch = 200
Figure 3: Critical behavior of the gambler's loss. Data showing that the learning almost do not happen at all for o Ocrit, while above Ocrit the behavior is qualitatively similar. The optimal o can be tuned for using this phenomenon; however, more often one does not need to tune for o.
In general, 0 - < m - and so a corrupt data, in expectation, has gradient 2-1 times smaller than that of the clean data, and, within proper range of o, the lower the o is, the slower the model learns the corrupt labels, providing stronger robustness (in fact, one might go one step further with this qualitative analysis, see section B). This trend is what we observe in Section 4.1 when o lies above critical value. Moreover, notice that this is independent of and we expect the gambler's 's loss to help even when e is unknown, and this is shown in section 4.1 On the other hand, this analysis suggests that, if we train the gambler's ' loss with a subgradient methods such as Adam (Kingma & Ba 2014).
the constant factor in front of the gradient would be canceled by the preconditioner 1 1/v Egg and so the widening effect would not be observed. See Figure 2 The time takes for Adam to reach and to start exiting the plateau seem invariant to o, while using SGD, we see that the plateau is widened as we decrease o. For Adam, even if the widening effect is not changed, we see that at convergence, smaller does lead to higher performance using both optimizers. More plots and experiments using Adam are given in the appendix. Moreover, as shown in Yu et al. 20 Han et al. (2018), simply training on the raw loss functions features a gap stage that lasts only very shortly (~ 1 epoch), while the gambler's ss loss makes the network stays around its performance peak for a much longer time. The performance on the gambler's 's loss has barely dropped by the time the baseline almost dropped to random accuracy, and this makes it easy to detect the peak and perform carly-stopping.
2.3 TUNING FOR THE OPTIMAL O AND FIRST-ORDER PHASE TRANSITION
We note that there is a "good" range for hyperparameter o and a bad range. The good range is between critical value Gciit and M (non-inclusive) and the bad range is smaller than Ocrit. We conducted substantial experiments to observe that any o within the good range are seen to provide improved robustness against label noise, and so most of the cases we do not tune for but only make sure it is within the good range. Moreover, tuning for the optimal o is fast and can be done in linear time, and we discuss this and its connection to the physical first-order phase transition theory in the appendix.
3 RELATED WORK
Label Noise: Modern dataset often contains lot of labeling errors (Russakovsky et al. 2015
Schroff et al. 2010). Existing methods often focus on using surrogate loss function (Patrini et al. 2017) that is specific to the label noise problem at hand, or design special training scheme to
alleviate the negative effect of learning the data points with wrong labels (Han et al. 2018 Yu et al. 2019). In this work, we mainly compare with the following methods. F-Correction (FC) et al. (2017): this method smooths the hard labels with the noise transition matrix; we estimate the transition matrix as suggested by the paper; Co-teaching (CT) (Han al., 2018): this method trains two networks simultaneously. and update each with the other's predicted label to decouple the mistakes; Co-teaching+ (CT+) (Yu et al. 2019): this method improves the previous method by only propagating the labels the two networks disagree. Also, we use the subscript ES as a shorthand for early stopping. Since the early stopping criterion for these methods are not clear, we early stop at where the training accuracy is larger than for comparison, since this is when the learning on the noisy data has definitely started.
Early Stopping. Early stopping is an old problem in machine learning Prechelt 1998 Amari (9988 and studying this in the setting of label noise also appeared recently Li et al. (2019): Hu et al. 2019 In particular, we propose to early stop loss function called the gambler's loss function Ziyin et al.
5

---

Under review as a conference paper at ICLR 2020
Our analysis on this loss function allows us to propose very simple analytic function to predict a early stopping threshhold without using a validation set and is independent of the model (as long as it is neural network with enough capacity) or the task. Early stopping without validation set is also recently studied, but does not relate to the label noise problem (Mahsereci et al. 2017). It has been long noticed (Frénay & Verleysen 2013) Han al.) (0188) and recently theoretically proven Li et al., (2199) that early stopping can an effective way to defend against label corruption (but no actual method heuristic for early stopping is given); the best of our knowledge, this first early stopping criterion to deal with the problem.
(2019).
EXPERIMENTS
We design several experiments to evaluate 1) the robustness of gam- bler's loss noisy labels, 2) the effectiveness of our early stopping criterion, and 3) the performance of gambler's loss when applied to existing methods combat noisy labels.
4.1 ROBUSTNESS OF GAMBLER' ' LOSS TO LABEL NOISE
We first demonstrate that simple training on the gambler's loss pro- vides robustness against label noise (also see Section 2.3 For this and later sections, the exact experimental details are given in the Appendix. For demonstration, we choose simple CNN with 2 con- volutional layers followed by 2 fully connected layers. See Figure We see that with properly tuned o, the gamblers's loss alleviates the drop in performance caused by learning the corrupt labels. For example, in Figure 4a all three choices of o makes the accuracy plateau wider than traiming the raw loss (nll loss). At convergence, the final accuracy is much higher (~ 90%) than the baseline (~ 55%).
70-
0-0.55
MNIST: r 0.5
For the experiments on CIFAR1O, 10, the results are slightly subtler. Here we see the effect of not properly tuning o. As shown in (Ziyin M; higher (* M)
AcCurecy
al. 2019), the proper choice ranges from
30
encourages learning while very low o (~ 1) slows down learning. We notice that for a different corruption rate, different Gerit seem exist. Also see (category Gbler), we see that at convergence, training on gambler's loss achieves much higher accuracy than the baseline training on nll loss.
100
125
150
Epoce
(b) CIFAR1O: r 0.5
4.2 EARLY STOPPING
Now, we show that stopping at our predicted loss level succeeds stopping at point close to where the maximum accuracy is achieved. See Figure We choose architectures of very different capacity. The experiment the experiment on CIFAR- 10 done using ResNet-18. We also test on wide range of noise levels ranging from small, with corruption rate 0.2, to extremely large, with r = 0.85 (the intermediate ranges with larger figures are given in the appendix, see Figure 7. We see that our method predicts close to optimal early stopping point for all corruption rates we tested on. Also notice how the learning curves are different for the two different datasets. This suggests that our theory works even in the presence of very different dynamies. The fact that this early stopping criterion works suggests in return suggests the validity of our assumption. In these experiment, we do not tune for optimal o, they are all set to 9.9; since the results are qualitatively similar when ox Ocrit-
MNIST is done as in the previous section and
25-
5-0.05
100
125
:50
Epocs
(c) CIFAR10: r= 0.8
Testing accuracy through out training; raw refers training the default loss func- tion, i.e. the nll loss, raw plot- ted in red. We see that, when o > Ocrit, the gambler's loss pro- vides robustness on its against noisy labels.
Figure 4:
See Table I and also compare with other baselines in Table 2 We notice that doing this already achieves SOTA results on MNIST for all symmetric noise categories according to the results in Han et al. (2018) and Yu et al. (2019). Early stopping (AES) using our criterion compared with very standard carly stopping method using validation set: we split 6000 images from the training set to make a validation set, and we early stop when the validation accuracy stops to increase for consecutive epochs. There are also few other validation-bant based early stopping

---

Under review as a conference paper at ICLR 2020
Mry
(a) MNIST: v 0.2
(b) 0.85
(c) CIFARIO: r 0.2
(d) r 0.85
Figure 5: Early stopping on MNIST (Ist row) and CIFARIO (2nd row). refers to corruption rate. The horizontal line is the predicted early stopping point. We see that this point corresponds to where the testing accuracy (blue solid line) at maximum.
Table 1: Robustness the Gambler's ss loss to noisy labels. We see that simply using gambler's 's loss on label noise problems improve. For the two categories using carly stopping, the number parenthesis refers to the avg. number of epochs to stop. Using gambler's 's loss stops at better accuracy, has smaller variance, and takes much shorter time.
Performance Comparisons
Early Stopping Comparisons
Dataset
nll loss
Gblers
VES
Ghlers
MN r 0.2
84.7
96.7 = 0.2
95.1+0.4 (95)
98.8 = 0.1 (17)
MN r 0.5
5.1
91.2 0.7
7..7 (115)
98.0 0.0 (18)
MN r 0.65
2.5
85.9 1.1
5.5 (115)
97.2 (15)
MN r 0.8
76.1 0.3 1.0
93.5 (15) 2.1 (117)
MN r 0.85
1.7
71.0 1.4
15.0 (110)
8 (13)
IMDB r =0.1
70.5 0.5
0..0 (46)
74.0 + 0.4
IMDB 0.15
6.3
6.2
6..0
69.1 0.1 (12)
IMDB r 0.2
0.1
0.3
59.6+
65.3 = 0.7(12)
IMDB r 0.25
5..4
5.1
5.2
02.499.4(9)
IMDB r= 0.3
53. 0.3
5..1
58.2 0.3(
criterion, but they are shown perform qualitatively and quantitatively similar (Prechelt 1998), so we only compare with this method. This early stopping is called VES (validation early stopping) and our method is called AES (analytical early stopping). We also introduce warmup technique for AES, where we set o= m for the first five epochs; this technieques demonstrates consistent gain throughout, and we denote this LAES. We see that our proposed method significantly outperforms the baseline early stopping method; this is because a small validation set may have large variance and this problem is more serious when label noise present (since validation sets are often split from the train set). In the VES vs. AES comparison, we fix o 9.99 and is not directly comparable to the left parts in the table. In the left part, We also notice that training and stopping on the gambler's ' loss is especially effective on tasks with extreme corruption rate (r 0.85). with around 70% accuracy improvement over the baseline and 28% improvement over the runner-up SOTA methods. We also conduct experiment sentiment analysis and is a binary classification problem. We use a standard LSTM with 256 hidden dimension, and with 300-dimensional pretrained GloVe word embeddings (Pennington et al. 2014). Again, we notice that, 1), using the gambler's loss consistently improve on the nll loss, and 2) using AES consistently improves early stopping validation set (by about 2 7% in absolute accuracy). Also see appendix section Mor experiments on really small corruption rate r = 0.02-0.08 on IMDB.
the IMDB dataset (Maas al. 2011). which is standard NLP task
4.3 BENCHMARKING AGAINST OTHER METHODS
In this section, we mainly use LAES the proposed method, since it consistently outperforms AES (for example, see the CIFAR-10 experiments, and see section D. See table 2 On MNIST, using our method always achieves SOTA performance. On CIFAR- TO (CF10), for the CIFARIO experiments, we do not compare with CT+ since it is a modified version of CT and we notice that it fails in the same range as CT (r = 0.7, 0.8, 0.85). Our method especially strong when the noise level is extreme (r ~ 0.7, 0.8, 0.85), agreeing with our finding in the previous section. We also note that LAES consistently outperforms AES. CT seems to outperform our method a little in the range r 0.1 - 0.6, but CT requires twice as much parameters, and takes at least twice as much time to train due to its special training scheme. We also emphasize that the improvement over purely training on the nill loss is significant, ranging from 10 30% in absolute accuracy. We note that also tried to
7

---

Under review as a conference paper at ICLR 2020
Table 2: On MNIST. using our method always achieves SOTA performance. On CIFAR- 10 (CFI0), our method is especially strong when the noise level is extreme (r~ ~ 0.7, 0.8, 0.85), agrecing with our finding in the previous section. (*NA means that the ES stopping point is not reached by the algorithm.)
Dataset
FC
CT
CT+
CT
CT+ES GbleTLAES
MN r 0.2
98.8 0.1
9.0
97.8 0.1
98.6 0.2
97.6 0.1
99.0 0.1
MN r 0.5
7.0
9.1
9.2
97.6
9.2
98.4 0.2
MN r 0.65
2.1
0.1
9.1
9.2
9.2
97.6 0.3
MN r -.88
0.1
7..
6.1
82.1
7.3
95.0 = 0.5
MN r 0.85
110
0.2
1.01
66.5 0.5
NA
88.0 1.0
all loss
CT
CT'S
GbliSAES
GblrSLAES
CF10 r 0.2
58.1
71.0 0.6
69.5
67.5
68.2 0.0
CF10 r 0.5
0.2
6.2
6.85
5.4
60.1 0.2
CF10 r 0.6
0.1
0.2
6.3
0.5
57.1 0.5
CF10 r 0.7
0.4
0.2
49.1
4.5
50.6 + 0.5
CF10 r 0.8
0.5
2.10
25.8 = 1.0
37.0
40.7 +: 2.0
CF10 r 0.85
13.0 = 0.5
1.1
NA
18.9 = 1.0
27.0 + 2.1
combine method with CT but not very successful, but CT throw away large portion of the training dataset, and changes the underlying distribution, making the AES criterion not applicable.
5 CONCLUSION AND DISCUSSION
In this work, we have proposed simple and effective method deal with label noise problems. This paper promotes the gambler's loss as a noise robust loss function, shown to improve performance of existing classifier model, while the early stopping criterion provides simple criterion for early stopping when the corruption rate is known. Depending on the problem, one may decide to use the complete framework together (gambler's loss training + early stopping). or, when the assumption is not well satisfied, only use the gambler's ss loss to improve the model's 's robustness against a label- corrupted learning task. This urges for an understanding for why the gambler's 's loss actually works. We plot in training accuracy for MNIST with corruption rate 0.8 in Figure We see that the gambler's loss seems to work in a fundamentally different way from the standard n// loss. Training on all loss achieves 100% accuracy on the training set, with its loss approaching 0; however, the gambler's loss does not do so, its training accuracy stops increasing at around 36%, which is slightly above e. We note that this observation also holds for various corruption rates, and for different optimizers such as SGD. Comparing with T we see that the gambler's 's loss achieves about 76% testing accuracy, while the baseline drops to 19% at this time. The vertical line also shows where our early stopping criterion is reached, at this point is training accuracy is ~ 20% with testing accuracy 92%. We hypothesize that the main reason for gambler's loss's outperforming the baselines is that automatically chooses a subset of the training set to learn on, thus avoiding large part of the dataset that is corrupted.
This result may provide insight into the current debate on the generalization--emooiiee
100
mY
effect (Zhang et al.) 2017, Wu
90-
2.0
et al. 2017] Hu et al. 2019: Arpit et al.) 2017]. A key ar-
80-
gument from Zhang et al. (2017) was "deep neural networks easily fit random labels...: explicit regularization may improve generalization performance, but is neither necessary nor by itself sufficient for controlling generalization error." Our results above, however, shows that this is not the case when a different loss function is chosen. While the nll loss encourages learning and memorizing the training set, the gambler's loss does not seem to do so. Our findings suggests that the key to understand- ing the memorization effect might be on the loss function itself, not the architecture of neural networks or explicit regulariza- tion; this is also what is suggested by Abiiodun et al. (2018). We hypothesize that it may be seen as a natural generalization of the n!/ loss function, since it only involves adding task inde- pendent term to the original n// loss, and using o > Oerit seems always beneficial. We would like to study the memorization effect using the gambler's 's loss in more detail. Applications to real label noise are also ripe avenues for future work.
weang seerggooooo scorste
-1.5
NOC obterss
Iosa golers
50
40-
0.5
30-
20
Figure 6: Training accuracy and train- ing loss MNIST with corruption rate 0.8 with Adam. o 9.7. Training with gambler's loss prevents memorization of noisy labels. At convergence, nt/ loss reaches 19% testing accuracy, while the gambler's loss stays around 76%.
weal-world large-scale datasets with
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Oludare Isaac Abiodun, Aman Jantan, Abiodun Esther Omolara, Kemi Victoria Dada, Nachaat Ab- dFlatif Mohamed, and Humaira Arshad. State-of-the-art in artificial neural network applications: A survey. Heliyon, 4(1 4(11))=00933) 2018.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Comput., 10(2):251- 276, February 1998. ISSN 0899-7667. doi: 10.1 10.116/00997669930017716 URL http: // //dx. dox. /ttp:///...112///777668111221990
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 233 242. JMLR. org, 2017.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N. Chang. Sungbok Lee, and Shrikanth Narayanan. lemocap: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335-335, 2008. URL http
//dblp. uni www.trrrrrreeeee. htt.eeeedd/ab/oooraaaaa//a/
htm wttettttttaaeeeee #BussOBLRMKCING
Mostafa Dehghani. Aliaksei Severyn, Sascha Rothe, and Jaap Kamps. Avoiding your teacher's
mistakes: Training neural networks with controlled weak supervision. In arXiv, 2017. URL
/tttps://axxi..or//aaa/
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learning systems, 25(5):845-869 2013.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Jeff Howe. Crowdsourcing:
Why the Power of the Crowd Driving the Future Business. Crown
Publishing Group, New York, NY, USA, edition, 2008. ISBN 0307396207, 9780307396204.
Wei Hu, Zhiyuan Li, and Dingli Yu. Understanding generalization of deep neural networks trained with noisy labels. arXiv preprint urXiv:1905.11 1368, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR. abs/1412,6980, 2014. URL http://dblp... http//lppp.niiieeeeeee umi-triers wtt.:ttrreeedddb//eeeee /ttp///////ooooaa/ooooorrr
corr1412. htm. #KingmaB14
Ivan Krasin, Tom Duerig, Neil Alldrin, Andreas Veit, Sami Abu-El-Haija, Serge Belongie, David Cai, Zheyun Feng, Vittorio Ferrari, Victor Gomes, Abhinav Gupta, Dhyanesh Narayanan, Chen Sun, Gal Chechik, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from http:://gihuu..oo/opennmmt
2016.
L.D. Landau and E.M. Lifshitz. Statistical Physics. Number v. 5. Elsevier Science 2013. ISBN 9780080570464. PRLattps:: /ttp///oo.ssoooo
/tLpatt/////:sss....... httpbbooks..oggle.coo www.le.coogooooooooooooo http://oe..../////////o.........
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is prov- ably robust label noise for overparameterized neural networks. arXiv preprint 1rXiv:1903. 2019.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Mceting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http
7/ wwtp://wwwweb..r//atttooo....
Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig, Early stopping without a validation set, arXiv preprint arXiv: (0t.:1773309880 2017.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions increasing complexity. arXiv preprint arXiv: 1604, 2019.

---

Under review as a conference paper at ICLR 2020
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952. 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 55-69.
Springer, 1998.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 252, 2015.
Florian Schroff, Antonio Criminisi, and Andrew Zisserman. Harvesting image databases from the web. IEEE transactions on pattern analysis and machine intelligence, 33(4):754-766, 2010.
L. Wu, Z. Zhu, and W. E. Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes. ArXiv e-prints, June 2017.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7164-7173, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017. URL https: http:///axxvvo.ooo. org/as//
1611. 03530]
Liu Ziyin, Zhikang Wang, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. arXiv preprint 2019.
10

---

_o11_ o(m-11) for MNIST and Under review as a conference paper at ICLR 2020
A A BRIEF INTRODUCTION TO THE GAMBLER'S LOSS
Let f()) denote the model and f()) is the m + 1-dimensional normalized prediction, i.e.,
m+1
Σ
(8)
The gambler's loss for classification problem takes the form
m++1
max ((f(z),))
Σ
Pk.y log f(x)x fx))
(9)
where Pk,z (r) is the true label. Notice that if o co, the loss function is equivalent to the KL divergence. The empirical target is usually a loss, where Py. = 1 for the true label y and zero for k y; thus the loss function takes the following simple form (and we change max & to min -f);
S log [(x))x + 1 {z))mm)
min (()((()))))))))
Σ k-1
(10)
log f(x) - (z)
(11)
and this reduces to the cross entropy (or nll loss) if co.
is also useful to notice the following relation. Let δ be the predicted probability on the true label y. and let & denote the prediction made on all the wrong classes added altogether, / be the predicted confidence score by the gambler's loss. To relate this to the definition above (Equation [T]. we note that this is equivalent defining ý normalization condition in Equation
f(x) hs Einy f()) I == f(x) (=)m+1, and, therefore, by the
m f(x)) + f(x) =1.
p+ k -+)+
(12)
and this condition will be used to prove Theorem
B
TRAINING SCHEDULE
Recall equation 7
0-1 m
E[v [[Vlogg()))) ~ Vg(3)
V
V (z))
(13)
We can go step further and plug in g(z), = fz), + f
consider the effect the we say that the reseravation score f(z)m+1 is prone to saturation, and learning on the label class j is prevented
prediction for /' (x) joy jemtl, and f(x) If [V(x)) (+)mll
by such saturation. We obtain:
( ( V
E[V B(Vlogg(x)))) V m m-1 v ( jemtt jemt1
(14)
where the first term is the gradient for the correct label, and the second term he gradient for the incorrect labels and the third term is the gradient for the reservation score. Roughly estimating, to prevent saturation delaying learning on correct labels, we want 1 1 ---1 CIFAR we have m = 10, and this condition translates to o > 8.88... which agree in magnitude to the experimental threshold ocra & 8.2. This high-level agreement suggests that qualitative analysis is good.
On the other hand, if want the saturation f(x)m+1 to prevent learning on the corrupt labels completely, then we should have am--IJ m-I and for m = 10, this translates to < 4. This
p-1 6-1
suggests the following training scheme for 10 problems:

---

Under review as a conference paper at ICLR 2020
1. start training with 8.89
2. when the early stopping criterion is met, either stop, or switch to o <
In fact, similar training scheme is used in (Ziyin al. 2019) in order to achieve SOTA results, while unaware of this analysis. For CIFARIO, 10, they started with o= 9.9 to for the first fifty epochs, and then switched to 2 afterwards to achieve the best performance.
C MATHEMATICAL SYMBOLS
N remmber of impul dimension
M number of output dimension
D dataset, [D] size of the dataset
D EXPERIMENT DETAILS
We use Pytorch as the framework of implementation. Code to this paper will be released at http://atssssee
E THEOREM PROOF
In this section we prove theorem We first show that À = O. Intuitively speaking, this simply means that gambler betting randomly will not make money, and so the better strategy is to reserve money in the pocket, and so it suffices to show that for any solution p, k. the solution p' = p. k 0. i achieves better or equal doubling rate. For mislabeled point (we drop ) the loss is log(
M !)) but
&
M o, and log( NT ( log( Aal) and we have that optimal solution always have k= 0. Now, we find the optimal solution to
1 1- + - log Φ+
1-1
{(p) = clog
(15)
by taking the derivative with respect to p:
oe
0
-1
(1)
(1 -
(16)
(o- 1)p+ +1
1-
and then setting equal to 0
0-1 (o 1)p+1
+ (1-c) -1
(p)
=
(17)
1-
is the Poptimat
60
Poptimal
(18)
0
and then plugging into the original equation [9]:
1-c
(e, min e (1- log
(19)
0-1
F ADDITIONAL EXPERIMENTS
In this section, we give experiment results Gambler Early Stopping. See Figure 7
12

---

Under review as a conference paper at ICLR 2020
15
08
(a) MNIST: r= 0.2
(h) r 0.5
80
230
18
238 8
20
X0
21
(c) = 0.8
(d) r= 0.85
6.11
225
(e) CIFARIO: r 0.2
(f) y= 0.5
215
125
118
p 0.8
(h) r= 0.85
Figure 7: Early stopping on MNIST (Ist and 2nd row) and CIFARIO (3rd and 4th row). r refers to corruption rate. The horizontal line is the predicted early stopping point. We see that this point almost always corresponds to where the testing accuracy (vertical blue solid line) is at maximum.
13

---

5 Under review as a conference paper at ICLR 2020
G DEFINITION OF TRANSITION MATRIX
Transition matrix is used to replace some correct labels with wrong labels because the dataset used in our experiments such as MNIST and CIFAR are clean. There are two different types of noise transition matrix and explanation follows.
Symmetric Flipping: Each label has an uniform possibility to transform to one of rest class. In most cases of manual label work, correct label has equal possibility to be any other class label. is the number of class and is the rate of label that will be modified as a wrong label
6
8
1"
1
6
1
1-n
13"
M :=
8
(20)
1-n
I-n
1-n.
Pair Flipping: Each class can only be corrupted as specific class. For example, it's more likely to label as 9 than any other class except 9, the rate of label that will be modified as a wrong label:
0
0 1
8
M :=
0
-
(21)
:
0
1 8
H CNN ARCHITECTURE
In this section, we show the architectures that we used but did not describe in the main text.
Table 3: architecture of the neural network used in MNIST
CCNN on MNIST
28 x 28 Gray Image
20 x 5 conv, ReLU
2 2 max- max-pool, stride 2
50 x 5 conv, 20 ReLU
2 x max-pool, stride 2
dense 800 500
dense 500 11
Table 4: architecture of the neural network used in cifar10
CNN on cifar10
32 x 32 RGB Image
5 conv, 128 IReLU
5 5 conv, 128 LReLU 2 2 max-pool, stride 2 dense 128 128
dense 128 11
I EFFECT OF TUNING AND CRITICAL BEHAVIOR
In this section, give experiments the critical behavior discussed in 2.3 First Figure
14

---

Under review as a conference paper at ICLR 2020
137
(a) critical behavior around critical behavior around (c) critical behavior around (d) critical behavior around o= 8.2 at epoch = 200 0 8.2 at epoch 200 o= 8.2 at epoch = 20
0 = 8.2 at epoch 200
Figure 8: critical behavior of the gambler's loss. We that the learning almost do not happen at all for o < Ocrit, while above Oerit the behavior is qualitatively similar. The optimal o can be tuned for using this phenomenon; however, more often one does need tune for
In Figure 2
we show that lower value of provides stronger robustness label noise consistently; this raises the question about whether we can continue to lower to its lower limit (= 1) to achieve the best performance or the answer is no, but the reason is very surprising. We empirically find that first order phase transition exists when tuning o (such as transition of water to ice). In particular, we find that there exists a critical value Ocrit such that, when > Ocrit, the robustness to label noise increases we lower when < Ocrit, the learning proceeds badly: the change in performance is discontinuous. This is characteristic 2013). See Figure 3] In this setting, have 0.5 for MNIST, and pin down the Ocrit to lie between 8.21 and 8.22: qualitatively different behaviors are found for the two different phase. See appendix for plots ranging from 10.0 to 5.0. A qualitatively similar behavior observed using Adan and on CIFAR1O. 10, and using other architectures. Clearly studying this critical behavior very interesting but beyond the scope of this work. The above observation, however, suggests the following heuristic to tune hyperparameter o:
physical first order phase transition (Landau Lifshitz)
1. Start with o very close M (for reference, for MNIST, 9.5 seems good starting point; for CIFARIO, o a 9.9; CIFARI00 o 99.9).
If the learning curve starts similarly as before, then decrease o until the learning stops abruptly, and use the previous
This procedure for tuning does not rely on a validation set at all. Also, as show in in Figure 2 for = 8.3, 8.5, the performance is very similar when is close to Ocrit, there is need to be infinitesimally close to the critical value. Unless otherwise noted, we tune our hyperparameter this way in the experiment section. Moreover, One can prove that M will always work at least as good as the baseline, and so Ocr should always < M.
Using Adam can even pin down Ocris for this problem to lie hetween 8.2100 and 8.2101
15

---

Under review as a conference paper at ICLR 2020
00.3
12
563
14
220
14
3.4
8
05
214
68.
314
1/
0831
15
12
(a) 0= 10.0
(b) 8.5
(c) 0 = 8.0
(d) 7.5
714
302
330 155
15)
25
125
159
25
56
300 150
25
15
159
(e) 0
7.0
(f) 6.5
(g) 0 = 6.0
(h) 5.0
Figure 9: Gambler's loss is used alone without early stopping method. Initially, the performance gets better with the decreasing. While the performance gets worse with the o decreasing when o is less than 8.5, which indicates there is critical point for o locates in between 8.0 and 8.5. The most suitable o is just larger than the critical point
5
03
39
300
155
75
125
(a) 8.35
(b) 8.3
(c) 8.25
(d) 8.2
15-
235
m
12
12
(e)
8.15
(1) 8.1
(g)
8.05
(h) 8.0
Figure 10: Under higher resolution hyperparameter o, it's for that the performance will get better when o gets closer to critical point from right side on number axis. While the performance will dramatically get terrible when is just lower than the critical points. What's more, the critical point the range of [8.2, 8.25]
16

---

Under review as a conference paper at ICLR 2020
Figure 11: put all curve of Figure in one Figure
17

---

Under review as a conference paper at ICLR 2020
0.7
2.5
0.6
training loss
2.0
train loss
0.3
1.5
0.
1.0
75
35
70
test accuracy
65
lest acc
30
25
60
20
50
80
100
20
epoch
Epochs
(a) Training IMDB dataset (r 0.1) with LSTM with pretrained GloVe embedding.
(b) Training of 5-layer CNN noisy openimage dataset (r unknown).
Figure 12: Three stage phenomenon across different datasets and architectures
J MORE EVIDENCE
THREE STAGE PHENOMENON
The three stage learning curve observed in section actually quite universal when significant level of label noise is present. Of course, the exact details of the curve really depends on the architecures and the tasks. See Figure I2 The left pictures shows training of an LSTM with hidden size 300 with pretrained GloVe embedding (Pennington et al. 2014) the IMDB dataset (Maas et al. 2011). Notice that the performance peak (around epoch 5) also coincides with the plateau, as we expected.
On the right, try 5-layer CNN the openimage dataset (Krasin et al. 2016). Two lines two independent runs. This is a very noisy dataset with many mis images that do not belong to any category in the dataset. In this dataset, the noisy are not artificially introduced by flipping or symmetric corrupt, and so the corruption rate is not known a priori. However, we still see the three stage phenomenon due the existence of strong label noise in the dataset. The plauteau is wide and casily observable in this case, also notice that the end of plateau corresponds to the performance peak. Moreover, this plot actually shows one more advantage of the our method: by observing the learning curve by naked eyes, one identify that the end of plateau occurs at loss ~ 2.2, and plug this back into our early stopping criterion, one may estimate the corruption rate be roughly 0.6 0.7.
K SOME EXPERIMENTS ON ASYMMETRIC NOISES
In this section, we present some experiments on asymmetric noise. In particular, here we consider the pairflip kind of asymmetric noise. The definition is given in section[]] which is the same as was given in Han et al. 2018). See Table While using gambler's 's loss seems to always improve performance at convergence, performing early stopping is tricky in this case, since the criterion in Theorem relies on the fact that the labels are symmetrically corrupted. In this sense, in order to perform early stopping, notice that the problem has effective dimension of 2 (because the
18

---

Under review as a conference paper at ICLR 2020
Table 5: Robustness of the Gambler's loss asymetric noise.
We set o 2.0 this experiment.
Dataset
all loss
Gblers
Gblers
MN pairflip r 0.3
71.8 + 0.5
74.7 0.8
96.3 1.5
MN pairflip 0.4
59.6 1.0
63.0 1.0
90.6 2.0
MN pairflip r 0.45
55.2 a 2.1
60.3 # 1.0
81.3 + 5.1
Table 6: Robustness of the Gambler's loss noisy labels, in comparison with the upperbound.
Dataset
n/l loss
Gblers
Gblers
GblerSLAES
Upper Bound
MN 0.2
84.7 = 0.5
96.7 + 0.2
98.8 a 0.1 (17)
99.0 = 0.1
98.9 = 0.1
MN -0.5
55.1
91.2 0.7
98.0 0.0 (18)
98.4 0.2
98.4 0.1
MN r 0.65
8.19 2.5
0.2 (15)
973
98.2 = 0.2
MN 0.8
1.0
76.1 0.3
93.5 0.3 (15)
9.5
98.0 0.1
MN r 0.85
0.7
71.0 1.4
8.72 (13)
88.0 1.0
97.9 = 0.2
corruption binary), and we set in this case. like before, the warmup schedule is for 5 epoch. We see that using Gambler's loss consistently improves the performance at convergence, and using the early stopping criterion gives very strong performance gain over that.
L FULL RESULTS ON MNIST
might be useful to also compare with an experimental upperbound at each corruption rate. See T The upper bound is computed by only training on Delean for 100 epochs. We see that while at convergence, performing early
upper bound is always better than training on the gambler's stopping on the gambler's loss might even perform better or good as the upperbound (e.g. when r 0.2, 0.5) because the upper bound, training until the very end, might overfit to the dataset, while early stopping prevents such overfitting.
M MORE RESULTS ON IMDB
In this section, give more results IMDB, baseline on all categories we compared with. In this section, o = 1.9 for all experiments. In the main body, we have included the results for in the range 0.1 0.3. Here, give the results in the range r 0.02 0.08. Again, we see that the improvement over nlll loss and over VES is consistent, with the gain increasing the corruption rate increases.
that the proposed method outperforms the
Table 7: Additional Resutls IMDB dataset.
Dataset
n// loss
VES
Gblers
IMDB r 0.02
78.5
79.7 79.7(25)
79.8(14)
IMDB 0.04
75.4
77.1 (25)
78.6 (13)
IMDB -00.6
73.9
74.4 (20)
77.6 (13)
IMDB r ==0.8
73.1
72.9 (32)
75.1 (25)
19


Reviews:
[{"review_id": "rJepBpantB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updated review: Thanks for your comments. I feel the latest version of the paper is better than the previous version. \nHowever, as stated by other reviewers as well, the claims of the paper are quite ambiguous. Another example from the author response is the point about how Chapter 6 of the Elements of Infomation Theory is related to Gambler's loss. This is not clear to me. I would not object to accepting the paper but I find it difficult to recommend accept for this paper. Perhaps the authors can be more clear in their claims. \n\n----------------------------------------------------------------------------------------------------------------------------------------------------------\n\nSummary: The paper focusses on the problem of noisy labels in supervised learning with deep neural networks. The paper, in turn, proposes an early stopping criterion for handling label noise. The early stopping criterion is dependent on a new loss function that is defined as the log of true label + weight on a reservation option? The paper shows that when the labels are corrupted then the propose early stopping criterion does better than early stopping criteria obtained via the validation set. \n\n\\The first section of the paper establishes that when label noise is present in the dataset, then there are three stages to training a deep neural network. \nThe learning stage where the highest accuracy on the test set is achieved. \nThe gap stage where test set accuracy goes down. \nMemorization stage corresponds to when a deep neural network memorizes corrupt labels and test accuracy goes completely down. \nI cannot understand figure 1(a). The y-label says accuracy but it seems that the plot is about loss. What dataset was this and what architecture of DNN was used? The plot shows that the DNN achieved a 100% accuracy in 5 epochs. Is this result meaningful? Before establishing a hypothesis based on this should the hypothesis not be tested on multiple datasets.\nThe paper says that these stages are persistent across multiple architectures and datasets and as proof the paper says ‘we verified that’. Why can’t the reader see the experiments? By across datasets does the paper mean MNIST and CIFAR? By across architecture does the paper mean the two architectures mentioned in the appendix one each for MNIST and CIFAR respectively? \n\nThe paper makes the assumption that label noise is symmetrically corrupted. Why and where does such an assumption hold? What happens to the proposed method if that is not true. \n\nAssumption 2: During the gap stage the model has learned nothing about the corrupt data points. \nHow is that even possible? \n\nEquation 1: So the loss function proposed is log(f(x)_y + (1/o) f(x)_m+1) .  What is y here? The true label? Why is y called a point mass? Is this different from the cross-entropy loss + log loss on m+1 ?\n\nI do not understand equations 2 to 5. \n\nFor figure 3 again what datasets were used?\n\n“ Making random bet will help with making money and a skilled gambler will not make such bets” \nWhy does making random bet help with making money? If random is good how can a skilled gambler exist in such a game? What is this skill?\n\nk denotes the sum of probability of predicting anything that is not y or m+1 (it does not denote prediction). \n\nIn the experiments section what was the symbol for the rate of corruption changed from epsilon to r. Are they different? \n \nWhat is nll? \n\nIt seems that gamblers loss best shines when the corruption rate is as high as 80% . That is 80 percent of the data is corrupted. Does this mean that if I trained with only 20% of the non-corrupt data I would still get a 99% accuracy on MNIST (even without gamblers loss)? A comparison of this sort would have been useful.  \nOne astonishing result the paper presents is that with gambler’s loss even with 80% corrupt labels a 94% test accuracy is possible on MNIST dataset. I think this is significant, this raises the question that is it required to label all the data points ina dataset to achieve high accuracy or is it possible to achieve just as much with only 20% of the labels?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxBYBr6Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nUpdate after rebuttal:\n\nThe good:\nThe rebuttal and updated paper address many of my concerns. Most importantly, the updated paper demonstrates the three-stage phenomenon on Open Images and adds experiments on IMDB showing that the Gambler's loss with AES helps a lot. The LAES iteration introduced in the updated paper alleviates my concern about performance drop compared to the CT baseline at certain corruptions on CIFAR-10.\n\nThe bad:\n- From Figure 12, it looks like the three-stage phenomenon doesn't hold on IMDB. Does AES provide additional benefit beyond the Gambler's loss on IMDB? This needs to be clarified with the way Figure 12 turned out.\n- There is a serious missing citation [1] that should be included as a baseline. The proposed method in [1] is at least superficially similar to the Gambler's loss and also makes use of the fact that it is easier to fit clean labels than noisy labels. My apologies for not noticing this earlier.\n\nOverall:\nI would still suggest acceptance, because the three-stage phenomenon is an interesting find that the authors make good use of. In light of the missing citation, though, I cannot raise my score.\n\n\n[1]: Zhilu Zhang, Mert R. Sabuncu. \"Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels\". NeurIPS 2018.\n\n-----------------------------------------------------------------------\nSummary:\nThis paper proposes a method to alleviate label noise. It opens with the observation of three distinct stages when training in the presence of label noise. Importantly, there is a ‘gap’ stage during which the network has not begun memorizing noisy labels and early stopping is ideal. The authors then observe that the Gambler’s loss (Ziyin et al., 2019) elongates the gap stage and propose an analytic early stopping (AES) criterion for identifying when to stop training.\n\nThe analysis of the AES criterion, e.g. in Figure 5, and the observation of a phase transition when tuning the o hyperparameter are quite interesting, and the latter observation is of practical value when using the AES criterion.\n\nThe AES criterion seems to be well-motivated, and the empirical evaluation of the Gambler’s loss with and without early stopping is good. The results are strong on MNIST but somewhat weak on CIFAR-10. Specifically, the improvements on CIFAR-10 only appear for large corruption rates (0.7+), and performance is lower than the baselines for other corruption rates. This is a worrying problem, because it calls into question the value of the method on larger problems. However, seeing as this is a distinct approach from the baselines and that it demonstrates some promise, I recommend borderline accept. The authors could raise my score by demonstrating more consistent gains on another larger-than-MNIST CV dataset or an NLP/speech dataset. Other points of concern that I have are listed below.\n\nMajor points:\nAt the top of page 3, the authors say that the idealized gap assumption “holds well for simple datasets such as MNIST and on datasets with very high corruption rate, where our method achieves best results, and less so on more complicated datasets such as CIFAR10”. The idealized gap assumption is behind the AES criterion, but Figure 5 suggests that the AES criterion works well on CIFAR-10, so what do the authors mean when they say the assumption doesn’t work as well on CIFAR-10? Is this just referring to the results?\n\nSaying traditional label noise correction methods are “of no use when one is not aware of the existence of label noise” seems unfair. The FC method and others do not require foreknowledge of the corruption rate and do not harm performance in the absence of label noise, so they can also be said to automatically correct label noise.\n\n“FC, however, requires knowing the whole transition matrix, and is outperformed significantly by our method.”\nThis is not quite true, because Patrini et al. propose an estimate of the transition matrix as part of the Forward correction. Did you use the estimated or true transition matrix for the FC method? It would be good to clarify this in the paper.\n\nMinor points:\nThere are a few grammatical errors and typos in the paper:\n\n“or explicit regularization, this is also what is suggested by Abiodun et al. (2018)” (run-on sentence)\\\n\n“CIFAR10” should be “CIFAR-10”", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJleZsO2KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new loss function for dealing with label noise, claiming that the loss function is helpful in preventing overfitting caused by noisy labels. Some experiments show the effectiveness.\n\nThe theory part is unclear to me. (1 Why the derivative of Eq. (1) is Eq. (2)? The notation of f, f_\\theta, and f_w has been abused. The notation is confusing without explanation. It seems Eq. (2) is not correct and the following is not convincing. (2 Why a small gradient will slower the model to fit the data? This is not clear and maybe not true. (3 The assumptions \\hat{p}+\\hat{k}+\\hat{l}=1 is very strong to me. The events should be dependent. This makes all the theoretical analyses pseudo and not convincing at all. The authors may spend more effort to make the part clear, reasonable, and convincing.\n\nMany claims are very ambiguous. For example, \"the key point is that it always holds on some degree\", on which degree and why always holds? \"In some cases, it even leads to a better memorization phenomenon\", in which cases and why lead to better memorization phenomenon?\n\nSome claims are even wrong. For example, \"Traditional methods in label noise often involves introducing a surrogate loss function that is specialized for the corrupted dataset at hand and is of no use when one is not aware of the existence of label noise\" This is just for some specific methods, not for the most of them.\n\nTypo: \"We verify that . Therefore,\"\n\nOverall, I cannot understand why the proposed loss function works and cannot recommend acceptance for the current version.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on mitigating the effect of label noise. They provide a new class of loss functions along with a new stopping criteria for this problem. The authors claim that these new losses improves the test accuracy in the presence of label corruption and helps avoid memorization. The reviewers raised concerns about (1) lack of proper comparison with many baselines (2) subpar literature review and (3) state that parts of the paper is vague. The authors partially addressed these concerns and have significantly updated the paper including comparison with some of the baselines. However, the reviewers were not fully satisfied with the new updates. I mostly agree with the reviewers. I think the paper has potential but requires a bit more work to be ready for publication and can not recommend acceptance at this time. I have to say that the authors really put a lot of effort in their response and significantly improved their submission during the discussion period. I recommend the authors follow the reviewers' suggestions to further improve the paper (e.g. comparing with other baselines) for future submissions
Decision: Reject