Detecting Noisy Training Data with Loss Curves
Authors: Geoff Pleiss
Tianyi Zhang
Ethan R. Elenberg
Kilian Q. Weinberger
Venue:  (2020)
Keywords: Deep learning
noisy data
robust training

Abstract: This paper introduces a new method to discover mislabeled training samples and to mitigate their impact on the training process of deep networks. At the heart of our algorithm lies the Area Under the Loss (AUL) statistic, which can be easily computed for each sample in the training set. We show that the AUL can use training dynamics to differentiate between (clean) samples that benefit from generalization and (mislabeled) samples that need to be “memorized”. We demonstrate that the estimated AUL score conditioned on clean vs. noisy is approximately Gaussian distributed and can be well estimated with a simple Gaussian Mixture Model (GMM). The resulting GMM provides us with mixing coefficients that reveal the percentage of mislabeled samples in a data set as well as probability estimates that each individual training sample is mislabeled. We show that these probability estimates can be used to down-weight suspicious training samples and successfully alleviate the damaging impact of label noise. We demonstrate on the CIFAR10/100 datasets that our proposed approach is significantly more accurate and consistent across model architectures than all prior work.

Under review as a conference paper at ICLR 2020
DETECTING LABEL NOISE FROM Loss CURVES
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper introduces a new method to discover mislabeled training samples and mitigate their impact on the training process of deep networks. At the heart of our algorithm lies the Area Under the Loss (AUL) statistic, which can be easily computed for each sample in the training set. We show that the AUL reliably re- veals differences in training dynamics between (clean) samples that benefit from generalization and (mislabeled) samples that need be "memorized". We demon- strate that the estimated AUL score conditioned on clean vs. noisy is approxi- mately Gaussian distributed and can be well estimated with simple Gaussian Mixture Model (GMM). The resulting GMM provides us with mixing coefficients that reveal the percentage of mislabeled samples a data set as well as probability estimates that each individual training sample is mislabeled. We show that these probability estimates can be used to down-weight suspicious training samples and successfully alleviate the damaging impact of label noise. We demonstrate the CIFAR10/100 datasets that our proposed approach is significantly more accurate and consistent across model architectures than all prior work.
INTRODUCTION
The success of supervised machine learning is inherently dependent on the reliability of the labels in the training data. Most algorithms assume that the training labels are clean and can be trusted blindly. However, creating real-world data sets that do not suffer from substantial label noise is extremely challenging. Human annotators, especially on crowdsourced platforms, can be prone to making labeling mistakes. "Weakly-labeled" datasets (Xiao et al., 2015 Joulin et al, [016 Li et al., 7a) which collect labels through proxy variables or web scraping, are especially likely to mislabeled samples. In fact, even the celebrated MNIST (LeCun & Cortes 2005) and ImageNet (Deng et al., 2009) data sets famously contain multiple mislabeled images (see seelFigure [for suspicious examples detected by proposed method). In practice, it is often too costly perform manual curation or quality control to remove such examples on large datasets.
In this paper we develop framework that provides multiple ways to handle label noise. Our method exploits the recent paradigm of training deep neural networks via stochastic gradient de- scent (SGD) (Huang al.] 2019 He al. 2016). These overparameterized networks achieve zero training eтror even when labels are assigned randomly (Zhang et al. (017)) essentially mem- orizing the entire data set. Contrary
classical machine learning wisdom, subsequent empirical and theoretical analyses (Arpit et al. 2017 Allen-Zhu et al.) 2019 Arora et al. 2019) show that de- spite their capacity overfit, these models can achieve impressive generalization performance. The implicit regularization of SGD permits memorization only if it is necessary to classify particular training point correctly. We refer to this (latent) variable as the memorization need of sample.
Although prior works often attempt to detect label noise through various methods involving valida- tion loss on rotating hold-out sets (Chen al., 2019), training auxiliary neural networks (Xiao al.] 2015 Jiang et al.) 2017). or meta-learning (Ren et al. 2018), we argue that the model's training dynamics can be far more revealing. For example, data points with high memorization need will exhibit noticeably different loss trajectories than those with low memorization need Rigure 2}.
We capture these differences in loss trajectories by introducing a simple, informative training statis- tic: the Area Under the Loss Curve (AUL). As its name suggests, the AUL can be trivially computed for a whole training set or any single training sample. smooths the loss trajectory across all train- ing epochs, capturing the general learning behavior of the samples for which it is computed. We

---

Under review as a conference paper at ICLR 2020
MNIST: High AUL Images
ImageNet-100 High AUL Images
7
3
1 0
9
0
0
0
f
5 8
7 0
Boods Tanser
7
1
d
4
Twench Foot
Drilling Plarerm
Figure 1: Images from MNIST (left) and the first 100 classes ImageNet (right) with the largest Area Under the Loss Curve (AUL). The high-AUL images are mislabeled extremely ambiguous. MNIST AULS are from a LeNet model and ImageNet AUL. are from DenseNet-12 121 model.
observe that in the presence of label noise, the distribution of AUL scores tends to be bimodal and can be estimated with a I-dimensional, 2-component Gaussian mixture model (GMM). This pro- vides with three useful ways to manage label noise: a) the mixing coefficients naturally estimate the fraction of noisy samples in the training data set; b) the membership likelihood of the higher- mean Gaussian estimates the probability that sample is mislabeled; c) the clean-label probabilities can be used to weight samples order to mitigate the influence of label noise in training.
Despite its compelling simplicity (the AUL score and GMM fit require only few lines of Python), it is currently by far the most reliable and accurate estimator of label noise. We evaluate our method on standard benchmark datasets with 2 models of synthetic label uncertainty. We demonstrate em- pirically that our method well-approximates the overall percentage of noisy data and detects mis- labeled points with high recall across wide range of noise levels. Furthermore, we AUL to achieve state-of-the-ard results of 11.1% and 40.2% classification error with 40% class-dependent label noise on CIFARIO and CIFARI00, respectively. Finally, we show that the AUL statistic is consistent across network architectures and training runs.
2 RELATED WORK
Learning with Weak or Noisy Labels. One popular approach to mitigate the effects of label noise design more robust training objectives. /Zhang & Sabuneu (0018) propose novel loss function by generalizing the cross entropy loss. Others interpolate target labels with predicted labels (Reed)
2018) or modify the loss term with an estimated noise transition matrix (Sukhbaatar et al. 2015 Patrini et al. 2016)] Hendrycks et al., 2018). Li al. (201bb) train network on small set of trusted data and guide learning through knowledge distillation. Another line of work either reweights or filters training samples to discourage learning from mislabeled examples. Attempted metrics to select training data includes training loss (Han et al. 2018: Shen & Sanghavi clean samples (Ren et al., 2018 MentorNet (Jiang et al., (017) learns a netw-ketwork to output example weights. Malach & Shalev-Shwartz (201)) train two networks and identify examples with prediction discrepancies. Our work also mitigates the effects of mislabeled samples through sample weights; however, in contrast to most prior work we extract these weights from the training dynamics.
al. 2014 Ma al.
2019), test/validation loss (Chen et al. 2019), and gradient information
Analysis of Neural Network Training. Recently, Neyshabur et al. (019)) and Allen-Zhu et al. (2019) both bound the generalization error of overparameterized neural networks assuming uncor- rupted labels. Another line of work analyzes the training dynamics of SGD under various assump- tions on the loss and network architecture (Mandt et al., 2017 /Zou et al.) 2019 Li & Liang 2018). Several recent works attempt to explain the observation that deep networks can overfit to random training labels (Zhang et al. 2017). For example, AArit et al. (0117) measure the gradient norms across epochs and empirically show that the number of clean data points with high loss sensitivity decreases throughout training. AAroa et al.] (2019) prove generalization bounds and convergence rates of GD for two-layer ReLU networks
function of the training data and labels. Other recent papers prove that GD combined with either early stopping (Li et al. (019)) or regularization (Hu et al. 2019) generalizes robustly in the presence of label noise. These analyses provide both the-
2

---

Under review as a conference paper at ICLR 2020
Distribution of Lass During Training (ResNet-20, CIFARTO, 40% Mishabeled) Upoch (AUD-291)
Spoch (AUC-0.M)
Epoch (AUC-085)))
Epoch (AUC-088)
Epoch (AUC-131)
1.6
2.0
Misihelal
Closs
1.0
1.4
0.5
1.0
1.
1.0
0.0
0.0
0.0
Look
Lose
Lios
Lisa
Loo.
Distribution of AUL During Training Net-22,, CIFARIO, 40% Mislabeled)
Epoch (AUCC-074)
(AD2C-0.54)
(SUCAUC00
Epoch 225 (ACC-0055)
(AUC-0.94)
3.6
0,015
9,9100
0.0075
0.006
0.008
0.006
Clas
Midaasiee
Jiint
0.4
9.004
0.010
9,0050
0.004
0.2
0.005
0,0025
0.002
0.0
0.000
0,0000
0.000
0.000
1000
AUL
AU2
AUL
ANL
AUL
A comparison of training loss and training AUL proxies for identifying mislabeled samples. Top: Histograms of training sample loss over the course of training. The distributions for clean and mislabeled samples are very separable the beginning of training, but become less sep- arable towards the end of training. Bottom: Histograms of training sample AUL. The distributions for clean and mislabeled samples remain separable even after the network has converged.
Figure 2:
oretical and empirical evidence that deep networks follow two-stage training procedure: models first learn generalizable parameters on the clean data before memorizing the noisy samples.
THE LOss CURVE AND MEMORIZATION NEED
In this section demonstrate that training loss dynamics can be used identify data points with high memorization need. We generate classification datasets with various amounts of memorization need by changing the labels of subset of data points. We then introduce metric Area Under the I Loss Curve (AUL) to identify these mislabeled (i.e. high memorization need) points. The AUL metric exhibits several desirable properties: high detection accuracy, stability over the course of training, and invariance to model architecture.
Rate of loss reduction indicates memorization need. To illustrate the training dynamics of high memorization need data, we train ResNet-20 on the CIFAR10 dataset where 40% of the training labels have been replaced with random labels. Following standard practice Huang et al. (2017). we use data augmentation, which is why the training loss does not diminish to zero for all samples. The learning rate is initialized to 0.1 and reduced by a factor of 10 at epochs 150 and 225. Since there is no relation between the incorrect labels and their corresponding images, these data points by design have a high memorization need. In Figure (top) we plot histograms of the training loss at various stages of training for both the correctly and incorrectly labeled data points. We make several observations. First, after the network has converged (epoch 300, top right histogram), most data points (both correctly and incorrectly labeled) have low training loss. This result is consistent with the observations of Zhang et al. (2017), suggesting that the network is memorizing these incorrectly labeled points. More notably however are the loss curves earlier stages of training. At 75 and 150 epochs (top middle-left and top middle histograms) the training loss for mislabeled points is much higher than that of the correctly labeled points. fact, the incorrectly labeled points tend to have a larger loss than at the beginning of training (epoch 1, top left histogram). Consistent with Arpit al. (2017); Arora al. (2019); y Li et al. (2019), this suggests that although all points achieve low loss. samples with high memorization need require more training iterations achieve low loss.
Capturing rate of loss change through AUL. Given that loss dynamics indicative of the memorization need, our goal is to capture this signal with robust metric. One challenge is that loss curves for individual samples rather noisy result of stochastic optimization. In In/Figure 3 (left)
3

---

Under review as a conference paper at ICLR 2020
Example Loss Trajectory (ReaNet-20, CIFARIO)
Example AUL Trajectory (ReaNet-20, CIFARI0)
Clei
600
Mishbooed
4
AUL
400
2
200
50 100
150
200
250
300
Epoch
Epoch
Figure 3: Left: Loss trajectories epochs for single clean sample and single mislabeled example (randomly selected), On average, the clean sample has a lower loss than the mislabeled example. However, the loss decreases over time for both examples, and the trajectories and non-monotonic. Right: AUL trajectory for the same examples. Integrating loss over training time reduces trajectory noise, making the samples' different learning dynamics more separable. ResNet AUL
noisy
DenseNet AUL
Label
Labul
Clean
Cleon
0.010
Mislaboled
Mislaboled
Donalty
Depth
Depih
10
14
0.005
50
110
0.000
400
500
700
100
200
AUL
AUL
Figure 4: Distribution of AUL (CIFARIO, 40% label noise) computed on ResNets and DenseNets of different depth. The modes of the distribution are consistent across networks.
plot loss curves for example training points one clean and one mislabeled. The variance of the loss is quite high, and there are several training epochs where the mislabeled point achieves lower loss than the correctly labeled point. The second challenge is that training loss becomes less indicative of memorization need in later epochs. In Figure 2 we observe that all samples have low training loss by epoch 300. To assess the discriminating ability of the loss, we compute the Area under the ROC Curve (AUC) at various training stages. Although the final training loss still separates clean and mislabeled samples with an AUC 81%, it is far less separable than at earlier epochs. In other words, the signal in the training loss varies throughout and becomes weaker over time.
One simple way to improve the signal to noise ratio to integrate training loss over time. As seen Figure 3 the losses of correctly-labeled samples decay early during training while the mislabeled samples' losses do not decay until later epochs. Therefore, clean samples will in expectation have lower loss than mislabeled samples any given epoch. We formalize this intuition in metric which we refer to as Area Under the Loss Curve, or AUI. Formally, for a given sample (x,
AULx. (((o((x))))
where 7' is the total number of epochs, L (-..) :) is the loss function, and Φ; is the model at epoch i. In Figure (center, right) we observe that the AUL dynamics are far less noisy than loss dynamics. The clean sample achieves a lower AUL than the mislabeld sample at every epoch. Moreover, the curves remain separable even after the loss signal decays in later epochs. Figure 2] (bottom row) shows the AUL histograms of clean and mislabeled samples at several stages of training. The AUC score of the AUL 94% at epoch and remains this high through epoch 300. This contrasts with the loss, which reaches its best predictive accuracy at epoch 75 (AUC=0.91 and decreases thereafter. While AUL remains a separable metric throughout the whole training cycle (including learning rate drops), in practice find it sufficient to compute the AUL before the first scheduled learning rate drop.
Stability and consistency of AUL and training dynamics. Unlike test-timm sistttmme predictions, which vary with network architecture and even with network initialization, training dynamics relatively
4

---

Under review as a conference paper at ICLR 2020
Motic
AUL (Trais Point)
Matric-
Ave. Valid
Metric
Loo (Trais Point))
Lows (Valid, Puints)
RN-14
0.99 0.55 9.96 94 0, 99 0.95
0.93 0.90 0.90
0.64
0.44 0.92
1
0.44
9.81
0.8L
0:85 0.83
0,55 0:45 0,45 0,53 0.55 0.52
1.0
RN-12
0.99
9.992.0000 98. 0.99
0.9
01
0.92
64
0:81
0
0.80
0.55
0.54
0.5 0.51 0.56 0.54
0.9
RN-50
1698
92.0
0.90
0/93
0.
0.78
0.84
81
0.54
0.8
RN-11
1.58
0.99
0.59
0.91 10
98 9.98
2.91
0:76
0.9 0.91
0.81 [88 0.61
0: 75
0.81
0.5
0:45 0.5 0.52
0.42
0.45 0.46
DN-10
0.94
0.99
0.92
0.91
0.94 0.92
0.84 0.
0.7
0.85 0.79
0.81 0.78 0.79 0.75
1
0:K1 0.75
0.53 0.51 0,46
0.59
0.46
9.7
DN-40
0.98
0.59
0.90
0.94
0.0
0.65
1
0.95
0.85
0.65
0.85
0.81
0.81
0.55 0.56 0.5
0.45
0.59
0.59
0.6
DN-100
1:98
0.99
50 92.0
2.920
0.x 0
0.044.
0.5
8N-14
XN-32
50
RN RN-110
DN-10
DN-100
R0N 14
32
RN-10
DN-10
DN-40
DN-100
RX-14
RN-32
RN-50
RN-110
DN-19
DN 40
DN-100
RN-14
2N-32
9N-59
RN-110
DN-10
DN-40
DN-100
38
DN
XXX
28
30
Figure 5:
Pearson's correlation of AUL and other metrics across various network architectures
Res=R,,Net, DN=DenseNet) after 150 epochs. AUL produces a very consistent ranking of the training data. Training loss produces less consistent rankings. Furthermore, the loss on held-out data points (from the test set) is also much less correlated across networks than AUL.
consistent across networks. In Figure 4 we plot the AUL distribution after training ResNet and DenseNet models of various depth (up to the first learning rate drop). Each network is trained on the CIFARIO 10 dataset with standard data augmentation, where the same 40% of samples are mislabeled. The AUL distributions are remarkably similar for networks of different depths. The modes are more sharply peaked for deeper networks, yet the modes are around roughly the same AUL values. To further demonstrate the consistency of training dynamics, we compute the Pearson's correlation coefficient of the dataset's AUL scores computed between different network architectures (Figure 5] left). Most pairs of networks achieve a correlation of 0.98 or 0.99. These results suggest that the AUL. metric captures properties that are depesetdd rather than model-dependent.
Conversely, the testing predictions of different models are less correlated than training dynamics. In Figure 5] (middle left) we measure the inter-network correlation of loss validation set samples. We average each sample's loss over all training epochs which - similarly to AUL reduces the impact of noise from any one epoch. The averaged validation loss achieves correlations between 0.8 and 0.95. While this is still a high correlation, we note that it is not as consistent as AUL. Finally, we observe that integrating loss values over time improves the consistency of the metric. Figure (middle right) plots the inter-network correlation training loss at epoch 150 (the last epoch before the learning rate dropped). Most networks achieve a correlation between 0.79 and 0.85- roughly 10 percentage points less than training AUL. There is even less inter-network correlation for validation loss (Figure 5) right). At epoch 150 the correlation is on average 0.55, which is roughly 40 percentage points less than averaged validation loss. The loss of individual sample any given epoch is likely impacted by stochastic training noise, and integrating reduces this noise.
4 INCORPORATING AUL DYNAMICS INTO ROBUST TRAINING
Though AUL is relatively invariant to network architecture choices, its range depends on both the dataset and the training length. Next introduce simple algorithms that convert the relative AUL metric into interpretable probabilities for (a) estimating the amount of noise in a dataset, (b) identi- fying which samples are likely to be mislabeled, and (c) reweighting samples to reduce overfitting.
4.1 ESTIMATING NOISE AND IDENTIFYING MISLABELED SAMPLES
For x, a given input from the training dataset, let y be its ground-truth label (which is unobserved), and let y be the potentially incorrect observed label. We assume that the observed labels in the dataset follow some unknown noise model p(ÿ | y). One goal is to estimate the amount of label noise in the training dataset: i.e. p( y). A more challenging goal is identifying which samples are most likely to be mislabeled. More specifically,
want to estimate p(ū
x) for all x.
As mentioned in the previous section, AUL is a relative metric and so its values cannot be directly converted into probability p(ğ # y I x) that sample is mislabeled. Therefore, we instead estimate the conditional AUL distributions for clean and mislabeled samples, i.e. p(AULx.5 y y) and p( (UUlx.. y # ?) respectively. plu # y xU) is estimated from these conditional
5

---

Under review as a conference paper at ICLR 2020
Norse 010
20% Nouse (AUC-099 0.95)
40% Noire (AUC-0.94)
50% Nouse (AUC- 0.94)
80% Noise (KUC- 0..63 0.923
Clas
Mishaaled
0.015
200
4000
5000
200
000
4000
500
200
400
5900
200
500
AUL
Figure 6: AUL distribution on CIFARIO (ResNet-32) with varying amounts of mislabeled data. The AUL distribution is bimodal, with the two modes corresponding to clean and mislabeled samples.
distributions using Bayes' rule. Without knowing which samples are clean, estimating these two conditional distributions an unsupervised problem.
Approximating the AUL distribution with a Gaussian mixture. We observe in Figure that the AUL distribution p(AULx,5) is distinctly bimodal when label noise is present. The networks in this figure are ResNet-32 models trained on CIFAR1O, and the AUL is measured before the learning rate drop (epoch 150). When no noise is present (far left plot), the distribution is unimodal with a long tail of high AUL samples. Intuitively, the mode corresponds to the loss for "easy" or "average" examples while the tail corresponds to "hard" examples. For noisy datasets however, we observe that the separability of AUL creates a distribution with two distinct modes corresponding to p(AULx, y y) and p(AULx.j amount of noise increases (e.g. with 80% noise the distance between the two modes a quarter of the distance between 20% noise modes), they are always separable.
* 7). Though the two modes become closer as the
We identify these two modes by fitting the AUL distribution with a mixture two Gaussians. The low-mean Gaussian (blue) approximates the clean-conditional AUL distribution p(AULx.ū x y 7), while the high-mean Gaussian (orange) approximates the mislabeled distribution p( p(AULx.g y + 7). In practice the actual conditional distributions are not necessarily Gaussian, yet find that this simple model is nevertheless quite good for the applications in [Section 5) From this Gaussian mixture. it easy to estimate the label noise and mislabeled samples. Let ²x.ÿ be the probability that sample (x. V) belongs to the high-mean (i.e. mislabeled) Gaussian distribution. 2x,ÿ is the model's
estimate of ply + y xULx,)) and Fx,77 [xx]]] is the estimate of dataset noise.
4.2 IMPROVING TRAINING WITH CORRUPTED LABELS
A high-capacity model trained on a noisy dataset is in danger of overfitting, memorizing misla- beled samples may hurt the generalization of correctly labeled samples. Next we describe how to remedy this by using AUL probabilit estimates down- down-weight mislabeled samples during training.
Assume that we have trained a neural network and computed AULx, for every training sample (x,, V.). Since this network was trained on noisy data has likely learned features that are specific to the mislabeled examples and will not generalize well. Therefore we propose to train a new model where the samples are reweighed using the AUL probability estimates. More formally, let
P(ù, o y AULx, )) be the probability that sample is mislabeled based on the Gaussian mixture fit. We train a network from scratch where the loss for each minibatch M is computed as
M
C(M)
Σ
r.L (xi, W))
where (x,; Te) are the samples M.
5 RESULTS
We measure the efficacy of AUL dynamics at estimating dataset noise, identifying mislabeled sam- ples, and improving training with noisy data. In order quantitatively measure the performance,

---

Under review as a conference paper at ICLR 2020
Estimated Ami, Noise Uniform Noise Modic
Estimaated Ami. of Notis Flip Noisc Model
Mislabelee Sample Identificarion
Mislaheled Sample Mdeniiication Pair Flip Noise Model
Uniform Noise Model
1.0
0.5
1.0
Latinaedd Noire
0.8
0.4
PrecisiomPoeee Roaall
0.9
0.6
0.3
0.8
0.4
0.2
0.7
Metric
02
0.1
0.6
Rocel
0.0
0.0
0.5
0.0
0.5
1.0
0.0
0.2
0.4
0.00
0.25
0.50
0.75
1.00
0.0
0.2 0.4
Actual %6 Nosse
Actual Noise
Actual Noise
Actual
Figure 7: Identifying mislabeled data in noisy variants of CIFARIO using AUL from trained ResNet-32 models. Left: predicted amount of noise in training set (uniform noise model, far left; pair flip noise model, middle left). Right: prediction/recall curves for identifying mislabeled data (uniform, middle right; pair flip, far right).
Table 1: Test-error comparison of noisy-data training methods on datasets with corrupted train- ing labels. All methods train ResNet-32 the CNFFATO/CIFARID 100 datasets with varying amounts/types of label noise. (Uniform: mislabeled samples are assigned a random incorrect la- bel. Pair Flip: mislabeled samples are assigned a to class-specific incorrect label.)
Dattset
CIFARIO Unifoem
CIFARIOO Uniffrm
Noise Model
Uniform
Uniform 1140
Unitorm 0.50
Pair Flip 0.40
Unitormn
Unitoem
Caidorn 0.80
Pair flip
6 Mistabelad
0.20
0.50
0.20
0,40
0.60
0.40
Standad
23.2 + 0.2
40.3
= 0.5
57.1
7.3++0.5
40.3 + 0.2
50.3 0.3
02.0 0.7
7
01.1+007
61.8 +.6
Randou
16.5 0.7
22.8
1.6
35.8
5.6
2
43.0==0..1
+1.2
63.5 +.0
83.7 1.8
Read-Hard
2.44::0044
37.4= 0.0
0..0::0..5
6.88110.
37.2
68.6 0.3
58.9 0..9::00.5
703 6:4
80.8 1.6
5888.00
MemocNers
13.3
06,0
42.3
75.7 1..7411
Co-Teaching
0..2+002
13.6=02
18.3+002
80.
169
0.0
6..7
2:.
Diverged
23.6
Diverged
Diverped
INCV
1..2:002
1..0:10.5
::001:00:
AUL
0.2
0.2
16.0
0.3
34.21..4
39.9
50.6
70.5
40.2
Onclle
9.
9.7
1.
0.1
0..0
00.1 $1.
8..
corrupt the labels of the CIFARIO and CIFARI00 datasets according to two predefined noise mod- els. The uniform noise model assumes that discrepancies between true labels and observed labels are random. A given label can be changed to any other class label with equal probability. The pair flip noise model assumes that mislabeling mistakes depend on the true label. If an observed label incorrect, then it will belong to only one other possible class. More formally, if p is the percentage of mislabeled points, we have
(1
p)
(1
p)
y y
PUnifoem 7 y)
P
y *
Prup (T v)
=
p
y = (y 1)
0
o.w.
These noise models, which have been well-studied in prior work (Han et al. 201 8
Hendryeks et al., 2018), represent two extremes: uniform class-independent and pair flip is fully dlassddependent.
We first test the ability of AUL to estimate the amount of mislabeled data in the training set. We construct noisy versions of CIFARIO and CIFAR 100 by applying the two noise models with varying amounts of noise on 45,000 training examples. The uniform noise model is applied with 10%-90% of samples mislabeled, and the pair flip model is applied with 10% 40%. To compute AUL we train 32-layer ResNet (He et al. (016)) on each corrupted dataset. The networks are trained for 150 epochs using SGD with a constant learning rate of 0.1 and a batch size of 64. This corresponds to the standard training hyperparameters for ResNets before the first learning rate drop (which typically occurs at epoch 151). We identify potentially mislabeled samples by fitting the AUL distribution with a mixture of 2 Gaussians, as described in Section 4.1 Results are averaged over trials.
Noise estimates. Figure 7 displays the estimated noise for the corrupted CIFAR1O datasets (uni- form noise- left, pair flip- middle left). We observe that AUL tends to overestimate high in the low- and high-noise settings (e.g. below 0.2 and above 0.8 in the uniform noise setting). Neverthe- less, the AUL-based estimates are able to closely recover the true amount of noise in most settings. Analogous plots for the corrupted CIFAR 100 datasets can be found in Appendix A
Identifying mislabeled samples. After fitting the Gaussian mixture to the AUL distribution, we classify examples as mislabeled if pūū + y AUlx,)) > 0.5. Figure 7 displays the precision
7

---

Under review as a conference paper at ICLR 2020
and recall of mislabeled sample identification on the corrupted CIFARIO datasets (uniform noise- middle right, pair flip- right). (See Appendix A for CIFÁR 100 plots.) The predictions achieve a recall of roughly 90% on most noise models, suggesting that the AUL distribution model identifies most mislabeled points. The AUL. estimates have the lowest precision in the small noise regime- around 60% to 70%. However, this corresponds to relatively few misclassifications since the number of predicted mislabeled samples is small in this setting. In the other noise settings the model achieves precision around 90%. It is worth emphasizing that the AUL model achieves this high performance without any supervision or prior knowledge about the noise model.
Robust training with corrupted labels. Next we evaluate the performance of models trained on corrupted datasets with the reweighting scheme proposed in infSection 4.2 In all experiments, we train 32-layer ResNets on the CIFARIO and CIFAR 100 variants described above. We assume that we do not have access to a clean validation set, and so all models are trained without early stopping.
To set up a sensible lower bound for performance, we train network that follows Standard training procedure (i.e. no method for combating mislabeled samples). The Oracle baseline- which is only trained correctly-labeled portion of data represents a sensible upper bound. We also compare to a Random (Ren et al., 2018) baseline, where each sample is assigned a random weight. The weights are drawn from rectified normal distribution Ren et al. (2018) and are drawn at every epoch. Reed-Hard (Reed et al. 2014) and D2L (Ma et al. 2018) interpolate the one-hot target label with the predicted label. MentorNet-PD (Jiang et al. 201 7 learns a weighting scheme for training examples using a L.TT.. Co-teaching (Han et al. 2018) simultaneously train two networks that inform each other about which training examples to keep. INCV (Chen et al.) 2019) is an extension of co-teaching that first filters training data through iterative cross-validation.
We train all networks on 45,000 training examples. We compute the AUL of these samples by training a network for 150 epochs with a learning rate of 0.1. AULs are converted to sample weights using the Gaussian mixture described in Section 4.2, The standard, oracle, and reweighted networks are trained using the procedure He et al. (216)) 300 epochs of SGD with an initial learning rate of 0.1 that is dropped by a factor of 10 at epochs 150 and 225. All other methods are trained using hyperparameterr that are defined in their publicly available implementations.
Table Jjisplays the test set performance for the methods on the corrupted CIFARIO and CIFAR100 datasets. We observe several trends. First, there is large performance gap between the Oracle and Standard baselines, suggesting that the standard networks overfit to the mislabeled examples. On CIFARIO the gap is 14% with 20% uniform noise, and 62% with 80% noise. Most of the baseline methods reduce this performance gap. Our proposed reweighting scheme (AUL) achieves the best error all dataset variants, especially on the high uniform noise and pair flip datasets. Most notably, AUL. is able to recover the oracle performance on low-noise CIFAR10 and surpass oracle performance on low-noise CIFAR 100. We hypothesize that the AUL reweighting not only identifies mislabeled examples but also reduces the impact of difficult or ambiguous examples.
6 DISCUSSION AND CONCLUSION
In this paper we introduce the AUL statistic, which reveals differences in training loss dynamics to reliably identify noisy labels. We observe that this statistic is far more stable across architec- tures than for example the validation loss. This finding is compatible with the commonly made observation that even networks with different initializations vary noticeably in their predictions a welcome side-effect for model ensembling (Caruana et al. 2004), but a nuisance when detecting outliers. While the AUL. score should be highly effective to increase classifier robustness and ana- lyze or improve the label quality of data sets, we believe that the its ability to detect samples with high memory need may open applications far beyond the scope of this paper. As future work, we are planning to investigate the use of the AUL score to estimate sample "hardness" for the use in curriculum learning (Bengio et al., 2009) and to detect mis- piieiieeisssons on unsupervised data in the context of semi-supervised learning (Chapelle et al. 2009).
Due incomplete release, we cannot train MentorNet Jiang al. 2017 in few noise settings.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li. and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. In NeurIPS, 2019.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparametterzze two-layer neural networks. In ICML, 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio. Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look at memorization deep networks. ICMI, pp. 233-242, 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, ACM, 2009.
Rich Caruana, Alexandru Niculescu-Mizzi Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. Proceedings of the twenty-firs international conference on Machine learn- ing, pp. 18. ACM, 2004.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2b0o) reviews]. IEEE Transactions on Neural Networks, 20(3):542-54,, 2009.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kuntai Li, and Li Fei-Fei. Imagenet: hierarchical image database. CVPR, pp. 248-255, 2009.
large-scale
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. NeurIPS, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. In NeurlPS, 2018.
Wei Hu, Zhiyuan Li, and Dingli Yu. Understanding generalization of deep neural networks trained with noisy labels. In NeurlPS, 2019.
G. Huang, Z. Liu, G. Pleiss, L. Van Der Maaten, and K.Q. Weinberger. Convolutional networks with dense connectivity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. doi: N0.1109/TPAMI..0199 18284.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks, In CVPR, pp. 4700 4700-4708, 2017.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentoret: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, 2017.
Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual fea- tures from large weakly supervised data. In ECCV, pp. 67 84, 2016.
Yann LeCun and Corinna Cortes. The mnist database handwritten digits. 2005.
Mingchen Li. Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust label noise for overparameterized neural networks, 2019. URL https:// arxi v. www.orggaaa abs / 903. 680
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data, 2017a.
http://wwww.....aa/////aaaa/ abs/1708..
62

---

Under review as a conference paper at ICLR 2020
Yuanzhi Li and Yingyu Liang. Learning overparameteriize neural networks via stochastic gradient descent on structured data. In NeurlPS, 2018.
Yuncheng Li. Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, and Jia Li. Learning from noisy labels with distillation. ICCV, pp. 1928-1936, 2017b.
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewickrema, and James Bailey. Dimensionality-drieen learning with noisy labels. ICML, 2018.
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In NIPS, 2017.
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent approxi- mate bayesian inference. The Journal of Machine Learning Research, 18:4873-4907, 2017.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization neural networks. In ICLR, 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making neural networks robust to label noise: loss correction approach. ArXiv, bbss/160900668,, 2016.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and An- drew Rabinovich. Training deep neural networks noisy labels with bootstrapping. CoRR, abs/1412.6596, 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. In ICMI., 2019.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. In ICLR Workshop, 2015.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. In ICLR, 2019.
10

---

Under review as a conference paper at ICLR 2020
A ADDITIONAL RESULTS
A. AVERAGE LOSS AND AUL TRAJECTORIES
To further visualize the difference between the loss and AUL metrics, we plot the distribution of loss and AUL as a function of training epoch in Figure 8, The losses and AULs are computed using ResNet-20 model trained on CIFARIO with 40% uniform noise. We see that the loss distributions for clean and mislabeled data points become less separable as training increases. Conversely, AUL remains separable throughout the course of training.
Avg. Loss Trajectory (ResNet-20, CIFARIO, 40% Mislabeled)
Avg. AUL Trajectory (ResNet-20, CIFARIO, 40% Mislabeled)
Clean
800
Mislaheled
3
600
2
400
200
0
100
Epoch
Epach
Figure 8: Distribution of loss (left) and AUL (right) over the course network training. Shaded
region represents 1 standard deviation.
A.2 NOISE ESTIMATES AND MISLABELED SAMPLE IDENTIFICATION FOR CIFAR
In this section we replicated the experiments from Section 5 on the CIFARIO dataset. Figure (left, middle left) displays the noise estimation for the CIFAR 100 dataset the uniform and pair flip models, respectively. We do note that the estimates for the pair flip noise models tend to be larger than the actual amount of noise. Nevertheless, we find that similarly to CIFARIO the model is able to accurately recover the amount of noise in the dataset. [Figure 9 (middle right, right) displays the identification precision/recall for the CIFAR100 dataset. We again find that the recall in the upper 90%. The precision tends to be lower with less noise, but rises above 80% when there more than 30% noise.
Mislabolod Sump le Identification Unitorm Noise Model
Mislabeled Sumple Identification Pair Top Noise Modol
Estimated Ané.
Estimttdd Flip Model
Unifanm
1.0
0.5
Batimist 54 Naike
0.8
0.4
ProcisionRReall
0.31
0.6
0.3
0.4
0.2
Matri Pasision
0.4
0.1
0.50
1.00
0.0
0.4
Actual 5 Nokse
Actual % Nouse
Astual % Noise
Actual % Noso
Identifying mislabeled data in noisy variants of CIFAR100 using AUL from trained ResNet-32 models. Left: predicted amount of noise in training set (uniform noise model, far left; pair flip noise model, middle left). Right: prediction/recall curves for identifying mislabeled data (uniform, middle right; pair flip, far right).
Figure 9:
11

---

Under review as a conference paper at ICLR 2020
A.3 ADDITIONAL HIGH-AUL IMAGES
WELL 3E0
tsteal
WEF BOR HETTOM
Hamoow
Toy Terner
Manetzrrem
Home Theater
Spotted Sulanender
Altespinscher
Bunow
Born
Lampehade
Toy
Tenoor
Trish Waner Spaniel
Afficaa Greys
Staewinder
Suntal
Microwave
+240
Deimetic
Holo Tiee
Figure 10: Additional images from the first 100 classes of ImageNet with large AUL when trained on a DenseNet-122 model.
12


Reviews:
[{"review_id": "r1e5gNqptr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method of detecting label noise through calculating the Area Under the Loss statistic, which is based on the learning trajectory.  \n\nI believe this paper is NOT the first one to point out the different training behavior of clean and noisy samples in the label noise problem. While the author mentioned that Shen & Sanghavi used the training losses for selecting data, Shen & Sanghavi also observed the training loss for good and bad samples are different across time. Therefore, it would be good to give correct credit to previous work, and not over-emphasizing the contribution. \n\nPlease proofread the paper, and correct the typos. For example, on page 3, there are 2 typos in the last three lines in the paragraph “rate of loss reduction indicates memorization need”. \n\nCan you explain why you normalize the loss of each batch by c ? If z_i s are all small within a batch, this loss function does not reweight the bad samples in the correct way.\n\nCan you explain why you use Pearson’s correlation instead of Spearman’s correlation to evaluate the consistency of the ranking? This sounds surprising to me. \n\nAlso, the cited paper “Understanding generalization of deep neural networks trained with noisy labels. “ by Hu et al. is not in NeurIPS 2019. Please correct this citation error if possible. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eybeV2YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed methodologies to identify mislabeled training examples. Specifically, the authors proposed to examine the training loss of each example to identify examples with high training losses. To mitigate the randomness in the training loss, the authors proposed an AUL (area under the loss) metric, which is less noisy that directly examining the loss. The authors proposed to use Gaussian mixture to approximate the AUL distribution, and then estimate the probability of mislabeling. The authors proposed to downweight training examples with high likelihood of being mislabeled. The proposal is examined in numerical studies.\n\nWhile I agree the method has a good potential to be significant, in its current form, I have several concerns that prevent me from recommending acceptance.\n1. How can we be sure that examples with large training loss are mislabeled examples, rather than examples that are inherently difficult to classify? For example, if the dataset consists of 99% bright images and 1% dark images, then it could be possible that dark image training examples have a higher loss. This question becomes more relevant when the model does not have sufficient complexity, or does not have the correct form of complexity.\n2. If we cannot distinguish mislabeled examples with examples that are hard to classify, then it seems to me that a simple downweighting based on AUL could potential lead to the opposite outcome. Intuitively, we should upweight those difficult to classify examples to improve generalizability.\n3. It is somewhat surprising to me that the AUL distribution is a nicely looking bimodal shape. Could it be due to the fact that the wrong labels are randomly generated with equal probabilities from other classes? What it we have a different error distribution, e.g., the label has p probability of being correct, and (1-p) probability of being k, where k is a constant?\n4. Given the above concerns, I feel the numerical studies are insufficient. The authors may consider 1) including more noise distributions, 2) presenting the AUL distributions to check whether they are truly bimodal, and 3) trying more datasets and more model architectures.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1g0BwJdKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. This work suggests a surprisingly elegant approach to identify data with corrupted labels. The proposed AUL (area under the loss curve) makes intuitive sense, and exploiting the fact that noisy examples are hard to learn at the early stage of training, leading to higher AULs than clean examples.\n2. The paper is well-written, with plenty of visualization to aid the understanding.\n3. Experimental results suggest advantage of the proposal. Evaluation seem to suggest the using AULs is more robust against large corruptions compared with SOTA.\n\nCons:\n1. The biggest concern is the generalizability of AULs to datasets that are more challenging than CIFAR10 and CIFAR100. This has been the concern not only for this work, but also for this general research area. For instance, on Tiny-ImageNet dataset (e.g., used in SELFIE: Refurbishing Unclean Samples for Robust Deep Learning (ICML19), which could btw be used in the benchmark), a clean dataset only gives ~55% test error, compared with about ~10% on CIFAR10 and ~30% on CIFAR100. The robustness of AULs needs to be tested on those harder problems as well. \n2. AULs are based on the assumption of a clear separation of two types of examples -- clean and noisy. It ignores the difficulty of training examples. This raises the question of where to place hard-clean examples between those two modes.\n3. It usually helps to include a real-world noisy datasets where its noisy corruption is unknown. One would be curious to know the noisy level estimate.\n\nQuestions:\n1. It seems from the experiments that MoGs are fit per training example in order to compute weights for the cost?\n2. The computational cost of AUL-based training is not mentioned. Is it 2x, 3x, considering there are distinct stages of training before and after AUL estimation?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new, stable metric, called Area Under Loss curve (AUL) to recognize mislabeled samples in a dataset due to the different behavior of their loss function over time. The paper build on earlier observations (e.g. by Shen & Sanghavi) to propose this new metric as a concrete solution to the mislabeling problem. 

Although the reviewers remarked that this is an interesting approach for a relevant problems, they expressed several concerns regarding this paper. Two of them are whether the hardness of a sample would also result in high AUL scores, and another whether the results hold up under realistic mislabelings, rather than artificial label swapping / replacing. The authors did anecdotally suggest that neither of these effects has a major impact on the results. Still, I think a precise analysis of these effects would be critically important to have in the paper. Especially since there might be a complex interaction between the 'hardness' of samples and mislabelings (an MNIST 1 that looks like a 7 might be sooner mislabeled than a 1 that doesn't look like a 7). The authors show some examples of 'real' mislabeled sentences recognized by the model but it is still unclear whether downweighting these helped final test set performance in this case. 

Because of these issues, I cannot recommend acceptance of the paper in its current state. However, based on the identified relevance of the problem tackled and the identified potential for significant impact I do think this could be a great paper in a next iteration. 
Decision: Reject