Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
Authors: Xiaobo Xia
Tongliang Liu
Bo Han
Mingming Gong
Jun Yu
Gang Niu
Masashi Sugiyama
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Learning with noisy labels
Sample selection
Uncertainty

Abstract: In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled data during training. However, losses are generated on-the-ﬂy based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks learn patterns ﬁrst; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the conﬁdence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection. In this way, we also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem to be mislabeled at ﬁrst glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.

Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
Anonymous Author(s)
Affiliation Address email
Abstract
In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. There are actually possibilities of large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks "learn patterns first"; (b) it belongs this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the confidence intervals of losses derived from distribution-fre concentration inequalities, but not losses themselves, are used for sample selection. In this way, also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As result, we can better explore underrepresented data that are correctly labeled but seem be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.
2
4
6
underrepresented group of data and has been selected yet. In
8
10
11
12
13
14
15
18
Introduction
17
Learning with noisy labels is one of the most challenging problems in weakly-supervised learning, since noisy labels are ubiquitous in the real world [36, 65, 40, 1, 61]. For instance, both crowdsourcing 20 and web crawling yield large numbers noisy labels everyday [12]. Noisy labels can severely impair
18
21 the performance of deep neural networks with strong memorization capacities [67, 69, 42, 30].
22 To reduce the influence noisy labels, a lot of approaches have been recently proposed [38, 29, 31, 68, 71, 55, 56, 46, 33, 25, 34, 47, 60, 49, 19, 17, 14]. They can be generally divided into two main categories. The first one is to estimate the noise transition matrix [41, 44, 15, 11]. which denotes the probabilities that clean labels flip into noisy labels. However, the noise transition matrix is hard to be
23
24
26 estimated accurately, especially when the number of classes is large [65]. The second approach is sample selection, which is our our focus in this paper. This approach is based selecting possibly clean examples from a mini-batch for training [12, 62, 50, 65, 23, 50, 51]. Intuitively, if we can exploit less noisy data for network parameter updates, the network will be more robust.
28
30 major question sample selection is what criteria can be used to select possibly clean examples.
31 At the present stage, the selection based on the small-loss criteria is the most common method, and effective many cireumstances [12, 16, 65, 52, 62). Specifically, since deep networks learn patterns first [2]. they would first memorize training data of clean labels and
32 has been verified
34 then those of noisy labels with the assumption that clean labels are the majority a noisy class. Small-loss examples thus be regarded as clean examples with high probability. Therefore, in
36 each iteration, prior methods [12, 52] select the small-loss examples based on the predictions of the current network for robust training.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Due
356
Caan Plart
322
3 3.00
622
90 Pelaaced Diu Nelo Frea Inboorece
Easch
Figure 1: Illustrations of uncertainty of losses, Experiments are conducted on the imbalanced noisy MNIST dataset. Left: uncertainty of small-loss examples. At the beginning of training (Epochs I and 2). due to the instability of the current prediction, the network gives a larger loss to the clean example and does not select for updates. If we consider the mean of training losses at different epochs, the clean example can be equipped with a smaller loss and then selected for updates, Right: uncertainty of large-loss examples. Since the deep network learns easy examples the beginning of training, it gives large loss to clean imbalanced data with non-dominant labels, which causes such data unable to be selected and severely influence generalization.
38
However, such a selection procedure debatable, since arguably does not consider uncertainty in selection. The uncertainty comes from two aspects, First, this procedure has uncertainty about small-los: examples. Specifically, the procedure uses limited time intervals and only exploits the losses provided by the current predictions. For this reason, the estimation for the noisy class posterior unstable [63], which causes the network predictions to be equally unstable. thus takes huge risks to only use losses provided by the current predictions (Figure 1, left). Once wrong selection is made, the inferiority of accumulated errors will arise [65]. Second, this procedure has uncertainty about
39
40
41
42
43
44
45 large-loss examples. To be specific, deep networks learn easy examples at the beginning of training, but ignore some clean examples with large losses. Nevertheless, such examples are always critical for generalization. For instance, when learning with imbalanced data, distinguishing the examples with non-dominant labels are more pivotal during training [35]. Deep networks often give large losses to such examples (Figure I, right). Therefore, when learning under the realistic scenes, e.g., learning with noisy imbalanced data, prior sample selection methods cannot address such an issue well.
46
47
48
50
51 To relieve the above issues, we study the uncertainty of losses in the sample selection procedure to combat noisy labels. To reduce the uncertainty of small-loss examples, we extend time intervals and utilize the mean of training losses at different training iterations. In consideration of the bad influence
52
53
54 of mislabeled data on training losses, we build two robust mean estimators from the perspectives of seft truncation and hard truncation w.r.t. the truncation level, respectively. Soft truncation makes the mean estimation more robust by holistically changing the behavior of losses. Hard truncation makes the mean estimation more robust by locally removing outliers from losses. To reduce the uncertainty of losge-loss examples, we encourage networks to pick the sample that has not been selected in a conservative way. Furthermore, to address the two issues simultaneously, we derive concentration inequalities [5] for robust mean estimation and further employ statistical confidence bounds [3] to consider the number of times example was selected during training.
56
58
60
61
62
The study of uncertainty of losses in learning with noisy labels can be justified as follows. statistical learning, it is known that uncertainty is related to the quality of data [48]. Philosophically, we need variety decrease for selected data and variety search for unselected data, which share a common objective, i.e., reduce the uncertainty of data to improve generalization [37]. This is our original intention, since noisy labels could bring more uncertainty because of the low quality of noisy data. Nevertheless, due to the harm of noisy labels for generalization, between variety decrease and search. Technically, our method is specially designed for handling noisy labels, which robustly uses network predictions and conservatively seeks less selected examples meanwhile to reduce the uncertainty of losses and then generalize well.
63
64
66
need to strike good balance
88
89
70
71
Before delving into details, we clearly emphasize our contributions in two folds. First, we reveal prior sample selection criteria in learning with noisy labels have some potential weaknesses and discuss them in detail. The new selection criteria are then proposed with detailed theoretical analyses. Second,
72
73
74 we experimentally validate the proposed method on both synthetic noisy balanced/imbalancee datasets and real-world noisy datasets, on which it achieves superior robustness compared with the state- of-the-art methods in learning with noisy labels. The rest of the paper is organized as follows. In Section 2, we propose robust learning paradigm step by step. Experimental results are discussed
76
78 in Section 3. The conclusion is given in Section 4.
2

---

79 2 Method
In this section, we first introduce the problem setting and some background (Section 2.1). Then we 81 discuss how to exploit training losses at different iterations (Section 2.2). Finally, introduce the 82 proposed method, which exploits training losses at different iterations more robustly and encourages networks to pick the sample that is less selected but could be correctly labeled (Section 2.3).
80
84 2.1
Preliminaries
Let X' and y be the input and output spaces, Consider a k-class classification problem, i.e., y = [k], where [k:] = [I,...,k) In learning with noisy labels, the training data are all sampled from corrupted distribution X' V. We are given sample with noisy labels, i.c., S = {(x, ()). where y is the noisy label. The aim is to learn a robust classifier that could assign clean labels to test data by only exploiting training sample with noisy labels.
85
86
88
89
Let f : + R be the classifier with learnable parameters w. At the i-th iteration during training. 91 the parameters of the classifier I can be denoted w,. Let f: R x y + IR be surrogate loss
90
92 function for k-class classification. We exploit the sofimax cross entropy loss this paper. Given an 93 arbitrary training example (x, :)., at the i-th iteration, we can obtain loss &, i.e., l; [([[[;;;) ū)-
94 Hence, until the 1-th iteration, we can obtain training loss set [, about the example (x, 9). i.e., L (e).
96
In this paper. we assume that the training losses in Le conform to Markov process, which is to represent a changing system under the assumption that future states only depend on the current state (the Markov property) [43]. More specifically, at the i-th iteration, if we exploit an optimization algorithm for parameter updates (e.g., the stochastic gradient descent algorithm [4]) and omit other
97
96
99
dependencies (c.g., 5), we will have P(w. w........ Wo) = P(w,/w;-- 1), which means that the future state the classifier ff only depends on the current state. Furthermore, given a training example and the parameters of the classifier f. we can determine the loss of the training example as discussed.
100
101
102
103
Therefore, the training losses in L, will also conform to a Markov process.
104
2.2
Extended Time Intervals
105
As limited time interval cannot address the instability issue of the estimation for the noisy class posterior well [42], we extend time intervals and exploit the training losses at different training iterations for sample selection. One straightforward idea is use the mean of training losses at different training iterations. Hence, the selection criterion could be
106
107
108
109
is intuitive and reasonable to use such selection criterion for sample selection, since the operation of averaging can mitigate the risks caused by the unstable estimation for the noisy class posterior, following better generalization. Nevertheless, such a method could arguably achieve suboptimal classification performance for learning with noisy labels. The main reason is that, due to the great harm of mislabeled data, part of training losses are with too large uncertainty and could be seen as outliers. Therefore, it could be biased to use the mean of training losses consisting of such outliers [10), which further influences sample selection. More evaluations for our claims are provided in Section 3.
110
111
113
115
116
117
2.3
Robust Mean Estimation and Conservative Search
We extend time intervals and meanwhile exploit the training losses at different training iterations more robustly. Specifically, we build two robust mean estimators from the perspectives of soft truncation and hard truncation [7]. Note that for specific tasks, it is feasible to decide the types of robust mean estimation with statistical tests based on some assumptions
119
121
[8]. We leave the analysis as future work. Two distribution-free robust mean estimators are introduced follows.
122
123 Soft truncation. We extend classical M-estimator from [7] and exploit the widest possible choice of the influence function. More specifically, give a random variable X, let us consider non-decreasing
3

---

125 influence function v ; R + IR such that
(XX) = log(1 1 + X X2
(2)
126
The choice of d is inspired by the Taylor expansion of the exponential function, which can make the estimation results more robust by reducing the side effect of extremum holistically. The illustration 128 for this influence function is provided in Appendix A.I. For our task, given the observations on training losses, i... Lt ......,e),, we estimate the mean robustly follows:
((G)).
We term the above robust mean estimator (3) the soft estimator.
131
Hard truncation. We propose a new robust mean estimator based on hard truncation. Specifically, given the observations on training losses LT, we first exploit the K-nearest neighhor (KNN) algorithm [27] to remove some underlying outliers L,. The number of outliers is denoted by (o < <t,, which
132
133
134
can be adaptively determined as discussed in [70]. Note that we can also employ other algorithms,
135
g.g principal component analysis [45] and the local outlier factor [6], identify underlying outliers
in L. The main reason we employ KNN is because of its relatively low computation costs [70].
137
The truncated loss observations on training losses are denoted by LE-,,- We then utilize L.-,, for the mean estimation. As the potential outliers are removed with high probability, the robustness of the estimation results will be enhanced. We denote such an estimated mean as p.g. We have
138
139
jhh
Σ 4.
(4)
BELL-
140
The corresponding estimator (4) is termed the hard estimator.
141
We derive concentration inequalities for the soft and hard estimators respectively. The search strategy
142
for less selected examples and overall selection criterion are then provided. Note that we do need
143
to explicitly quantify the mean of training losses. We only need to sort the training examples based
144
on the proposed selection criterion and then use the selected examples for robust training.
Theorem 1.
Let Zn = {21,'- 2n} 1 be observation set with mean μız and variance 0². By exploiting the necddccrgg influence function (2) = llog(1 + 2 + 2² /2). For any c > 0, we have
146
o ( + logse 14
Σ 1=1
(2z)
n²
(5)
n ṭ²
147
with probability at least 1 2c.
148
Proof can be found in Appendix A.I.
149
Theorem 2. Let Zn {21, ...,, 2n3 be (not necessarily time homogeneous) Markov chain with
150
mean Hs, taking values in a Polish state space A1 X An, and with minimal mixing time Tmin- The truncated set with hard truncation is denoted by Zn with n. n. If [/2 upper bounded by Z.
151
152
For any > 0 and >> 0, we have
1
Σ
Mg S
2 2mmn log
2 - + 2Tmin log 62
22n
2n
(6)
61
153
with probability at least
E2.
Proof can be found in Appendix A.2. For our task, let the training loss be upper-bounded by L. The value of L can be determined easily by training networks on noisy datasets and observing the loss distribution [1].
155
157
Conservative search and selection criteria. this paper, we will use the concentration inequalities (5) and (6) to present conservative search and the overall sample selection criterion. Specifically, we exploit their lower boundy and consider the selected number examples during training. The selection of the examples that are less selected is encouraged.
159

---

Algorithm CNLCU Algorithm.
1: Input 01 and 82, learning rate 7/, fixed T, epoch 75 and max, iteration imaxi
for -1,2..., T do
2: Shuffle training dataset 5;
for /. ,,..., m do
3: Fetch mini-batch S from S:
4: Obtain S = arg minl (())) (01,,
I/ calculated with Eq. (7) or Eq. (8)
5: Obtain S2 arg m (02,
I/ calculated with Eq. (7) Eq. (8)
6: Update 01 01 1)Ve(0,, S2);
7: Update B2 = 02 (1)
end
I Update R(T) = min
To T.
end
9: Output 0j and Oz.
Denote the number of times one example was selected by mi(n, S 1). Let circumstance with soft truncation, the selection criterion is
For the 1 21'
162
(² (2g(
= ja
f4g
163
Ej
E2 22
for the situation with hard truncation, by rewriting (6), the selection criterion
Gx P1 2 2T/2 + 22t))
log(4t)
(8)
(1-
n2e
164
Note that we directly replace with ny. If an example is rarely selected during training, n, will be far less than n, which causes the lower bounds to change drastically. Hence, we do not use the mean of all training losses, but use the mean of training losses in fixed-length time intervals. More details about this can be checked in Section 3.
185
186
167
168
For the selection criteria (7) and (8), we can that they consist of two terms and have one with a minus sign. The first term in Eq. (7) (or Eq. (8) is to reduce the uncertainty of small-loss examples, where we robust mean estimation
169
170
training losses. The second term, i.e., the statistical confidence bound, is to encourage the network to choose the less selected examples (with small n2). The two terms are constraining and balanced with 0² Twin To avoid introducing strong assumptions on the underlying distribution of losses [8]. we tune a and Tmin with noisy validation set. For the mislabeled data, although the model has high uncertainties on them (i.e., small n;) and tends to pick them, the overfitting to the mislabeled data is harmful. Also, the mislabeled data and clean data are rather hard to distinguish in some cases as discussed. Thus, we should search underlying clean data in conservative way. In this paper, we initialize and Tiin with small values. This way can reduce the adverse effects of mislabeled data and meanwhile select the clean examples with large losses, which helps generalize. More evaluations will be presented in Section 3.
171
172
173
174
175
176
178
180
The overall procedure of the proposed method, which combats noisy labels by concerning uncertainty (CNLCU), is provided in Algorithm 1. CNLCU works in a mini-batch manner since all deep learning training methods are based on stochastic gradient descent. Following [12], we exploit two networks with parameters f and 0g respectively to teach each other. Specifically, when a
181
182
183
5 is formed (Step 3), we let two networks select small proportion of examples in this mini-batch with Eq. (7) or (8) (Step 4 and Step 5). The number of instances is controlled by the function R(T), and two networks only select R(T") percentage of examples out of the mini-batch. The value of R(T) should be larger at the beginning of training, and be smaller when the number of epochs goes large, which can make better use of memorization effects of deep networks [12] for sample selection. Then, the selected instances are fed into its peer network for parameter updates (Step 6 and Step 7).
194
185
186
188
189
Experiments
180
In this section, we evaluate the robustness of our proposed method to noisy labels with comprehensive experiments on the synthetic balanced noisy datasets (Section 3. 1), synthetic imbalanced noisy datasets (Section 3.2), and real-world noisy dataset (Section 3.3).
192
5

---

194
3.1
Experiments on Synthetic Balanced Noisy Datasets
185
Datasets. We verify the effectiveness of our method on the manually corrupted version of the following datasets: MNIST [22], F-MNIST [58], CIFAR-IO [21], and CIFAR-100 [21], because these datasets are popularly used for the evaluation of learning with noisy labels in the literature
196
197
198
[12, 65, 54, 23]. The four datasets are baass-balanced. The important statisties of the used synthetic
199
datasets are summarized in Appendix B.1.
200
Generating noisy labels. We consider broad types of label noise: (1). Symmetric noise (abbreviated
201
as Sym.) [53, 31, 26]. (2) Asymmetric noise (abbreviated as Asym.) [32, 57, 52]. (3) Pairflip noise
202
(abbreviated as Pair.) [12, 65, 71]. (4). Tridiagonal noise (abbreviated as Trid.) [68]. (5). Instance
203
noise (abbreviated as Ins.) [9, 56]. The noise rate is set to 20% and 40% to ensure clean labels are
204
diagonally dominant [32]. More details about above noise are provided in Appendix B.I. We leave
205
out 10% of noisy training examples as a validation set.
Baselines. We compare the proposed method (Algorithm 1) with following methods which focus on sample selection, and implement all methods with default parameters by PyTorch, and conduct all the experiments on NVIDIA Titan Xp GPUs. (1). S2E [62]. which properly controls the sample selection process so that deep networks can better benefit from the memorization effects. (2). MentorNet [16], which learns curriculum to filter out noisy data. We use self-paced MentorNet in this paper. (3). Co-teaching [12), which trains two networks simultaneously and cross-updates parameters of peer networks. (4). SIGUA [13], which exploits stochastic integrated gradient underweighted ascent to handle noisy labels. We use self-reaching SIGUA in this paper. (5). JoCor [52], which reduces the diversity of networks improve robustness. Other types of baselines such as adding regularization are provided in Appendix B.2. Note that we do not compare the proposed method with some state- of-the-art methods, e.g., SELF [39] and DivideMix [24].
207
209
210
211
212
213
214
215
216
because their proposed methods are aggregations of multiple techniques. We mainly focus on sample selectionin in learning with noisy labels. Therefore, the comparison is not fair. Here, we term our methods with soft truncation and hard truncation as CNLCU-S and CNLCU-H respectively.
217
218
219
Network structure and optimizer. For MNIST, F-MNIST, and CIFAR-10, 10, we use 9-layer CNN structure from [12]. Due to the limited space, the experimental details on CIFAR-1O0 are provided in Appendix B.3. All network structures we used here are standard test beds for weakly-supervised learning. For all experiments, the Adam optimizer [20] (momentum=0.9) is used with an initial learning rate of 0.001, and the batch size is set to 128 and we run 200 epochs. We linearly decay learning rate to zero from 80 to 200 epochs as did in [12]. We take two networks with the same architecture but different initializations as two classifiers as did in [12, 65, 52], since even with the same network and optimization method, different initializations can lead to different local optimal
221
222
223
224
225
226
227
[12). The details of network structures can be checked in Appendix C. 228
229
6² and Tuin, we determine them in the range with a noisy validation set. Here, we assume the noise level is known and set R(T) 1 TV r} with T,=10. If 7 is not known in advanced, it can be inferred using validation sets [29, 66]. As for performance measurement, we use test accuracy, i.e., test accuracy (# of correct prediction) / (# of testing). All experiments are repeated five times. We report the mean and standard deviation of experimental results.
For the hyper-parametee
10-2, 1,,10-,110000
230
min T
232
233
234
235
Experimental results. The experimental results about test accuracy are provided in Table 1, 2, and 3. Specifically, for MNIST, as can be seen, our proposed methods, i.e., CNLCU-S and CNLCU-H, produce the best results in the vast majority of cases. In some cases such as asymmetric noise, the baseline S2E outperforms ours, which benefits the accurate estimation for the number of selected small-loss examples. For F-MNIST, the training data becomes complicated. S2E cannot achieve the accurate estimation in such situation and thus has great performance like it got on MNIST. Our methods achieve varying degrees of lead over baselines. For CIFAR-10, our methods once again outperforms all the baseline methods. Although some bascline, e.g., Co-teaching, can work well in some cases, experimental results show that it cannot handle various noise types. In contrast, the proposed methods achieve superior robustness against broad noise types. The results mean that our methods can be better applied to actual scenarios, where the noise is diversiform.
237
239
241
243
245
Ablation study. We first conduct the ablation study to analyze the sensitivity of the length of time intervals. In order to avoid too dense figures, we exploit MNIST and F-MNIST with the mentioned noise settings representative examples. For CNLCU-S, the length of time intervals is chosen in
247

---

Noise type Method/Noose ratio
Sym.
Asym.
Pair.
Trid.
Ins.
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
98.46
95.62
99.05
98.45
98.56
94.22
99.02
97.23
97.93
94.02
S2E
10.06
+.0.1
10.02
-.0.26
+0.32
+-0.79
+:009
1.22
1.26
2
95.04
92.08
96,32
90.86
93.19
90.93
96.42
93.28
94.65
90.11
MentorNet
+0.0
+0.42
#0.17
+0.97
:0.17
1..54
1:009
1.337
10073
1.26
97.53
95.62
98.25
95.08
96.05
94.16
98.05
96.18
97.96
95.02
Co-teaching
#0.12
10.30
+:008
+0.43
1096
+1.37
1:006
+0.85
::0.09
+0.39
92.31
91.88
93.96
62.59
93.77
86.22
94.92
83.46
92.90
86,34
SIGUA
+1.10
+00.22
+0.82
+-0.15
1,40
+
+.083
+
+.82
+
98.42
98.04
98.05
94.55
98.01
96.85
98.45
96.98
98.62
96.07
JoCor
+0.14
+-007
+0.37
+1008
+-019
+0.43
+
+0.25
+-0006
+0.31
98.82
98.31
98,93
97.67
98.86
97.71
99.09
98.02
98.77
97.78
CNLCU-S
10.03
10.05
1000
+0022
1.06
1.064
10.04
10.17
1.008
10.2
98.70
98.24
99.01
98.01
98.44
97.37
98.89
97.92
98.74
97.42
CNLCU-H
0.06
1-006
0.04
-.003
0.11
+.032
+0.15
0.00
0.01
0.39
Table I: Test accuracy (%) on MNIST over the last ten epochs. The best two results are in bold.
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noise ratio
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
89.99
75.32
89.00
81.03
83.66
67.09
89.53
77.29
88.65
79.35
S2E
+
+5.84
+0.95
1.99
1.3
+
+
3.99
+2.12
33.0
90.37
86.53
89.69
67.21
87.92
83.70
88. 74
85.63
87.52
83.27
MentorNet
#0.17
+0.65
+0.19
+2
1.08
+0.49
+0.3
+0.5
+0.15
11.4
91.48
88.80
91.03
68.07
90.77
86.91
91.24
89.18
90.60
87.90
Co-teaching
+0.00
+0029
+0.14
+4.58
+0023
1.071
+0.11
0.
+0.12
+
87.64
87.23
76.97
45.96
69.59
68.93
79.97
76.14
76.92
74.89
SIGUA
11229
1-0.72
+-2.59
+.3.0
1:5.75
+-2.80
+.3.23
4.224
+:5.09
91.97
89,96
90.95
79.79
91.52
87.40
92.01
89.42
91.43
87,59
JoCor
+.0.3
1-0.19
+-0.21
+-2.9
+:024
+
+
+..33
+
+
92.37
91.45
92.57
83.14
92.04
88.20
92.24
90.08
91.69
89.02
CNLCU-S
10.15
10.28
+0.15
1:1.77
0.26
10.44
10.17
10.3
0.10
1.02
92.42
91.60
92.60
82.69
91.70
87.70
92.33
90.22
91.50
88.79
CNLCU-H
+0.21
+-0.19
+.1.88
+.0.33
0..18
+.669
+
+0071
+0,21
+.22
Table 2: Test accuracy on F-MNIST over the last ten epochs. The best two results are in bold.
the range from 3 to 8. For CNLCU-H, the length of time intervals is chosen in the range from 10 to 15. Note that the reason for their different lengths is that their different mechanisms. Specifically, CNLCU-S holistically changes the behavior of losses, but does not remove any loss from the loss set.
250
252
We thus do not need too long length time intervals. As a comparison, CNLCU-H needs to remove discussed. The length should be longer guarantee the number of examples available for robust mean estimation. The experimental results are provided in Appendix
some outliers from the loss
254
B.4 which show the proposed CNLCU-S and CNLCU-H are robust to the choices of the length of time intervals. Such robustness to hyperparameters means our methods can be applied in practice and does not need too much effect to tune the hy perpaaaaameees.
256
257
258
Furthermore, since our methods concern uncertainty from two aspects, i.e., the uncertainty from both small-loss and large-loss examples, conduct experiments to analyze each part of our methods. Also, as mentioned, we compare robust mean estimation with non-robus mean estimation when learning with noisy labels. More details are provided in Appendix B.4.
280
3.2
Experiments on Synthetic Imbalanced Noisy Datasets
263
Experimental setup. We exploit MNIST and F-MNIST. For these two datasets, we reduce the number of training examples along with the labels from "0" to "4" to 1% of previous numbers. We term such synthetic imbalanced noisy datasets as IM-MNIST and IM-F-MNIS respectively. This setting aims to simulate the extremely imbalanced circumstance, which common in practice. Moreover, we exploit asymmetric noise, since these types of noise can produce more imbalanced case [41, 32].
264
285
286
288
Other settings such as the network structure and optimizer are the same as those in experiments on synthetic balanced noisy datasets.
7

---

Noise type Method/Noose ratio
Sym.
Asym.
Pair.
Trid.
Ins.
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
80.78
69.72
84.03
75.04
81.72
61.50
81.44
64.39
79.89
62.42
S2E
1.88
4.3.4
1101
1.22
+.093
4.663
+
+22.2
+0026
13.11
80.92
74.67
80.37
71.69
77.98
69.39
78.02
71.56
77.02
68.17
MentorNet
+0.48
+..17
10026
1.00
+0
1.77
+0029
+0.9
+0,71
2.55
82.35
77.96
83.87
73.43
80.94
72.81
81.17
74.37
79.92
73.29
Co-teaching
10.6
10.39
+0.24
10.62
+0046
+0092
+0060
1..64
=0.57
1162
78.19
77.67
75.14
52.76
74.41
61.91
75.75
74.05
74.34
67.98
SIGUA
+0.22
+-0.41
+0.36
+-0.68
+0,81
+
+0.53
+0.41
+
+1.34
80.96
76.65
81.39
69.92
80.33
71.62
79.03
74.33
78.21
71.46
JoCor
+022
+-0.33
+.074
+.663
+-020
+.00
+
+1009
+.034
+
83.03
78,25
85.06
75.34
83.16
73.19
82.77
74.37
82.03
73.67
CNLCU-S
10.11
1..70
10.17
1.033
1.225
11.25
1.032
1.33
1.037
1.0
83.03
78.33
84.95
75.29
83.39
73.40
82.52
74.79
81.93
73.58
CNLCU-H
0..47
+-0.50
+0227
+.080
+.066
1.53
1.0.7
1.13
0.055
1
Table 3: Test accuracy (%) on CIFAR-IO over the last ten epochs. The best two results are in bold.
270 As for performance measurements, we use test accuracy. In addition, we exploit the selected ratio of training examples with the imbalanced classes, i.c., selected ratio=(# of selected imbalanced labels / # of all selected labels). Intuitively, a higher selected ratio means the proposed method can make better use of training examples with the imbalanced classes, following better generalization [18].
271
273
274
Experimental results. The test accuracy achieved on IM-MNIST and IM-F-MNIST is presented in Figure 2. Recall the experimental results in Table 1 and 2, we can see that the imbalanced issue is catastrophic to the sample selection approach when learning with noisy labels. For IM-MNIST, as can be seen, all the baselines have serious overfitting in the early stages of training. The curves of test accuracy drop dramatically. As a comparison, the proposed CNICCU-S and CNLCU-H can give try to large-loss but less selected data which are possible to be clean but equipped with imbalanced labels. Therefore, our methods always outperform baselines clearly. In the case of Asym. 10%, our methods achieve nearly 30% lead over baselines. For /M- F-NNIST, we can also see that our methods perform well and always achieve about 5% lead over all the baselines. Note that due to the huge challenge of this task, some baseline, e.g., S2E, has a large error bar. In addition, the baseline SIGUA performs badly. It is because SIGUA exploits stochastic integrated gradient underweighted ascent on large-loss examples, which makes the examples with imbalanced classes more difficult to be selected than them in other sample selection methods.
275
278
277
279
281
283
285
287
The selected ratio achieved on IM-MNIST and IM-F-MNIST is presented Table 4. The results explain well why our methods perform better on synthetic imbalanced noisy datasets, i.e., our methods can make better use of training examples with the imbalanced classes. Note that since we give a try to large-loss but less selected data in conservative way, the selected ratio is still far away from the class prior probability on the test set, i.e., 10%. However, a little improvement of the selection ratio can bring considerable improvement of test accuracy. These results tell us that, in the sample selection approach when learning with noisy labels, improving the selected ratio of training examples
289
291
293
with the imbalanced classes is challenging but promising for generalization. This practical problem deserves to be studied in depth.
295
296
3.3
Experiments on Real-world Noisy Datasets
287
Experimental setup. To verify the efficacy of methods in the real-world scenario, we conduct experiments on the noisy dataset Clothing / [59]. Specifically, for experiments on Clothing /M, we use the IM images with noisy labels for training and 10k clean data for test respectively. Note that we do not use the 50k clean training data in all the experiments. For preprocessing, we resize the
298
299
300
301
image 256x256. crop the middle 224 x24 as input, and perform normalization. The experiments
302
on Clothing/// /M are performed once due to the huge computational cost. We leave 10% noisy training data as a validation set for model selection. Note that we do not exploit the resampling trick during training [24]. Here, Best denotes the test accuracy of the epoch where the validation accuracy was
303
304
305
optimal. Last denotes test accuracy of the last epoch. For the experiments on Clothing M, we use a ResNet-18 pretrained on ImageNet as did in [52]. We also use the Adam optimizer and set the batch
307
size to 64. During the training stage, we run 15 epochs in total and set the learning rate 10-1
5 x 10-4, and 5 x 10-5 for 5 epochs each.
8

---

Dataset
IM-MNIST
IM-F F.NIIST
Method/Noose ratio
10%
20%
30%
40%
10%
20%
30%
40%
0.13
0.11
0.09
0.05
0.13
0.17
0.16
0.12
S2E
10.12
+0.05
10002
+00.11
+0.04
0.03
+0.02
0.04
0.10
0.15
0.12
0.13
0.12
0.15
0.09
0.14
MentorNet
+0.0
+0.02
0000
+0.02
+0.01
0003
+0.01
0000
0.09
0.07
0.05
0.12
0.17
0.04
0.13
0.07
Co-teaching
+0.03
+0.02
1000
10.01
+:000
00000
+0.04
0.01
0.04
0.04
0.01
0.02
0.03
0.02
0.04
0.00
SIGUA
+0.00
+0.00
+
+-0.00
+
00000
+0.00
+0000
0.11
0.08
0.07
0.05
0.13
0.13
0.07
JoCor
+0.04
+0.01
+.003
+-0022
+0.0
-0.04
+0003
+0.0
0.60
0.37
0.39
0.38
0.35
0.39
0.36
0.30
CNLCU-S
10.11
10.09
10.04
10.06
10.03
0004
10.03
0.02
0.57
0.32
0.37
0.32
0.34
0.35
0.32
0.28
CNLCU-H
00.13
+0.01
0.00
t.005
+0.02
0.06
+..04
0..3
Table Selected ratio (%) on IM-MNIST and IM-F-MNIST. The best two results are bold.
S2E
MentorNet
Co-teaching
SIGUA
JoCor
CNLCU-S
CNLCU-H
Asymm
Asym.
Asym.
Asym.
IM MNNST
DM1 MANSST
Figure 2: Test accuracy vs. number of epochs on IM-MNIST and IM-F-MNIST. The error bar for standard deviation in each figure has been shaded.
309
Experimental results. The results on Clothing M are provided in Table 5. Specifically, the proposed methods get better results than state-of-ther -ftt----ttt--tttt- methods Best, which achieve improvement of +1.28% and +0.99% over the best bascline JoCor. Likewise, the proposed methods outperform all the baselines on Last. We achieve results are a bit lower than some state-of-art methods, e.g., [64] and [46], because of the following reasons. (1). We follow [52] and ResNet-18 18 as backbone. The state-of-art methods [64, 46] use ResNet-50 as a backbone. Our aim is to make the experimental results directly comparable with previous papers [52] in the same area. (2). We only focus the sample selection approach and do not employ other advanced techniques, e.g., introducing the prior distribution [46] and combining semi-supervised learning [24, 39, 28].
311
improvement of +1.01% and +0.54% over JoCor. Note that the
313
315
316
317
Methods
S2E
MentorNet
Co-teaching
SIGUA
JoCor
CNLCU-S CNLCU-H
Best
67.34
68.36
69.37
62.89
70.09
71.37
71.08
Last
65.90
67.42
68.62
58.73
69.75
70.76
70.29
Table 5: Test accuracy (%) on Clothing MM. The best two results are in bold.
318
Conclusion
319
320
In this paper, we focus on promoting the prior sample selection in learning with noisy labels, which starts from concerning the uncertainty losses during training. We robustly use the training losses at different iterations to reduce the uncertainty of small-loss examples, and adopt confidence interval estimation reduce uncertainty of large-loss examples. Experiments are conducted on benchmark datasets, demonstrating the effectiveness of our method. We believe that this paper opens up new
321
322
323
325
possibilities in the topies of using sample selection to handle noisy labels, especially in improving the robustness of models imbalanced noisy datascts.

---

References
327
328
[1]
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Conoo, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, pages 312-321, 2019.
[2] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look memorization in deep networks. In /CMI., pages 233-242, 2017.
331
332
333
[3] Peter Auer. Using confidence bounds for exploitation-explorriion trade-offs. Journal of Machine Learning Research, 3NNov):397-422, 2002.
334
335
[4] Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pages
338
421-436. Springer, 2012.
337
[5] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
338
[6] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng. and Jörg Sander. Lof: identifying density-based local outliers. In SIGMOD, pages 93-104, 2000.
340
341
77] Olivier Catoni. Challenging the empirical mean and empirical variance: a deviation study. In Annales de I'IHP Probabilités el statistiques, volume 48, pages 1188-1185, 2012.
[8] Arijit Chakrabarty and Gennady Samorodnitsky. Understanding heavy tails in a bounded world or, is a truncated heavy tail heavy or not? Stochastic models, (8(1):0991133 2012.
344
345
99] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance-and label-dependent label noise. In ICML, 2020.
346
347
Has Diakonikolas, Daniel M Kane, and Ankit Pensia. Outlier robust mean estimation with subgaussian rates via stability. arXiv preprint arXiv:2007. 15618, 2020.
348
349
[1]] Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In NeurIPS, pages 5836-5846, 2018.
350
351
352
[12] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pages 8527-8523 8577, 2018
353
354
355
[13] Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In ICML, pages 4006-4016, 2020.
356
[14] Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization
359
by controlling label-noiss information in neural network weights. In ICML, pages 4071-4081,
2020.
[15] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NewrlPS, 2018
362
363
[16] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, pages 2309-2318 2018.
384
385
386
[17] Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In /CML, pages 4804 4815, 2020.
367
368
[18] Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
369
370
371
[19] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlnl: Negative learning for noisy labels. In ICCV, pages 101-110, 2019.
10

---

373
[20]
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
374
375
[21] Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
376
[22]
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The MNIST database of handwritten digits. http://www..cccc.cc/////////////
378
[33]
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICMI, pages 3763-3772, 2019.
379
380
[24] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2020.
381
382
[55]
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In AISTATS, 2020.
383
[26]
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. 2021.
385
386
[27] Yihua Liao and V Rao Vemuri. Use of k-nearest neighbor classifier for intrusion detection. Computers & security, (1(5):439-445 2002.
[28] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez- Early- learning regularization prevents memorization of noisy labels. In NeurIPS, 2020.
389
390
[29]
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting, IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-461, 2016.
391
392
300 Michal Lukasik, Srinadh Bhojanapalli,, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In ICML, pages 6448-6458, 6458,, 2020.
383
[31] Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-drveee learning with noisy labels. In ICMI.,
385
pages 3361-3370, 2018.
[32] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Normalized loss functions for deep learning with noisy labels. In ICML, pages 2020.
398
400
[33] Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". In NeurlPS, pages 960-970, 2017.
401
402
[34] Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with iistance-dependers noise. Machine Learning. 107(8-10):1511- 10):1561-15555, 2018
403
404
[35]
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314. 2020.
405
406
407
[36] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of neural networks against noisy labels. In NeurlPS, 2020.
[37] David S Moore. Uncertainty. On the shoulders of giants: New approaches to numeracy, pages 95-137, 1990.
410
411
[38] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurIPS, pages 1196-1204, 2013.
412
413
[39] Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling. ICLR, 2020.
414
415
416
[40] Kento Nishi, Yi Ding, Alex Rich, and Tobias Höllerer. Augmentation strategies for learning with noisy labels. arXiv preprint arXiv:2103.021300 2021.

---

418
[41] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: A loss correction approach. In CVPR, pages 1944-1952, 2017.
419
421
[42] Geoff Pleiss, Tianyi Zhang, Ethan R Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. In NeurIPS, 2020.
422
[43] Jeffrey S Rosenthal. Faithful couplings of markov chains: now equals forever. Advances in Applied Mathematics, 18(3):372-311, 1997.
424
[44] Jun Shu, Qian Zhao, Zengben Xu, and Deyu Meng. Meta transition adaptation for robust deep learning with noisy labels. arXiv preprini arXiv:2006.05697, 2020.
426
[45] Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakom, and LiWu Chang. A novel anomaly detection scheme based on principal component classifier. Technical report, 2003.
428
[46] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, 2018.
430
[47] Kiran K Thekumparampil. Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In NeurIPS, pages 10271-10282, 2018.
432
433
[48]
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
434
435
[49] Qizhou Wang, Jiangchao Yao, Chen Gong, Tongliang Liu, Mingming Gong, Hongxia Yang, and Bo Han. Learning with group noise. In AAAI, 2021.
438
437
[50] Xiaobo Wang, Shuo Wang, Jun Wang, Hailin Shi, and Tao Mei. Co-mining: Deep face recognition with noisy labels. In ICCV, pages 9358-9367, 2019.
438
439
[51] Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pages 8688 8896, 2018.
441
[52] Hongxin Wei, Lei Feng. Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, pages 13726-13735, 2020.
443
[53] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. In NeurlPS, 2020.
444
445
[54] Songhua Wu, Xiaobo Xia. Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML, 2021.
446
447
448
[55] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS, pages 6835 6846, 2019.
419
450
451
[56] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS, 2020.
452
453
454
[57] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2021.
455
456
[58) Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 07777777777777770 2017.
458
[59] Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive
noisy labeled data for image classification. In CVPR, pages 2691-2699, 2015.
460
[60] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. I. ___mi: A novel iefommation-theoree loss function for training deep nets robust to label noise. In NeurIPS, pages 6222-6233 2019.
12

---

462
[61] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021.
463
464
[62] Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In ICML, pages 10789-10798, 2020.
[63]
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. In NeurlPS, 2020.
467
469
[64]
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, pages 7017-7025, 2019.
470
[65] Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? In ICMI., 2019.
472
475
[66] Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In CVPR, pages 4480-4489, 2018.
475
476
[67]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
477
478
[68]
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In ICML, 2021.
479
669
Zhilu Zhang and Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurIPS, pages 8778-8788 2018.
481
482
700 Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1-7, 2019.
[71]
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, pages 1 11477-11457, 2020.
485
13

---

Checklist
486
487
The checklist follows the references. Please read the checklist guidelines carefully for information on
488
how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or
489
[N/A]. . You are strongly encouraged to include justification to your answer. either by referencing the appropriate section of your paper or providing a brief inline description. For example:
490
491
Did you include the license to the code and datasets? [No] proprietary.
The code and the data are
493
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
495
496
1. For all authors...
497
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
498
(b) Did you describe the limitations of your work? [Yes]
500
(c) Did you discuss any potential negative societal impacts of your work? [No]
501
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
502
503
2. If you are including theoretical results...
504
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
505
(b) Did you include complete proofs of all theoretical results? [Yes]
506
3. If you ran experiments..
507
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code and instructions are provided in the supplemental material. The used datasets can be publicly downloaded. Besides, the code for generating noisy labels is provided.
508
509
510
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3.
512
513
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 3.1 and 3.2.
514
515
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See "Baselines" in Section 3.1.
517
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
518
(a) If your work uses existing assets, did you cite the creators? [Yes] We use MNIST, F F-MNIST, CIFAR-JO, CIFAR- 100, and Clothing IM in this paper. We cite the creators, which can be checked in Section 3.
520
521
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
523
524
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
525
526
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
527
528
5. If you used crowdsourcing or conducted research with human subjects...
$29
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review
532
Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
534
14


Reviews:
[{"review_id": "PNpntsRlKhQ", "reviewer": "Reviewer_mZp6", "summary": "The authors discuss potential weakness of previous sample selection criteria and propose new selection criteria based on the uncertainty of losses (not losses themselves).  Some theoretical analysis on the criteria is provided.  Then, they experimentally validate the proposed algorithms (CNLCU-S and CNLCU-H), showing that the proposed algorithms outperform some baselines in learning with noisy labels.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Distinguishing between mislabeled examples and hard examples is a fundamentally difficult problem and worthwhile to investigate.\n\nThe key idea is to incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses.  This idea is definitely reasonable, but is not totally new.  Although the details are different, batch selection based on uncertainty has been actively discussed in this field.  For example, refer to ActiveBias (NeurIPS 2017).\n\nThe assumption that the loss conforms to a Markov process is not reasonable.  Thus, Theorem 2 based on this assumption may not work well with a real loss function.\n\nMy main concern is on the experiments.  The authors argue that they did not compare the proposed algorithms with the state-of-the-art algorithms (e.g., SELF and DivideMix), because they focus on sample selection mechanisms.  I understand the authors' intention.  However, I believe that this is not acceptable for this top-tier conference, because the proposed algorithms should be proven to improve the cutting edge performance.  Sample selection alone may not be very convincing, because combining various techniques is a clear trend these days.\n\nAccording to a nice survey, https://arxiv.org/abs/2007.08199, there are recent sample selection methods, which are not covered in this paper.  Please see Section III.E.  Even though the authors want to focus on sample selection, it would be necessary to cover recent advances more comprehensively.\n\nAlso, it is common to include at least two real-world noisy datasets, e.g., WebVision, Clothing1M.\n\nTypo:\nLine 217, selectionin\n\n===================\n\nAfter the author feedback: the authors have successfully addressed my concern on the experiments (lack of the comparison with the state-of-the-art algorithms). Thus, I am willing to raise my rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think that there is no potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "B1upq4Gs1vV", "reviewer": "Reviewer_1zBg", "summary": "This paper proposes a CNLCU strategy that incorporates the uncertainty of losses by adopting interval estimation.  In learning with noisy labels, the sample selection approach regards small-loss data as correctly labeled during training.  This paper gives large-loss data a try to explore underrepresented data that are correctly labeled but seem to be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types. This paper opens up new possibilities in the topics of using sample selection to handle noisy labels.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In learning with noisy labels, the sample selection based on small-loss criteria is the most common method. This paper reveals the limitation of small-loss criteria and proposes a new selection criterion based on uncertainty to give large-loss data a try. The theoretical analyses are detailed and the claims are well supported by experimental results.  This paper adopts interval estimation instead of point estimation, where the lower bounds are used for sample selection. In this paper, the writer focuses on promoting the prior sample selection. However, this paper is not well organized and there is quite a bit of room for improvement in the representation. The overall writing is not fluent and the theorems are lack of interpretation. Besides, the penultimate paragraph in the introduction is a little bit confusing and figure 2 is worthy of improvement.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "8L5prE0fQBG", "reviewer": "Reviewer_PRnD", "summary": "This paper introduces the uncertainty of losses into the sample selection approach for learning with noisy labels. The authors discuss how to effectively exploit underlying clean examples that are less selected during training. Experimental results verify the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The sample selection approach is popular and useful in learning with noisy labels. Motivated by the fact that deep networks first memorize the training data with clean labels, the training data with small losses can be seen as clean data and used for optimization. This paper considers the uncertainty of losses in the sample selection procedure, which consists of two aspects: the uncertainty of small-loss data and the uncertainty of large-loss data. The authors use the concentration inequalities and confidence bounds to study these uncertainties at the same time. The experiments in this paper are comprehensive. The results are also promising, especially when the training data are noisy and imbalanced. \n\nStrength\n- Strong motivation. This paper is motivated by experimental results, which show that sometimes the loss values cannot fully reflect the purity of an example. When the data are noisy and imbalanced, the losses of mislabeled data and clean imbalanced data are almost equal and hence cannot be used directly for sample selection. Besides, the loss achieved at a certain epoch may not be trusted. The results are clear and well motivate this paper. \n- Great writing. The logic of this paper is smooth. The authors first reveal the issues of prior effects and then present the proposed method step by step. Most of the technical details are described clearly. \n- Theory. This paper provides theoretical analyses for two robust mean estimators, which could guide the process of sample selection during training. \n- Experiments. The experiments of this paper are comprehensive. The authors consider five types of synthetic label noise and real-world label noise, following good experimental results. Detailed experimental analyses and ablation study are also provided. \n\nWeakness/Question\n- Learning with noisy imbalanced data is challenging. Expect for the results (Figure 1 Right) on the MNIST dataset, would the results on the other datasets be similar to the case on MNIST?\n- The methods CNLCU-S (H) are built on Co-teaching. It seems that the proposed methods can also build on other sample selection methods. Could the authors further add such discussions and results?\n- The proposed method relies on a noisy validation set to find suitable hyperparameters, i.e., $\\sigma$ and $\\tau$, which makes the methods more complex. \n- The authors use ResNet-18 for the experiments on Clothing1M. I agree with the claims of the authors. The proposed method cannot be competitive with DivideMix. However, most of the existing methods use a ResNet-50 network. It would be better if the results with ResNet-50 can be provided. \n- The paper discusses the weaknesses of existing methods on sample selection at a high level (Section 1). Could the authors provide a more detailed algorithm flow like Algorithm 1 to explain this more clearly?\n\nIt is expected that the above issues/concerns could be addressed. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed method introduces extra hyperparameters into training, which cannot be determined adaptively. Although the results look very promising, it is better if such issues can be addressed. \n\nThe authors should discuss the negative societal impacts of this work like the possible model bias from the data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission considers the problem of learning from noisy labels, and propose to incorporate an interval estimate of training loss to the sample selection approach. The intuition is to distinguish between mislabelled data and under-represented data. Theoretical and empirical results shows the promise of the method.

Three reviewers have carefully considered the strengths and weaknesses of the paper. During the author rebuttal period, the authors put in significant effort to address some concerns about empirical results raised by the reviewers. The reviewers appreciated the clarifications, and one reviewer increased their score. There was unfortunately no substantive discussion among the reviewers post rebuttal.

There are many high quality papers submitted to NeurIPS each year, and this year is no exception. Many papers are on the borderline, with reviewers finding the problems important, but also found weaknesses in the paper. Unfortunately this paper could not be accepted into this year's program.
Decision: Reject