Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
Authors: Xiaobo Xia
Tongliang Liu
Bo Han
Mingming Gong
Jun Yu
Gang Niu
Masashi Sugiyama
Venue: ICLR 2022 Poster (2022)
Keywords: Learning with noisy labels
Sample selection
Uncertainty

Abstract: In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled data during training. However, losses are generated on-the-ﬂy based on the model being trained with noisy labels, and thus large-loss data are likely but not certain to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks learn patterns ﬁrst; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the conﬁdence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection. In this way, we also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem to be mislabeled at ﬁrst glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.

Published as a conference paper at ICLR 2022
SAMPLE SELECTION WITH UNCERTAINTY OF LOSSES FOR LEARNING WITH NOISY LABELS
Xiaobo Xia' Tongliang Liu'
Bo Han
Mingming Gong" Jun Yu
Gang Niu Masashi Sugiyama 5.6
'TLL Lab, The University of Sydney "oong Kong Baptist University
The University of Melbourne *University of Science and Technology of China
5 RIKEN AIP "The University of Tokyo
ABSTRACT
In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fj fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certain to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks "learn patterns first"; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the confidence intervals of losses derived from distribution-fee concentration inequalities, but not losses themselves, are used for sample selection. In this way, also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.
INTRODUCTION
Learning with noisy labels is one of the most challenging problems in since noisy labels are ubiquitous in the real world (Mirzasoleiman et al., 2020; Yu et al., 2019; Nishi et al., 2021; Arazo et al., 2019; Yang et al., 2021a; Bai & Liu, 2021). For instance, both crowdsourcing and web crawling yield large numbers of noisy labels everyday (Han et al., 2018). Noisy labels can severely impair the performance of deep neural networks with strong memorization capacities (Zhang et al., 2017; Zhang& & Sabuneu, 2018; Pleiss et al., 2020; Lukasik et al., 2020; Chen et al., 2022).
learning,
To reduce the influence of noisy labels, lot of approaches have been recently proposed (Natarajan et al., 2013; Liu & Tao, 2016; Ma et al., 2018; Yang et al., 202 Zheng et al., 2020; Xia et al., 2019; 2020; Tanaka et al., 2018: Malach & Shalev-Shward
2017: Li et al., 2020b; Menon et al., 2018;
Thekumparampil et al., 2018; Xu et al., 2019; Kim et al., 2019; Jiang et al., 2020; Harutyunyan et al., 2020). They can be generally divided into two main categories. The first one is estimate the noise transition matrix (Patrini et al.. 2017: Shu et al., 2020; Hendrycks et al., 2018: Yang et al.. 2021c; Wu al., 2022), which denotes the probabilities that clean labels flip into noisy labels. However, the noise transition matrix is hard to be estimated accurately, especially when the number of classes is large (Yu et al., 2019). The second approach is sample selection, which our focus in this paper. This approach is based on selecting possibly clean examples from a mini-batch for training (Han et al., 2018; Wang et al., 2018: Yao et al., 2020a; Wang et al., 2019; Yu et al., 2019; Lee et al., 2019; Wang et al., 2019; Yao et al., 2022). Intuitively, if we can exploit less noisy data for network parameter updates, the network will be more robust.
A major question in sample selection is what criteria can be used to select possibly clean examples. At the present stage, the selection based on the small-loss criteria is the most common method, and has been verified to be effective in many circumstances (Han al., 2018; Jiang et al., 2018; Yu et al.,
Corresponding author

---

Published as a conference paper at ICLR 2022
Con
1.50
MMsan VilliostelMaaa
1.5
1.00
0.75
0.0
025
Gaa
Milbbcied
Chan Belanced Gear inbudanced
Figure 1: Illustrations of uncertainty of losses. Experiments are conducted on the imbalanced noisy MNIST datasct. Left: uncertainty of small-loss examples. At the beginning of training (Epochs I and 2), due to the instability of the current prediction, the network gives a larger loss to the clean example and does not select it for updates. If we consider the mean of training losses at different epochs. the clean example can be equipped with a smaller loss and then selected for updates, Right: uncertainty of large-loss examples. Since the deep network learns easy examples the begimning of training, gives large loss to clean imbalanced data with non-dominant labels, which causes such data unable to be selected and severely influence generalization.
2019; Wei et al., 2020; Yao et al., 2020a). Specifically, since deep networks learn patterns first (Arpit et al., 2017), they would first memorize training data of clean labels and then those of noisy labels with the assumption that clean labels are of the majority in noisy class. Small-loss examples can thus be regarded as clean examples with high probability. Therefore, in each iteration, prior methods (Han et al., 2018; Wei et al., 2020) select the small-loss examples based on the predictions of the current network for robust training.
However, such selection procedure debatable, since arguably does not consider uncertainty in selection. The uncertainty comes from two aspects, First, this procedure has uncertainty about small-loss examples. Specifically, the procedure uses limited time intervals and only exploits the losses provided by the current predictions. For this reason, the estimation for the noisy class posterior unstable (Yao al., 2020b), which causes the network predictions be equally unstable. It thus takes huge risky to only use losses provided by the current predictions (Figure 1, left). Once wrong selection is made, the inferiority of accumulated errors will arise (Yu et al., 2019). Second, this procedure has uncertainty about large-loss examples. To be specific, deep networks learn casy examples at the beginning of training, but ignore some clean examples with large losses. Nevertheless, such examples always critical for generalization. For instance, when learning with imbalanced data, distinguishing the examples with non-dominant labels are more pivotal during training (Menon et al., 2020; Wei et al., 2021). Deep networks often give large losses such examples (Figure 1, right). Therefore, when learning under the realistic scenes, e.g., learning with noisy imbalanced data, prior sample selection methods cannot address such issue well.
To relieve the above issues, we study the uncertainty of losses in the sample selection procedure to combat noisy labels. To reduce the uncertainty of small-loss examples, we extend time intervals and utilize the mean of training losses at different training iterations. In consideration of the bad influence of mislabeled data on training losses, we build two robust mean estimators from the perspectives of soft truncation and hard truncation w.r.t. the truncation level, respectively. Soft truncation makes the mean estimation more robust by holistically changing the behavior of losses. Hard truncation makes the mean estimation more robust by locally removing outliers from losses. To reduce the uncertainty of large-loss examples, we encourage networks to pick the sample that has not been selected in a conservative way. Furthermore, to address the two issues simultaneously, concentration inequalities (Boucheron et al., 2013) for robust mean estimation and further employ statistical confidence bounds (Auer, 2002) consider the number of times example was selected during training.
derive
The study of uncertainty of losses in learning with noisy labels can be justified as follows. In statistical learning, it is known that uncertainty is related to the quality of data (Vapnik, 2013). Philosophically. need variety decrease for selected data and variety search for unselected data, which share common objective, i.c., reduce the uncertainty of data to improve generalization (Moore, 1990). This our original intention, since noisy labels could bring more uncertainty because of the low quality of noisy data. Nevertheless, due to the harm of noisy labels for generalization, we need to strike a good balance between variety decrease and search. Technically, our method is specially designed for
2

---

Published as a conference paper at ICLR 2022
handling noisy labels, which robustly uses network predictions and conservatively seeks less selected examples meanwhile to reduce the uncertainty of losses and then generalize well.
Before delving into details, we clearly emphasize contributions in two folds. First, reveal prior sample selection criteria in learning with noisy labels have some potential weaknesses and discuss them in detail. The new selection criteria are then proposed with detailed theoretical analyses. Second, we experimentally validate the proposed method on both synthetic noisy balanced/imballne and real-world noisy datasets, on which it achieves superior robustness compared with the state- of-the-art methods in learning with noisy labels. The rest of the paper is organized as follows. In Section 2. we propose our robust learning paradigm step by step. Experimental results are discussed in Section 3. The conclusion is given in Section 4.
datasets
METHOD
In this section, we first introduce the problem setting and some background (Section 2.1). Then we discuss how to exploit training losses at different iterations (Section 2.2). Finally, introduce the proposed method, which exploits training losses at different iterations more robustly and encourages networks to pick the sample that is less selected but could be correctly labeled (Section 2.3).
2.1 PRELIMINARIES
Let X' and ) be the input and output spaces. Consider a k-class classification problem, i.e., y = [k]]
where [k] = f1....,kk k)- In learning with noisy labels, the training data are all sampled from corrupted distribution on X' V. We are given a sample with noisy labels, i.e., S = {(x, ū)), where ÿ is the noisy label. The aim is to learn robust classifier that could assign clean labels to test data by only exploiting a training sample with noisy labels.
Let f RR be the classifier with learnable parameters w. At the i-th iteration during training, the parameters of the classifier f can be denoted as w,. Let é R x y - IR be a surrogate loss function for k-class classification. We exploit the softmax cross entropy loss in this paper. Given an arbitrary training example (x, y). at the i-th iteration, we can obtain a loss L,, i... L = (W])),)) Hence, until the 1-th iteration, we can obtain training loss set L, about the example (x, 9). i.e., L, = (e,,..,, E.).
In this paper, we assume that the training losses in Lt conform Markov process, which is to represent a changing system under the assumption that future states only depend on the current state (the Markov property). More specifically, at the i-th iteration, if we exploit an optimization algorithm for parameter updates (e.g., the stochastic gradient descent algorithm (Bottou, 2012)) and omit other dependencies (e.g., S), we will have P(w.(W:-1..... Wo) P(w,/W,-1) which means that the future state of the classifier only depends on the current state. Furthermore, given a training example and the parameters of the classifier f, we can determine the loss of the training example as discussed. Therefore, the training losses in L,, will also conform to a Markov process.
2.2 EXTENDED TIME INTERVALS
As limited time interval cannot address the instability issue of the estimation for the noisy class posterior well (Pleiss et al., 2020), we extend time intervals and exploit the training losses at different training iterations for sample selection. One straightforward idea is use the mean of training losses at different training iterations. Hence, the selection criterion could be
(I)
is intuitive and reasonable to use such a selection criterion for sample selection, since the operation of averaging can mitigate the risks caused by the unstable estimation for the noisy class posterior, following better generalization. Nevertheless, such a method could arguably achieve suboptimal classification performance for learning with noisy labels. The main reason is that, due to the great harm of mislabeled data, part of training losses are with too large uncertainty and could be seen as outliers. Therefore, could be biased use the mean of training losses consisting of such outliers (Diakonikolas et al., 2020), which further influences sample selection. More evaluations for our claims are provided in Section
3

---

Published as a conference paper at ICLR 2022
2.3 ROBUST MEAN ESTIMATION AND CONSERVATIVE SEARCH
We extend time intervals and meanwhile exploit the training losses at different training iterations more robustly. Specifically, we build two robust mean estimators from the perspectives of soft truncation and hard truncation (Catoni, 2012). Note that for specific tasks, it is feasible to decide the types of robust mean estimation with statistical tests based on some assumptions (Chakrabarty & Samorodnitsky, 2012). We leave the analysis future work. Two distribution-fiee robust mean estimators are introduced as follows.
Soft truncation. We extend classical M-estimator from (Catoni, 2012) and exploit the widest possible choice of the influence function. More specifically, give a random variable X, let us consider non-decreasing influence function R IR such that
(x) = log(1 X + x2/2),
(2)
The choice of v/ is inspired by the Taylor expansion of the exponential function, which can make the estimation results more robust by reducing the side effect extremum holistically. The illustration for this influence function is provided in Appendix A.1. For our task, given the observations training losses, i.e., L = (li,...,
(.....,f)., fe), we estimate the mean robustly as follows:
p. = Σ ((L)).
We term the above robust mean estimator (3) the soft estimator.
Hard truncation. We propose a new robust mean estimator based on hard truncation. Specifically, given the observations on training losses L,, we first exploit the K-nearest neighbor (KNN) algorithm (Liao & Vemuri, 2002) to remove some underlying outliers in L. The number of outliers denoted by lo(lo < t), which can be adaptively determined as discussed in (Zhao et al., 2019). Note that can also employ other algorithms, e.g., principal component analysis (Shyu et al., 2003) and the local outlier factor (Breunig et al., 2000), to identify underlying outliers in L, The main reason employ KNN is because its relatively low computation costs (Zhao et al., 2019).
The truncated loss observations on training losses are denoted by Li-t.- We then utilize L. for the mean estimation. As the potential outliers are removed with high probability, the robustness of the estimation results will be enhanced. We denote such an estimated mean fun. We have
Σ 1
jhn - t- EEELi-to
The corresponding estimator (4) is termed the hard estimator.
We derive concentration inequalities for the soft and hard estimators respectively. The search strategy for less selected examples and overall selection criterion are then provided. Note that we do not need to explicitly quantify the mean of training losses. We only need to sort the training examples based on the proposed selection criterion and then use the selected examples for robust training.
Theorem Let Zn = f......, 2n} be an observation set with mean P2 and variance 0². By exploiting the non-decreasing influence function (2) log(1 = +22 22) For any > O, we have
1 n
0² o² logse
Σ (2x)
11
(5)
n 0²
with probability at least 1 2c.
Proof can be found in Appendix A.I.
Let Zn = {Z1,... Zn} be a (not necessarily time homogeneous) Markov chain with mean ps,, taking values in a Polish state space A₁ ... An and with minimal mixing time Tuin- The truncated set with hard truncation is denoted by Zn with no < n. If [2 I is upper bounded by Z. For any E > 0 and 62 > O, we have
Theorem
1
Σ
U2 <
2
2mmin log
2 - +
2Zn
2Tmia log
2n
(6)
To 2,EZ 2mm
no
E1
E2
with probability at least
61 E2
4

---

Published as a conference paper at ICLR 2022
Proof can be found in Appendix A.2. For task, let the training loss be upper-bounded by L. The value of L can be determined easily by training networks on noisy datasets and observing the loss distribution (Arazo et al., 2019).
Conservative search and selection criteria. this paper, we will use the concentration inequalities (5) and (6) to present conservative search and the overall sample selection criterion. Specifically, we exploit their lower bounds and consider the selected number of examples during training. The selection of the examples that less selected is encouraged.
Denote the number of times one example was selected by ne(re. S 1). Let e circumstance with soft truncation, the selection criterion is
1 2r
For the
(² + 1og(2
(7)
74
Let 61 62
for the situation with hard truncation, by rewriting (6), the selection criterion is 2F'
G Pr 2 27/2min + 22ta)
log(4t)
(8)
to-t
n2t
Note that directly replace with ny. If an example is rarely selected during training, ne will be far less than n, which causes the lower bounds to change drastically. Hence, we do not use the mean of all training losses, but the mean of training losses in fixed-length time intervals. More details about this can be checked in Section 3.
For the selection criteria (7) and (8), we can see that they consist of two terms and have one term with a minus sign. The first term in Eq. (7) (or Eq. (8)) is to reduce the uncertainty of small-loss examples, where we use robust mean estimation on training losses. The second term, i.e., the statistical confidence bound, is to encourage the network to choose the less selected examples (with small n,). The two terms are constraining and balanced with o² or Tmin- To avoid introducing strong assumptions on the underlying distribution of losses (Chakrabarty & Samorodnitsky, 2012), we tune a and Tmin with a noisy validation set. For the mislabeled data, although the model has high uncertainties them (i.e., small n²) and tends to pick them, the overfitting the mislabeled data is harmful. Also, the mislabeled data and clean data are rather hard to distinguish in some cases as discussed. Thus, we should search underlying clean data in conservative way. In this paper, initialize a and Timin with small values. This way can reduce the adverse effects of mislabeled data and meanwhile select the clean examples with large losses, which helps generalize. More evaluations will be presented in Section 3.
The overall procedure of the proposed method, which combats noisy labels by concerning uncertainty (CNLCU), is provided in Algorithm 1. CNLCU works in a mini-batch manner since all deep learning training methods are based on stochastic gradient descent. Following (Han et al., 201 1), we ex- ploit two networks with parameters 0, and 0g respectively to teach each other. Specifically, when a mini-batch S is formed (Step 3), we let networks select a small proportion of examples in this mini-batch with Eq. (7) (8) (Step 4 and Step 5). The number of instances is controlled by the function R(T), and two networks only select R(T) percentage of examples out of the mini-batch. The value of R(T) should be larger at the beginning of training, and be smaller when the number of epochs goes large, which can make better use of memo- rization effects of deep networks (Han et al., 2018) for sample selection. Then, the selected instances are fed into its peer network for parameter updates (Step 6 and Step 7).
Algorithm 1 CNLCU Algorithm.
1: Input 0, and 02, learning rate 7), fixed t, epoch Th and Tmax. iteration tmaxi for 1, ......... max do
2: Shuffle training dataset S;
for f t= 1,..., ''maa do
3: Fetch mini-batch S from $;
Obtain 5, = arg min 155 E (01,
// calculated with Eq. (7) Eq. (8)
5: Obtain S₂ = arg g
8 (02, S');
// caleulated with Eq. (7) Eq.
6: Update 0₃ = 01 - V((0,, S2);
7: Update 02 = 02 V/ SI):
end
8: Update R(T) = min {
T T.T
T
end 9: Output 01 and 02.
5

---

Published as a conference paper at ICLR 2022
3 EXPERIMENTS
In this section, evaluate the robustness of our proposed method noisy labels with comprehensive experiments on the synthetic balanced noisy datasets (Section 3.1), synthetic imbalanced noisy datasets (Section 3.2), and real-world noisy dataset (Section 3.3).
3.1 EXPERIMENTS ON SYNTHETIC BALANCED NOISY DATASETS
Datasets. We verify the effectiveness of our method on the manually corrupted version of the follow- ing datasets: MNIST (LeCun et al.). F-MNIST (Xiao et al., 2017), CIFAR-IO (Krizhevsky, 2009), and CIFAR-100 (Krizhevsky, 2009), because these datasets are popularly used for the evaluation of learning with noisy labels in the literature (Han et al., 2018; Yu et al., 2019; Wu et al., 2021; Lee et al., 2019). The four datasets are class-balanced. The important statistics of the used synthetic datasets are summarized in Appendix B.1.
Generating noisy labels. We consider broad types of label noise: (1). Symmetric noise (abbreviated as Sym.) (Wu et al., 2020: Ma et al., 2018). (2) Asymmetric noise (abbreviated as Asym.) (Ma et al., 2020; Xia et al., 2021; Wei et al., 2020). (3) Pairflip noise (abbreviated as Pair.) (Han et al., 2018; Yu al., 2019; Zheng et al., 2020). (4). Tridiagonal noise (abbreviated as Trid.) (Zhang et al., 2021). (5). Instance noise (abbreviated as Ins.) (Cheng et al., 2020; Xia et al., 2020). The noise rate is set to 20% and 40% to ensure clean labels are diagonally dominant (Ma et al., 2020). More details about above noise are provided in Appendix B.I. We leave out 10% of noisy training examples as a validation set.
Baselines. We compare the proposed method (Algorithm 1) with following methods which focus on sample selection, and implement all methods with default parameters by PyTorch, and conduct all the experiments on NVIDIA Titan Xp GPUs. (1). S2E (Yao et al., 2020a), which properly controls the sample selection process so that deep networks can better benefit from the memorization effects. (2). MentorNet (Jiang et al., 2018), which learns a curriculum to filter out noisy data. We use self-paced MentorNet in this paper. (3). Co-teaching (Han et al., 2018), which trains two networks simultaneously and cross-updates parameters of peer networks. (4). SIGUA (Han et al., 2020), which exploits stochastic integrated gradient underweighted ascent to handle noisy labels. We use self-teaching SIGUA in this paper. (5). JoCor (Wei et al., 2020), which reduces the diversity of networks to improve robustness. To avoid too dense tables, we provide results of other sample selection methods and other types of baselines such as adding regularization. All results are presented in Appendix B.2. Here, we term our methods with soft truncation and hard truncation CNLCU-S and CNLCU-H respeciieelly.y
Network structure and optimizer. For MNIST, F-MNIST, and CIFAR-1O, we use 9-layer CNN structure from (Han et al., 2018). Due to the limited space, the experimental details on CIFAR- 100 provided in Appendix B.3. All network structures we used here are standard test beds for weakly- supervised learning. For all experiments, the Adam optimizer (Kingma & Ba. 2014) (momentum=0.9) used with an initial learning rate of 0.001, and the batch size is set to 128 and we run 200 epochs. We linearly decay learning rate to zero from 80 to 200 epochs as did in (Han et al., 201 8). We take two networks with the same architecture but different initializations as two classifiers as did in (Han et al., 2018; Yu et al., 2019; Wei et al., 2020), since even with the same network and optimization method, different initializations can lead to different local optimal (Han et al., 2018). The details of network structures can be checked in Appendix C.
For the ppprr-parameters x² and Tiin= we determine them in the range (10-1, 10-2, 10-3, 10-1) with a noisy validation set. Note that the use of hyperparameters aims to reduce the dependency on strong assumptions and thus make our methods perform well in practice. We provide more details about this in Appendix D. Here,
assume the noise level is known and set R(T)= 1 min 7. T, 7 with 10= If T is not known in advanced, it can be inferred using validation sets (Liu & Tao, 2016; Yu et al., 2018). As for performance measurement, use test accuracy, i.c., test accuracy= (# of correct prediction) / (# of testing). All experiments are repeated five times. We report the mean and standard deviation of experimental results.
Experimental results. The experimental results about test accuracy are provided in Tables 1, 2, and 3. Specifically, for MNIST, as can be seen, our proposed methods, i.c., CNLCU-S and CNLCU-H. produce the best results in the vast majority of cases. In some cases such as asymmetric noise, the baseline S2E outperforms ours, which benefits the accurate estimation for the number of selected

---

Published as a conference paper at ICLR 2022
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noose
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
98.46
95.62
99.05
98.45
98.56
94.22
99.02
97.23
97.93
94.02
S2E
-006
4:0.9
+-0.02
1:0,2
+..32
d:0.79
1.00
1.22
1.22
4-2.99
95,04
92.08
96.32
90.86
93.19
90.93
96.42
93.28
94.65
90.11
MentorNet
0003
+0.42
+
+,,97
+-0.17
+..5
+.009
1.33
+0.73
+126
97.53
95.62
98.25
95.08
96.05
94.16
98.05
96.18
97.96
95.02
Co-teaching
00.2
+0.30
1:000
+0.43
0096
1.37
0.00
+0.8
+0.09
1.339
92.31
91.88
93.96
62.59
93.77
86.22
94,92
83.46
92.90
86.34
SIGUA
=1.10
10.9
+0.82
=0.15
11.0
1.75
+0.83
-2,98
1.82
+3.51
98.42
98.04
98.05
94.55
98.01
96.85
98.45
96.98
98.62
96.07
JoCor
-0.14
+0.07
+-037
+.008
+-019
+..43
+
+0025
+-006
+0.31
98.82
98.31
98.93
97.67
98.86
97.71
99.09
98.02
98.77
97.78
CNLCU-S
-0.03
+.005
+0006
+0022
+0.06
+.66
+0,04
+0.17
+0.08
+.0.5
98.70
98.24
99.01
98.01
98.44
97.37
98.89
97.92
98.74
97.42
CNLCU-H
0.06
1.006
1.0004
10003
10.19
1.0.32
1.015
10.05
10.16
1.0.9
Table I: Test accuracy (%) on MNIST over the last ten epochs.
The best two results are in bold.
Noise type
Sym.
Asym.
Pair.
Tri..
Method/Noise
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
89.99
75.32
89.00
81.03
88.66
67.09
89.53
77.29
88.65
79.35
$2E
-2.07
5.884
+0.95
1.93
1.32
+
+
+..97
+2.12
3
90.37
86.53
89.69
67.21
87.92
83.70
88.74
85.63
87.52
83.27
MentorNet
0.17
+.665
+
2
1.00
+
+
+0059
+5.15
1.44
91.48
88,80
91.03
68.07
90.77
86.91
91.24
89.18
90.60
87,90
Co-teaching
2010
10.29
10.14
1:458
-00.33
1:0.71
10.11
10.36
10.12
10.45
87.64
87.23
76.97
45.96
69.39
68.93
79.97
76.14
76.92
74.89
SIGUA
1.29
10.72
12.59
1:340
155775
1:2.80
1:3.23
1:422
1..09
14.84
91.97
89,96
90.95
79.79
91.52
87.40
92.01
89.42
91.43
87.59
JoCor
=0.13
+
2.
+0024
+..58
17.17
+0.33
+0.71
+.09
92.37
91.45
92.57
83.14
92.04
88.20
92.24
90.08
91.69
89.02
CNLCU-S
=0.15
+0.28
-0015
1:77
0.26
+..44
+0.17
+0.34
+0.00
92.42
91.60
92.60
82.69
91.70
87.70
92.33
90.22
91.50
88.79
CNLCU-H
0021
1.019
1.0.18
1:0.33
1018
1.0.69
1.026
100.1
10.21
11122
Table 2: Test accuracy on F-MNIST over the last ten epochs. The best two results are in bold.
small-loss examples. For F-MNIST, the training data becomes complicated. S2E cannot achieve the accurate estimation in such situation and thus has no great performance like got on MNIST. Our methods achieve varying degrees of lead over baselines. For CIFAR-1O, our methods once again outperforms all the baseline methods. Although some baseline, e.g., Co-teaching, can work well in some cases, experimental results show that cannot handle various noise types. In contrast, the proposed methods achieve superior robustness against broad noise types. The results mean that our methods be better applied to actual scenarios, where the noise is diversiform.
Ablation study. We first conduct the ablation study to analyze the sensitivity of the length of time intervals. In order to avoid too dense figures, we exploit MNIST and F-MNIST with the mentioned noise settings as representative examples. For CNLCU-S, the length of time intervals is chosen in the range from 3 to 8. For CNLCU-H, the length of time intervals is chosen in the range from 10 to 15. Note that the reason for their different lengths is that their different mechanisms, Specifically, CNLCU-S holistically changes the behavior of losses, but does not remove any loss from the loss set. We thus do not need too long length of time intervals. As a comparison, CNLCU-H needs to remove some outliers from the loss set as discussed. The length should be longer to guarantee the number of examples available for robust mean estimation. The experimental results are provided B.4, which show the proposed CNLCU-S and CNLCU-H are robust to the choices of the length of time intervals. Such robustness to hyperparameters means our methods can be applied in practice and does not need too much effect to tune the hyperparameters.
Appendix
Furthermore, since our methods concern uncertainty from two aspects, i.e., the uncertainty from both small-loss and large-loss examples, we conduct experiments to analyze each part of our methods. Also, as mentioned, we compare robust mean estimation with non non-robust mean estimation when learning with noisy labels. More details are provided in Appendix B.4.
7

---

Published as a conference paper at ICLR 2022
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noose
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
80.78
69.72
84.03
75.04
81.72
61.50
81.44
64.39
79.89
62.42
S2E
-088
3.994
+1.01
1:1224
+0093
4-4.63
+..59
+:2.22
0.226
1.111
80.92
74.67
80.37
71.69
77.98
69.39
78.02
71.56
77.02
68.17
MentorNet
-048
+.17
+-0.26
1.00
+.031
1.77
+.029
+0093
+0.71
+-2.5
82.35
77.96
83.87
73.43
80.94
72.81
81.17
74.37
79.92
73.29
Co-teaching
0016
10.39
1:0.4
0.66
00
+:092
+.66
00.64
+0.57
1.66
78.19
77.67
75.14
52.76
74.41
61.91
75.75
74.05
74.34
67.98
SIGUA
=0.22
+0.41
+0.36
+0.68
-0.8
5:227
+0.53
+0.41
+0.39
11.34
80.96
76.65
81.39
69.92
80.33
71.62
79.03
74.33
78.21
71.46
JoCor
-0.25
+0.43
+-0.74
+166
+-0.20
+1.05
+...3
+1009
+-034
+1.27
83.03
78.25
85.06
75.34
83.16
73.19
82.77
74.37
82.03
73.67
CNLCU-S
+0.70
+-0.17
+-0.32
+0.25
1.25
+0.32
+.33
+0.37
1.0
83.03
78.33
84.95
75.29
83.39
73.40
82.52
74.79
81.93
73.58
CNLCU-H
0.44
1.0.50
1.0.27
1.880
10.68
1.53
1.071
1.13
10.25
11.39
Table 3: Test accuracy (%) on CIFAR-IO over the last ten epochs. The best two results are in bold.
3.2 EXPERIMENTS ON SYNTHETIC IMBALANCED NOISY DATASETS
Experimental setup. We exploit MNIST and F-MNIST. For these two datasets, we reduce the number of training examples along with the labels from "0" to "4" to 1 %% of previous numbers. We term such synthetic imbalanced noisy datasets IM-MNIST and IM-F-MNIST respectively. This setting aims to simulate the extremely imbalanced circumstance, which is common practice. Moreover, we exploit asymmetric noise, since these types of noise can produce more imbalanced case (Patrini et al., 2017; Ma et al., 2020). Other settings such as the network structure and optimizer are the same as those in experiments synthetic balanced noisy datasets.
As for performance measurements, we use test accuracy. In addition, we exploit the selected ratio of training examples with the imbalanced classes, i.e., selected ratio=(# of selected imbalanced labels / of all selected labels). Intuitively, a higher selected ratio means the proposed method can make better use of training examples with the imbalanced classes, following better generalization (Kang et al., 2020).
Experimental results. The test accuracy achieved on IM-MNIST and IM-F-MNIST is presented in Figure 2. Recall the experimental results in Tables and 2, we can see that the imbalanced issue is catastrophic to the sample selection approach when learning with noisy labels. For IM-MNIST, as can be seen, all the baselines have serious overfitting in the early stages of training. The curves of test accuracy drop dramatically. As a comparison, the proposed CNLCU-S and CNLCU-H can give a try to large-loss but less selected data which are possible to be clean but equipped with imbalanced labels. Therefore, our methods always outperform baselines clearly. the case of Asym. 10%, our methods achieve nearly 30% lead over baselines. For IM-F-MNIST, we can also see that our methods perform well and always achieve about 5% lead over all the baselines. Note that due to the huge challenge of this task, some bascline, c.g.. S2E, has large error bar. In addition, the baseline SIGUA performs badly. It is because SIGUA exploits stochastic integrated gradient underweighted ascent on large-loss examples, which makes the examples with imbalanced classes more difficult to be selected than them in other sample selection methods. Due to the limited space, the selected ratio achieved on IM-MNIST and IM-F-MNIST is presented in Appendix B.5, which explain well why our methods perform better than multiple baselines.
3.3 EXPERIMENTS ON REAL-WORLD NOISY DATASETS
Experimental setup. To verify the efficacy of our methods in the real-world scenario, we conduct experiments on the noisy dataset Clothing/ M (Xiao al., 2015). Specifically, for experiments on Clothing IM, we use the 1M images with noisy labels for training and 10k clean data for test respectively. Note that we do not use the 50k clean training data in all the experiments. For preprocessing, we resize the image to 256x256. crop the middle 224x224 as input, and perform normalization. The experiments on Clothing IM are performed once due to the huge computational cost. We leave 10% noisy training data validation set for model selection. Note that do not exploit the resampling trick during training (Li et al., 2020a). Here, Best denotes the test accuracy of the epoch where the validation accuracy was optimal. Last denotes test accuracy of the last epoch. For
8

---

Published as a conference paper at ICLR 2022
MentorNet
Co-teaching
SIGUA
CNLCU-S
CNLCU-H
Asym, 10%-
Asym. 20%
Asym. 30%
Asym. 40%-
IM MNISST
Taos
MM MNUST
Figure 2: Test accuracy vs. number of epochs IM-MNIST and IM-F-MM-SS The error bar for standard deviation in each figure has been shaded.
the experiments on Clothing M,, we use ResNet-18 and ResNet-50 which are pretrained on ImageNet. We also use the Adam optimizer and set the batch size 64. During the training stage, we run 15 epochs in total and set the learning rate 8 x 10 10-$, x 10 and 5 x 10 for 5 epochs each.
Experimental results. The results on I are provided in Table 4. Specifically, the proposed methods get better results than state-of-the-art methods on Best. With ResNet-18, we achieve improvements of +1.28% and +0.99%. With ResNet-50, we achieve improvements of +2.51% and +2.166% Likewise, the proposed methods outperform all the baselines on Last. We achieve improvements of + +1001 % and +0.54% with ResNet-18 18, and improvements +2.47% and +2.05% with ResNet-50. All these results verify the effectiveness of the proposed methods.
Methods
S2E
MentorNet
Co-teaching
SIGUA
JoCor
CNLCU-S
CNLCU-H
Best (R-18)
67.34
68.36
69.37
62.89
70.09
71.37
71.08
Last (R- (R-18)
65,90
67.42
68.62
58.73
69.75
70.76
70.29
Best (R-50)
68.03
67.25
67.94
65.37
69.06
71.57
71.22
Last (R-50)
66.25
66.59
67.05
60.77
68.41
70.88
70,46
Table 4:
Test accuracy (%) on Clothing IM. "R-18" (resp. "R-50") means that exploit ResNet-18 (resp. ResNet-50). The best two results are in bold.
Combining with semi-supervised learning. For combating noisy labels in real-world noisy datasets, the state- of-the-art methods, e.g., DivideMix (Li al., 2020a), always employ the semi-supervised learning technology. As our methods mainly focus on sam- ple selection, to make the comparison fair, we combine our methods with semi-supervised learning. The sample selection procedure in DivideMix is replaced by our methods. Other settings are kept the same. Following prior works (Ma et al., 2020), the experiments are conducted on three real-world noisy datasets, ie.. Food-101 (Bossard et al., 2014), WebVision (Mini) (Li et al., 2017), and Clothing IM (Xiao et al., 2015). The results are provided in Table As can be seen, the proposed methods superior and can be used improve the cutting edge performance.
Methods
Food-101
WebVision (Mini)
Clothing
DivideMix
86.73
77.32
74.76
DivideMix-S
86.92
77.53
74.90
DivideMix-II 86.88
77.48
74.82
Table 5: The test accuracy (%) three real-world datasets. DivideMix-S (resp. DivideMix-H) means that our CNLCU-S (resp. CNLCU-H) is combined with the advanced techniques in DivideMix. The best two results are in bold.
4 CONCLUSION
In this paper, we focus on promoting the prior sample selection in learning with noisy labels, which starts from concerning the uncertainty losses during training. We robustly use the training losses at different iterations to reduce the uncertainty of small-loss examples, and adopt confidence interval estimation reduce the uncertainty of large-loss examples. Experiments are conducted on benchmark datasets, demonstrating the effectiveness of our method. We believe that this paper opens up new possibilities in the topics of using sample selection to handle noisy labels, especially in improving the robustness of models imbalanced noisy datasets.

---

Published as a conference paper at ICLR 2022
5 ETHICS STATEMENT
This paper doesn't it raise any ethics concerns. This study doesn't it involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimination/iia/aainss concerns, privacy and security issues, legal compliance, and research integrity issues.
ACKNOWLEDGMENT
XBX supported by Australian Research Council Projects DE-190101473. TLL is partially supported by Australian Research Council Projects DE-190101473, IC-190100031, and DP-220102121. BH was supported by the RGC Early Career Scheme No. 22200720 and NSFC Young Scientists Fund
No. 62006202. MMG is supported by Australian Research Council Projects DE-210101624. JY sponsored by CAAl-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-000 100)6B........ GN and MS are supported by JST AIP Acceleration Research Grant Number JPMJCR20U3, Japan. MS is also supported by the Institute for and Beyond, UTokyo.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In ICML, pp. 312-321, 2019.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In ICML, pp. 233 242, 2017.
Peter Auer. Using confidence bounds for exploitation-exploraioo trade-offs. Journal of Machine Learning Research, ((NNov):397-42) 2002.
Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In ICCV, pp. 9312-9321, 2021.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-mining discriminative compo- nents with random forests. In ECCV, pp. 446-461, 2014.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp. 421-436. Springer, 2012.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
Markus Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In SIGMOD, pp. 93- 93-104, 2000.
Olivier Catoni. Challenging the empirical mean and empirical variance: deviation study. In Annales de l'THP Probabilités et statistiques, volume 48, pp. -188, 2012.
Arijit Chakrabarty and Gennady Samorodnitsky. Understanding heavy tails in a bounded world or, is a truncated heavy tail heavy or not? Stochastic models, 28(1):109-143 2012.
Haw-Shiuan Chang, Erik Learned-Milles and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In NeurIPS, pp. 1002-1012. 2017.
Ling-Ho Hao Chen, He Li. and Wenhao Yang. Anomman: Detect anomaly on multi-view attributed networks. arXiv preprint arXiv:2201.0288 2022.
Peng Chen, Xinghu Jin, Xiang Li, and Lihu Xu. A generalized catoni's m-cstimator under finite a-th moment assumption with a E (1, 2). arXiv preprint arX5v:2010.05008, 2020.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML., pp. 1062-1070, 2019.
10

---

Published as a conference paper at ICLR 2022
Jiacheng Cheng. Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance-am and sabel-dependent label noise. In ICML, 2020.
Ilias Diakonikolas and Daniel M Kane. Recent advances in algorithmic high-dimensional robust statistics. arXiv preprint arXiv:191.05911,, 2019.
Ilias Diakonikolas, Daniel M Kane, and Ankit Pensia. Outlier robust mean estimation with subgaus- sian rates via stability. arXiv preprint urXiv:2007 15618, 2020.
Evarist Giné, Rafal Latala, and Joel Zinn. Exponential and moment inequalities for u-statistics. In High Dimensional Probability 1I, pp. 13-38. Springer, 2000.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 8527-8537, 2018.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In ICML, pp. 1806-4016, 2020.
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information in neural network weights. In ICML, pp.
2020.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data train deep networks on labels corrupted by severe noise. In NeurIPS, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pp. 2309-2318, 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In ICML, pp. 4804-4815, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlul: Negative learning for noisy labels. In ICCV, pp. 101-110, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 112.v:1412.6980, 2014.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The MNIST database of handwritten digits. http:///am..eccu..coo//e////i///
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. ICML., pp. 3763-3772, 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In ICLR, 2020a.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameteriieed neural networks. AISTATS, 2020b.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 02862, 2017.
Yihua Liao and Rao Vemuri. Use of k-nearest neighbor classifier for intrusion detection. Computers & security, 21(5):439 448, 2002.
Liu Liu, Tianyang Li, and Constantine Caramanis. High dimensional robust m-estimation: Arbitrary corruption and heavy tails. arXiv preprint arXiv:1901 0003378777, 2019.
11

---

Published as a conference paper at ICLR 2022
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447 461, 2016.
Eyal Lubetzky and Allan Sly. Cutoff for the ising model on the lattice. Inventiones mathematicae, 191(3):719 2013.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In ICML, pp. 6448-64880 (488-665,,48 2020.
Yueming Lyu and Ivor W Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2020.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-driven learning with noisy labels. In ICMI, pp. 3361-3370, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICML, pp. 6543-6553 2020.
Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how update". In NeurIPS, pp. 960-970, 2017.
Aditya Krishna Menon, Brendan Van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent noise. Machine Learning, (07(8810):151-1110 2018.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 2020.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of neural networks against noisy labels. In NeurlPS, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, 2018.
David S Moore. Uncertainty. On the shoulders of giants: New approaches numeracy, pp. 95-137, 1990.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurlPS, pp. 1196-1204, 2013.
Kento Nishi, Yi Ding, Alex Rich, and Tobias Höllerer. Augmentation strategies for learning with noisy labels. arXiv preprint 2 2021.
Laura Niss and Ambuj Tewari. What you see may not be what you get: Ucb bandit algorithms robust to e-contamination. In UAI, pp. 450-459, 2020.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pp. 1944-1952, 2017.
Daniel Paulin et al. Concentration inequalities for markov chains by marton couplings and spectral methods. Electronic Journal of Probability, 20, 2015.
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Weinberger. Identifying mislabeled data using the area under the margin ranking. In NeurlPS, 2020.
Gareth O Roberts, Jeffrey S Rosenthal, et al. General state space markov chains and meme algorithms. Probability surveys, 1:20-71, 2004.
Jun Shu, Qian Zhao, Zengben Xu, and Deyu Meng. Meta transition adaptation for robust deep learning with noisy labels. arXiv preprint arXiv:200600669, 2020.
Mci-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakom, and LiWu Chang. A novel anomaly detection scheme based principal component classifier. Technical report, 2003.
12

---

Published as a conference paper at ICLR 2022
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gi Gie-Gil Lee. Learning from noisy labels with deep neural networks: A survey. arXiv preprint arXiv:2007.08199, 2020.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, 2018.
Kiran K Thekuumprrampl, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans to noisy labels. In NeurlPS, pp. 10271-10282, 2018.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Xiaobo Wang, Shuo Wang, Jun Wang, Hailin Shi, and Tao Mei. Co-mining: Deep face recognition with noisy labels. In ICCV, pp. 906889937,,000
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp. 8688-8696, 2018.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: joint training method with co-regularization. In CVPR, pp. 13726-13735, 2020.
Tong Wei, Jiang-Xin Shi, Wei-We Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. arXiv preprint arXiv:2108. 11221008.11500 11569, 2021.
Kirk M Wolter. Introduction variance estimation, volume 53. Springer, 2007.
Pengxiang Wu, Songzhu Zheng. Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. In NeurlPS, 2020.
Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML, 2021.
Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classification with instance-dependent label noise. Clear, 2022.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable label-noise learning? In NeurlPS, pp. 6835 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurlPS, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXi:: 2017.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1005v01505.0085: 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. I. dmi: A novel informatiin-theerrtie loss function for training deep nets robust to label noise. In NeurlPS, pp. 6222-6233, 2019.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021a.
Shuo Yang. Songhua Wu, Tongliang Liu, and Min Xu. Bridging the gap between few-shot and many- shot learning via distribution calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021b.
13

---

Published as a conference paper at ICLR 2022
Shuo Yang, Erkun Yang. Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependene label-noise transition matrix using dnns. arXiv preprint arXi::2105. 2021c.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin- Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In ICML, pp. 10789-10798, 2020a.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix in label-noise learning. NeurlPS, 2020b.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. In NeurlPS, 2021.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-prior estimation for positive-umlabeldd learning. In ICLR, 2022.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? In ICML, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, Kayhan Batmanghelich, and Dacheng Tao. An efficient and provable approach for mixture proportion estimation using linear independence assumption. In CVPR, pp. 1480-4489, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In 20211,20211.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, pp. 3778-8785, 2018.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1-7, 2019
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, pp. 11447-11457, 2020.
14

---

Published as a conference paper at ICLR 2022
APPENDIX
The appendices organized as follows. Section A details the proofs for our theorems. Section B presents more experimental results. Section C shows the details of used network structures. Section gives more justifications for our theoretical analyses. Section E provides more discussions on learning by considering uncertainty. Section provides more discussions on sample selection methods.
A
PROOF OF THEORETICAL RESULTS
A. PROOF OF THEOREM 1
For the cireumstance with soft truncation, μ!= 1 n Ei= (zi). As suggested in (Catoni, 2012), can exploit jن and j such that
S 1s S
(9)
to derive a bound for P1z. For some positive real parameter a, we define
r(we) Σ v la(z [i,]]
(10)
Let us introduce the quantity
(0) = an Σ v [a(z; - -0]-
(11)
With the exponential moment inequality (Giné et al., 2000) and the C, inequality (Mohri et al., 2018),
we have
exp(anr(0)} S {1 aluz 0) + a0 + (Az - -)}
(12)
s exp(naluu= 0) na + (Nz 0)
In the same way,
< exp(-naauu, - 0) + na? + (1z - 0)
(13)
we define for any us E IR the bounds
B0) = μ2= - aala? + -4=-00² 0)9 log(E 1
(14)
an
and
(0) = - 0 aooz + (uz - [)² log(E-
(15)
an
From (Chen et al., 2020) (Lemma 2.2), we obtain that P(r(0) > B (0))
and P(r(0) < (+))
(16)
Let ū be the largest solution of the quadratic equation (_ (0) and Ø be the smallest solution of the quadratic equation B,, (0). Also, to guarantee the solution of the quadratic equation, we assume
4a202 + 4log(E-1) S 1.
(17)
From (Chen et al., 2020) (Theorem 2.6), we then have
log(e 1 em
2 P2g
a0
(18)
and
log(r
0x7 +
(19)
S f1
15

---

Published as a conference paper at ICLR 2022
With probability least 1-2c, we have 2Iz S fb= S ps. We can choose 2. Then we have
o² + logíc
AA: - Mzl
(20)
n
which holds with probability least 1-2c.
We exploit the lower bound and let 28 Then we have
(² + log(2))
(21)
n4 o²
where ny denotes the number of times that the example selected in the time intervals.
y- logil -x+x721
Figure 3: The illustration the influence function for the soft estimator.
Here, we provide the graph of the used influence function for the soft estimator, which explains the x /2) more clearly. The illustration is presented in
mechanism of the function y log(1 Figure 3. As can be seen, when x is large and may be outlier, the influence function can reduce its negative impact for mean estimation. Therefore, we exploit such an influence function for robust mean estimation, which brings better classification performance.
A.2 PROOF OF THEOREM 2
Lemma ((Paulin et al., 2015)) Let Zn, = (21....,, be a (not necessarily time homogeneous) Markov chain with mean μs, taking values in a Polish state space A1 ... Ano with a mixing time T(V) (for 0 < v S 1). Let
(
Twin inf r(U)
(22)
0sv<1
For some n E IR suppose that f + IR satisfies the following inequality:
f(a) - f(b) <
n1 * bil.
(23)
for every a, b E Then for any e 2 0, we have
-2c
(Z.) F.)(()) 2)) 2
(24)
17
The detailed definition of the mixing time for the Markov chain can found (Paulin et al., 2015; Roberts et al., 2004). Let ff be the mean function. Following the prior work mean estimation (Liu et al., 2019; Diakonikolas et al., 2020; Diakonikolas & Kane, 2019; Niss & Tewari, 2020), without loss of generality, we assume μl, for the underlying true distribution, and |z,] is upper bounded by Z. Then we can set to 4Z/n for Eq. (23). Combining the above analyses, we can revise Eq. (24) follows:
2Z
2
P
25
>
2Tain log
S C1, E1
(25)
12
16

---

Published as a conference paper at ICLR 2022
and
2Z
2n
P
max
|2:|
2
2 log
S <2
(26)
iEin]
71
62
for 61 0 and 62 0. If we remove the potential outliers Z,, from Zn Therefore, have
no
no
2.E2n Vano
FiEZn
2
2:E
11 11
(27)
Σ
+ no max 22 sEini
11
2Zn.
2n
27
mm log
+
mmm log (
71 no
C1
n
which holds with probability at least 1 For our task, we exploit the concentration inequality. Let él = 62 = L. Next we can obtain
E2.
1
and the losses bounded by
27
Mn - 11 s t -- to
2L
(/2 log(48) + v 4rmmn log(
(28)
2v 2Tmin L(t + 22 Iog(4t)
t.
with the probability at least 1 1. In practice, it is easy to identify the value of L. For example, can training deep networks on noisy datasets to observe the loss distributions. Then, we exploit the lower bound such that
2x 2T [t + V21a)
)lo((4)) 74
E = Ph
(29)
(t (t-t,)) -.tt
for sample selection.
B COMPLEMENTARY EXPERIMENTAL ANALYSES
# of training
# of testing
# of class
size
MNIST
60,000
10,000
10
28x28x
F-MNIST
60,000
10,000
10
28x28x1
CIFAR-10 10
50,000
10,000
10
32x32x3
CIFAR-100 50,000
10,000
100
32x32x3
Table 6: Summary of synthetic datasets used in the experiments.
B. THE DETAILS OF DATASETS AND GENERATING NOISY LABELS
For the details of datasets, the important statistics of the used datasets summarized in Table 6.
For the details of generating noisy labels, we exploit both dlas-deeendeent and instance deppedeed label noise which include five types of synthetic label noise to verify the effectiveness of the proposed method. Here, we describe the details of the noise setting as follows:
(1). Class-dependent label noise:
Symmetric noise: this kind label noise is generated by flipping labels each class uniformly to incorrect labels of other classes.
17

---

Published as a conference paper at ICLR 2022
(8)
(b)
(c)
(d)
Figure 4: Synthetic class-dependent transition matrices used in our experiments on MNIST. The noise rate is set 20%.
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noise
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
98.66
98.21
96.14
98.06
97.50
98.84
97.62
97.25
95.75
NPCL
1.00
10.11
10.02
11.21
10.24
10.18
10.05
10.29
10.63
11.04
98.58
98.37
98.89
97.67
97.98
97.21
98.74
97.65
96.83
94.97
INCV
10.06
10.06
10.26
10.21
1.0.99
10.17
10.32
1.04
11.35
98.76
94.92
98.63
88.65
98.66
68.44
98.93
76.44
97.63
87.90
APL
=006
+0.31
+0005
1,77
+0.10
+
+.004
+3.04
+0.73
+1.
94.77
92.16
96.73
91.05
93.25
71.02
94.06
70.28
93.17
77.45
CDR
20.17
+0.73
+0.19
0076
3:.89
+0.92
=4.01
+0.9
+3.04
Table 7: Test accuracy (%) on MNIST over the last ten epochs.
Asymmetic noise this kind of label noise is generated by flipping labels within of similar classes. In this paper, for MNIST, flipping 2-77, 3-8, 5646. For F-MNIST, flip- ping SCHIRT-+SHIRYS PULLOVER-CCOK
SANDALS SNEAKER. For CIFAR-IO. flipping TRUCK-AUTOMOBILE BIRD-AAIRPLANE DEER HORSE, CAT+DOG, For CIFAR- -00, the 100 classes are grouped into 20 super-classes, and each has 5 sub-classes. Each class is then flipped into the next within the same super-class.
Pairflip noise: the noise flips each class its adjacent class.
the noise corresponds to spectral of classes where adjacent classes are casier be mutually mislabeled, unlike the unidirectional pair flipping. can be implemented by two consecutive pair flipping transformations in the opposite direction.
Tridiagonal noise:
(2). Instance-dependent label noise:
the noise is quite realistic, where the probability that an instance is mislabeled depends on its features. We generate this type of label noise to validate the effectiveness of the proposed method as did in (Xia et al., 2020; Yao et al., 2021).
Instance noise:
We use synthetic noisy MNIST as an example and plot the noise transition matrices in Figure 4. The noise rate is set to 20%.
B.2 COMPARISON WITH OTHER BASELINES
We focus on the sample selection approach in learning with noisy labels. In the main paper (Section
3.1), we fairly compare our methods with the baselines which also focus on sample selection. We add two other sample-selectie the sample selection methods can be divided into three categories: (a) "Multi-network Learning", (b) "Multi-round Learning", and (c) "Hybrid Approach". As our method belongs to (a), we exploit INCV
baselines here. As suggested by the related survey (Song et al., 2020).
(Chen et al., 2019) which belongs to (b), and NPCL (Lyu & Tsang, 2020) which belongs (c), to make the comparison more comprchensive.
Here, also evaluate other types of baselines. We exploit APL (Ma et al., 2020) and CDR (Xia et al., 2021), which add implicit regularization from different perspectives. The experiments are conducted MNIST and F-MNIST. Other experimental settings the as those in the main paper. The experimental results are provided in Tables 7 and 8, which show that the proposed methods superior these baselines with respect classification performance.
18

---

Published as a conference paper at ICLR 2022
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noose
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
91.65
86.72
91.20
70.65
91.07
85.03
90.99
88.79
90.77
86.34
NPCL
00.37
+-2.30
+-0.66
d:2,83
+00.2
1:194
+.015
1:1.0
-0.85
4-2.88
91.66
85.39
91.35
65.82
90.26
85.73
91.76
88.42
89.54
86.21
INCV
=0.31
1.14
+-0.70
+.4322
+:0.88
+.92
+
1.55
1.79
+
91.73
89.06
90.13
80.34
90.22
78.54
90.84
86.53
90.96
85.55
APL
0020
+:041
1:017
+:063
+0.80
1:4.3
+0222
0076
+0.77
22286
85.62
71.83
89.78
79.05
85.72
69.07
86.75
73.63
85.92
73.14
CDR
1.37
+0.41
1133
1066
2.331
1..1
22882
1.43
13.12
Table Test accuracy on F-MNIST over the last ten epochs.
B.3
EXPERIMENTS ON SYNTHETIC CIFAR-100
For CIFAR-10O, we use 7-layer CNN structure from (Yu et al., 2019: Yao et al., 2020a). Other experimental settings are the same as those the experiments on MNIST, F-MNIST, and CIFAR-IO.
The results provided in Table 9. We can see the proposed method outperforms all the baselines.
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noosse
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
44.59
25.78
42.18
26.81
42.99
26.96
43.16
27.72
43.13
27.12
S2E
-033
+5.44
+1.73
+
1.54
+
+0.93
+3.56
+0.67
3
43.15
37.62
41.03
28.27
40.06
27.17
42.20
31.74
40.54
33.09
MentorNet
004
+.089
+:0.2
+:041
-0.37
+:0.2
+..30
+-088
+0009
+..5
45.17
40.95
42.76
30.27
42.50
30.07
44.41
34.96
42.23
35.87
Co-tcaching
=025
+0.52
+.034
+0033
0..39
17
+0.41
+0.35
+0.52
1.44
42.03
40.53
36.67
26.71
36.48
26.73
39.21
32.69
39.19
33.51
SIGUA
10.3
10.49
10.25
11,42
-0.33
1..33
10.40
110336
10,32
10.43
45.93
41.56
42.89
29.19
42.12
30.12
44.98
34.23
44.28
35.60
JoCor
10.57
10.37
11.42
10.35
10.65
10.27
1.133
10.59
10.99
46.09
42.11
43.06
30.47
43.08
30.33
45.19
35.49
44.80
36.23
CNLCU-S
0022
+.0.0
+:0.28
0..37
0.99
+:0.74
+.090
1.30
+0,70
+:04
46.27
42.05
43.21
30.55
43.25
30.79
45.02
35.24
45.02
36.17
CNLCU-H
1.887
1.0.93
1.0.72
0.75
1:0.86
1.06
1.9.3
1.107
1:5.4
Table Test accuracy (%) on CIFAR-100 over the last epochs. The best results are in bold.
B.4 EXPERIMENTS FOR ABLATION STUDY
We conduct the ablation study to analyze the sensitivity of the length of time intervals. The results are shown Figure and 6. As we can seen, the proposed method, i.e., CNLCU-S and CNLCU-H are robust to the choices of hyperparameters.
Sym.
Asym.
Phir.
Trid.
Ins.
MNSST
F--NNIST-
Figure 5: Illustrations of the hyperparameter sensitivity for the proposed CNLCU-S. The error bar for standard deviation in each figure has been shaded.
Note that in this paper, we concern uncertainty from two aspects, i.c., the uncertainty about small-loss examples and the uncertainty about large-loss examples. Here, we conduct ablation study to show the
19

---

Published as a conference paper at ICLR 2022
Sym.
Asym.
Pair.
Trid.
Ins.
MVSST
-F-MVSS
Figure 6: Illustrations of the hyperparameter sensitivity for the proposed CNLCU-H. The error bar for standard deviation in each figure has been shaded.
effect of removing different components to provide insights into what makes the proposed methods successful. The experiments conducted MNIST and F-MNIST. Other experimental settings are the same as those in the main paper (Section 3.1). J. Note that we employ two networks to teach each other following (Han al., 2018). Therefore, when we do not consider uncertainty in sample selection, the proposed methods will reduce to the baseline Co-teaching (Han et al., 2018).
To study the effect of concerning uncertainty about small-loss examples, remove the concerns about large-loss examples, i.c., the network is not encouraged to choose the less selected examples for updates. We express such a setting as "without concerning about large-loss examples" (abbreviated as w/o cl). To study the effect of concerning uncertainty about large-loss examples, we remove the concerns about small-loss examples, i.e., we only exploit the predictions of the current network. We express such setting as "without concerning about small-loss examples" (abbreviated w/o cs). Besides, we express the setting which directly uses non-robust mean as Co-teaching-M.
The experimental results of ablation study are provided in Tables 10 and 11. As can be seen, both aspects of uncertainty concerns can improve the robustness models. Therefore, combining two uncertainty concerns, can better combat noisy labels. In addition, robust mean estimation is superior to the non-robust mean learning with noisy labels.
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noise
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
98.82
98.31
98.93
97.67
98.86
97.71
99.09
98.02
98.77
97.78
CNLCU-S
10.03
10005
10.06
+0.22
10.06
1.0.4
10.04
10.17
10.08
-.25
98.02
96.83
98.50
96.25
98.22
96.08
98.64
97.25
98.17
97.13
CNLCU-S w/o
+0,08
+:0299
0.04
+0.13
+:0.13
-00.55
-..31
+.0.24
4.0.00
0.40
98.15
97.12
98.36
96.39
98.04
96.12
98.74
97.30
98.11
97.32
CNLCU-S w/o
+0220
0.22
=0007
+0.48
0..24
0.66
0.05
+0.52
10.15
0043
98.70
98.24
99.01
98.01
98.44
97.37
98.89
97.92
98.74
97.42
CNLCU-H
+000
+0.06
+0.0
+0.03
19.19
-0.3
00.5
=0000
10.16
0.33
98.06
96.92
98.39
96.51
97.04
95.62
98.33
97.41
98.01
96.15
CNLCU-H w/o cl
+0.13
+-0.33
+0004
+0,57
+0.87
+.0.3
0.447
+-0.92
+
-.288
98.19
97.05
98.76
97.17
97.26
96.31
98.29
97.65
98.34
96.49
CNLCU-H w/o cs
+0.22
+.449
0.559
+0.60
1.19
+
00.17
+0.92
+0.36
97.72
97.78
98.27
95.42
96.22
95.01
97.92
96.64
98.02
96.03
Co-teaching-M
+0.08
1:032
1:0.03
10.42
+0.10
10.65
0014
10.77
10.04
0055
97.53
95.62
98.25
95.08
96.05
94.16
98.05
96.18
97.96
95.02
Co-teaching
10.12
1:0.00
120.8
10.43
10.96
11.37
1000
10.85
10.09
-0.39
Table 10: Test accuracy (%) on MNIST over last ten epochs.
B.5 RESULTS ON SYNTHETIC IMBALANCED NOISY DATASETS
In the main paper, have provided the test accuracy on synthetic imbalanced noisy datasets in Figure 2. Due to the limited space, we provide the selected ratio achieved on IM-MNIST and IM-F. MNIST in Table 12. The results explain well why our methods perform better synthetic imbalanced noisy datasets, i... our methods can make better use of training examples with the imbalanced classes. Note that since we give try to large-loss but less selected data in conservative way, the selected
20

---

Published as a conference paper at ICLR 2022
Noise type
Sym.
Asym.
Pair.
Trid.
Ins.
Method/Noise
20%
40%
20%
40%
20%
40%
20%
40%
20%
40%
92.37
91.45
92.57
83.14
92.04
88.20
92.24
90.08
91.69
89.02
CNLCU-S
+0.15
+-0.28
1..77
+0.26
+:0.44
0.11
+-0.34
1.010
=1.02
91.77
89,40
91.25
72.93
91.53
87.31
91.31
89.50
91.09
88.45
CNLCU-S w/o
+0.35
+
0.300
+2663
+0.17
+-0.59
0.55
+-0.32
+0.13
-0.57
91.85
90.76
91.94
80.99
91.28
87.31
91.39
89.29
90.98
88.73
CNLCU-S w/o
+0.33
1.228
0.00
22.74
+0020
0.77
0.00
10.51
+:.43
0.62
92.42
91.60
92.60
82.69
91.70
87.70
92.33
90.22
91.50
88.79
CNLCU-H
+0.21
+0.19
10.18
+0.43
+0.18
10669
0.26
+0071
+0.21
91.70
90.05
91.08
71.35
91.03
87.22
91.59
90.01
90.80
88.31
CNLCU-H w/o cl
+0.04
+0.31
+0006
+2
+
+-0.72
+.007
+.0.24
+
=1 -1.09
91.82
90.92
92.45
80.73
91.21
87.49
92.08
89.72
91.21
88.62
CNLCU-H w/o cs
+0.13
+0.42
+0025
1.63
+7.17
0.11
+0.24
+
91.33
89.05
91.14
71.03
90.85
86.95
91.50
89.18
90.74
88.25
Co-tcaching-M
10.18
1:073
1:0.90
1:3.73
10061
1:0.19
0.46
1.0.44
1,06
-0.92
91.48
88.80
91.03
68.07
90.77
86.91
91.24
89.18
90.60
87.90
Co-teaching
10.10
1.0.29
10.14
1.4.58
10.23
100.71
0.11
1..36
10.12
1044
Table 11: Test accuracy (%) on F-MNIST over last ten epochs.
Dataset
IM-MNIST
IM-F-MNIST
Method/Noise
10%
20%
30%
40%
10%
20%
30%
40%
0.13
0.11
0.09
0.05
0.13
0.17
0.16
0.12
S2E
+0.12
t0.05
+0.02
+0.01
+0.04
+0.0
+00.0
0
0.10
0.15
0.12
0.13
0.12
0.15
0.09
0.14
MentorNet
+000
+0.02
+0.0
+0002
+0.01
0.03
=0001
0.00
0.09
0.07
0.05
0.12
0.17
0.04
0.13
0.07
Co-teaching
10003
40.02
10000
100001
+t0.05
10000
10.04
1000
0.04
0.04
0.01
0.02
0.03
0.02
0.04
0.00
SIGUA
+0000
40.00
+.000
+.0.00
+0.000
1.00
+00.00
1.00
0.11
0.08
0.07
0.06
0.05
0.13
0.13
0.07
JoCor
#0.0
t0.01
+0.03
10.02
+0000
0004
10.03
1000
0.60
0.37
0.39
0.38
0.35
0.39
0.36
0.30
CNLCU-S
10.11
10.09
1000
10.06
+0.03
0.04
1..03
1.00
0.57
0.32
0.37
0.32
0.34
0.35
0.32
0.28
CNLCU-H
10.3
10.01
10007
10.05
=10.02
0.00
10.04
10.03
Table 12:
Selected ratio (%) on IM-MNIST and IM-F-MNIST. The best two results in bold.
ratio still far away from the class prior probability on the test set, i.c., 10%. However, little improvement of the selection ratio can bring considerable improvement of test accuracy. These results tell us that, in the sample selection approach when learning with noisy labels, improving the selected ratio of training examples with the imbalanced classes is challenging but promising for generalization. This practical problem deserves to be studied in depth.
B.6 EXPERIMENTS WITH HIGH NOISE LEVELS
In the main paper, we set the noise rates to be smaller than 50%, which ensures that clean labels in noisy classes diagonally dominant in all label noise settings. To demonstrate the effectiveness of our method with high noise levels, we conduct experiments on MNIST and F-MNIST with 50% 60%, and 70% symmetric noise. The experimental results are presented in Table 13. As can be seen, the proposed methods can achieve the best performance in almost all cases.
C COMPLEMENTARY EXPLANATION FOR NETWORK STRUCTURES
Table 14 describes the 9-layer CNN (Han et al., 2018) used on MNIST, F-MNIST, and CIFAR-IO. Table 15 describes the 7-layer CNN (Yu et al., 2019) used on CIFAR-1OO. Here, LReLU stands for Leaky ReLU (Xu et al., 2015). The slopes of all LReLU functions in the networks are set to 0.01. Note that that the l/yer CNN is standard and common practice in weakly supervised learning. We decided to use these CNNs, since then the experimental results are directly comparable with previous approaches in the same area, i.e., learning with noisy labels.
21

---

Published as a conference paper at ICLR 2022
Dataset
MNIST
F-MNIST
Method/Noise
50%
60%
70%
50%
60%
70%
APL
84.97-12.97
75.68
70.11::0.52
76.80+13.21
72.77-14.37
68.39:17.17
CDR
76.85:12.46
5..22::1992
54.22:10.94
13.41=1881
45.82:+2.77
41.33#3.69
INCV
96.14-1-0.3
94.12+11.13
92.13-40.75
84.92-0.60
82.79-11.64
80.12:+0.26
NPCL
97.05:1:.14
94.7710.73
93.58:1.22
86.07
83.29
19.39:1.558
S2E
87.76:12..39
79.241:.779
50.30:1.822
62.20=3.94
59.88=1.277
45.88+7.96 7.99
MentorNet
91.14-10.17
90.11=0.377
08.72-0.46
86.51=0.11
85.91=0.444
83.27-10.55
Co-teaching
95.60:10.38
05.44:1.30
90.11
88.72:40.14
87.92-0.34
85.92:10.72
SIGUA
91.35-122.2
88.62-11.93
86.08
83.39:13.29
19.36
72.14-14.28
JoCor
97.14
96.47: 0,46
95.01
89.1640.27
87.93-0.61
86.99=0.92
CNLCU-S
97.7810.07
9..7310.31
0..02:10.38
89.5610.25
08.56
87.5411.42
CNLCU-H
97.64::0.25
96.45=00.55
95.30=10.26
89.77=0.29
88.07=0.62
8..40=1.05
Table 13: Test accuracy on MNIST and F-MNIST with high noise levels over the last ten epochs. The best two results are in bold.
CNN MNIST
CNN F-MNIST
CNN CIFAR-10
28x28 Gray Image
28x28 Gray Image
32x32 RGB Image
3x3 conv, 128 L
conv,
3x3 conv, 128 LReLU
CNN on CIFAR-T00
3x3 conv, 128 LReLU
32x32 RGB Image
2x2 max-pool dropout, p = 0.25
3x3 conv, 64 ReLU
3x3 conv, 64 ReLU
3x3 conv, 256 LReLU
2x2 max-pool 3x3 conv, 128 ReLU
3x3 conv, 256 LRRl.U
3x3 conv, 256 LReLU
3x3 conv, 128 ReLU
2x2 max-pool dropout, p = 0.25
2x2 max-pool
3x3 conv, 196 ReLU
3x3 conv, 512 LReLU
3x3 conv, 196 ReLU
3x3 conv, 256 |Rel.U
2x2 max-pool dense 256-+100
3x3 conv, 128 RReLUU
avg-pool dense 128 10
dense 128 10
dense 128-+10
Table 15: The CNN on CIFAR-100.
Table 14: The CNN on MNIST, F-MNIST, and CIFAR- 10.
D NEEDED ASSUMPTIONS WITHOUT NOISY VALIDATION SETS
In the main paper, a noisy validation set is exploited to determine o² and Tmin- We discuss, if we do not use the noisy validation set, what assumptions we will need.
In statistical learning, it is important problem for estimating the variance of a data distribution. However, it is hard to obtain an accurate estimation from a finite sample. To reduce the estimation errors, some kinds of assumptions are often needed, e.g., random sampling assumptions, replacement assumptions, and strict distribution assumptions. We suggest the readers refer (Wolter, 2007) for more details.
For the accurate estimation of Tmin. we have to assume that the Markov chain can exhibit a cutoff, which means that the total variation distance decreases very rapidly in a small interval (Paulin et al., 2015). Readers can check the Figure of (Lubetzky & Sly, 2013) for better understanding.
E FURTHER DISCUSSION ON LEARNING BY UNCERTAINTY
We discuss related work named ActiveBias (Chang et al., 2017) on learning with uncertainty, although it studies the predicted probability, rather than the loss. In general, the differences between our work and ActiveBias lie in the different problems, different focuses, and different technical implementation. In more detail, we summarize the main differences between the two works in the following:
ActiveBias mainly focuses on mining hard examples to help generalization. Although mentions the label-noise problem, it does not consider comprehensive noise settings. Also,
22

---

Published as a conference paper at ICLR 2022
the baselines SGD, ADAM, and other optimization methods, but not the methods specially designed for learning with noisy labels. In contrast, our work mainly focuses on learning with noisy labels and has more comprehensive experiments, which explore the performance different label-noise methods under noisy supervision.
ActiveBias focuses on using the hard examples, but not too difficult examples, which is stated in the original paper. In the imbalanced setting of this paper, clean imbalanced data are too difficult to distinguish with the predictions of models (Figure I, right) and therefore are too difficult examples. Our methods aim to make of such examples. Experiments show that such examples are of great importance for generalization.
The introduction of uncertainty in ActiveBias relies on the estimation of the prediction variance but does not consider the bad influence of mislabeled data/outliers this estimation. Our methods use the loss distribution and the side effect of mislabeled data is considered by robust mean estimators.
F RELATED WORK ON SAMPLE SELECTION
In this paper, we focus on the procedure of using the small-loss criterion for sample selection, which very commonly used in learning with noisy labels and shared by the state-of-the-an methods. We review prior effects in this line. Existing methods (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019; Wei et al., 2020; Lyu Tsang, 2020) focus on the class-balanced problem. Based on memorization effect of deep networks, they select examples with small losses for network updates and drop examples with large losses. The main reason that clean examples are more likely to have small losses, but mislabeled examples cases. Distinctly, these methods ignore the clean examples with large losses, following suboptimal generalization. When the training dataset is noisy and extremely imbalanced, prior methods weak. is because since they cannot select clean hard examples within non-dominant classes effectively based losses, which is shown in the main paper. Unfortunately, such examples are critical for generalization. We are the first one to focus on this important problem and tackle by considering the uncertainty of losses.
more likely have large losses in class-balanced
23


Reviews:
[{"review_id": "Qa5TBF_0KGK", "reviewer": "Reviewer_5G3L", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper is well written and easy to follow. \n2. The goal of this paper is well-motivated, with a detailed discussion about the uncertainty of losses in sample selection.\n3. The technical steps are clearly explained with theoretical justification.\n\nCons:\n\nAs shown in Table5, the difference between DivideMix-S/H and the original DivideMix is a little bit small. It will be more convincing if the authors could report the error bar of these experiments.\n", "summary_of_the_paper": "This paper discusses the potential weaknesses in previous sample selection criteria in learning with noisy labels. And then propose a new selection criterion by incorporating the uncertainty of losses, together with theoretical justification. Experiments on both synthetic noisy balanced/imbalanced datasets and real-world noisy datasets validate the effectiveness of the proposed approach.", "summary_of_the_review": "From my perspective, this work studies an important problem in previous sample selection criteria in learning with noisy labels. And the proposed method is clearly explained and theoretical analyses have been provided. Sufficient experiment results validate the effectiveness of this approach, although the results in semi-supervised learning seem not very convincing. Overall, I tend to accept this paper.\n\n\n################# After rebuttal ##################\\\nThanks for the responses. I still tend to accept this paper and keep my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JCdrntPWNd8", "reviewer": "Reviewer_45Q2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed approach is justified theoretically\n- Many datasets and comparison methods used and thus the proposed approach is also validated empirically\n\nWeaknesses\n- I could not find a major weakness in the paper. It would be interesting to see the work extended to other types of data (NLP, Speech...)\n\nTypos\n- 2.3, Soft truncation : « given a random variable X » probably\n", "summary_of_the_paper": "This paper proposes a novel algorithm to improve the sample selection in the case of noisy labels training by incorporating the uncertainty of losses. To select samples, the authors propose to use the lower bounds of the confidence intervals derived from concentration inequalities instead of using point estimation of losses. The authors introduce the Soft truncation estimator and the Hard truncation estimator and propose lower bounds for both estimators. The authors introduce an algorithm, the CNLCU Algorithm based on using either the lower bounds calculated for the Soft truncation estimator or the Hard truncation estimator. \n\nThe authors validate empirically their results on four benchmark datasets (MNIST, F-MNIST, CIFAR-10, CIFAR-100) and use a diverse set of possible noise functions. The method proposed almost always outperforms all other comparable approaches.\n", "summary_of_the_review": "The authors propose a novel algorithm, justified theoretically and validated empirically to improve the sample selection in the case of noisy labels training.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "G7dJJNUlxXB", "reviewer": "Reviewer_7WR9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning with noisy labels is an important issue and has attracted much attention. This paper proposes a novel sample selection method, namely CNLCU, to optimize model training. CNLCU is based on the probabilistic constraint relaxation and can select ‘good’ samples with time intervals.\n\nOverall, the method proposed in the paper is technically sound. The authors provide complete derivations in the appendix, which shows the correctness of the features of the proposed method. The experimental design is reasonable, which follows the conventions in many noisy label learning studies. The comparison with well-selected baselines in different noise levels shows the advantages of the proposed method. The paper is well organized and clearly presented.\n\nWeaknesses:\n- It would better if the noise ratios cover a wider range rather than only 20% and 40% are selected. Usually, evaluations also should be made under heavily noisy situations where noise ratios are large than 50%. This will make the conclusion more convincing.\n- The evaluation matric as well as the analysis should be diverse. CNLCU outperforms in terms of mean accuracy. E.g., if the dataset is imbalanced, the accuracy may be not enough to comprehensively describe the performance. I also notice that on some datasets, the performance of CNLCU appears larger variances, which should be further explained.\n- This paper assumes that the training losses in L_t conform to a Markov process. This assumption may not be always true in reality. The position of W in parameter space depends on the starting points and search paths, but not merely the last positions. Some support evidences are better to be shown. Or, some statistical operations should be taken on the correlation of the W sequence during the training process.\n", "summary_of_the_paper": "This paper proposes a probabilistic-based sample selection approach CNLCU to distinguish whether the training sample is mislabeled or ignored. The experimental results show that the proposed method outperforms most up-to-date sample selection methods.", "summary_of_the_review": "Although some minor issues are better to be further addressed, CNLCU is indeed novel and technically sound. The experimental results show the potential superior performance of CNLCU. Thereby, I rate this paper with marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AkSDp6buVPy", "reviewer": "Reviewer_LJ1a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of label noise + (extreme) class imbalance is an interesting yet under studied direction with a potentially significant impact to many applications (e.g., worst-group generalization, fairness), which the paper does a good job brining attention to. The paper is well written and overall easy to follow, despite that some clarification may help readers better understand the proposed methods (see minor concern below).\n\nSome of my major concerns are:\n\n1. To implement the proposed method, it seems that naively one will need to keep track of the history of losses of every sample in the training set. For soft truncation (Eq. 7), one can simply keep a per-sample running mean and selection times in memory; for hard truncation (Eq. 8), it seems much more complicated. Can authors add some details of the space complexity overhead to use the proposed method? These can be argued as one major disadvantage when compared to other regularization-based LNL methods (Appendix B.2).\n2. The hard truncation criterion is derived based on a Markovian assumption. Does it contradict with some popular tricks training DNN, such as temporal averaging, or momentum? For instance, it seems that all models in the paper are trained using ADAM where momentum is used hence the Markovian assumption is broken. Can the authors please comment?\n3. The experiments mostly are designed with relatively low noise (20% and 40%). Sample selection can suffer typically from high-noise settings and the so-called self-confirmation trap. It would be good to compare the proposed method to other baselines in such settings, since so far all methods perform quite comparably across all datasets and noise types where the proposed method outperforms other slightly by 1-2% (Table 1, 2, 3, 9).\n\nA minor concern/suggestion:\n\n1. The paper only discussed related work very briefly in Sec 3.1 (Baselines). A dedicated related work section should help improve the paper: How do other sample selection consider different reasons for samples having large losses? Have they not considered at all? Any other paper that challenged the small-loss/large-loss assumption in any way?\n2. How is Step 4/5 in Algorithm 1 optimized? At a first glance, this amounts to solving a combinatorial problem in each mini-batch? Perhaps the authors has a better way to do so?\n3. How are hparams (sigma and tau_min) tuned given a *noisy* validation set?", "summary_of_the_paper": "The paper proposes a novel sample selection method for learning with noisy labels (LNL). Based on the typical \"small-loss\" assumption, the motivation is to consider uncertainty about large-loss samples in order to distinguish the two confounding cases: truly mislabeled samples, or clean yet underrepresented samples that are less frequently selected or learned by the model so far. The proposed method explores robust mean estimation to summarize the per-sample loss trajectory, through soft truncation and hard truncation. Concentration inequalities are used to obtain the final selection criteria to perform conservative search. Results are overall promising, in particular showing very good results dealing with label noise + extreme class imbalance (Figure 2).", "summary_of_the_review": "The proposed method is a novel method concerning an interesting yet under studied direction with a potentially significant impact to many applications (e.g., worst-group generalization, fairness), which the paper does a good job brining attention to. Results are overall promising, in particular showing very good results dealing with label noise + extreme class imbalance (Figure 2). Some concerns should be addressed during open discussion with authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4OU9ZIMYz27", "reviewer": "Reviewer_jAuy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: (1) Provide comprehensive analysis on one category of noisy label classification method. (2) experiment dataset is diverse covering many commonly used datasets and setups.\n\nWeakness: (1) the improvement in classification performance is marginal. (2) A critical state-of-the-art is not provided (see below). (2) Experiment results could be improved (need a t-test proving the experiment results are significant). (3) The paper only considers one category of noisy-label classification methods. There are many other categories that may provide better performance than the proposed methods (e.g., early learning, error bounded approach, etc.).\n", "summary_of_the_paper": "The paper performs a comprehensive analysis on a sub-category of noisy-label classification techniques. Following that, the paper proposed a new improved method to handle sample selection-based noisy-label classification techniques.", "summary_of_the_review": "I like this paper. It discusses the weakness of some noisy-label learning methods. The paper also conducts a solid experiment (except missing one critical comparison -- MentorMix). MentorMix is also a sample selection-based approach. It has demonstrated superior performance in dealing with the noisy labels. Because MentorMix uses mixup to enrich data, it should be robust against data imbalance. I would like to see a comparison of MentorMix and the proposed method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript discusses weaknesses in previous sample selection criteria in learning with noisy labels, and proposes a new selection criterion by incorporating the uncertainty of losses, together with theoretical justification. To select samples, the manuscript uses the lower bounds of the confidence intervals derived from concentration inequalities instead of using point estimation of losses. By incorporating uncertainty about large-loss samples, the method is able to distinguish: truly mislabeled samples, or clean yet underrepresented samples that are less frequently selected or learned by the model so far. Experiments are performed on four benchmark datasets (MNIST, F-MNIST, CIFAR-10, CIFAR-100) and use a diverse set of possible noise functions. 

Reviewers agreed on several positive aspects of the manuscript, including:
1. This manuscript addresses weaknesses in previous sample selection methods and potentially impacts various applications (e.g., worst-group generalization and fairness);
2. The technical steps are clearly explained with theoretical justification;
3. Many datasets and comparison methods used and thus the proposed approach is also validated empirically.

Reviewers also highlighted several major concerns, including:
1. The space complexity of the proposed method; to implement the proposed method, it seems that naively one will need to keep track of the history of losses of every sample in the training set;
2. The validity of Markov process assumption on the training losses when momentum is used in optimization. Also, the position of the parameter in parameter space depends on the starting points and search paths, but not merely the last positions.
3. The experiments mostly are designed with relatively low noise (20% and 40%);
4. The evaluation matric as well as the analysis should be diverse. The performance of the proposed approach appears to have larger variances, which should be further explained.

Many of the major concerns have been addressed during the rebuttal including: experiments with 50%, 60%, and 70% symmetric noise, extension to NLP data by showing results on the NEWS dataset, and further discussion on the space complexity and Markov process assumption.
Decision: Accept (Poster)