Label Noise-Robust Learning using a Confidence-Based Sieving Strategy
Authors: Reihaneh Torkzadehmahani
Reza Nasirigerdeh
Daniel Rueckert
Georgios Kaissis
Venue: Accepted by TMLR ()

Abstract: In learning tasks with label noise, improving model robustness against overfitting is a pivotal challenge because the model eventually memorizes labels, including the noisy ones. Identifying the samples with noisy labels and preventing the model from learning them is a promising approach to address this challenge. When training with noisy labels, the per-class confidence scores of the model, represented by the class probabilities, can be reliable criteria for assessing whether the input label is the true label or the corrupted one. In this work, we exploit this observation and propose a novel discriminator metric called confidence error and a sieving strategy called CONFES to differentiate between the clean and noisy samples effectively. We provide theoretical guarantees on the probability of error for our proposed metric. Then, we experimentally illustrate the superior performance of our proposed approach compared to recent studies on various settings, such as synthetic and real-world label noise. Moreover, we show CONFES can be combined with other state-of-the-art approaches, such as Co-teaching and DivideMix to further improve model performance.

Published in Transactions on Machine Learning Research (09/2023)
Label Noise-Robust Learning using Confidence-Based
Sieving Strategy
Reihaneh Torkradehmahani Techmical University of Munich
reihaneh. haa.eeee.errrdddeemmmmeiieeeeee
Reza Nasirigerdeh
reaa.nasirigendeh@@taa.com
Technical Universily of Munich
Helmholtz Munich
Daniel Rueckert Technical University of Munich
Maniel..ueckert&tumm.c
Imperial College London
Georgios Kaissis
g. Blunn de
Technical Universily of Munich
Helmholtz Munich
Reviewed on OpenReview: https: // openreview. net/ forum? id- 3taI004C77
Abstract
In learning tasks with label noise, improving model robustness against. overlitting a pivotal challenge because the model eventually memorizes labels, including the noisy ones. Identifying the samples with noisy labels and preventing the model from learning them is promising approach to address this challenge. When training with noisy labels, the per-class confidence scores of the model, represented by the class probabilities, can be reliable criteria for assessing whether the input label is the true label or the corrupted one. In this work, we exploit this observation and propose a novel discriminator metric called confidence error and a sieving strategy called CONFES to differentiate between the clean and noisy samples effectively. We provide theoretical guarantees on the probability of error for our proposed metric. Then, we experimentally illustrate the superior performance of our proposed approach compared to recent studies various settings, such synthetic and real-world label noise. Moreover, show CONFES can be combined with other state-of-the-art approaches, such as Co-teaching and DivideMix to further improve model performance".
1 Introduction
The superior performance of deep neural networks (DNNs) in numerous application domains, ranging from medical diagnosis (De Fauw et al., 2018; Liu et al., 2019) to autonomous driving Grigorescu et al. (2020) mainly relies on the availability of large-scale and high-quality data (Sabour et al., 2017; Marcus, 2018). Supervised machine learning in particular requires correctly annotated datasets to train highly accurate DNNs. However, such datasets are rarely available in practice due to labeling errors (leading to label noise) stemming from high uncertainty (Beyer et al., 2020) or lack of expertise (Peterson et al., 2019). In medical applications, for instance, there might be a disagreement between the labels assigned by radiology experts and those from the corresponding medical reports (Majkowska et al, 2020; Bernhardt al., 2022), yielding datasots with noisy labels. Hence, it is indispensable to design and develop robust learning algorithms that
"The code available https: //
httpss//github.. httpt/ttttuuegee/// wot.uabb..oo/eeeeeeeeeee http://riaanatttorrrddaaahaaaale

---

Published in Transactions on Machine Learning Research (09/2023)
are able to alleviate the adverse impact of noisy labels during training. Throughout this paper, we will refer to these methods as label noise learning methods.
In the literature, there are different types of label noise including symmetric noise, pairflip noise, and iistance-dependntt noise. In symmetric noise a sample is allocated a random label, while in pairflip noise the label of sample is flipped into the adjacent. label (Patrini et al., 2017; Xia et al., 2020; Bai et al., 2021). In real-world scenarios, a corrupted label assigned to a sample depends on the feature values and the true noise (Liu, 2021; Zhang et al., 2021b). Training DNNs in
label of the sample, known as bustance-dependee the presence of label noise can lead to memorization of noisy labels and consequently, reduction in model generalizability (Zhang et al., 2021a; Chen et al., 2021b).
Some of the existing studies for dealing with label noise focus on learning the noise distribution. Patrini et al. (2017); Berthon et al. (2021); Yao et al. (2021); Xia et al. (2019); Yao et al. (2020) model the noise distribution as a transition matrix, encapsulating the probability of clean labels being flipped into noisy oncs and leverage loss correction to attemuate the elfect of the noisy samples.
Other studies (Cheng et al., 2020; Xia et al., 2021; Wei et al., 2020) learn the cleon label distribution and capitalize on regularization or selection of reliable samples to cope with the noisy labels. A main challenge in this line of work, also known as sample siewing (or sample selection), is to find reliable criterion (or metric) that can efficiently differentiate between clean and noisy samples. The majority of the previous studies (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019) employ the loss value to this end, where the samples with small loss values are considered to likely be clean ones (small-loss trick). A prior study (Zheng et al., 2020) proposes confidence-based criterion and shows the label is likely noisy if the model confidence in that label is low. Our work lies in this category of confidenco-based sieving metrics.
Since learning noise distributions is challenging, it is rarely used in practice. Sample sieving methods, on the other hand, have multiple unsolved problems: They might not always be capable of effectively filtering out noisy labels without supplementary assistance (e.g., additional model in Co-teaching). Morcover, their performance might not be salisfactory in the presence of certain types of noises such as instance-dependent or higher levels of noise. This motivates us to develop new metrics and learning algorithms that are more robust against various types and levels of label noise with minimal additional computational overhead.
Contributions. Our main contributions can be summarized as follows:
We introduce novel metric called confidence error to efficiently discriminate between clean and noisy labels. The confidence error metric is defined as the difference between the softmax outputs/ /logits of the predicted and original label of a sample. Moreover, we provide theoretical bound on the probability of error for the proposed metric. Our theoretical analysis and observations indicate there exists a clear correlation between the confidence error value and the probability of being clean. That is, a sample with a lower confidence error has much higher probability to be clean sample than a noisy one.
We then integrate the confidence error criterion into learning algorithm called CONFidence Error Sieving (CONFES) to robustly train DNNs in the instance-dependent, symmetric, and pairflip label noise settings. The CONFES algorithm computes the confidence error associated with training samples at the beginning of cach epoch and only incorporates a subset of training samples with lower confidence error values during training (i.e., likely clean samples).
We validate our findings experimentally showing that CONFES significantly outperforms the state- of-the-art learning algorithms in terms of accuracy on typical benchmark datasets for label noise learning including CIFAR-10/ 10/1-110110 (Krizhevsky et al., 2009) and Clothing (Xiao et al,, 2015). The superiority of CONFES becomes particularly pronounced in scenarios where the noise level is high or when dealing with more intricate forms of noise such as instance-dependent noise.
We moreover demonstrate that combining CONFES with other learning algorithms including Co- teaching (Han et al., 2018), JoCor (Wei et al., 2020), and DivideMix (Li et al., 2020) provides further accuracy gain, illustrating synergy between CONFES and the existing research endeavors in the field of learning with label noise.
2

---

Published in Transactions on Machine Learning Research (09/2023)
2
Related Work
Overcoming the memorization of noisy labels plays a crucial role in label noise learning and improves model generalization by making the training process more robust to label noise (Zhang et al., 2021a; Arpit et al., 2017; Natarajan et al., 2013). The research community mainly tackled the memorization problem by adjusting the loss function (known as loss correction), using coppicit/erplica regularization techniques, or refining the training data and performing sample sieving.
Adjusting the loss function according to the noise transition probabilities is an effective method for decreasing the adverse impact of noisy samples during the training but comes at the cost of accurate estimation of the transition matrix (Patrini et al., 2017). Previous studies have paved the way for this non-trivial estimation in different ways. For instance, T-Revision (Xia et al., 2019) estimates the transition matrix without requiring anchor points (the data points whose associated class is known almost surely), which play an important role in the effective learning of the transition matrix. Dual-T (Yao et al., 2020) first divides the transition matrix into two matrices that are easier to estimate and then aggregates their outputs for a more accurate estimation of the original transition matrix.
Another line of work improves model generalization by introducing regularization effects suitable for learning with noisy labels. The regularization effect may be injected implicitly using methods such as data angmentation and inducing stochasticity. For example, Mirup (Zhang et al., 2018) augments the training data using a convex combination of a pair of examples and the corresponding labels to encourage the model to learn a simple interpolation between the samples. SLN (Stochastic Label Noise) (Chen et al., 2021a) introduces a controllable noise to help the optimizer skip sharp minima in the optimization landscape.
Although the implicit regularization techniques have been proven effective in alleviating overfitting and improving generalization, they are insufficient to tackle the label noise challenge (Song al., 2022). Thus, the community came up with explicit regularization approaches such as ELR (Early-Learning Regularization) (Liu et al., 2020) and CDR (Xia et al., 2021). ELR is based on the observation that at the beginning of training, there is an early-learning phase in which the model learns the clean samples without overfitting the noisy ones. Given that, ELR adds a regularization term to the Cross-Entropy (CE) loss, leading the model output toward its own (correct) predictions at the early-learning phase. Similarly, CDR first groups the model parameters into critical and non-critical in terms of their importance for generalization and then penalizes the non-critical parameters.
A completely different line of work is sample sieving/selection which aims to differentiate the clean samples from the noisy ones and employ only the clean samples in the training process. The previous works in this direction exploit loss- based or confidence- based metrics as the sample sieving criteria. MentorNet (Jiang et al., 2018) uses an extra pre-trained model (mentor) to help the main model (student) by providing it with small-loss samples. The decoupling algorithm (Malach & Shalev-Shwarlu, 2017) trains two networks simultaneously using the samples on which the models disagree about the predicted label. Co-teaching (Han et al., 2018) tross-trains two models such that each of them leverages the samples with small-loss values according to the other model. Co-teaching+ (Yu et al., 2019) improves Co-teaching by considering clean samples as those that not only have small loss but also those on which the models disagree. JoCoR (Wei et al., 2020) first computes a joint-loss to make the outputs of the two models become closer, and then it considers the samples with small loss as clean samples. The utilization of two models in MentorNet, decoupling, Co-teaching, Co-teaching+, and JoCoR leads to a computational inefficiency that is twice as high compared to traditional training methods. LRT (Zheng et al., 2020) employs the likelihood ratio between the model confidence in the original label and its own predicted label and then selects the samples according to their likelihood ratio values. Our work is closely related to LRT as both employ a confidence- based metric for sample sicving. However, our metric is an absolute metric that captures the difference between the model's confidence in the given label and the predicted label. In contrast, LRT is a relative metric that is more sensitive to the model's quality.
Our study belongs to the category of sample selection methods and capitalizes on model confidence to discriminate between the clean and noisy samples akin to LRT, without the need for training an additional model as required by methods like Co-teaching.
3

---

Published in Transactions on Machine Learning Research (09/2023)
3 CONFES: CONFidence Error based Sieving
We first provide brief background on the training process for the classification task. Then, we introduce the proposed confidence error metric and provide a theoretical analysis of its probability of error. Afterward, we present the CONFES algorithm, which capitalizes on confidence error for effective sample sieving.
3.1 Background
We assume classification task on a training dataset = [ E X, Y) where the number of samples and X and Y are the feature and label (class) space, respectively. The neural network model F(Xs: 0) E Rm*k is k-class classifier with trainable parameters 0 that takes mini-batches batches Xo of size input. In real life, a sample might be assigned the wrong label (e.g., due to human error). Consequently, clean (noise-free label) training datasets might not be available in practice. Given that, we assume y = { and D = (1, /i= indicate the noisy labels and noisy datasct, respectively. The training process is conducted by minimizing the empirical loss (e.g,, cross-entropy) using mini-batches of samples from the noisy dataset:
m min ((F(Xi;)) 0); Po) = min 1 m Σ 0),
(1)
where L the loss function and (X,, Y%) is a mini-batch of samples with size m from the noisy datasel D. Table provides a comprehensive summary of all the notations used in the theoretical analysis, along with their respective definitions.
Table 1: Summary of notations and their definitions
Notation
Defesition
Notation
Defenition
Sample i with features and possibly noisy label &n
8/
Predicted label for a semple i Total mumber of classes/labele
Total number of samples
k
F(:0)
A classifier with weights 0 Softmax activation function
H"(-)
The optimal Bayes classifier
a()
C1O
Model coufidence label I
PA()
True conditional proisability for label Loes function (e-g., crues entrupy)
Pイ(-
Nuisy conditional probability for label
2(-)
TV
The probability that label I flipped label j
The best prediction Baye's optimal classifier Sieving threshold
The second-best prediction of Baye's optimal classifier Coufidence error for sample
Ec()
Maximum approcimstion error of the classifier code///sympootic notation (the upper bound of complexity)
A placeholder variable
0
8,
Tsybakov noise condition variables e (0. 1), A7> 0)
In the presence of label noise, the efficiency of the training process mainly depends on the capability of the model to distinguish between clean and noisy labels and to diminish the impact of noisy ones on the training process. this study, we propose an elegant metric called confidence error for efficient sieving of the samples during training.
3.2 Confidence Error as the Sieving Metric
Consider sample (z,, 9i) from the noisy dataset D. The k-class/labé classifier F(x;; 8) takes x; as input and computes the weight value associated with each class as output. Moreover, assume a(.) is the softmax activation function such that (F((,;; 0)) [0, 1 takes classifier's output and computes the predicted probability for each class. We define the model confidence for a given label I E (1,..., k} associated with sample as the prediction probability assigned to the label:
( -0 (FFx; 0))
(2)
The class with the maximum probability is considered as the predicted class, i.e. vh. for sample s: y arg max g(d) (Fzx;
(3)
jE(1, ..., kb
The confidence error Eo(s) for sample s is defined as the difference between the probability assigned to the predicted label o and the probability associated with the original label ū::
Ee(s) ((v)) ((O)
(4)
4

---

Published in Transactions on Machine Learning Research (09/2023)
where (s) E [0. 11. In other words, the confidence error states how much the model confidence in the original class is far from the model confidence in the predicted class. The confidence error of zero implies that the original and predicted classes are the same.
3.3 Probability of Error
In the following, we thcorctically prove the probability that confidence error wrongly identifies noisy labels as clean ones and vice versa is bounded. Presume H* is a Bayes optimal classifier that predicts the correct label according to the true conditional probability (x) = Pr(y = j|æ]]- Consider v= (x) = arg max, (x as the H' best prediction and w arg m Pa(x) as its second best prediction. Define P5(x) as the noisy conditional probability, and the maximum approximation error of the classifier F:
k p²)x)= = Prlū = - = Prlj
h
Σ Prlp - - P((x) = Tt Pi(x), = max ag
[ (c0)-Ps(w)
1=1
where Ny represents the probability that the label I is flipped to label j. Presume the true conditional probability P meets the multi-class Tsybakov noise condition (Zheng et al., 2020), which guarantees the presence of margin (region of uncertainty) around the decision boundary separating different classes. This implies that the true conditional probabilities are sufficiently apart, and there is a reasonable level of distinguishability between the classes.
Lemma 1. Given the true conditional probability P satisfying the multi-class Tsybakov noise condition, there exists a = min min, Tyg (x) + Enps Ty P((x)) such that
Pr j = H (z), (00 (x) < < (0(0(e))
(5)
for constants μ E (0. 1], B,7 0, and < pmmin Tzp-
Proof. The proof can be found Zheng ct al. (2020).
In simple terms, Lemma 1 states that if a label is noisy and the model confidence in that label low, the label has limited probability of being correct. The probability of correctness is determined by e, the maximum approximation error for the model, which tends to be small in practical scenarios (Zheng et al., 2020). In other words, Lemma implics that the error bound for model confidence is small in practice.
Now, we provide the error bound for our proposed metric. We consider two possible error cases: (I) the label is noisy according to the optimal Bayes classifier H but our metric recognizes it as clean, and (II) the label is clean based on H", but our metric identifies as noisy. In the following theorem, we show the probability of making any of these two errors is bounded.
Theorem Given that the true conditional probability P satisfics the multi-class Tsybakov noise condition for constants μ E (0, 1], B,7 0, and < fr miny Tjj, we have:
(0) Tyy Pull)) Euso Tyy P((x)) ). then: Case (I): Let the threshold max {
Pr + H1) (x), Eo 7) < s 3((c)) + t..
(6)
Case (II): Let the threshold a = min { (o - Typ Pm
Σ Cuzo Fig Pix((())) }. then:
Eo 1 Bo)
al (
(7)
Proof. In Case (I), the predicted label is either the same as the Bayes optimal classifier's not:
Pr [g + 11 (x), (o( Ø) a Pr + (x) Eo(a, 0) < a, (x)x) = v'I+
(8)
Pr + H" (x), Ec 9) S a, H² +
5

---

Published in Transactions on Machine Learning Research (09/2023)
Simplifying the terms and using the definition of H (2) yields:
Pr + (z), 0) s a] - Pr(Pw(()) S { sclPptoooo
(9)
By substituting the definition of confidence error, we have:
Pr Pr + (2), al Pr (Pu(t) (
s aa +
(10)
Then, we set Pr [7 o U, + v/ v and substitute (x) with p.r (x) -e based on the definition of maximum approximation error:
Pr L + 11 (z), Eo S aa S Pr P00 (x) +
(11)
Next, we expand the pyr term:
O + H (x). (c(( sals
( Po)
Eny
Thy P((a)) sa 100+000000 (x) +E + +
(12)
and simplify the resulting inequality as follows:
Pr (H), Ec(r, 0) s a S.PP Pr PP (10 + Eito Tly * Pi(xt)
+ (13)
Tyylv'
TyU
Then, we substitute the defined threshold a:
Pr H)), 1) S Sa S Pr
S +
+
(14)
Ty's
Utilizing the multi-class Tsybakov noise condition completes the proof for the first case:
e
Pr (x), Ø) S al a 8
+ +
(15)
To'''
Similarly, we calculate the bound for Case (II) as follows:
Pr [ū H² Ec(a.))
> Pr H² Ec(a, (x) +'I+
(16)
Pr [ (x), Ec(x, > a, (x) y v')
Upon simplifying the terms and substituting the definition of confidence error, we have:
Pr [r (1), aa
(17)
Pr { 0 a + Pr -=) = (00) =a]
Substituting the definition of yields the following expression:
Pr j = H'(xx,) Ec(a, 0) > a) = Pr (P((x) & -a Pz(x) - c] +0 (18)
(80)
Then, we expand the Ps term,
Pr j = (x), Ec(x, 1) > 2
(19)
Pr
Pu (x) s Pp(c)), (v') - ( Pg())( + Σ
Tig * Pi(x) -
17

---

Published in Transactions on Machine Learning Research (09/2023)
and simplify it as follows:
(i)'(())
-Ei4g Tig PP(())
Pr 5 - (2)) > a] s Pr Pw(x) Pull)s Pp(()) s
(20)
TW
Substituting the threshold a based on its definition and employing the multi-class Tiybakov noise condition results in the following inequality:
7
Pr 1- Ec(x, > af S
(21)
T0g
which completes the proof of Theorem 1.
This theorem establishes that the probability of error in distinguishing between clean and noisy samples is bounded if we utilize the confidence error as the sieving criterion. the following, we present CONFES, which capitalizes on the confidence error metric for sieving the samples label noise scenarios.
3.4 CONFES algorithm
Previous studies (Bai et al., 2021; Liu et al., 2020) show that deep neural networks tend to memorize noisy samples, which can have a detrimental effect on the model utility. Therefore, it crucial to detect the noisy samples and alleviate their adverse impact, especially in the early steps of training. The CONFES algorithm takes this into consideration by sieving the training samples using the confidence error metric and completely excluding the identified noisy samples during training. CONFES (Algorithm 1) consists of three main steps at each epoch: (1) Sieving samples, (2) building the refined training set, and (3) training the model.
Algorithm 1: Confidence error based sieving (CONFES)
Input: Noisy training dataset D (z,, }<) model Fø, number of training epochs T, initial sieving threshold a, number of warm-up epochs Tw, batch size m
Output: Trained model Fe
for i = 0..... T P-1 do
2
a; = max (a- - i- Twn 0) /+ Set sieving threshold
D = {s D Es(s) < aa /* Compute confidence error using equation sieve samples
siz....,,ize(( size(Di)) / Build new dataset (cttssttcclam(((uplliaaaa for mini-batch 3 {( 95)) D2 do
D = DD=Deen a [(I,, ūj) E Dj s.l.
/+ Train the model on new dataset
6
Update model Fø on mini batch 3 using equation
return Trained model Fo
In the sieving step, the confidence error for each training sample is computed using Equation 4; then, the samples whose confidence error is less than or equal to a, (sieving threshold at epoch i) are considered as clean, whereas the remaining samples are assumed to be noisy and excluded from training. The per-epoch sieving threshold a, computed using two hyper-parameters:: the initial sieving threshold a, and the number of warm-up epochs Tw,, where ag is lincarly reduced from a zero during Tu warm-up epochs. The idea of adaptive sieving threshold a; is based on the observation that generalization occurs in the initial epochs of training, while memorization gradually unfolds afterward (Stephenson et al., 2021; Liu et al., 2020). We capitalize on the warm-up mechanism using an adaptive sieving threshold by training the model on carefully selected subset of samples, which are potentially clean labels according to their confidence error values, instead of using all samples from the beginming of the training, laying a solid foundation for the learning process.
In the second step, a new training dataset is created by concatenating (0) only the identified clean samples and their augmentations (duplicates) such that this dataset becomes as large as the initial training set. This is based on the fact that sieving the clean samples results in reduction in the number of training samples
7

---

Published in Transactions on Machine Learning Research (09/2023)
due to the exclusion of noisy samples. Duplication of the clean samples accounts for this reduction and emphasizes learning the potentially clean samples. Moreover, in line with a previous study (Carlini et al., 2023), which indicates duplication is a very strong promoter of learning, duplicating clean samples produces a very strong learning signal which improves the algorithm overall. Finally, the model is trained on this augmented dataset.
4 Evaluation
We draw a performance comparison between CONFES and recent. baseline approaches on three label noise settings: symmetric, pairflip, and instance-dependent. In the following, we first describe the experimental setting and then present and discuss the comparison results. Moreover, we provide additional results regarding the effectiveness of the confidence error metric and CONFES algorithm in sieving the samples as well as the sensitivity of CONFESS to its hyper-paramettrs.
4.1 Experimental Setup
Datasets. We utilize the CIFAR-10/100 datasets (Krizhevsky al., 2009) and make them noisy using different types of synthetic label noise. Furthermore, we incorporate the Clothing dataset (Xiao al., 2015), a naturally noisy benchmark datasct widely employod in previous studies. CIFAR-10/100 100000 contain 50000 training samples and 10000 lesting samples of shape 32 x 32 from 10/100 classes. For the CIFAR datasets, label noise introduced in Xia
we perturb the training labels using symmetric, pairflip, and instance-dependee et al. (2020), but keep the test set clean. Following data augmentation, ppprppootttio/ppppprooossngggg procodure in previous works (Liu et al., 2020; Li et al., 2020), the training samples are horizontally flipped with probability 0.5, randomly cropped with size 32 32 and padding 4 4, and normalized using the mean and standard deviation of the dataset. ClothingIM (whose estimated noise level is approximately 38% (Wei et al., 2022; Song et al., 2019)) and 10k clean test images from 14 classes. Following prior studies (Liu el al., 2020; Li et al., 2020), the data augmentation steps performed on the clothingIM dataset include 256 x 256 resizing, 224 x 224 random cropping, and random horizontal flipping. In clothing M,, the number of samples for each class is imbalanced. We follow Li et al. (2020) and sample a class- balanced subset of the training dataset at each epoch.
real-world dataset of 1 million images of size 224 x 224 with noisy labels
State-of-the-art methods. On all considered datasets, we compare CONFES with the most recent related studies including (1) standard cross-entropy loss (CE), (2) Co-teaching (Han et al., 2018) that cross-trains trains two models and uses the small-loss trick for selecting clean samples and exchanges them between the two models, (3) ELR (Liu et al., 2020), an early-learning regularization method that leverages the model output during the early-learning phase, (4) CORES² (Cheng et al., 2020), a sample sieving approach that uses confidence regularization which leads the model towards having more confident predictions, (5) PES (Bai et al., 2021), a progressive early-stopping strategy, (6) SLN (Chen et al., 2021a) that improves regularization by introducing stochastic label noise, and (7) LRT, confidence based algorithm that leverages likelihood ratio values for sample selection. Co-teaching, CORES?, and LRT based on sample sicving, whereas ELR and SLN are regularization--osse methods. For all methods, the specific Hyper-paraneest are set according to the corresponding manuscript or the published source code if available.
Parameter Settings and Computational Resources. We conduct the experiments on a single GPU system equipped with an NVIDIA RTX A6000 graphic processor and 48GB of GPU memory. Our method is implemented in PyTorch v1.9. For all methods, epochs, and for co-teaching, works (Li et al., 2020; Bai et al., 2021), we train the PreActResNet-188 (He et al., 2016) model on CIFAR-10 and CIFAR-100 using the SGD optimizer with momentun of 0.9, weight decay of 5e-4, and batch size of 128. The initial learning rate is set to 0.02, which is decreased by 0.01 in 300 epochs using cosine annealing schoduler (Loshchilov & Hutter, 2017). For the ClotingIM dataset, we adopt the setting from Li ct al. (2020) and train the R0 model for 80 epochs. The optimizer is SGD with momentum of 0.9 and weight decay of 1e-3. The inl learning rate is 0.002, which is reduced by factor of 10 at epoch 40. At each
evaluate the average test accuracy on the last five report the average of this metric for the two networks. Following previous
8

---

Published in Transactions on Machine Learning Research (09/2023)
epoch, the model is trained on 1000 mini-batcher of size 32. Note that ResNet-50 has been pretrained on ImageNet (Deng et al., 2009).
4.2 Results
CIFAR-10/100 datasets. Tables 2 and 3 list lest accuracy values for different noise types and noise rates on CIFAR-10 and CIFAR-100 datasets respectively. According to these tables, CONFES outperforms the competitors for all considered symmetric, pairflip, and instance-dependent noise types. Similarly, CONFES delivers higher accuracy than the competitors for different noise rates. Moreover, as the noise level increases, the accuracy gap between CONFES and its competitors widens in favor of CONFES. Figure 1 illustraaes the test accuracy versus epoch for the different learning algorithms. As shown in the figure, CONFES is robust against overfitting because the corresponding test accuracy continues to increase as training moves forward, and stays at the maximum after the model converges. Some of the other algorithms such as SLN and ELR, on the other hand, suffer from the overfitting problem, where their final accuracy values are lower than the maximum accuracy they achieve.
Co-teaching
CORES
PES
SNL
MentorMix
CONFES(ours)
0.7
0.7
0.6-
0.6
0.6
Test sccusry
0.5
0.5
0.5
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0
50
100
150 Epoch
200
250
300
0
100
150 Epoch
200
250
300
50
100
150
200
250
300
Epoch
(a) Instaene-dependent
(b) Pairflip
(c) Symmetric
Figure Test accuracy for Res trained on CIFAR-100: CONFES is robust against overfitting, whereas some competitors including SLN and ELR suffer from overfitting: noise level 40%.
Table 2: Test accuracy on CIFAR-10 for different noise types with noise level 40%.
Method
Symmetric
Pairflip
Instance
CONFES (ours)
90.62=0.2
86.18=0.3
90.28=0.2
CE
66.61 20.4
59.25
40.1
66.04 =0.2
Co-teaching (Han et al., 2018)
87.42 =0.2
84.57
20.2
86 06.9010.1
ELR (Liu et al., 2020)
85.74 10.2
86.15
10.1
85.37 -.3
CORES (Cheng et al., 2020)
83.90 =0.4
58.38
#0.6
76.71 =0.4
LRT (Zheng al., 2020)
85.47 10.3
59.25
10.3
80.53
PTD (Xia et al., 2020)
72.05 +0.9
58.34
+0.8
65.97
=0.9
PES (Bai et al., 2021)
90.55 +0.1
85.56
+0.1
85.63
=0.5
SLN (Chen et al., 2021a)
83.69 10.2
85,26
10.5
67.71
20.4
Combining CONFES with state-of-the-art algorithms. Table 4 shows the accuracy values of CeTchh-- ing, JoCor, and DivideMix if confidence error is used the discriminator metric instead of the training loss. As shown in the table, the accuracy from these algorithms is enhanced by 2-5% compared to their baseline performance by combining them with CONFES, indicating that confidence error is not only effective as the main building block of the proposed CONFES algorithm but also combined with other state-of the art methods including DivideMix, which a complex method employing data augmentation, and guessing or refining the noisy labels rather than excluding them, which helps in utilizing the noisy samples and learning their feature information.
9

---

Published in Transactions on Machine Learning Research (09/2023)
Table 3: Test accuracy on C1EAR-100 for various label noise types with different noise rates.
(a) Iustanco-dependent
Method
20%
40%
60%
CONFES (ours)
73.5910.2
69.68.02
59.48.10.1
CE
63.16 +0.1
48.92
+0.3
30.65
+0.4
Co-tesching (Hau al., 2018)
71.12 10.3
66.55
1.03
57.18
10.2
ELR (Liu et al., 2020)
63.10 =0.2
49.15
+0.2
29.88
+006
CORES (Cheng et al., 2020)
64.55 #0.1
50.98
0.22
33.93
+0.5
LRT (Zheng et sl., 2020)
73.14 10.2
65.32
10.6
43.37
10.1
MentorMix (Jiang et al., 2020)
69.41 10.2
56.41
40.1
34.61
#0.1
PES (Rai et al., 2021)
71.65 +0.33
64.83
+0.2
41.10
+0.5
SLN (Chen et al., 2021a)
60.08 +:0.1
46.08
0.3.
29.77
+0.4
(b) Pairfiip
Method
20%
30%
CONFES (uurs)
73.12::0.1
7..
62.37:10.44
CE
64.31 10.3
35.77.10.1
45.62 10.4
Co-tesching (Han al., 2018)
69.59 +002
64.04
+0.4
35.42
+0.5
ELR (Láu al., 2020)
62.05 +0.5
54.44
+0.2
44.31 +0.3
CORES (Cheng et al., 2020)
63.85 +0.2
54.88
40.3
45.34=0.2
LET (Zheng et al., 2020)
71.70 =0.1
60.78
10.1
46.24
+022
MentorMix (Jinng et al., 2020)
69.65 10.1
62.01
10.1
50.97
10.2
PES (Bai et al., 2021)
71.73 10.4
68.28
10.3
39.18
10.2
SLN (Chen et al., 2021a)
61.82 +0.3
53.67
+0.2
45.72
+022
(c) Symmetric
Method
20%
40%
60%
CONFES (ours)
73.89-0.11
69.63.10.2
60.65.10.1
CE
63.46 +00.7
47.85
+..4
29.59
+-0.3
Co-teuching (Hau et al., 2018)
71.54 10.3
66.26
10.1
58.82
1001
ELR (Liu al., 2020)
63.59
48.33
#0.2
30.37
#0.1
CORES (Cheng al., 2020)
65.99 =0.5
52.26
+0.2
34.61
+0.2
LRT (Zheng et al., 2020)
73.72 10.1
66.52
10.2
50.86
10.4
MentorMix (Jiang et al., 2020)
71.52 10.2
61.96
10.2
44.38
10.3
PES (Bai el al., 2021)
71.42 +0.2
68.37
+0.2
60.38
+0.1
SLN (Chen et al., 2021a)
60.48 #0.1
46.98
+0.2
28.50 #0.2
Table 4: Test accuracy for CONFES combined with other approaches on CIFAR-100 with noise rate 40%.
Method
Symmetric
Pairflip
Instance
Co-teaching (Han et al., 2018)
66.26 +0011
35.42 +0.5
66.55:10.3
CONFES-Co-teseee
69.94=0.1
57.90+102
69.51=0.1
Improvement
+3.68
+2.48
+22.6
DivideMix (Li et al., 2020)
74.6310.2
74.910.1
66. 79.03.
CONFFS-DivideMiss
76.31+0.2
76.51+0011
69.03+0.1
Improvement
+1.68
+1.61
+2.24
JoCoR (Wei et al., 2020)
67.05.0.2
54.96.10.3
67. 67.2
CONFES-JoCoR
70.48=0.2
59.61+0.1
70.24=0.4
Improvement
+3.43
+4.65
+2.78
ClothingIM datasct. Table summarisos the performance of mothods on Clothing datasct. CORES² and PES provide slight or no accuracy gain compared to the baseline cross-entropy training. CONFES, on the other hand, outperforms the competitors including ELR and SLN.
Table 5: Test accuracy on Clothing dataset
Method
CE
ELR
CORES
PES
SLN
CONFES (owrs
Test accuracy
69. 89.21%
71.39%
69.50%
69.18%
72.80%
73.24%
10

---

Published in Transactions on Machine Learning Research (09/2023)
Effectiveness of confidence error. We design experiment to illustrate the effectiveness of the confidence error metric: we employ the SGD optimizer and cross-entropy loss function to train ResAetResNet on CIFAR-100, where 40% of the labels are made noisy using the instance-dependent noise. At the beginning of each epoch, the model computes the confidence error for all training samples and sorts them in ascending order by the confidence error value. The model considers the first 60% of the samples with lower confidence error values as clean and only incorporates them during training. This procedure repeated for 200 epochs. As shown in Figure 2, the distribution of confidence error values for the clean and noisy samples becomes more and more dissimilar as the training process proceeds. For instance, at epoch 50, sample with a high confidence error (e.g. near 1.0) is much more likely to be a noisy sample than clean one. Likewise, a sample with very low confidence error is probably clean. The extensions of this experiment to pairflip and symmetric label noise are available at Figures 6-7 in the Appendix. These observations are highly consistent with the theoretical analysis from Theorem 1, which states the probability of error (identifying noisy labels wrongly as clean and vice versa) for the confidence error metric bounded and low in practice.
30
Probability density
6
Clean
Noisy
10
Clean
Noisy
Noisy
Clean
20
4-
5
10
2-
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Confidence error
Confidence error
Confidence error
(a) Epoch 10
(b) Epoch 50
(c) Epoch 200
Figure 2: Distributions of confidence error values for clean and noisy samples progressively diverge from cach other as the training process continues. The experiment is conducted using PreAct-ResNeti and CIFAR-100 with noise level of 40%.
Why CONFES? We use previous experimental sctup and train the model with the naive cross-entropy method and CONFES algorithm to answer this question. Figures 3a and 3b show the model confidence for the noisy, clean, and predicted labels (averaged over the corresponding samples) with cross-entropy and CONFES, respectively. According to Figure 3a, the confidence over noisy labels is very low at the early stages of cross-entropy training. However, as the training proceeds, the model's confidence in noisy labols increascs. At the end of the training, the model confidence over predicted and noisy labels is close each other. This indicates that the model has been misled by the noisy samples, wrongly considering them as the true labels the other hand, utilizes the model confidence error to distinguish between clean
of the samples. CONFES, and noisy samples and exclude the identified noisy samples during training. This results in consistently low confidence for the noisy samples, but high confidence in clean and predicted labels throughout all training stages, as shown in Figure 3b. This observation shows the importance of identifying noisy samples efficiently and keeping confidence in them as low as possible as performed by the CONFES algorithm.
Furthermore, we employ the CONFES algorithm the same setting as Figures and 3 to calculate the confusion matrix and empirically examine the error made by CONFES in differentiating between the clean and noisy labels in practice. Figure 4 shows the confusion matrix for the CONFES algorithm. According to the figure, CONFES is effective in recognizing the noisy samples from the beginning to the end of the training, where it correctly identifies around 38% out of 40% of noisy samples. On the other hand, the algorithm wrongly identifies many clean samples as noisy in the early epochs (around 27%). However, as training moves forward, CONFES becomes more and more efficient in identifying the clean samples, where it correctly recognizes around 55% out of 60% of the clean samples the last epoch. Figure the Appendix, moreover, visualizes the number of clean and noisy samples that CONFES identifies correctly for different. noise types and noise rates, which are consistent with those from confusion matrices.
11

---

Published in Transactions on Machine Learning Research (09/2023)
Predicted label
Noisy label
Clean label
1.0
1.0
Average confidence
Average confidence
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
100
150
200
50
100
150
200
Epochs
Epochs
(a) Cross-entropy
(b) CONFES
Figure 3: Effectiveness of CONFES: Using naive cross-entropy training (a), the model confidence over noisy labels increases the training moves forward. implies that the model is misled by the noisy samples. CONFES (b), however, differentiates the noisy samples from the clean ones and excludes the identified noisy samples during training. This leads very low model confidence over the noisy samples but high confidence in clean and predicted labels throughout training.
Nolsy
True Nolsy 38.81%
False Clean 1.00%
Noisy
True Nolsy 39.03%
False Clean 0.78%
Noisy
True Noisy 38.7 22%
False Clean 1.09%
Nolsy
True Nolsy 38.05%
False Clean 1.77%
Cle
Nolsy
Clean
Nolsy
Cle an
False 5.22%
27.01%
33.18%
Noisy
Clean
Noisy
Clean
Noisy
Clean
Noisy
Clean
Accuracy=71.995
Accuracy=84.68%
Accuracy=88.75%
Accuracy=93.cm
(a) Epoch 10
(b) Epoch 50
(c) Epoch 100
(d) Epoch 200
Figure 4: Confusion matrix for the CONFES algorithm: In early epochs, CONFES correctly identifies the majority of noisy labels (around 38% out of 40%), but wrongly identifies many clean labels as noisy ones (about 27%). As training proceeds, the algorithm not only still remains effective in identifying the noisy labels (around 38% out of 40%) but also correctly recognizes the clean labels (about 55% out of 60%). The model Nest1tResNet18 trained on CIFAR-100 with instance-dependant label noise of rate 40%.
Sensitivity analysis of hyper-parameterss. The initial sieving threshold a and number of warm-up epochs Tw are the hyper-parameter of the proposed CONFES algorithun The per-epoch sieving threshold is computed using the aforementioned hyper paper-parameter... For CIFAR-100, we set a=0.2 and T,,=30 for noise types and noise rates. For CIFAR-10, the values of a and Tw are 0.1 and 25, respectively, for symmetric and instance-dependent noise types. For Clothing1M, and T., are set 0.05 and 3, respectively. We also investigate the sensitivity of CONFES to its syper-parameters using the CIFAR-100 dataset with noise rate of 40% for symmetrie, instance-dependentt, and pairflip noise settings. To analyze the sensitivity T.., set a = 0.2 and use four different values for warm-up epochs: Te E {5, 20, 30, 50). Similarly, we set T... 30 and employ four different values for sieving threshold: a E {0.1, 0.3, 0.5). As shown in Figure 5, the accuracy reductions using the suboptimal hyper T, = 30) are 1.6%, 2.3% and 4.1% for symmetric, instanco-dependent, and pairflip noise settings, respectively, at the worst case. This indicates that CONFES is relatively robust against hyper-parameter value choices, making it easy to employ tune by practitioners.
values compared to the optimal setting (a = 0.2 and
12

---

Published in Transactions on Machine Learning Research (09/2023)
0.70
0.70
Test Test accuracy
0.65
0.65
0.65
0.60
0.60
0.60
0.55
10
20
30
40
50
0.55
10
20
30
40
50
0.55
10
20
30
40
50
Warm-up epochs
Warm-up epochs
Warm-up epochs
(a) Symmetric (a = 0.2)
(b) Instance-deeendent( = 0.2)
(c) Pairflip(a = 0.2)
0.70
0.70
0.70
Test Test accuracy
0.65
0.65
0.65
0.60
0.60
0.60
0.55
0.55
0.55
0.2
0.3
0.4
0.5
0.5
0.3 0.4
0.5
Initial sieving threshold
Initial sieving threshold
Initial sieving threshold
(d) Symmetric(II. 30)
Instance-dependeut(Z. 30)
(I) Pairflip(T'. 30)
Figure 5: Sensitivity analysis of CONFES to its hyper-parameters (number of warm-up epochs)
and a (initial sieving threshold) for different noise types. The dataset is CIFAR-100 with a noise rate of 40%.
5 Discussion and Conclusion
We present the confidence error metric to effectively discriminate between noisy and clean samples in label noise learning settings. Moreover, we theoretically prove the probability of error for the proposed metric is bounded and experimentally show it is small in practice. We integrate the confidence error metric into a learning algorithm called CONFES, which refines the training samples by keeping only the identified clean samples and filtering out the noisy ones. Our experimental results verify the robustness of CONFES under different noise types such symmetric, pairflip, and instanco-dependent, especially when noise levels are high. We also demonstrate that confidence error be employed by other algorithms including Co-teaching and Mix to further improve the model accuracy.
CONFES versus baseline methods. According to the experimental results, CONFES outperforms all baseline methods in the considered symmetric, pairflip, and instant-dependent noise settings. As the noise rate increases, the efficiency of the CONFES algorithm becomes more apparent (c.g., noise rate of 60% in CIFAR-100). Moreover, CONFES is robust to overfitting unlike some of its competitors such as SLN and ELR. Tn terms of computational overhead, CONFES has one additional forward pass for constructing the refined dataset, which only includes clean samples according to the confidence error metric. However, methods such as Co-teaching (Han et al., 2018) employ two networks the training process, which makes them substantially less computationally efficient compared to our approach.
Although some methods such as PES (Bai et al., 2021) perform well in the presence of symmetric label noise, their accuracy decreases in more complex noise settings such as instance-dependent, which is not the case for CONFES. Additionally, the accuracy of some other baseline methods such LRT (Zheng et al., 2020) and MentorMix (Jiang et al., 2020) drastically reduces in a highly noisy setting (c.g., with 60% noise rate). Approaches such as ELR (Liu et al., 2020) and PES (Bai el al., 2021) work well for 'easy to classify' datasets such as CIFAR-10, but their efficiency reduces on more challenging datasets including CIFAR-100 and Clothing CONFES, on the other hand, outperforms the compared basclines different noise types (symmetric, instanco-dependent, and pairflip), with various noise levels (i.e., 20%, 40% and 60%), and on CIFAR-10, CIFAR-100 and the challenging Clothing1M datasets.
13

---

Published in Transactions on Machine Learning Research (09/2023)
CONFES versus LRT. Our work is related to the work from Zheng et al. (2020) which proposed confidence-based metric called likelihood ratio test. (TRRT) for sieving the clean samples. Our proposed confidence егтог metric has at least two advantages over the likelihood ratio: (1) Confidence error enables the algorithm to start performing the sample sicving in the early epochs of training. Using the sieving threshold 0; the algorithu only incorporates the samples with confidence error less than a; in the training instead of all samples. Applying a similar threshold to likelihood ratio in warm-up epochs delivers much lower accuracy than using all samples based on our observations. (2) The confidence error is a more efficient metric than the likelihood ratio for differentiating the clean samples from the noisy ones according results provided in Figure 10 in the Appendix, which are indeed consistent with the accuracy results provided in the Evaluation section. Moreover, we empirically compared the probability of error for confidence error and LRT on the CIFAR-100 dataset with different types and levels of noise. The results (Figure 9 in the Appendix) show confidence error has a much smaller error rate in identifying noisy samples compared to LRT, while its error rate in identifying clean samples slightly worse than LRT. In sample sioving, misidentifying noisy samples as clean ones (false negatives) is much more detrimental to utility than wrongly recognizing clean samples as noisy (false positives). The latter issue can be alleviated with techniques such as clean data duplication as employed by CONFES.
experimental
In the future, we can extend our work by incorporating techniques such as semi-supervisss learning to perform label correction. We can also automate the selection process for sample sieve size by utilizing soft clustering techniques to model confidence errors. This approach would eliminate the need for the initial sieving threshold Adaboost-like methodology. Leveraging the proposed confidence error metric and incorporating multiple weak classifiers might improve the efficiency of sicving but can be computationally expensive.
(a). Furthermore, we can employ ensemble learning techniques including
Acknowledgement
This work was supported by Google Ph.D. Fellowship to R.T., as well the German Federal Ministry of Education and Rescarch and the Bavarian State Ministry for Science and the Arts The authors of this work take full responsibility for its content.
References
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference machine learning, pp. 233-242. PMLR, 2017.
Yingbin Bai, Erkum Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
Mélanie Bernhardt, Daniel C Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C' Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew P Lungren, Aditya Nori, Ben Glocker, et al. Active label cleaning for improved dataset quality under resource constraints. Nature communications, 13(1):1 11, 2022.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International Conference on Machine Learning, pp. 825-834 836. PMLR, 2021.
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? Xiv preprinl
2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Pengfei Chen, Guangyong Chen, Junjie Ye, Jingwei Zhao, and Pheng-Ann Heng. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference on Learning Regresenlalions, 2021a.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Robustness of accuracy metric and its inspirations in learning with noisy labels. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021b.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. International Conference Learning Representations, 2020.
Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O' O'Donoghue, Daniel Visentin, al. Clinically applicable deep learning for diagnosis and referral in retinal discase. Nature medicine, 24(9):1342-133) 13500, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248-255. Icce, 2009.
Sorin Grigorescu, Bogdan Trasnca, Tiberiu Cocias, and Gigel Macesanu. A survey of doep learning techniques for autonomous driving. Journal of Pield Robolics, (7) 386, 2020.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural net works with extremely noisy labels. Advances in newral information processing systems, 31, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630 645. Springer, 2016.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp.
PMLR, 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning, pp. 4804-4815. PMLR, 2020.
Alex Krizhevsky, GeofTrey Hinton, et al. Learning multiple layers of features from tiny images. Cileseer, 2009.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Fernandez-Granda Early-learning regular- ization prevents memorization of noisy labels. Advances in neural informalion processing systems, 33: 10831-203443 2020.
Xiaoxuan Liu, Livia Faes, Aditya U Kale, Siegfried K Wagner, Dun Jack Fu, Alice Bruynseels, Thushika Mahendiran, Gabriella Moraes, Mohith Shamdas, Christoph Kern, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and mnaa-analysis. The lancet digital health, 1(6):e271-e2M) 2019.
Yang Liu. The importance of understanding instance-level noisy labels. arXiv e-prints, pp. arXiv-2102. 2021.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.no .nt, 2017.
Anna Majkowska, Sid Mittal, David F Steiner, Joshua J Reicher, Scott Mayer McKinney, Gavin E Duggan, Krish Eswaran, Po-Hsuan Cameron Chen, Yun Liu, Sreenivasa Raju Kalidindi, et al. Chest radiograph interpretation with deep learning models: assessment with radiologit-aadjdiiette reference standards and population-adjusted evaluation. Radiology, 294(2):421-433 2020.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Eran Malach and Shai Shalev-Shwartz. Decoupling' when to update' from' how to update". Advances in Neural Information Processing Systems, 30, 2017.
Gary Marcus. Deep learning: critical appraisal. arXiv preprint arXiv:1801.00691 2018.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1944-1952, 2017.
Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9617-9626, 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. How does early stopping help generalization against label noise? In ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning, 2019.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Newral Networks and Learning Systems, 2022.
Cory Stephenson, suchismita padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and SueYeon Chung. On the geometry of generalization and memorization in deep neural networks. In International Conference on Learning Representations, 2021.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with roguullaarzztion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 13726 13735, 2020.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable label-noise learning? Advonces in Neural Information Processing Systems, 32, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 13:7597-7610, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust carly-learning: Hindering the memorization of noisy labels. In International Conference Learning Representations, 2021.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy laboled data for image classification. In Proceedings of the IEEE conference on computer vision and pollern recognilion, pp. 2691-2699, 2015.
Yu Yao, Tongliang Tiu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noisc learning. Advances in neural information processing systems, 33:7260 7271, 2020.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. Advances in Neural Information Processing Systems, 34, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp- 7164 7173. PMLR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking gencralization. Communications of the ACM, 64(3):107-115, 2021a.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization ion. In International Conference on Machine Learning, pp. 2021b.
PMLR,
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error- bounded correction of noisy labels. In International Conference on Machine Learning, pp. 11457. PMLR, 2020.
17

---

Published in Transactions on Machine Learning Research (09/2023)
A
Appendix
Further experiments on the effectiveness of confidence error. We extended the experiments associated with Figure 2 of the main manuscript to the symmetric and pairflip label noise. Experiments are conducted using PreAct-ResNet18 and CIFAR-100 with noise level of 40%.
Probability density
6
Noisy
Probabllity density
10.0
Noley
Probability density
Noisy
15
7.5
4
10
5.0
2.5
0
0.0
0.0
0.5
1.0
0.0
0.5
Confidence error
Confidence error
Confidence error
(a) Epoch 10
(b) Epoch 50
(c) Epoch 200
Figure Distributions of confidence error values for pairflip label noise
Probability density
Clean Noisy
Probability density
Clean
Probability density
Clean
6
10
Noisy
Noisy
20
4
10
2
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Confidence error
Confidence error
Confidence error
(a) Epoch 10
(b) Epoch 50
(c) Epoch 200
Figure 7: Distributions of confidence error values for symmetric label noise
Additional details on the experimental setup. For all experiments on the CIFAR-10, CIFAR-100, and ClothingIM datasets, there are some general sapee-parameters such learning rate, batch size, and weight decay, specified in the original manuscript and summarized in Table 6. The method-specific hyper-parameters used in the experiments are set based on the corresponding manuscript the published source code: Co-teaching (Han et al., 2018)*, ELR (Liu et al., 2020) 1, CORES (Cheng et al., 2020) PES (Bai et al., 2021)$, SLN (Chen et al., 2021a) 9, DivideMix (Li et al., 2020) JoCoR (Wei et al., 2020) ** LRT (Zheng et al., 2020) MentorMix (Jiang et al., 2020) and PTD (Xia et al., 20
Instance-dependent label noise. In order to generate the instance-dependent label noise in the experi- ments, we followed the previous works Cheng et al. (2020); Yao al. (2020); Bai et al. (2021); Chen et al. (2021a) and employed the following algorithm proposed in Xia et al. (2020):
"https://gitaae
#
18

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm 2: Instance-dependern Label Noise Generation taken from Xia et al. (2020)
Input: Clean samples {( yt) Noise rate 7
Output: Noisy samples { D1 )=)
Sample instance flip rates q IR from the truncated normal distribution (r, 0.13, [0, 1])
Independently samples w1; my We from the standard normal distribution NO, 12)
for i 0, ...,, do
I; Wy: /+ Generate instance dependent flip rate
Py. = -0 /- control the diagonal entry of the instance-dependet transition natrix
qi softmax(p) /+ nake the aum of the off-diagonal entries of the yi-th rou to be
Py. = 1 - qi /* aet the diagonal entry to be 1-9:
Randomly choose a label from the label space according to the possibilities as noisy label y/ return Noisy samples {(z,, D1)
Table General training hyperparamcters (common for all methods of comparison).
CIFAR-10
CIFAR-100
ClothingIM
Model
PreActResNet-18
PreActResNet-188
Pretrained ResNet-50
Batch size
128
128
32
Learning rale (Ir)
2e-2
2e-2
2e-3
Ir scheduler
Cosine annealing
Cosine annealing
MultiStep
Ir decay factor
100
100
10 epoch 40
Weight decay
5e-4
5e-4
1e-3
Epochs
300
300
80
Truly clean
Truly noisy
30k
30k
Number of samples
20k
20k
10k
10k
10k
100
200
300
100
200
300
0
100
200
300
Epoch
Epoch
Epoch
(a)
Instance-dependent (60%)
(b) Symmetric (60%)
(c) Pairflip (10%)
30k
30k
Number of samples
30k
20k
20k
20k
10k
10k
10k
100
300
100
200
300
0
100
300
Epoch
Epoch
Epoch
(d) Instance-dependeent (10%)
(e) Symmetric (40%)
(l) Pairflip (30%)
Figure The number of clean and noisy samples that CONFES correctly identifies on CIFAR-100 with different noise types and noise rates. The dashed lines represent the total number of clean and noisy samples. CONFES consistently achieves high success rate in correctly distinguishing between clean and noisy samples.
19

---

Published in Transactions on Machine Learning Research (09/2023)
CONFES (ours)
LRT
0.6
0.6
0.6
Prinoisy, selected)
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
100
200
300
0
100
200
300
100
200
300
Epoch
Epoch
Epoch
(a) Instance-dependent (60%)
(b) Symmetric (60%)
(c) Pairflip (40%)
0.6
0.6
Prinoisy, selected)
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
100
200
300
0
100
200
300
100
200
300
Epoch
Epoch
Epoch
(d) Instance-dependent (10%)
(e) Symmetric (10%)
(I) Pairflip (30%)
0.6
Priclean, rejected)
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
100
200
300
0
100
200
300
100
200
300
Epoch
Epoch
Epoch
(g) Inslance-dependent (60%)
(h) Symmetric (60%)
(i) Pairflip (40%)
0.6
0.6
0.6
Prt rejected)
0.4
0.4
0.4
0.0
0.0
0.0
100
200
300
100
200
300
100
200
300
Epoch
Epoch
Epoch
(i) Instance-dependent (10%)
(k) Symmetric (10%)
Pairflip (30%)
Figure 9: Comparison between the probability of error (lower is better) for CONFES (based the proposed confidence error metric) and the LRT algorithm (Zheng et al., 2020) (based on the likelihood ratio metric)
the CIFAR-100 dataset with different noise types and noise rates. (noisy, selected) means the sample is noise but wrongly selected by the algorithm to be incorporated in training. Likewise, (clean, rejected) means the sample is clean but excluded by the algorithm during training. The probability of error lower for CONFES in the former case, whereas slightly higher in the latter case. Note that the former case is more detrimental the performance compared to the latter one.
20

---

Published in Transactions on Machine Learning Research (09/2023)
Probability density
4.
Clean Nolsy
10.0
Noisy
Clean
7.5
3.
2.5
0
0.0
0.0
1.0
0.0
0.5
Likelihood ratio
Confidence error
(a) LRT
(b) CONFES
Figure 10: Distributions of likelihood ratio employed in LRT(Zheng et al., 2020) and the proposed confidence error metric used in CONFES at epoch 200. The distributions confidence error for noisy and clean samples are more dissimilar than that of likelihood ratio, indicating that confidence error is a more effective metric than likelihood ratio for sieving the samples. The experimental setup the same as Figure 2.
21


Reviews:
[{"review_id": "fF00yBcpiY", "reviewer": "Reviewer_zGci", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths,\n1. The paper proposes a novel metric to differentiate clean labeled data and noisy labeled data, which has been demonstrated to be effective at differentiation empirically.\n2. An empirical effective method is proposed based on the metric\n3. The paper is well-written and well-organized. It is easy to follow.\n\nWeakness\n1. Some of the notations lack definition in the theoretical results, such as the \\mathfrak{O}\n2. The paper's motivation includes that two-model methods are less efficient, and this motivation may imply there is a trade-off between efficiency and accuracy. Such kind of results have not been presented in the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThis paper presents a new metric aimed at discerning between noisy and clean labels. The theoretical foundation of the metric is also established, demonstrating its effectiveness in expectation by showcasing a low error rate for the identified labels. Furthermore, empirical evidence is provided to validate the metric's efficacy. Additionally, the paper proposes a method that strategically filters out noisy examples early in the learning process. The experimental results serve to substantiate the efficiency of this proposed approach and its ability to improve overall learning performance by leveraging the metric to handle noisy data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Theoretically, the paper's proposed method is closely related to the LRT [Zheng et al. 2020]. The paper even uses part of their theoretical results to form one's own. So except for the empirical studies, what is the relationship between the current paper and LRT? Section 2 needs more discussion on this part. \n2. Any empirical results demonstrate the efficiency of the proposed method compared to those two-model methods? I understand a trade-off is possible if the proposed method is faster but less accurate. However, if the proposed method is quicker and more accurate, I am wondering why this happen.\n3. The clarity of presenting the theoretical results needs to be improved: please define all notations clearly before presenting the concrete results. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a9iCAQniOs", "reviewer": "Reviewer_zMtf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is simple and easy to implement.\n2. The paper studies theoretical properties of the proposed method.\n3. The paper shows that the proposed method has better empirical performance than many noisy learning baselines.\n4. The proposed method can be combined with existing baseline methods to achieve better performance.\n\nQuestions and Weaknesses:\n1. I am not very clear about the intuition behind the proposed metric. If the predicted class and the labeled class have almost equal confidence (but they are different classes), e.g., 0.5 and 0.5, then such a sample is always selected as a clean sample but the given label is used for training. Why is this a valid decision?\n2. For the CONFES algorithm, it starts with a higher threshold, which selects more data points and gradually decreases the threshold. In such a case, how many data points do the algorithm typically selects in the end? Could it be the case that many samples never get selected and trained?\n3. Suppose a noisy-labeled sample by chance have the predicted class to be the same as the given class, such a sample is selected and it seems likely that such a sample will remain to predict the given label and selected in every epoch. Do you observe this happening in practice and if it is the case, how to prevent it?\n4. Can you give more justifications for the duplication step of CONFES? Does duplication give better performance and why? \n\nMinor:\n1. Page 3: two right parentheses in \"The k-class/label classifier F(xi; θ))\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of noise label learning. The paper proposes a simple metric called the confidence error, which is essentially the difference between the predicted class probability and the label class probability, and uses such a metric to identify clean labels to construct the training data set. The paper shows theoretical properties of the proposed metric that under certain conditions, samples selected using the criteria have bounded probabilities to being noisy. The paper conducts empirical experiments and show that the proposed method achieve better performance compared to some baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my questions and concerns in the questions and weaknesses part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c9vDf1eELS", "reviewer": "Reviewer_dBHc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.The paper presents an innovative alternative metric, distinct from previously used loss values, that offers a more effective means of sieving training examples. \n2.Theorectically, It also establishes rigorous theoretical assurances regarding the probability of error. Experimentally, the proposed method has an impressive performance and can be equipped into other algorithms.\n3.The organization and presentation of the paper are well done.\n\nWeaknesses:\n1. It might be better to visualize the number of clean samples the proposed method sieves, since the core of the work is to use a new metric to select clean samples.\n2. The reason why the warm-up is used needs to be further explained. Why not only warm up the model and then set the threshold?\n3. The contributions seems a little incremental though it compare the method to LRT. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work leverages an insightful observation to introduce a groundbreaking discriminator metric known as \"confidence error\" and a filtering technique called CONFES, which significantly enhances the distinction between clean and noisy samples. It also establishes rigorous theoretical assurances regarding the probability of error associated with the novel metric. Furthermore, it conducts comprehensive experimental evaluations, wherein it demonstrates the remarkable performance superiority of its proposed methodology over recent investigations across different scenarios, encompassing both synthetic and real-world label noise.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Visualize the number of clean samples the proposed method sieves.\n2. Give more explanation to warm-up.\n3. Perhaps it would be beneficial to introduce an additional Adaboost-like algorithm to further enhance the novelty of the approach.\n", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After a major revision, significant additional evidence for the proposed method has been included.  This has been appreciated by all reviewers, resulting in a unanimous recommendation of acceptance.
Decision: Accept (poster)