PARS: PSEUDO-LABEL AWARE ROBUST SAMPLE SELECTION FOR LEARNING WITH NOISY LABELS
Authors: Arushi Goel
Yunlong Jiao
Jordan Massiah
Venue: ICLR 2022 Submitted (2022)
Keywords: learning with label noise

Abstract: Acquiring accurate labels on large-scale datasets is both time consuming and expensive. To reduce the dependency of deep learning models on learning from clean labeled data, several recent research efforts are focused on learning with noisy labels. These methods typically fall into three design categories to learn a noise robust model: sample selection approaches, noise robust loss functions, or label correction methods. In this paper, we propose PARS: Pseudo-Label Aware Robust Sample Selection, a hybrid approach that combines the best from all three worlds in a joint-training framework to achieve robustness to noisy labels. Specifically, PARS exploits all training samples using both the raw/noisy labels and estimated/refurbished pseudo-labels via self-training, divides samples into an ambiguous and a noisy subset via loss analysis, and designs label-dependent noise-aware loss functions for both sets of filtered labels. Results show that PARS significantly outperforms the state of the art on extensive studies on the noisy CIFAR-10 and CIFAR-100 datasets, particularly on challenging high-noise and low-resource settings. In particular, PARS achieved an absolute 12% improvement in test accuracy on the CIFAR-100 dataset with 90% symmetric label noise, and an absolute 27% improvement in test accuracy when only 1/5 of the noisy labels are available during training as an additional restriction. On a real-world noisy dataset, Clothing1M, PARS achieves competitive results to the state of the art.

Under review as a conference paper at ICLR 2022
PARS: PSEUDO-LABEL AWARE ROBUST SAMPLE SE- LECTION FOR LEARNING WITH NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Acquiring accurate labels on large-scale datasets is both time consuming and expensive. To reduce the dependency of deep learning models on learning from clean labeled data, several recent research efforts are focused on learning with noisy labels. These methods typically fall into three design categories to learn a noise robust model: sample selection approaches, noise robust loss functions, label correction methods. In this paper, propose PARS: Pseudo-Label Aware Robust Sample Selection, a hybrid approach that combines the best from all three worlds in toint-training framework achieve robustness noisy labels. Specifically, PARS exploits all training samples using both the raw/noisy labels and estimated/refurbiihhe pseudo-labels via self-training. divides samples into an ambiguous and a noisy subset via loss analysis, and designs label-dependent noise-aware loss functions for both sets of filtered labels. Results show that PARS significantly outperforms the state of the art on extensive studies on the noisy CIFAR- 10 and CIFAR- 100 datasets, particularly on challenging high-noise and low-resource settings. In particular, PARS achieved an absolute 12% improvement in test accuracy on the CIFAR- dataset with 90% symmetric label noise, and an absolute 27% improvement in test accuracy when only 1/5 of the noisy labels are available during training as an additional restriction. On real-world noisy dataset, Clothing MM, PARS achieves competitive results to the state of the art.
INTRODUCTION
Deep neural networks rely on large-scale training data with human annotated labels for achieving
good performance (Deng et al. 2009 Everingham et al. 2010). Collecting millions or billions
of labeled training data instances is very expensive, requires significant human time and effort,
and can also compromise user privacy (Zheng et al. 2020 Bonawitz et al. 2017). Hence, there
has been a paradigm shift the interests of the research community from large-scale supervised
learning KKizhevsky et al. 2017 He et al. 2016a Huang et al.
2017) to Learning with Noisy Labels
2016 Patrini et al. 2017] Tanno et al. 2019) and/or unlabeled data (Berthelot al. 2009h Sohn et al.; 2020). This is largely due the abundance of raw unlabeled data with weak user tags (Plummer et al.] 2015)] Xiao et af.) 2015) or caption descriptions (Lin al. (0144). However, is not trivial to build models that are robust to these noisy labels as the deep convolutional neural networks (CNNs) trained with cross-entropy loss can quickly overfit the noise in the dataset, harming generalization (Zhang al. 2016).
(LNL) (Natarajan et al.] 2013 Goldberger &
Most of the existing approaches on LNL can be divided into three main categories. First, several noise robust loss functions (Ghosh al. 2017 Wang et al., 2019a| Zhang & Sabuncu/ 20188) were proposed that are inherently tolerant to label noise. Second, sample selection methods (also referred to as loss correction in some literature) HHan et al.) 2018 Yu et al. 2019 Arazo et al., 2019) are popular technique that analyzes the per-sample loss distribution and separates the clean and noisy samples. The identified noisy samples are then re-weighted so that they contribute less in the loss computation. A challenge in this direction is to design reliable criterion for separation and hence prevent overfitting to highly confident noisy samples, a behavior known as self confiraoinntoo bias. Third, label correction methods attempt to correct the noisy labels using class-prototypes (Han et al.) 2019) or secudollbbeling techniques (Tanaka et al. 2018 Yi & Wu 2019). However, in order to correct noisy labels, we typically need extra (usually small) set of correctly labeled validation

---

Under review as a conference paper at ICLR 2022
labels. In particular, these methods can fail when the noise ratio is high and estimating correct labels or high-quality pseudo-labels is non-trivial.
More recently, the success of several state-of-the-ar LNL methods is attributed to leveraging Semi- Supervised Learning (SSL) based approaches (Li et al. 2020 Kim et al. 2019). Typically, a sample selection technique applied to separate clean and noisy labels in the training data, then the noisy labels are deemed unreliable and hence treated as unlabeled in a SSL setting. Following the recent SSL literature (Lee et al., 2013; Arazo et al. 2020), estimated pseudo-labels are usually used to replace the filtered noisy labels during training. These approaches have shown to be highly tolerant to label-noise. However, the noisy labels are always discarded favor of pseudo-labels in all the existing literature, but they may still contain useful information for training. Eseudo-labeling is in turn only applied to the filtered noisy subset while the rest of the raw labels are typically used as is, which makes it sensitive to the quality of the filtering algorithm.
In particular, motivated by a simple principle of making the most of the signal contained in the noisy training data, we design PARS, short for Pseudo-Label A Aware Robust Sample Selection. Our contributions are as follows:
1. PARS proposes a novel, principled training framework for LNL. trains on both the original labels and pseudo-labels. Unlike previous works, instead of filtering and then discarding the low-confident noisy labels, PARS uses the entire set of original labels, and applics self-training with pseudo-labeling and data augmentation for the entire dataset (rather than the filtered noisy data only).
2. PARS is able to learn useful information from all the available data samples through label-dependent noise-aware loss functions. Specifically, in order to prevent overfitting to inaccurate original labels (or inaccurate pseudo-labels), PARS performs simple confidence- based filtering technique by setting a high threshold on their predicted confidence, and applies robust/negative learning (or positive/negative learning) accordingly.
3. We perform extensive experiments on multiple benchmark datasets i.e. noisy CIFAR- 10, noisy CIFAR-100 and Clothing I.. Results demonstrate that PARS outperforms previous state-of-the-att methods by a significant margin, in particular when high level of noise is present in the training data. We also conduct sufficient ablation studies to validate the importance of our contributions.
4. We design novel low-resource semi-stperrvsedd LNL setting where only small subset data is weakly labeled (Section 4.3). We show significant gains over sfate-of-the-as approaches using PARS. This setting is particularly interesting when it is hard to obtain large-scale noisy labeled data. In particular, we find that surprisingly none of the existing LNL methods outperform baseline SSL model (FixMatch) (Sohn et al. 2020) that is not even designed to handle label noise, and yet PARS can achieve up to an absolute 27% improvement in test accuracy in a controlled high-noise low-resource setting.
2 RELATED WORK
In recent literature on LNL, methods typically fall into three design categories learn noise robust model: noise robust loss functions, sample selection approaches, or label correction methods.
Noise robust loss function based methods propose objective functions that are tolerant label noise. A commonly used loss function that is identified to be robust to noisy labels is Mean Absolute Error (MAE) (Ghosh et al.) 2017). h. /Wang et al. (2019a) proposed Improved MAE which is weighted version of MAE. Zhang& & Sabuncu (2018) proposed Generalized Cross Entropy Loss (GCE) which a generalization of MAE and Categorical Cross Entropy loss. More recently, Wang et al. 2019b) designed a Symmetric Cross Entropy (SCE) loss which is similar in spirit to the symmetric KL- divergence and combines the cross-entropy loss with the reverse cross-entropy. Although SCE is robust to noisy labels, Ma et al. (2020) proposed a normalized family of loss functions which are shown to be more robust than SCE for extreme levels of label noise. Kim et al. (2019) and Kim et al. (2021) designed framework that alternates between positive learning accurate/clean labels and negative learning on complementary/wwenn loss function during training take into account the noise distribution by modeling noise transition
labels. Loss correction approaches explicitly modify the
2

---

Under review as a conference paper at ICLR 2022
matrix (Patrini et al. 2017; Tanno al. 2019; Xia et al., 2019 Goldberger & based on label-dependent weights (Natarajan et al. 2013
2016
Another family of methods focus primarily sample selection, where the model selects small-loss samples as "clean" samples under the assumption that the model first fits to the clean samples before
memorizing the noisy samples (also known the early-learning assumption) (Arpit et al. 2017] Zhang et al. 2016] Liu et al.] 2020). Han et al. (2018) and Yu et al. (2019) proposed Co-teaching where sample selection is conducted using two networks to select clean and noisy samples, and then the clean samples are used for further training. MentorNet (Jiang et al. 2018) is a student-teacher framework where a pre-trained teacher network guides the learning of the student network with clean samples (whose labels are deemed "correct"). In the decoupling training strategy proposed by Malach &Shalev-Swartzz (2017), ; two networks are trained simultaneously and guide each other on when and how to update. One limitation of these approaches is that they ignore all the noisy/unclean samples during training and only leverage the expected expected-to-
clean samples for improving performance. [Li et al. (2020) proposed DivideMix where sample selection is conducted based on per-sample loss distribution, and then the noisy samples are treated as unlabeled data in SSL setting (Berthelot et al. 2019b). Nishi et al. (2021) further investigated the potential of using augmentation strategies in LNL, one used for loss analysis and another for learning, thus improving generalization of DivideMix.
Compared to the above, label correction aims to improve the quality of the noisy labels by explicitly correcting the wrong labels. Tanaka et al. (2018) and Yi & Wu (2019) predicted the correct labels either as estimates of label probabilities (soft labels) as one-hot class labels (hard labels). Arazo iet al. (019)) combined label correction with iterative sample selection by first filtering clean labels from noisy labels modeling a two-component Beta Mixture, and then estimating to correct labels for those noisy samples. Tanaka et al. (2018) combined label correction with additional regularization terms that were proved to be helpful for LNL. [Song et al. (2019) also used label replacement to refurbish a subset of labels, thereby gradually increasing the number of available training samples. Liu et al. 2020) computed soft labels as model predictions and then exploit it to avoid memorization.
METHODOLOGY
3.1 PRELIMINARIES
with x E X C Rd as the input features and y E y (1....,I$ K} as the corresponding label, we aim to learn a classifier p):;0== X' )' parametrized by f. For clean labeled set, i.e. where y' is the true label for the i-th training sample x², we can learn the model parameters A by minimizing the Cross Entropy (CE) loss:
For a K-class classification problem with noisy labels, let D
min
Σ (Z.Y)ED
Lcr(p(a;0), 0),))
(1)
However, in the presence of noisy labels where y² is possibly incorrect for the i-th training sample w., the cross-entropy loss can quickly overfit to the noise in the dataset (Zhang al. 2016).
Robust Learning, Variants to the CE loss have been proposed to improve the classification perfor- mance under label noise. Examples of such noise robust loss functions include Mean Absolute Error AMAE (Ghosh et al. 2017). Symmetric Cross Entropy LSCE (Wang et al.] (2019b). Normalized Cross Entropy LNCE or Active Passive Loss LAPl. (Ma et al., (020). We may simply replace the LCe with any of these noise robust loss, denoted by L in the optimization Equation (1) to perform
Robust Learning:
LR. E {LMAE, LSCE, LNCE, CAPL) For the sake of the limited space, definitions and further discussion about the robust losses deferred to Appendix(A) A.1
Positive and Negative Learning. A particular variant to the CE loss that is designed to handle noisy/wrong labels is called Negative Learning (Kim et al.) 2019). Given a sample with a correct label (x. y). Positive Learning (PL) is equivalent to optimizing the standard CE loss in Equation (I. so that we optimize the output probabilities corresponding to the true label to be close to 1:
K
Lpl(p(æ; 0), v) == (ce(p(x;0).)) =
Σ lu log [p(:; 0)
(3)
3

---

Under review as a conference paper at ICLR 2022
Strong Data Augmenbation
Posiiue learning
Negative Leaminn
Paeudo Labels
Labeled
Warmup
Robust
Thresholding
Ambiguaus Original Labels
Robust Learning
Labels
Noisy Original
Negative learing
Labels
Thresholding
Preute-Label
Sample
Generator
Pseude Labels
Figure 1: PARS: Pseudo-Label Aware Robust Sample Selection. PARS has stages: (1) First perform robust warm-up training. (2) We then iteratively obtain pscudo-label on the fly and perform a joint training with the original raw labels and sclf- toffntrainig with pscudo-labels, filtering the raw labels into ambiguous/noisy samples and the ascudo-labels into corrected/noisy samples. We apply aabeldependent noise-awaad loss functions in robust/negative and positive/negative learning framework for the raw labels and pseudo-labels respectively.
where the class label y is one-hot encoded, and [:]s denotes the k-th dimension of a vector. Now for a data example with wrong label (æ. 77. Negative Learning (NL) applies the CE loss function to fit the "complement" the label in Equation (I). so that the predicted probabilities corresponding to the given label is optimized to be far from 1:
K
Lxz(p(æ;0), 0). =
b1 log [1 p(x; 0))
(4)
Sample Selection. In order to apply the Positive and Negative Learning framework in LNI., Kim ct al. (2019) proposed to select samples based on simple threshold of the per-sample loss value, i.e.a data point (æ, y) is selected to be clean for PL if [p(æ; 0)l, > where is a noise-aware threshold. This simple yet effective procedure is based on the commonly used "small-loss" assumption that smaller-loss samples (typically in early learning stages) are usually easier to learn and hence expected
to be "clean", and it has been successfully applied in several other studies (Jiang et al. 2018 2020).
3.2 PARS: PSEUDO-LABER A AWARE ROBUST SAMPLE SELECTION
We propose PARS: Pseudo-Label Aware Robust Sample Selection. The method illustrated in Figure I (and Algorithm D in Appendix A.2). and each component is discussed in detail below. In particular, there are three major differences between PARS and previous works: 1) PARS does not treat the high-confidemt the relatively low yet non-negligible level of noise. 2) PARS does not discard the noisy (or low- confidence) labels, but rather applies negative learning framework to learn the "complement" of these labels (Kim et al.) 2019). This constitutes one major difference between the selective negative learning procedure of Kim al. 2019) and PARS, that Kim et al. (2019) only apply it during the warm-up stage and still choose to discard the filtered noisy labels in favour of pseudo-labels in a SSL manner, and unify the two stages and show improved performance. 3) We improve the convergence of PARS by self-training using pseudo-labels and data augmentation for the entire dataset proxy enhance clean labels and correct noisy labels.
samples as "clean", but rather applies robust loss functions to address
Robust Warm-up Training. In the presence of label noise, the model overfits to noise when trained with CE loss and produces overconfident and wrong predictions after few epochs. This problem is more severe with high levels of label noise. To overcome this, we perform "robust warm-up" using robust loss for a few epochs for the model to produce reliable and confident predictions. Specifically, in the warm-up up training phase, we minimize:
1
min
Σ
Lau (r(:;())))
(5)
D
(R.D)ED
where LRL can be any loss from Equation (2). In our experiments, we test different loss functions for warm-up training (for more details see Appendix A.3).
Robust Sample Selection, As shown in Figure 2a after the robust warm-up training, we observe that there is weak trend that high-confidence samples tend have clean labels. Given the model
4

---

Under review as a conference paper at ICLR 2022
04
013
Carfoaes
(a) Robust Warmup CRL
(b) Robust/Negative Learning
(c) PARS (ours)
Figure 2: Confidence Probabilities for different stages of training on CIFAR- 10 dataset with 50% symmetric noise. PARS (ours) improves separation of the clean and noisy labels as compared to the model after the robust warm-up training.
parameters Ó the current iteration and a confidence threshold E [o, 1]. we divide the entire dataset set DA and noisy set Dx of samples by
D (or similarly, minibatch B C D) into an ambiguous n thresholding the maximum of the predicted confidence probabilities over all classes:
DA = ( (I. D max p( 0)
>
DN = DD
(6)
A notable difference is that our confidence-basee thresholding is label-free (i.e. thetshheshoding of the maximum confidence over all classes), while the widely used loss-based thresholding Kim et al. (2019) depends on the ground-truth (noisy) label (ieehhhrsholling the confidence corresponding to the given class). Our approach has two advantages: 1) our selective criterion depends on the predicted confidence probabilities alone, therefore it is straightforward to apply cases where ground-truth labels are unavailable (e.g.in a semi-supervised setting in Section 4.3): 2) our strategy helps to reduce the self-confirmation bias propagating the mistake of overfitting possibly noisy samples, because our model only selects samples that it is confident to make a prediction on regardless of the correctness of the raw label.
Notably, previous works such as Jiang et al. (2018 1018:1 Li et al. 2020) usually maintain two divergent networks (one for sample selection and one for the main classification task) in order to avoid the self-confirmation bias. Our method proves to be highly resilient to the overfitting issue with a single network simultaneouuly updated to perform both sample selection and classification, which drastically simplifies implementation and training complexity.
Label-Dependent Noise-Aware Loss Functions. Despite the robust sample selection of dividing samples into ambiguous and noisy set, propose to learn useful information from all the available training data instead of discarding the raw labels of the noisy samples as typically done in previous
literature (Jiang et al. 2018 Kim et al. 2019 Li et al. 2020).
Specifically, we propose to use two separate loss functions. tailored for either the ambiguous or noisy labels respectively, which prevents overfitting to noise and enables further filtering of labels effectively (Figure (b)). For the ambiguous set, DA, we train with the robust learning loss, LRL, (same loss used during the warm-up training), where
deem that the given label in this set may still contain non-negligible noise. For the noisy set, DN, we train with the negative learning loss, LVL, learn the complement of the given raw labels for the identified least-confident highly-noisy samples. The overall label-depender noise-aware loss function for the proposed robust/negative learning framework using the given raw labels is thus:
Craw (0)
1
Σ
(nt.p((r;0).)) Ax
1
Σ
(x
(7)
DA
DDNl
(D.y)EDA
(a.
where An is the weight for the negative learning loss term, and DA, DN are defined as in Equation (
We set a high threshold 0..5.0.55 throughout our experiments) on the predicted confidence in the robust sample selection (Equation (6) since we observe that this yields better performance as shown
We choose the term "ambiguous" rather than the typically used term "clean" (Jiang et al. 2018 2020) because there is still significant amount of noise in the expected-to-he clean samples when simple thresholding method is used select samples (Arazo al. 2019.
5

---

Under review as a conference paper at ICLR 2022
in Appendix A.4 We will show that, by using the noisy labels in the negative learning setting, we able to consistently improve the model performance (Table
Self-Training with Pseudo-Labels. After the robust warm-up training. the initial convergence of the model leads to reliable predictions which can be used as labels to further improve model convergence via self-training. Therefore, in parallel to using the given raw labels during training, we use pseudo- labels generated by the network itself to guide the learning process. Given the model parameters i at the current iteration, for each sample (x, -) E DD we generate the underlying pseudo-labed corresponding to the model's most confident class, and then randomly sample complementary "wrong" pseudo-label E from the remaining classes following Kim et al. (2019):
arg max
p((x; 0)
BE (l..... {z).
Similarly to how we use label-dependent noise-aware loss functions for the given raw labels in the ambiguous and noisy sets, we train with the pseudo-labels from the two sets using different losses too. Specifically, we apply positive/negative learning for the generated pseudo-labels in the ambiguous/noisy set respectively:
Cpecado(f) (0) =
1
Σ (p)(p(x;0), +AN (DE,))DA
1
Σ
(N)(p(x;0), (9)
(DA
DDN
(so.)EDN
where AN is the weight for the negative learning loss term (equal the weight in Equation 77) Da, Dy are defined as in Equation (6), 2, 2 are defined as in Equation (8). We find that it suffices to train with the standard CE loss for self-training with pseudo-labels from the ambiguous set, without the need for noise robust learning.
One caveat of applying self-training that, the model's own predictions may become unreliable when most of the samples are primarily guided by the self-training, encouraging the network to predict the same class to minimize the loss. We observe that this is particularly the case under high levels of label noise. To reliably apply self-training, we regularize Equation (9) with confidence penalty (Tanaka al. 2018 Arazo et al. 2019)
Plk Greg Σ 1p log [T((D:)),,
(10)
where [p] is the prior probability distribution for class k, and [A(D; 0),, is shorthand for the mean predicted probability of the model for class & across all samples the dataset D. In our experiments, we assume a uniform distribution for the prior probabilities (i.e.[p), 1/ K). while approximating h(D; 6) by h:B: 8) using mini-batches B D as suggested by [Tankka al. (2018). The generated pseudo-label, become reliably confident with help of the regularization, which guides the robust sample selection to resemble the true distribution more closely (Figure 2c).
PARS: Pseudo-Label Aware Robust Sample Selection. After the robust warm-up training stage (Equation (5)) our final model, PARS, is then trained using both the original labels and the self- generated pseudo-labels with robust sample selection. The final loss is given by:
min Exotat (0) == araw (0) + Apppecddo (0) + AR
(11)
where Lrawr Epecndor Creg are defined as in Equations (7). (9) and (10 respectively, and Äs AR are the weights for the self-training loss and confidence penalty term.
Data Augmentation. For image classification tasks in LNL, data augmentation techniques have recently been shown to significantly improve generalization (Nishi et al. (02T).. We adapt such data augmentation strategies to further improve PARS for benchmarking image classification per- formance in our experiments. Specifically, we apply weak augmentation staandard random flip, crop, normalization) to all data samples in the robust/negattiv learning with original raw labels in computing Equation (7). We also apply strong augmentation Ce.g.AutAAugmeen (Cubuk et al.) 2019), RandAugment (Cubuk et al., (0200) to all data samples in the positive/negative learning with pseudo-labels in computing Equations 9 and (0)). We do not incorporate strong augmentation for training with the original labels as it leads to poor performance in presence of high label noise, as also observed by Nishi et al. (2021). In this way, PARS can be seen as example of how to extend the work of Nishi et al. (2021) 1b by combining weak and strong data augmentation effectively with noisy labels and pseudo-labee labeling,
effective technique to further advance the state of the art in LNI..
6

---

Under review as a conference paper at ICLR 2022
Datasets
CIFAR-I 10
CIFAR- -AR
Noise Type
Sym
Asym.
Sym.
Alg. Noise Ratio
20%
50%
80%
90%
40%
20%
50%
80%
90%
EntropE Loss Low
86.8
79.4
62.9
42.7
85.0
62.0
46.7
19.9
10.1
Bootstrap (Reed et al. 2014
86.8
79.8
63.3
42.9
62.1
46.6
19.9
10.2
F-correction
79.8
63.3
42.9
87.2
46.6
19.9
Mixup Zhang of al
10
95.6
87.1
71.6
52.2
67.8
57.3
30.8
14.6
Co-teaching-
et
2019
89.5
85.7
67.4
47.9
65.6
51.8
27.9
13.7
P-correction
Y) &
2019
92.4
89.1
77.5
58.9
57.5
31.1
15.3
Meta-Learnin:
(L el
2019
92.9
89.3
77.4
58.7
89.2
68.5
59.2
42.4
19.5
M-correction Arazo
2015
92.0
86.8
69.1
48.2
24.3
DivideMix Li
2020
96.1
94.6
93.2
76.0
93.4
77.3
74.6
60.2
31.5
ELR+ (Liu al
20220
95.8
94.8
93.3
78.7
93.0
77.6
73.6
60.8
33.4
DM AesC WS- (Nish al.
95.4
66.4
DM AMM-AuDDesc-No WSSSAW (Nishi el
2021
96.3
95.6
93.7
35.3
94.6
79.6
77.6
61.8
17.3
Robust Loss Laz (warm-up tramming only)
84.0
76.4
74.1
54.5
79.0
63.9
59.8
42.1
18.9
PARS (ours)
96.3
95.8
94,9
95.4
75.3
69.2
53.1
Table Test Accuracy (%) for all the baselines and our proposed method, PARS, on CIFAR-10 and CIFAR-100 100 with symmetric noise (noise ratio ranging 20% to 90%) and asymmetric noise (noise ratio 40%) on CIFAR-10. The results above the "DivideMix" row are taken from LimlLiie et al. 2020).
Alg. Dataset
Clothing
Cross-Entros Entropy Loss Cop
69.21
F-correction (Patrini et al.
2017
69.84
M-correction Araza ef al
2019
71.00
Juint-Optim lamaka
ml
2015
72.16
Meta-Learning (L al.
2019
73.47
P-enrrection &Y Wu
2119
73.49
DivideMix (Li [L
2020)
74.76
ELR+ (Li
74.81
DM-AugDese- WAW (Nisti ct 2021
WAW
74.72
DM-AugDess-WSS DW-S--W-WWSSWWWW (Nishi el 2021
75.11
Robust Loss Cat. (warm-up training only)
71.80
PARS (ours)
74.61
Table Test Accuracy (%) for baselines our proposed method, PARS, on Clothing IM. The results above the "DivideMix" row are taken from Li el al. (2020).
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
We perform extensive experiments on two benchmark datasets, CIFAR-10 and CIFAR- 100 (Krizhessky et al., 2009) with synthetic label noise. The CIFAR-10 and CIFAR- datasets each contain 50,000 training images and 10,000 test images of size x32 pixels with 10 and 100 classes respectively. Both of these datasets have clean label annotations, hence we inject synthetic noise to train the models, and test the performance using the original clean labels. Following previous work (Arazo et al.) 2019 Li et al. 2020), we evaluate with two types of noise: symmetric (uniform) and asymmetric (clas--coniitional) For symmetric noise, the labels are generated by flipping the labels of a certain percentage of training samples to any other class labels uniformly. We use noise ratios of {0.2, 0.5, 0.8, 0.9) which are the symmetric noise probabilities defined in (Patrini al. 2017 Li et al. 2020). While for asymmetric noise labels, flipping of labels only happens for a certain set of classes (Patrini et al. 2017 Li et al. 2020) such truck - automobile. bird airplane, deer horse, and car->sog. For asymmetric noise, we use the noise rate of 40%.
We also test our method on the Clothing dataset (Xiao al. 2015)) which large-scale real- world noisy dataset containing 1 million images from online shopping websites. The labels are collected by extracting tags from the surrounding texts and keywords, and can thus be noisy. As the images vary in size, they are resized to 256x250 pixels. The implementation details for all the experiments are given in Appendix A.3
4.2 COMPARISON WITH STATE THE ART ON LEARNING WITH NOISY LABELS
In Table I compare our final method PARS with several existing of-te-of-the-ar methods LNL. For this setting, we evaluate our models following previous work (Liu et al., 2020 Li et al. 2020), where noisy labels are present for a large number of data samples. We evaluate with different
7

---

Under review as a conference paper at ICLR 2022
Dataset
CIFAR-10
CIFAR- 100
# Labels
4000
10000
Noise Type
Sym.
Sym.
Alg. Noise Ratio
50%
90%
50%
90%
FixMatcch'
[Sohn et al. 2020
82.7
54.9
55.6
22.0
DivideMix
Li ct al. 2020
66.0
37.1
25.2
8.0
AugDese-WS!
Nishi et al 2021
73.3
56.1
31.1
10.1
DivideMix+
et al. 2020)
74.3
40.7
10.6
7.2
AugDese-WSS
Nishi et al. 2021
73.2
57.5
30.1
7.6
PARS (ours)
95.3
70.3
76.3
48.9
Table 3: Test Accuracy (%) for all the baselines and our proposed method. PARS, on CIFAR-10/100 datasets with 50% and 90% symmetric noise with a subset of training labels. + denotes that methods are adapted to include unlabeled data while training. { denotes re-training with the open-source code.
levels of symmetrie label noise ranging from 20%- 90% for CIFAR-I0/ noise for CIFAR-10 with 40% noise ratio. All of the reported results are the best test accuracy of CIFAR-10/100 across all epochs, following the same logic of Li et al. (2020). As shown, compared to all the existing methods, PARS consistently outperforms on CIFAR-10 10 dataset with different types and levels of label noise. Specifically, we observe an increase over the best reported results by Liu et al. 2020)) by approx 14.9% and by approx 1.7% on Nishi et al. (2021) for 90% symmetric label noise, which shows the robustness of our method to high levels of label noise. On a more challenging setting of CIFAR-100 dataset, we improve over the fatte-of-thearr work Nishi et al. (202T) by an absolute 12% improvement in test accuracy in the presence of 90% symmetric label noise. Notably, compared to training with robust loss function alone (i.eppeforming warm-up training thoroughly). PARS is able to achieve up to 40% absolute improvement in test accuracy. These experimental results strongly advocate for PARS, particularly the proposed training framework of using label-dependent noise-aware loss functions on the ambiguous/noisy original raw labels (and corrected/noisy pseudo-labels) with robust/negative learning (and positive/negative self-learning).
and on asymmetric label
On a noisy large-scale real dataset, Clothing Table 2 shows the best accuracy for PARS and all previous works. In particular, PARS achieve improvements over the the robust training baseline with LR by 2.8%. %. Compared to the state of the art, PARS is able to achieve comparable results with DivideMix (Li et | 2020) and DM-AuD WS- WAW (Nishi et al.) (02T). Another variant method proposed by Nishi et al. (022)) DM- DM-AugDesc- WS- NSW, seemed to obtain the best performance overall due to the use of strong augmentation a warm-up training phase. is important to note that both methods from Nishi et al. (202T) are based on DivideMix (Li et al. 2020)), whereas PARS is stand-alone novel training procedure which is able to achieve significant improvements. This paves way to designing other methods for ad advancing the state of the art in LNL. We will leave to future work the investigation of how to optimize the augmentation strategies in order to further improve the warm-up training of PARS.
4.3 LOW-RESOURCE SEMI-SUPERVISED LEARNING WITH NOISY LABELS
We also evaluate our proposed method on a novel low-resource LNL in a semi-supervised setting. We randomly take subset of 4000/10000 samples from the clean labeled CIFAR- 1IFAR-1001100 datasets, and then introduce 50% and 90% symmetric noise in these labels, and include all the remaining samples as unlabeled data for training. As none of the previous LNL works evaluate in a low-resource setting, we decide to compare to state-of-the-art semi-supervised learning method (FixMatch (Sohn et al. 2020)) as baseline, as well as two state-of-the-ar methods from LNL (DivideMix (Li al. 20200, AugDesc-WS SS (Nishi et al. 202T)). Note that both DivideMix and AugDesc- -WS are methodologically motivated by the semi- supervisd literature, where the entire training set is filtered into clean and noisy set, and then the noisy samples are simply treated as unlabeled data in semi-supervised manner. For fair comparison, we also extend these two methods by combining the unlabeled data with the filtered noisy data in their semi-supervised training framework, and term them as "DivideMix+" and "AugDese-Ve WWS+ respectively to signify the difference. All baselines are retrained using the open-sourced code. For PARS, after the warm-up training on the subset of labeled data, we predict pseudo-labels on the unlabeled subset, perform sample selection on the entire dataset (both labeled and unlabeled), and train using optimization objective Equation (II).
As shown in Table the existing LNL models both perform significantly worse than the SSL bascline, Match This is surprising due the fact that Fix Match was not designed with consideration of how
8

---

Under review as a conference paper at ICLR 2022
Dataset
CIFAR-I0 10
CIFAR- 100
Noise Type
Sym.
Asymn
Sym.
Alg. Noise Ratio
20%
50%
80%
90%
40%
20%
50%
80%
90%
Cross- Entropy Loss Lee
Cross-Entropy
86.8
79.4
62.9
42.7
85.0
62.0
46.7
19.9
10.1
Robust Loss Laz, (warm-up training only)
84.0
76.4
74.1
54.5
79.0
63.9
59.8
42.1
18.9
PARS wlo Epecudo
87.6
86.6
79.7
59.1
84.3
66.4
62.0
45.7
21.4
PARS wio L.1,.
Crow or Eysendo
95.6
94.6
94.3
92.4
92.3
74.3
67.0
51.3
PARS (fuil model)
96.3
95.8
94,9
93.6
95.4
77.7
75.3
69.2
53.1
Table 4: Test Accuracy (%) reported the ablations of PARS the CIFAR-10//00 datasets.
to handle label noise. Further, both of the existing LNL models perform similarly, sometimes worse, when the unlabeled data are introduced to the training set, meaning that they cannot effectively exploit the signal contained in all available data. However, our method, PARS. significantly outperforms the baselines in all datasets and noise settings by margin, and achieves an absolute improvement of up to 27% in test accuracy (CIFAR-100 with 90% noise). We deem that this is attributed to the ability of PARS exploiting the pseudo-labels with the positive/negative learning framework, which is robust to a variety of levels and types of label noise in this low-resource LNL setting.
4.4 ABLATION STUDY
Table Jnnayyess the results of different components in PARS and how each learning stage contributes to the final model performance. First, we empirically test the performance of the "warm-up training only" stage (without the sample selection or losses on either the original or pseudo labels). This is empirically equivalent to replacing the standard cross-entropy loss LCE by noise robust loss LRI and optimizing Equation (5) thoroughly. With such noise robust loss function, we achieve good performance compared to using cross-entropy, especially the presence of high levels label noise. However, the warmed-up model performs far worse than the "full model" of PARS, emphasizing the importance of robust sample selection and using label-dependent noise-aware losses on both the original and pseudo labels
have proposed.
Next, we investigate the importance of different components in PARS. 1) training with pseudo- labels is an important part of PARS. Compared to the full model, removing the of pssudo-labels entirely (i.e." "PARS w/o Lpsenda") ") leads to significant reduction in model performance. This implies that adding the self-training losses PARS critical improve the model convergence after the robust warm-up training. 2) Motivated by the fact that there is exploitable signal contained in the filtered noisy set after robust sample selection, we include information from all the original and pseudo labels to enhance performance of PARS. We design a training procedure where we discard the filtered noisy labels and ignore the negative learning losses on both the original and pseudo labels (i.e. "PARS w/o L in either Craw or Lpseudo") "). Overall, there is consistent reduction in model performance across all noise types and levels as compared to the full model. To the best of our knowledge, this is the first evidence in literature that the filtered noisy labels by sample selection strategy can be effectively included (rather than discarded) in order to consistently boost the model performance in LNL.
5 CONCLUSION
In this paper, we proposed a novel method for training with noisy labels called PARS: Pseudo- Label Aware Robust Sample Selection, which takes advantage of self-training with pseudo-labels and data augmentation, and exploits original and pseudo labels with label-dependent noise-aware loss functions with help of robust sample selection. We conduct several experiments on multiple benchmark datasets to empirically evaluate the strengths of our proposed method in lcarning with noisy labels. In particular, PARS gains significant improvements over the sfate-of-the-arr methods in challenging situations such as when high levels of noise are present. Furthermore, we also evaluate our method under a novel low-resource/semiisupe supervised learning with noisy labels setting, where only subset of noisy labels are available during training, and PARS outperforms the baseline models by significant margin. We believe this is a ubiquitous setting in the real world when it may be difficult to obtain large-scale weak labels potentially due to data privacy concerns. In the future, we are interested in exploring further this semi-supervised learning setting with noisy labels, and extending our model to other domains such audio and NLP.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning, pp. 312-321. PMLR, 2019.
Eric Arazo, Diego Ortego. Paul Albert, Noel E O'Connor, and Kevin McGuinness, Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEFE, 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International Conference Machine Learning, pp. 233-2422
PMLR, 2017.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint 1911vv:9111099000 2019a
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi- -upeevupereieed learning. arXiv preprint arXiv: 1905.0290202 022499 2019b.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Ieee, 2009.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International joaenaatooaJoounall of computer vision, 88(2): 303-338, 2010.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. 2016.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. 2018
arXiv preprint arXiv:
Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5138-5147, 2019.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 201 20.
10

---

Under review as a conference paper at ICLR 2022
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference PMLR, 2018.
on Machine Learning, pp.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning, pp. 4804-4815. PMLR, 2020.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 101-110, 2019.
Youngdong Kim, Juseung Yun, Hyounguk Shon, and Junmo Kim. Joint negative and positive learning for noisy labels. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 9442-9451, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Dong-Hyun Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5051-5055 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels semi- supervised learning. In International Conference on Learning Representations, 2020.
Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez- Granda Early-learning regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
lya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /608 /00.0033,, 2016.
Xingjun Ma. Hanxum Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Normal- ized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, pp. 6543-6555 PMLR, 2020.
Eran Malach and Shai Shalev-Shwartz. Decoupling" " when to update" from" how to update". arXiv preprint arXiv: 17X.::1706006633,, 2017.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26:1 2013.
Kento Nishi, Yi Ding, Alex Rich, and Tobias Hollerer. Augmentation strategies for learning with noisy labels. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 8022-8031, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, higf-peeformance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
11

---

Under review as a conference paper at ICLR 2022
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944-1952, 2017.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentent models. In Proceedings of the IEEE international conference computer vision, pp. 2641-26499 2015.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6558 2014.
Kihyuk Sohn, David Berthelot. Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi sami-supee learning with consistency and confidence. arXiv preprint arXiv:2001.07685 2020.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. International Conference on Machine Learning, pp. 5907-5915. PMLR, 2019.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018.
Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silber- man. Learning from noisy labels by regularized estimation of annotator confusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11244-11253, 2019.
Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil M Robertson. Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude's 's variance matters. arXiv preprint 1rXiv:1903. 111900..2144. 12141, 2019a.
Yisen Wang, Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE International Conference on Computer Vision, 2019b.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32:6838 6849, 2019.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 2691-2699 2015.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7017-7025, 2019.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7174-7173. PMLR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arxiv preprint (in iclr 2017). arXiv preprint 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.09412. 2017.
Zhilu Zhang and Mert R Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Huadi Zheng, Haibo Hu, and Ziyang Han. Preserving user privacy for machine learning: local differential privacy federated machine learning? IEEE Intelligent Systems, 35(4):5-14, 2020.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 NOISE ROBUST LOss FUNCTIONS
In this section, we discuss the choice of two noise robust loss functions that are used our experi- ments.
Wang et al. 2019bb and Ma ct al. (2020) design variants of Cross-Entropy (CE) loss that are noise- tolerant. Wang et al. (2019b) propose the Symmetric Cross Entropy (SCE) Loss where they combine the Reverse Cross Entropy (RCE) loss with the CE loss, as
K (a 0) log(lv&s)
LSCR
W loglu(a; 0)] 3.
Σ k=1
(12)
k=1
LCE
CRCE
where a and / are hyperparameters to control the effect of the two loss functions and K are the total number of classes. Equation (22 is similar to the symmetric KL-divergence and as y does not represent the true label distribution (due to noisy labels) the RCE loss term minimizes the entropy between the true and predicted distribution, p(æ; f),, as this reflect the true distribution some extent. As argued by Ma et al. (2020), SCE loss is only partially robust to noisy labels and they introduce normalized version traditional loss functions for classification, e.g. Cross-Entropy (CE), Mean Absolute Error (MAE), Focal Loss (FL) and Reverse Cross Entropy (RCE).
For instance, the general normalized form for the Cross Entropy loss named Normalized Cross Entropy (NCE) is formulated as
LNCE Lar(p(æ; xx; 0). Ek Lon(p(a;;
(13)
Furthermore, they empirically show that although robust, these normalized loss functions suffer from underfitting. To deal with this, they use a combination of "Active" and "Passive" losses. Active Losses (CE, Normalized CE, FL and Normalized FL) maximize the network's ' output probability the class position only specified by the label, whereas Passive losses (MAE, Normalized MAE, RCE, Normalized RCE) minimize the probability at atleast one other class position. Combining the passive loss with the normalized active loss such as normalized cross entropy loss helps deal with the denominator increasing significantly in eq. (13) Hence, Active Passive Loss (APL) can leverage both the robustness and convergence advantages. Formally.
LAPL C.C + B.Brrassive
(14)
Four possible combinations satisfying the APL principle are considered the original work aLNCE + 3LMAE, aLNCE BLRCE, aLNFI, + 3LMAE and oLNFL RCRC.
A.2 ALGORITHM OF PARS
The detailed algorithm of PARS, Pseudo-Label Aware Robust Sample Selection, is summarized in Algorithm
13

---

Under review as a conference paper at ICLR 2022
Algorithm PARS: Pscudo-Label Aware Robust Sample Selection
1: Input:
Noisy labeled dataset 2> {(z, 2) ()) Ji=i confidence threshold т. self-training loss weight Äs, negative learning loss weight AN. confidence penalty weight AR. current epoch e, total epochs E, warm-up epoch we (we E), original model parameters 0 and model parameters after warmup 0.
2: while e < We do
3:
mino [D]
Cn
Cnr.
D Robust Warm-up training as in Equation 5
4: end while
5: for e We to E do
6: Lines 7-8, Robust Sample Selection
7:
Da (a..)) v) E D maxe (p(:;)) ] >
Ambiguous Label Set D Noisy Label Set
8:
DN D DA
9:
Line 10, Noise-Aware training with filtered raw labels
10:
(r)w(() = PA E(ew)eD, Cau (p(=;; 0), + Av ( Eia.)eDD (p(a;));
px
11:
Lines 12-13, Noise-Aware self-training with filtered pseudo-labels
12:
Predict positive pseudo-label, 2 and negative pseudo-label. 2 given in Equation 8
Epacnda = TeAT E(or)ED, (ple 0))
AN
[PN] Em (xi(p(a:0),
14:
Calculate Confidence Penalty (re given in Equation
10
15:
Liorat (0) = Craw (0) + Ap (0) (0)+ ARCreg
Final Loss
16:
end for
return Leoret (0)
A.3 IMPLEMENTATION DETAILS
backbone model following previous works (Li et al. 2020). All networks are trained using SGD with momentum 0.9, weight decay 5e-4 and batch size of 128. The models trained for 100 epochs. For the learning rate schedule, we use a cosine learning rate decay (Loshchilov & Hutter 2016) which changes the learning rate rcoos( 16T) where is the initial learning rate, the current training step and T are the total number of training steps. The initial learning is set to 0.03, Às to 1.0, Av to 0.1, AR to 1.0, and the confidence threshold for sample selection is set to 0.95 unless stated otherwise. For the robust warm-up training, we train with the combination of Normalized Cross Entropy (CNCr) and Mean Absolute Error (CMAA) The weights for the NCE and MAE are 1.0 for CIFAR-1O. and 10.0 and 1.0 respectively for C1FAR-100. The warm-up for CIFAR- 10 is done for 25 epochs if the noise ratio below 0.6, and 40 epochs if the noise ratio is above 0.6. For CIFAR-100, the warm-up is done for 50 epochs. For all experiments three times the original batch size for the self-training loss. The confidence based thresholding (T 0.95) is also performed this pseudo-labeles batch of data samples for the positive and negative learning. We apply Exponential Moving Average to the model parameters with decay of 0.999. We leverage RandAugment (Cubuk et al. 2020) for strong augmentations
CIFAR-10/100. We use PreAct Resnet-18 (He et al. 2016b)
CIFAR-10 and CIFAR- 100 datasets, we sample data with
the data samples, and keep the number of augmentations fixed to 2 for fair comparison. All our models are trained using the PyTorch framework (Paszke al. 2019). The code will be made publicly available for future research purposes.
The backbone network used for the Clothing 1M dataset ResNet-50 initialized with pre-trained ImageNet weights following previous work (Li et al. 2020). We train the network using SGD with momentum of 0.9, weight decay of 0.001 and batch size of 32. The warm-up is done for 10 epochs with the Symmetric Cross Entropy loss (LSCr) with the same hyper-parameter settings as proposed in the paper (Wang et al., 2019b). The initial learning rate is set to 0.002, and decayed by factor of 10 every 25 epochs. We set Àg to 1.0 and xy to 0,01, Ag to 1.0. For strong augmentations, we use RandAugment (Cubuk et al. 2020 as in CIFAR datasets with the number augmentations fixed to 4.
Clothing
Choice of Robust Loss CRL. In each experiment, we test several different robust loss functions (Equation ( and then pick the best performing LRL based the validation loss for PARS. For fair comparison, we design a baseline that trained using the selected robust loss function alone in each experiment, which is equivalent performing the warm-up training thoroughly without the
14

---

Under review as a conference paper at ICLR 2022
rest of PARS (Equation (5). Comparison of this baseline and the final PARS can demonstrate the importance of different components of our proposed method, PARS, independently of the choice of a robust loss function. Note that studying which noise robust loss is best suited for each dataset and/or noise setting beyond the scope of this paper, which we leave for future work.
A.4 ADDITIONAL EXPERIMENTS
Convergence Test Accuracy. Figure 3 we compare the test accuracies for our final model, PARS trained on all the original and pseudo labels vs. the model ablation without any negative learning losses on both the original and pseudo labels (PARS w/o negative learning). PARS outperforms the model trained without negative learning and achieves stable and better performance. This demonstrates the effectivvness of training with negative learning losses.
MMMW
90
Accurayy
85-
80
75-
PARS|ours)
PARS negative learning
10
Epochs
Figure 3: Test accuracy curves after robust warmup training CIFAR- 10 dataset with 40% asymmetric noise.
Noise Robust Loss for Corrected Pseudo Labels. As discussed in the main paper, the corrected pseudo labels are trained with a positive learning loss unlike the ambiguous original labels (which are trained with a noise robust loss). Here, we empirically evaluate the performance of the model on the CIFAR-10 dataset with 50% symmetric label noise when trained with noise robust loss for corrected pseudo labels. This model achieves the test accuracy of 92.9%, much lower than the accuracy achieved of 95.8% with the proposed PARS model. This shows that the model underfits when trained with the noise robust losses with clean/corrected pseudo labels. This is in line with what observed in previous work (Ma al. (2020)).
Ablation study on Confidence Threshold, model PARS with 90% symmetric noise on the CIFAR- 10 dataset at different confidence threshold for sample selection. The model accuracy drops below threshold 0.95 due to the poor quality of pseudo-labels (and original labels) in the separated corrected (or ambiguous) label sets. Setting a higher threshold keeps only the high-confident and reliable predictions and hence leads better performance.
Figure shows the test accuracy for our final
33.6
Text
82.8
92.6
0.5
0.6
0.7
0.8
6.9
Phrushold
Figure 4: Test accuracy of PARS at different threshold rates on CIFAR-10 dataset with 90% symmetric noise.
15


Reviews:
[{"review_id": "w0gFSYGws4w", "reviewer": "Reviewer_yGpG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths__\n-\tThe paper address problems arising from the three main approaches used to tackle noisy labels. They discuss about the shortcomings of sample selection algorithms, loss functions designed for noisy distribution and label correction/pseudo-labelling algorithms. While all of them reduce noisy labels in successive iterations, a common problem stems from confirmation bias of the network due to higher noise in labels. The authors study and explain various ways to address this issue and propose a noise aware loss mechanism that adjust the weight for ambiguous and noisy labels, reducing the training error from prior approaches.\n-\tFor sample selection mechanism, the authors use the confidence of all classes for thresholding and decide based on the highest probability value of all present classes. This is different from prior works which use the probability value of raw label (which could be noisy pseudo-label) class for thresholding and deciding if the sample is noisy or clean. The authors’ proposed mechanism only uses the network’s output for separating the samples into ambiguous and noisy set and can reduce noise propagation from one iteration to the other.\n-\tThe use of different loss functions for different sample set based on the noise level is also intuitive to adjust the impact they have during the training process. Dealing with ambiguous and noisy labels with independent losses allows adjustment of feedback from these labels better. This is one of the valid methods to handle learning from noisy labels which this paper sheds more light on and their results in CIFAR-10/100 validates the claim. Clothing1M scores are similar to prior work which hints that this loss formulation is equally valid as previous methods.\n-\tThe proposed method shows better resilience to noisy labels when learning in a low-resource semi-supervised setting and significantly outperforms prior works. Compared to FixMatch which is dedicated to learn using semi-supervised approach on low resource, the proposed approach is able to handle unlabelled data better and reduce impact from noise.\n\n__Weakness__\n-\tThe authors specify they use pseudo-labels from both D_A and D_N in Eq 9 to train the network. Is this pseudo-label based self-training done together with noise aware training with raw labels? Or is it trained in alternating iterations with either of those losses? \n-\tContinuing on previous point, a large gain is seen with the use of pseudo-labels in training which suggests the model’s prediction after warm-up is better than the raw labels. In that case, the raw labels might differ from the pseudo-labels significantly. The small performance gap by removing negative learning in both pseudo and raw loss shows its more dependent on the positive learning using perceived labels. In this case, the network is being trained using both raw label and pseudo-label for the same sample at the same time, which could be different from each other. Is there a detailed explanation for this occurrence?\n-\tHave the authors tried not using raw labels after warm-up and only use the pseudo-labels for all loss formulations? This is interesting as biggest gain comes from pseudo-labels after the initial warm-up.\n-\tFrom appendix A.4, using noise robust loss on pseudo-labels performs lower than using positive learning from Eq 9. This finding is also interesting as the pseudo-label would also be assumed to have certain ambiguity to it that could benefit from the noise robust loss. Is the model underfit caused by pseudo-labels being very accurate that positive learning has better feedback than a toned-down robust loss?\n-\tSince the approach shows more impact in semi-supervised setting, perhaps including more study on that similar to FixMatch would benefit the paper and the readers more.\n", "summary_of_the_paper": "This paper proposes a unified approach to handle noisy labels for training neural networks and utilize all the training data to learn effectively when noise is present. The authors utilize the assumed correct and noisy labels with different loss functions adjusted using different weights in order to adjust the impact on training. The authors show how their approach improves over CIFAR-10 and CIFAR-100 with high noise ratio and demonstrate competitive results in Clothing1M dataset. The authors also show strong results in semi-supervised setting, outperforming FixMatch significantly.", "summary_of_the_review": "The paper provides a unified algorithm to learn from noisy label and shows convincing results on various datasets. The results on semi-supervised setting are also appreciated and can be useful for future research in this direction.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gjmhpa3KdLR", "reviewer": "Reviewer_NCaX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- S1: A novel combination of the existing successful approaches with a combination of novel contributions in some components (e.g., sample selection)\n- S2: Good empirical gains over LNL SOTA methods (Table 3)\n\nWeaknesses\n- W1: Fig 1 is not clear. As the method seems having the order of operations, the figure without the order of operations is not clear. In addition, it is not clear how the pseudo-label generator is used to label the noisy labeled data.\n- W2: Empirical gain in small noise regime (SYM 20%-50% in CIFAR-10/100) are small.\n- W3: No empirical gain against SOTA in large dataset (Clothing1M)\n- W4: There are large gains in large noise regime (more than 80%). But this set-up may not be very realistic.", "summary_of_the_paper": "The paper combines three branches of approaches (1. sample selection, 2. noise robust loss, 3. label correction) to address label noise in classification in a single framework. Specifically, the method includes (1) warm up phase, (2) a novel label-free sample selection, (3) noise aware loss (as a standard technique) and (4) self-training with pseudo labels along with the given labels. The proposed method outperforms prior arts in CIFAR-10/100, especially in high noise regimes (80-90%). More interestingly, in the small sample training regime, gains by the proposed method increase significantly in evaluations with CIFAR-10/100.", "summary_of_the_review": "The proposed method is a novel combination of existing successful components for LNL. Although the proposed method is somewhat novel, the empirical gain does not assert the benefit of the proposed approach. In addition, some of the presentation (e.g., Fig 1) is not clear to understand the proposed method as it omits the order of the procedure.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2T_WkjXcuOV", "reviewer": "Reviewer_mAAg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper proposes an easy-to-implement method PARS for LNL task and it obtains certain improvements under the traditional LNL setting especially with a large noise ratio and a low-resource semi-supervised LNL setting. The ablation study also shows the contributions of each of the two main improvements in this paper.\n2. The paper is well organized and easy to follow, with clear explanations of the algorithm details.\n\nWeaknesses:\n1. This paper mainly makes use of two existing techniques, i.e., negative learning and FixMatch (a semi-supervised learning method) to build a better framework. While the motivation of adopting negative learning is relatively clear, I think the authors should further explain the motivation of applying the pseudo-labeling for the entire datasets instead of only the noisy data, the reason of using strong data augmentation in this step, and the difference compared to FixMatch. Since the pseudo-labeling contributes a lot to the results as shown in Table 4, further discussions and analyses are needed to figure out why the pseudo-labeling is so important in PARS? Are the pseudo labels more accurate than original labels or does the consistency between the model predictions of a sample with weak and strong augmentations matter? I think some in-depth analyses lack here. Besides, the authors could try replacing the strong augmentation with weak augmentation to show its advantage with experimental results. \n2. In my opinion, when the noise ratio becomes large, from the perspective of PARS, the LNL task becomes more similar to a semi-supervised learning task, thus PARS (part of which inherits from FixMatch) would achieve a high performance as shown in Table 1. However, when the noise ratio is small, i.e., 20% or 50%, PARS doesn’t perform the best on CIFAR-100 dataset. I think the reason of this situation should be further discussed. Also, on Clothing1M dataset, is the fact that PARS doesn’t perform the best possibly related to the noise ratio in this dataset? \n3. Some claims or components of PARS are not evaluated. (1) In Equation 6, a label-free conﬁdence-based thresholding is proposed and the authors claim that it would reduce the self-conﬁrmation bias. Are there any empirical results that would support this claim? (2) I think an additional ablation study is needed for validating the effectiveness of Equation 10.\n", "summary_of_the_paper": "This paper proposes a hybrid framework PARS for Learning with Noisy Labels (LNL) task. The framework jointly leverages the original noisy labels and estimated pseudo labels of all samples for model training. Specifically, for samples whose maximum classification probabilities are higher than a threshold, their original/pseudo labels are used in robust/positive learning, while for the remaining samples, their original/pseudo labels are used in negative learning. When using pseudo labels, strong augmentations are also applied to the samples. Experiments conducted on three public datasets with the traditional and a new low-resource semi-supervised LNL settings show certain improvements over existing methods.", "summary_of_the_review": "This paper unifies several well-performed methods for LNL task to build a better framework, which achieves certain improvements especially under high-noise scenarios. However, some in-depth analyses about the effectiveness of the pseudo-labeling in PARS and the inferior performances under some settings still lack in this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper a framework of learning with noisy labels named PARS that combines three types of approaches, i.e., sample selection, noise robust loss, and label correction.  The framework leverages both original noisy labels and estimated pseudo labels of all samples for improving the training performance, and the empirical studies demonstrated competitive results on CIFAR datasets especially in high-noise and low-resource settings. 

Reviewers raised some major concerns about the weaknesses. For example, empirical gain in small noise regime are small or negligible, and no empirical gain against SOTA in large dataset with real-world noise (Clothing1M).  While large gains in large noise regime (more than 80%), such setting may not be very realistic and there also  lack of in-depth analysis on the sources of the gain (e.g., it is unknown if the gain is mainly because of using a better SSL or other factors since LNL becomes more similar to SSL when noise is very high). For technical novelty perspective, while the proposed approach is new, the overall novelty may not be very significant as this paper mainly combines existing techniques, e.g., negative learning and FixMatch (a semi-supervised learning method) in the proposed learning approach. 

Authors have made great efforts for addressing the reviewers’ concerns partly, but some major concerns on the technical novelty and empirical studies remain.  Therefore, the paper is not recommended for acceptance in its current form. I hope authors found the review comments and discussions useful and constructive, and like to see it accepted in the near future after these issues are fully addressed.
Decision: Reject