PARS: PSEUDO-LABEL AWARE ROBUST SAMPLE SELECTION FOR LEARNING WITH NOISY LABELS

1 INTRODUCTION . Deep neural networks rely on large-scale training data with human annotated labels for achieving good performance ( Deng et al. , 2009 ; Everingham et al. , 2010 ) . Collecting millions or billions of labeled training data instances is very expensive , requires significant human time and effort , and can also compromise user privacy ( Zheng et al. , 2020 ; Bonawitz et al. , 2017 ) . Hence , there has been a paradigm shift in the interests of the research community from large-scale supervised learning ( Krizhevsky et al. , 2017 ; He et al. , 2016a ; Huang et al. , 2017 ) to Learning with Noisy Labels ( LNL ) ( Natarajan et al. , 2013 ; Goldberger & Ben-Reuven , 2016 ; Patrini et al. , 2017 ; Tanno et al. , 2019 ) and/or unlabeled data ( Berthelot et al. , 2019b ; a ; Sohn et al. , 2020 ) . This is largely due to the abundance of raw unlabeled data with weak user tags ( Plummer et al. , 2015 ; Xiao et al. , 2015 ) or caption descriptions ( Lin et al. , 2014 ) . However , it is not trivial to build models that are robust to these noisy labels as the deep convolutional neural networks ( CNNs ) trained with cross-entropy loss can quickly overfit to the noise in the dataset , harming generalization ( Zhang et al. , 2016 ) . Most of the existing approaches on LNL can be divided into three main categories . First , several noise robust loss functions ( Ghosh et al. , 2017 ; Wang et al. , 2019a ; Zhang & Sabuncu , 2018 ) were proposed that are inherently tolerant to label noise . Second , sample selection methods ( also referred to as loss correction in some literature ) ( Han et al. , 2018 ; Yu et al. , 2019 ; Arazo et al. , 2019 ) are a popular technique that analyzes the per-sample loss distribution and separates the clean and noisy samples . The identified noisy samples are then re-weighted so that they contribute less in the loss computation . A challenge in this direction is to design a reliable criterion for separation and hence prevent overfitting to highly confident noisy samples , a behavior known as self-confirmation bias . Third , label correction methods attempt to correct the noisy labels using class-prototypes ( Han et al. , 2019 ) or pseudo-labeling techniques ( Tanaka et al. , 2018 ; Yi & Wu , 2019 ) . However , in order to correct noisy labels , we typically need an extra ( usually small ) set of correctly labeled validation labels . In particular , these methods can fail when the noise ratio is high and estimating correct labels or high-quality pseudo-labels is non-trivial . More recently , the success of several state-of-the-art LNL methods is attributed to leveraging SemiSupervised Learning ( SSL ) based approaches ( Li et al. , 2020 ; Kim et al. , 2019 ) . Typically , a sample selection technique is applied to separate clean and noisy labels in the training data , then the noisy labels are deemed unreliable and hence treated as unlabeled in a SSL setting . Following the recent SSL literature ( Lee et al. , 2013 ; Arazo et al. , 2020 ) , estimated pseudo-labels are usually used to replace the filtered noisy labels during training . These approaches have shown to be highly tolerant to label-noise . However , the noisy labels are always discarded in favor of pseudo-labels in all the existing literature , but they may still contain useful information for training . Pseudo-labeling is in turn only applied to the filtered noisy subset while the rest of the raw labels are typically used as is , which makes it sensitive to the quality of the filtering algorithm . In particular , motivated by a simple principle of making the most of the signal contained in the noisy training data , we design PARS , short for Pseudo-Label Aware Robust Sample Selection . Our contributions are as follows : 1 . PARS proposes a novel , principled training framework for LNL . It trains on both the original labels and pseudo-labels . Unlike previous works , instead of filtering and then discarding the low-confident noisy labels , PARS uses the entire set of original labels , and applies self-training with pseudo-labeling and data augmentation for the entire dataset ( rather than the filtered noisy data only ) . 2 . PARS is able to learn useful information from all the available data samples through label-dependent noise-aware loss functions . Specifically , in order to prevent overfitting to inaccurate original labels ( or inaccurate pseudo-labels ) , PARS performs a simple confidencebased filtering technique by setting a high threshold on their predicted confidence , and applies robust/negative learning ( or positive/negative learning ) accordingly . 3 . We perform extensive experiments on multiple benchmark datasets i.e . noisy CIFAR-10 , noisy CIFAR-100 and Clothing1M . Results demonstrate that PARS outperforms previous state-of-the-art methods by a significant margin , in particular when high level of noise is present in the training data . We also conduct sufficient ablation studies to validate the importance of our contributions . 4 . We design a novel low-resource semi-supervised LNL setting where only a small subset of data is weakly labeled ( Section 4.3 ) . We show significant gains over state-of-the-art approaches using PARS . This setting is particularly interesting when it is hard to obtain large-scale noisy labeled data . In particular , we find that surprisingly none of the existing LNL methods outperform a baseline SSL model ( FixMatch ) ( Sohn et al. , 2020 ) that is not even designed to handle label noise , and yet PARS can achieve up to an absolute 27 % improvement in test accuracy in a controlled high-noise low-resource setting . 2 RELATED WORK . In recent literature on LNL , methods typically fall into three design categories to learn a noise robust model : noise robust loss functions , sample selection approaches , or label correction methods . Noise robust loss function based methods propose objective functions that are tolerant to label noise . A commonly used loss function that is identified to be robust to noisy labels is Mean Absolute Error ( MAE ) ( Ghosh et al. , 2017 ) . Wang et al . ( 2019a ) proposed Improved MAE which is a re-weighted version of MAE . Zhang & Sabuncu ( 2018 ) proposed Generalized Cross Entropy Loss ( GCE ) which is a generalization of MAE and Categorical Cross Entropy loss . More recently , Wang et al . ( 2019b ) designed a Symmetric Cross Entropy ( SCE ) loss which is similar in spirit to the symmetric KLdivergence and combines the cross-entropy loss with the reverse cross-entropy . Although SCE is robust to noisy labels , Ma et al . ( 2020 ) proposed a normalized family of loss functions which are shown to be more robust than SCE for extreme levels of label noise . Kim et al . ( 2019 ) and Kim et al . ( 2021 ) designed a framework that alternates between positive learning on accurate/clean labels and negative learning on complementary/wrong labels . Loss correction approaches explicitly modify the loss function during training to take into account the noise distribution by modeling a noise transition matrix ( Patrini et al. , 2017 ; Tanno et al. , 2019 ; Xia et al. , 2019 ; Goldberger & Ben-Reuven , 2016 ) or based on label-dependent weights ( Natarajan et al. , 2013 ) . Another family of methods focus primarily on sample selection , where the model selects small-loss samples as “ clean ” samples under the assumption that the model first fits to the clean samples before memorizing the noisy samples ( also known as the early-learning assumption ) ( Arpit et al. , 2017 ; Zhang et al. , 2016 ; Liu et al. , 2020 ) . Han et al . ( 2018 ) and Yu et al . ( 2019 ) proposed Co-teaching where sample selection is conducted using two networks to select clean and noisy samples , and then the clean samples are used for further training . MentorNet ( Jiang et al. , 2018 ) is a student-teacher framework where a pre-trained teacher network guides the learning of the student network with clean samples ( whose labels are deemed “ correct ” ) . In the decoupling training strategy proposed by Malach & Shalev-Shwartz ( 2017 ) , two networks are trained simultaneously and guide each other on when and how to update . One limitation of these approaches is that they ignore all the noisy/unclean samples during training and only leverage the expected-to-be clean samples for improving performance . Li et al . ( 2020 ) proposed DivideMix where sample selection is conducted based on per-sample loss distribution , and then the noisy samples are treated as unlabeled data in a SSL setting ( Berthelot et al. , 2019b ) . Nishi et al . ( 2021 ) further investigated the potential of using augmentation strategies in LNL , one used for loss analysis and another for learning , thus improving generalization of DivideMix . Compared to the above , label correction aims to improve the quality of the noisy labels by explicitly correcting the wrong labels . Tanaka et al . ( 2018 ) and Yi & Wu ( 2019 ) predicted the correct labels either as estimates of label probabilities ( soft labels ) or as one-hot class labels ( hard labels ) . Arazo et al . ( 2019 ) combined label correction with iterative sample selection by first filtering clean labels from noisy labels modeling a two-component Beta Mixture , and then estimating to correct labels for those noisy samples . Tanaka et al . ( 2018 ) combined label correction with additional regularization terms that were proved to be helpful for LNL . Song et al . ( 2019 ) also used label replacement to refurbish a subset of labels , thereby gradually increasing the number of available training samples . Liu et al . ( 2020 ) computed soft labels as model predictions and then exploit it to avoid memorization . 3 METHODOLOGY . 3.1 PRELIMINARIES . For a K-class classification problem with noisy labels , let D = { ( xi , yi ) } n i=1 with x ∈ X ⊂ Rd as the input features and y ∈ Y = { 1 , . . . , K } as the corresponding label , we aim to learn a classifier p ( · ; θ ) : X → Y parametrized by θ . For a clean labeled set , i.e.where yi is the true label for the i-th training sample xi , we can learn the model parameters θ by minimizing the Cross Entropy ( CE ) loss : min θ 1 |D| ∑ ( x , y ) ∈D LCE ( p ( x ; θ ) , y ) . ( 1 ) However , in the presence of noisy labels where yi is possibly incorrect for the i-th training sample xi , the cross-entropy loss can quickly overfit to the noise in the dataset ( Zhang et al. , 2016 ) . Robust Learning . Variants to the CE loss have been proposed to improve the classification performance under label noise . Examples of such noise robust loss functions include Mean Absolute Error LMAE ( Ghosh et al. , 2017 ) , Symmetric Cross Entropy LSCE ( Wang et al. , 2019b ) , Normalized Cross Entropy LNCE or Active Passive Loss LAPL ( Ma et al. , 2020 ) . We may simply replace the LCE with any of these noise robust loss , denoted by LRL , in the optimization Equation ( 1 ) to perform Robust Learning : LRL ∈ { LMAE , LSCE , LNCE , LAPL } . ( 2 ) For the sake of the limited space , definitions and further discussion about the robust losses are deferred to Appendix A.1 . Positive and Negative Learning . A particular variant to the CE loss that is designed to handle noisy/wrong labels is called Negative Learning ( Kim et al. , 2019 ) . Given a sample with a correct label ( x , y ) , Positive Learning ( PL ) is equivalent to optimizing the standard CE loss in Equation ( 1 ) , so that we optimize the output probabilities corresponding to the true label to be close to 1 : LPL ( p ( x ; θ ) , y ) : = LCE ( p ( x ; θ ) , y ) = K∑ k=1 [ y ] k log [ p ( x ; θ ) ] k , ( 3 ) where the class label y is one-hot encoded , and [ · ] k denotes the k-th dimension of a vector . Now for a data example with a wrong label ( x , ȳ ) , Negative Learning ( NL ) applies the CE loss function to fit the “ complement ” of the label in Equation ( 1 ) , so that the predicted probabilities corresponding to the given label is optimized to be far from 1 : LNL ( p ( x ; θ ) , ȳ ) : = − K∑ k=1 [ ȳ ] k log [ 1− p ( x ; θ ) ] k . ( 4 ) Sample Selection . In order to apply the Positive and Negative Learning framework in LNL , Kim et al . ( 2019 ) proposed to select samples based on a simple threshold of the per-sample loss value , i.e.a data point ( x , y ) is selected to be clean for PL if [ p ( x ; θ ) ] y > γ where γ is a noise-aware threshold . This simple yet effective procedure is based on the commonly used `` small-loss '' assumption that smaller-loss samples ( typically in early learning stages ) are usually easier to learn and hence expected to be “ clean ” , and it has been successfully applied in several other studies ( Jiang et al. , 2018 ; 2020 ) . 3.2 PARS : PSEUDO-LABEL AWARE ROBUST SAMPLE SELECTION We propose PARS : Pseudo-Label Aware Robust Sample Selection . The method is illustrated in Figure 1 ( and Algorithm 1 in Appendix A.2 ) , and each component is discussed in detail below . In particular , there are three major differences between PARS and previous works : 1 ) PARS does not treat the high-confidence samples as `` clean '' , but rather applies robust loss functions to address the relatively low yet non-negligible level of noise . 2 ) PARS does not discard the noisy ( or lowconfidence ) labels , but rather applies a negative learning framework to learn the “ complement ” of these labels ( Kim et al. , 2019 ) . This constitutes one major difference between the selective negative learning procedure of Kim et al . ( 2019 ) and PARS , that Kim et al . ( 2019 ) only apply it during the warm-up stage and still choose to discard the filtered noisy labels in favour of pseudo-labels in a SSL manner , and we unify the two stages and show improved performance . 3 ) We improve the convergence of PARS by self-training using pseudo-labels and data augmentation for the entire dataset as a proxy to enhance clean labels and correct noisy labels . Robust Warm-up Training . In the presence of label noise , the model overfits to noise when trained with CE loss and produces overconfident and wrong predictions after a few epochs . This problem is more severe with high levels of label noise . To overcome this , we perform “ robust warm-up ” using a robust loss for a few epochs for the model to produce reliable and confident predictions . Specifically , in the warm-up training phase , we minimize : min θ 1 |D| ∑ ( x , y ) ∈D LRL ( p ( x ; θ ) , y ) , ( 5 ) where LRL can be any loss from Equation ( 2 ) . In our experiments , we test different loss functions for warm-up training ( for more details see Appendix A.3 ) . Robust Sample Selection . As shown in Figure 2a , after the robust warm-up training , we observe that there is a weak trend that high-confidence samples tend to have clean labels . Given the model parameters θ̂ at the current iteration and a confidence threshold τ ∈ [ 0 , 1 ] , we divide the entire dataset D ( or similarly , a minibatch B ⊂ D ) into an ambiguous1 set DA and noisy set DN of samples by thresholding the maximum of the predicted confidence probabilities over all classes : DA = { ( x , y ) ∈ D|max k [ p ( x ; θ̂ ) ] k > τ } , DN = D \ DA . ( 6 ) A notable difference is that our confidence-based thresholding is label-free ( i.e.thresholding of the maximum confidence over all classes ) , while the widely used loss-based thresholding Kim et al . ( 2019 ) depends on the ground-truth ( noisy ) label ( i.e.thresholding the confidence corresponding to the given class ) . Our approach has two advantages : 1 ) our selective criterion depends on the predicted confidence probabilities alone , therefore it is straightforward to apply to cases where ground-truth labels are unavailable ( e.g.in a semi-supervised setting in Section 4.3 ) ; 2 ) our strategy helps to reduce the self-confirmation bias of propagating the mistake of overfitting to possibly noisy samples , because our model only selects samples that it is confident to make a prediction on regardless of the correctness of the raw label . Notably , previous works such as Jiang et al . ( 2018 ) ; Li et al . ( 2020 ) usually maintain two divergent networks ( one for sample selection and one for the main classification task ) in order to avoid the self-confirmation bias . Our method proves to be highly resilient to the overfitting issue with a single network simultaneously updated to perform both sample selection and classification , which drastically simplifies implementation and training complexity . Label-Dependent Noise-Aware Loss Functions . Despite the robust sample selection of dividing samples into ambiguous and noisy set , we propose to learn useful information from all the available training data instead of discarding the raw labels of the noisy samples as typically done in previous literature ( Jiang et al. , 2018 ; Kim et al. , 2019 ; Li et al. , 2020 ) . Specifically , we propose to use two separate loss functions , tailored for either the ambiguous or noisy labels respectively , which prevents overfitting to noise and enables further filtering of labels effectively ( Figure 2b ) . For the ambiguous set , DA , we train with the robust learning loss , LRL , ( same loss used during the warm-up training ) , where we deem that the given label in this set may still contain non-negligible noise . For the noisy set , DN , we train with the negative learning loss , LNL , to learn the complement of the given raw labels for the identified least-confident highly-noisy samples . The overall label-dependent noise-aware loss function for the proposed robust/negative learning framework using the given raw labels is thus : Lraw ( θ ) : = 1 |DA| ∑ ( x , y ) ∈DA LRL ( p ( x ; θ ) , y ) + λN 1 |DN | ∑ ( x , ȳ ) ∈DN LNL ( p ( x ; θ ) , ȳ ) , ( 7 ) where λN is the weight for the negative learning loss term , andDA , DN are defined as in Equation ( 6 ) . We set a high threshold ( e.g.0.95 throughout our experiments ) on the predicted confidence in the robust sample selection ( Equation ( 6 ) ) since we observe that this yields better performance as shown 1We choose the term “ ambiguous ” rather than the typically used term “ clean ” ( Jiang et al. , 2018 ; 2020 ) because there is still significant amount of noise in the expected-to-be clean samples when a simple thresholding method is used to select samples ( Arazo et al. , 2019 ) . in Appendix A.4 . We will show that , by using the noisy labels in the negative learning setting , we are able to consistently improve the model performance ( Table 4 ) . Self-Training with Pseudo-Labels . After the robust warm-up training , the initial convergence of the model leads to reliable predictions which can be used as labels to further improve model convergence via self-training . Therefore , in parallel to using the given raw labels during training , we use pseudolabels generated by the network itself to guide the learning process . Given the model parameters θ̂ at the current iteration , for each sample ( x , · ) ∈ D , we generate the underlying pseudo-label z corresponding to the model ’ s most confident class , and then randomly sample a complementary “ wrong ” pseudo-label z̄ from the remaining classes following Kim et al . ( 2019 ) : z : = arg max k [ p ( x ; θ̂ ) ] k , z̄ ∈ { 1 , . . . , K } \ { z } . ( 8 ) Similarly to how we use label-dependent noise-aware loss functions for the given raw labels in the ambiguous and noisy sets , we train with the pseudo-labels from the two sets using different losses too . Specifically , we apply positive/negative learning for the generated pseudo-labels in the ambiguous/noisy set respectively : Lpseudo ( θ ) : = 1 |DA| ∑ ( x , · ) ∈DA LPL ( p ( x ; θ ) , z ) +λN 1 |DN | ∑ ( x , · ) ∈DN LNL ( p ( x ; θ ) , z̄ ) , ( 9 ) where λN is the weight for the negative learning loss term ( equal to the weight in Equation ( 7 ) ) , DA , DN are defined as in Equation ( 6 ) , z , z̄ are defined as in Equation ( 8 ) . We find that it suffices to train with the standard CE loss for self-training with pseudo-labels from the ambiguous set , without the need for noise robust learning . One caveat of applying self-training is that , the model ’ s own predictions may become unreliable when most of the samples are primarily guided by the self-training , encouraging the network to predict the same class to minimize the loss . We observe that this is particularly the case under high levels of label noise . To reliably apply self-training , we regularize Equation ( 9 ) with a confidence penalty ( Tanaka et al. , 2018 ; Arazo et al. , 2019 ) : Lreg ( θ ) : = K∑ k=1 [ p ] k log ( [ p ] k [ h ( D ; θ ) ] k ) , ( 10 ) where [ p ] k is the prior probability distribution for class k , and [ h ( D ; θ ) ] k is shorthand for the mean predicted probability of the model for class k across all samples in the dataset D. In our experiments , we assume a uniform distribution for the prior probabilities ( i.e . [ p ] k = 1/K ) , while approximating h ( D ; θ ) by h ( B ; θ ) using mini-batches B ⊂ D as suggested by Tanaka et al . ( 2018 ) . The generated pseudo-labels become reliably confident with help of the regularization , which guides the robust sample selection to resemble the true distribution more closely ( Figure 2c ) . PARS : Pseudo-Label Aware Robust Sample Selection . After the robust warm-up training stage ( Equation ( 5 ) ) , our final model , PARS , is then trained using both the original labels and the selfgenerated pseudo-labels with robust sample selection . The final loss is given by : min θ Ltotal ( θ ) : = Lraw ( θ ) + λSLpseudo ( θ ) + λRLreg ( θ ) , ( 11 ) where Lraw , Lpseudo , Lreg are defined as in Equations ( 7 ) , ( 9 ) and ( 10 ) respectively , and λS , λR are the weights for the self-training loss and confidence penalty term . Data Augmentation . For image classification tasks in LNL , data augmentation techniques have recently been shown to significantly improve generalization ( Nishi et al. , 2021 ) . We adapt such data augmentation strategies to further improve PARS for benchmarking image classification performance in our experiments . Specifically , we apply weak augmentation ( e.g.standard random flip , crop , normalization ) to all data samples in the robust/negative learning with original raw labels in computing Equation ( 7 ) . We also apply strong augmentation ( e.g.AutoAugment ( Cubuk et al. , 2019 ) , RandAugment ( Cubuk et al. , 2020 ) ) to all data samples in the positive/negative learning with pseudo-labels in computing Equations ( 9 ) and ( 10 ) . We do not incorporate strong augmentation for training with the original labels as it leads to poor performance in presence of high label noise , as also observed by Nishi et al . ( 2021 ) . In this way , PARS can be seen as one example of how to extend the work of Nishi et al . ( 2021 ) by combining weak and strong data augmentation effectively with noisy labels and pseudo-labeling , as an effective technique to further advance the state of the art in LNL .

Summary: This paper proposes a hybrid framework PARS for Learning with Noisy Labels (LNL) task. The framework jointly leverages the original noisy labels and estimated pseudo labels of all samples for model training. Specifically, for samples whose maximum classification probabilities are higher than a threshold, their original/pseudo labels are used in robust/positive learning, while for the remaining samples, their original/pseudo labels are used in negative learning. When using pseudo labels, strong augmentations are also applied to the samples. Experiments conducted on three public datasets with the traditional and a new low-resource semi-supervised LNL settings show certain improvements over existing methods.