Learning with Instance-Dependent Label Noise: A Sample Sieve Approach

1 INTRODUCTION . Deep neural networks ( DNNs ) have gained popularity in a wide range of applications . The remarkable success of DNNs often relies on the availability of large-scale datasets . However , data annotation inevitably introduces label noise , and it is extremely expensive and time-consuming to clean up the corrupted labels . The existence of label noise can weaken the true correlation between features and labels as well as introducing artificial correlation patterns . Thus , mitigating the effects of noisy labels becomes a critical issue that needs careful treatment . It is challenging to avoid overfitting to noisy labels , especially when the noise depends on both true labels Y and features X . Unfortunately , this often tends to be the case where human annotations are prone to different levels of errors for tasks with varying difficulty levels . Recent work has also shown that the presence of instance-dependent noisy labels imposes additional challenges and cautions to training in this scenario ( Liu , 2021 ) . For such instance-dependent ( or feature-dependent , instance-based ) label noise settings , theory-supported works usually focus on loss-correction which requires estimating noise rates ( Xia et al. , 2020 ; Berthon et al. , 2020 ) . Recent work by Cheng et al . ( 2020 ) addresses the bounded instance-based noise by first learning the noisy distribution and then distilling examples according to some thresholds.1 However , with a limited size of datasets , learning an accurate noisy distribution for each example is a non-trivial task . Additionally , the size and the quality of distilled examples are sensitive to the thresholds for distillation . ∗Equal contributions in alphabetical ordering . Hao leads experiments and Zhaowei leads theories . †Corresponding authors : Y. Liu and Z. Zhu { yangliu , zwzhu } @ ucsc.edu . 1The proposed solution is primarily studied for the binary case in Cheng et al . ( 2020 ) . Departing from the above line of works , we design a sample sieve with theoretical guarantees to provide a high-quality splitting of clean and corrupted examples without the need to estimate noise rates . Instead of learning the noisy distributions or noise rates , we focus on learning the underlying clean distribution and design a regularization term to help improve the confidence of the learned classifier , which is proven to help safely sieve out corrupted examples . With the division between “ clean ” and “ corrupted ” examples , our training enjoys performance improvements by treating the clean examples ( using standard loss ) and the corrupted ones ( using an unsupervised consistency loss ) separately . We summarize our main contributions : 1 ) We propose to train a classifier using a novel confidence regularization ( CR ) term and theoretically guarantee that , under mild assumptions , minimizing the confidence regularized cross-entropy ( CE ) loss on the instance-based noisy distribution is equivalent to minimizing the pure CE loss on the corresponding “ unobservable ” clean distribution . This classifier is also shown to be helpful for evaluating each example to build our sample sieve.2 ) We provide a theoretically sound sample sieve that simply compares the example ’ s regularized loss with a closed-form threshold explicitly determined by predictions from the above trained model using our confidence regularized loss , without any extra estimates . 3 ) To the best of our knowledge , the proposed CORES2 ( COnfidence REgularized Sample Sieve ) is the first method that is thoroughly studied for a multi-class classification problem , has theoretical guarantees to avoid overfitting to instance-dependent label noise , and provides high-quality division without knowing or estimating noise rates . 4 ) By decoupling the regularized loss into separate additive terms , we also provide a novel and promising mechanism for understanding and controlling the effects of general instancedependent label noise . 5 ) CORES2 achieves competitive performance on multiple datasets , including CIFAR-10 , CIFAR-100 , and Clothing1M , under different label noise settings . Other related works In addition to recent works by Xia et al . ( 2020 ) , Berthon et al . ( 2020 ) , and Cheng et al . ( 2020 ) , we briefly overview other most relevant references . Detailed related work is left to Appendix A . Making the loss function robust to label noise is important for building a robust machine learning model ( Zhang et al. , 2016 ) . One popular direction is to perform loss correction , which first estimates transition matrix ( Patrini et al. , 2017 ; Vahdat , 2017 ; Xiao et al. , 2015 ; Zhu et al. , 2021b ; Yao et al. , 2020b ) , and then performs correction/reweighting via forward or backward propagation , or further revises the estimated transition matrix with controllable variations ( Xia et al. , 2019 ) . The other line of work focuses on designing specific losses without estimating transition matrices ( Natarajan et al. , 2013 ; Xu et al. , 2019 ; Liu & Guo , 2020 ; Wei & Liu , 2021 ) . However , these works assume the label noise is instance-independent which limits their extension . Another approach is sample selection ( Jiang et al. , 2017 ; Han et al. , 2018 ; Yu et al. , 2019 ; Northcutt et al. , 2019 ; Yao et al. , 2020a ; Wei et al. , 2020 ; Zhang et al. , 2020a ) , which selects the “ small loss ” examples as clean ones . However , we find this approach only works well on the instance-independent label noise . Approaches such as label correction ( Veit et al. , 2017 ; Li et al. , 2017 ; Han et al. , 2019 ) or semi-supervised learning ( Li et al. , 2020 ; Nguyen et al. , 2019 ) also lack guarantees for the instancebased label noise . 2 CORES2 : CONFIDENCE REGULARIZED SAMPLE SIEVE Consider a classification problem on a set of N training examples denoted by D : = { ( xn , yn ) } n∈ [ N ] , where [ N ] : = { 1 , 2 , · · · , N } is the set of example indices . Examples ( xn , yn ) are drawn according to random variables ( X , Y ) ∈ X × Y from a joint distribution D. Let DX and DY be the marginal distributions of X and Y . The classification task aims to identify a classifier f : X → Y that maps X to Y accurately . One common approach is minimizing the empirical risk using DNNs with respect to the cross-entropy loss defined as ` ( f ( x ) , y ) = − ln ( fx [ y ] ) , y ∈ [ K ] , where fx [ y ] denotes the y-th component of f ( x ) andK is the number of classes . In real-world applications , such as human-annotated images ( Krizhevsky et al. , 2012 ; Zhang et al. , 2017 ) and medical diagnosis ( Agarwal et al. , 2016 ) , the learner can only observe a set of noisy labels . For instance , human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly . The label noise of each instance is characterized by a noise transition matrix T ( X ) , where each element Tij ( X ) : = P ( Ỹ = j|Y = i , X ) . The corresponding noisy dataset2 and distribution are denoted by D̃ : = { ( xn , ỹn ) } n∈ [ N ] and D̃ . Let 1 ( · ) be the indicator function taking 2In this paper , the noisy dataset refers to a dataset with noisy examples . A noisy example is either a clean example ( whose label is true ) or a corrupted example ( whose label is wrong ) . value 1 when the specified condition is satisfied and 0 otherwise . Similar to the goals in surrogate loss ( Natarajan et al. , 2013 ) , LDMI ( Xu et al. , 2019 ) and peer loss ( Liu & Guo , 2020 ) , we aim to learn a classifier f from the noisy distribution D̃ which also minimizes P ( f ( X ) 6= Y ) , ( X , Y ) ∼ D. Beyond their results , we attempt to propose a theoretically sound approach addressing a general instance-based noise regime without knowing or estimating noise rates . 2.1 CONFIDENCE REGULARIZATION . In this section , we present a new confidence regularizer ( CR ) . Our design of the CR is mainly motivated by a recently proposed robust loss function called peer loss ( Liu & Guo , 2020 ) . For each example ( xn , ỹn ) , peer loss has the following form : ` PL ( f ( xn ) , ỹn ) : = ` ( f ( xn ) , ỹn ) − ` ( f ( xn1 ) , ỹn2 ) , where ( xn1 , ỹn1 ) and ( xn2 , ỹn2 ) are two randomly sampled and paired peer examples ( with replacement ) for n. Let Xn1 and Ỹn2 be the corresponding random variables . Note Xn1 , Ỹn2 are two independent and uniform random variables being each xn′ , n′ ∈ [ N ] and ỹn′ , n′ ∈ [ N ] with probability 1N respectively : P ( Xn1 = xn′ |D̃ ) = P ( Ỹn2 = yn′ |D̃ ) = 1 N , ∀n ′ ∈ [ N ] . Let DỸ |D̃ be the distribution of Ỹn2 given dataset D̃ . Peer loss then has the following equivalent form in expectation : 1 N ∑ n∈ [ N ] EXn1 , Ỹn2 |D̃ [ ` ( f ( xn ) , ỹn ) − ` ( f ( Xn1 ) , Ỹn2 ) ] = 1 N ∑ n∈ [ N ] [ ` ( f ( xn ) , ỹn ) − ∑ n′∈ [ N ] P ( Xn1 = xn′ |D̃ ) EDỸ |D̃ [ ` ( f ( xn′ ) , Ỹ ) ] ] = 1 N ∑ n∈ [ N ] [ ` ( f ( xn ) , ỹn ) − ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] ] . This result characterizes a new loss denoted by ` CA : ` CA ( f ( xn ) , ỹn ) : = ` ( f ( xn ) , ỹn ) − ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] . ( 1 ) Though not studied rigorously by Liu & Guo ( 2020 ) , we show , under conditions3 , ` CA defined in Eqn . ( 1 ) encourages confident predictions4 from f by analyzing the gradients : Theorem 1 . For ` CA ( · ) , solutions satisfying fxn [ i ] > 0 , ∀i ∈ [ K ] are not locally optimal at ( xn , ỹn ) . See Appendix B.2 for the proof . Particularly , in binary cases , we have constraint f ( xn ) [ 0 ] + f ( xn ) [ 1 ] = 1 . Following Theorem 1 , we know minimizing ` CA ( f ( xn ) , ỹn ) w.r.t f under this constraint leads to either f ( xn ) [ 0 ] → 1 or f ( xn ) [ 1 ] → 1 , indicating confident predictions . Therefore , the addition of term −ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] helps improve the confidence of the learned classifier . Inspired by the above observation , we define the following confidence regularizer : Confidence Regularizer : ` CR ( f ( xn ) ) : = −β · ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] , where β is positive and ` ( · ) refers to the CE loss . The prior probability P ( Ỹ |D̃ ) is counted directly from the noisy dataset . In the remaining of this paper , ` ( · ) indicates the CE loss by default . Why are confident predictions important ? Intuitively , when model fits to the label noise , its predictions often become less confident , since the noise usually corrupts the signal encoded in the clean data . From this perspective , encouraging confident predictions plays against fitting to label noise . Compared to instance-independent noise , the difficulties in estimating the instance-dependent noise rates largely prevent us from applying existing techniques . In addition , as shown by Manwani & Sastry ( 2013 ) , the 0-1 loss function is more robust to instance-based noise but hard to optimize with . To a certain degree , pushing confident predictions results in a differentiable loss function that approximates the 0-1 loss , and therefore restores the robustness property . Besides , as observed by Chatterjee ( 2020 ) and Zielinski et al . ( 2020 ) , gradients from similar examples would reinforce each other . When the overall label information is dominantly informative that Tii ( X ) > Tij ( X ) , DNNs 3Detailed conditions for Theorem 1 are specified at the end of our main contents . 4Our observation can also help partially explain the robustness property of peer loss ( Liu & Guo , 2020 ) . will receive more correct information statistically . Encouraging confident predictions would discourage the memorization of the noisy examples ( makes it hard for noisy labels to reduce the confidence of predictions ) , and therefore further facilitate DNNs to learn the ( clean ) dominant information . ` CR is NOT the entropy regularization Entropy regularization ( ER ) is a popular choice for improving confidence of the trained classifiers in the literature ( Tanaka et al. , 2018 ; Yi & Wu , 2019 ) . Given a particular prediction probability p for a class , the ER term is based on the function −p ln p , while our ` CR is built on ln p. Later we show ` CR offers us favorable theoretical guarantees for training with instance-dependent label noise , while ER does not . In Appendix C.1 , we present both theoretical and experimental evidences that ` CR serves as a better regularizer compared to ER .

Summary: The paper introduces a noise-robust loss function CORES2, motivated by peer loss. The novel loss adds a regularization term that promotes confident prediction and pushes the model prediction away from the prior of the label. Using this loss function, the authors propose a dynamic sample sieve to separate the clean data and corrupted data on-the-fly, by the magnitude of CORES2 loss. The author's approach is to rule out samples whose losses are larger than an adaptive threshold. Importantly, the process of sieving successfully sieves out corrupted samples, both in a theory of 'better than random guess classifier' and in practice. The authors, then show that the proposed CORES2 can be decoupled under the instance-dependent noise setting. Then CORES2 is proved to be noise-robust, which means CORES2 is equivalent to minimizing the original cross-entropy loss. They also show a principle approach for finding the hyperparameters $\beta$. Further, a consistency loss is adopted after sample sieve on the corrupted samples. The author conducts extensive experiments, including CIFAR10, CIFAR100, and Clothing1M under different settings of noise. CORES2 achieves the SOTA results in all the experiments.