Learning with Instance-Dependent Label Noise: A Sample Sieve Approach

1 INTRODUCTION . Deep neural networks ( DNNs ) have gained popularity in a wide range of applications . The remarkable success of DNNs often relies on the availability of large-scale datasets . However , data annotation inevitably introduces label noise , and it is extremely expensive and time-consuming to clean up the corrupted labels . The existence of label noise can weaken the true correlation between features and labels as well as introducing artificial correlation patterns . Thus , mitigating the effects of noisy labels becomes a critical issue that needs careful treatment . It is challenging to avoid overfitting to noisy labels , especially when the noise depends on both true labels Y and features X . Unfortunately , this often tends to be the case where human annotations are prone to different levels of errors for tasks with varying difficulty levels . Recent work has also shown that the presence of instance-dependent noisy labels imposes additional challenges and cautions to training in this scenario ( Liu , 2021 ) . For such instance-dependent ( or feature-dependent , instance-based ) label noise settings , theory-supported works usually focus on loss-correction which requires estimating noise rates ( Xia et al. , 2020 ; Berthon et al. , 2020 ) . Recent work by Cheng et al . ( 2020 ) addresses the bounded instance-based noise by first learning the noisy distribution and then distilling examples according to some thresholds.1 However , with a limited size of datasets , learning an accurate noisy distribution for each example is a non-trivial task . Additionally , the size and the quality of distilled examples are sensitive to the thresholds for distillation . ∗Equal contributions in alphabetical ordering . Hao leads experiments and Zhaowei leads theories . †Corresponding authors : Y. Liu and Z. Zhu { yangliu , zwzhu } @ ucsc.edu . 1The proposed solution is primarily studied for the binary case in Cheng et al . ( 2020 ) . Departing from the above line of works , we design a sample sieve with theoretical guarantees to provide a high-quality splitting of clean and corrupted examples without the need to estimate noise rates . Instead of learning the noisy distributions or noise rates , we focus on learning the underlying clean distribution and design a regularization term to help improve the confidence of the learned classifier , which is proven to help safely sieve out corrupted examples . With the division between “ clean ” and “ corrupted ” examples , our training enjoys performance improvements by treating the clean examples ( using standard loss ) and the corrupted ones ( using an unsupervised consistency loss ) separately . We summarize our main contributions : 1 ) We propose to train a classifier using a novel confidence regularization ( CR ) term and theoretically guarantee that , under mild assumptions , minimizing the confidence regularized cross-entropy ( CE ) loss on the instance-based noisy distribution is equivalent to minimizing the pure CE loss on the corresponding “ unobservable ” clean distribution . This classifier is also shown to be helpful for evaluating each example to build our sample sieve.2 ) We provide a theoretically sound sample sieve that simply compares the example ’ s regularized loss with a closed-form threshold explicitly determined by predictions from the above trained model using our confidence regularized loss , without any extra estimates . 3 ) To the best of our knowledge , the proposed CORES2 ( COnfidence REgularized Sample Sieve ) is the first method that is thoroughly studied for a multi-class classification problem , has theoretical guarantees to avoid overfitting to instance-dependent label noise , and provides high-quality division without knowing or estimating noise rates . 4 ) By decoupling the regularized loss into separate additive terms , we also provide a novel and promising mechanism for understanding and controlling the effects of general instancedependent label noise . 5 ) CORES2 achieves competitive performance on multiple datasets , including CIFAR-10 , CIFAR-100 , and Clothing1M , under different label noise settings . Other related works In addition to recent works by Xia et al . ( 2020 ) , Berthon et al . ( 2020 ) , and Cheng et al . ( 2020 ) , we briefly overview other most relevant references . Detailed related work is left to Appendix A . Making the loss function robust to label noise is important for building a robust machine learning model ( Zhang et al. , 2016 ) . One popular direction is to perform loss correction , which first estimates transition matrix ( Patrini et al. , 2017 ; Vahdat , 2017 ; Xiao et al. , 2015 ; Zhu et al. , 2021b ; Yao et al. , 2020b ) , and then performs correction/reweighting via forward or backward propagation , or further revises the estimated transition matrix with controllable variations ( Xia et al. , 2019 ) . The other line of work focuses on designing specific losses without estimating transition matrices ( Natarajan et al. , 2013 ; Xu et al. , 2019 ; Liu & Guo , 2020 ; Wei & Liu , 2021 ) . However , these works assume the label noise is instance-independent which limits their extension . Another approach is sample selection ( Jiang et al. , 2017 ; Han et al. , 2018 ; Yu et al. , 2019 ; Northcutt et al. , 2019 ; Yao et al. , 2020a ; Wei et al. , 2020 ; Zhang et al. , 2020a ) , which selects the “ small loss ” examples as clean ones . However , we find this approach only works well on the instance-independent label noise . Approaches such as label correction ( Veit et al. , 2017 ; Li et al. , 2017 ; Han et al. , 2019 ) or semi-supervised learning ( Li et al. , 2020 ; Nguyen et al. , 2019 ) also lack guarantees for the instancebased label noise . 2 CORES2 : CONFIDENCE REGULARIZED SAMPLE SIEVE Consider a classification problem on a set of N training examples denoted by D : = { ( xn , yn ) } n∈ [ N ] , where [ N ] : = { 1 , 2 , · · · , N } is the set of example indices . Examples ( xn , yn ) are drawn according to random variables ( X , Y ) ∈ X × Y from a joint distribution D. Let DX and DY be the marginal distributions of X and Y . The classification task aims to identify a classifier f : X → Y that maps X to Y accurately . One common approach is minimizing the empirical risk using DNNs with respect to the cross-entropy loss defined as ` ( f ( x ) , y ) = − ln ( fx [ y ] ) , y ∈ [ K ] , where fx [ y ] denotes the y-th component of f ( x ) andK is the number of classes . In real-world applications , such as human-annotated images ( Krizhevsky et al. , 2012 ; Zhang et al. , 2017 ) and medical diagnosis ( Agarwal et al. , 2016 ) , the learner can only observe a set of noisy labels . For instance , human annotators may wrongly label some images containing cats as ones that contain dogs accidentally or irresponsibly . The label noise of each instance is characterized by a noise transition matrix T ( X ) , where each element Tij ( X ) : = P ( Ỹ = j|Y = i , X ) . The corresponding noisy dataset2 and distribution are denoted by D̃ : = { ( xn , ỹn ) } n∈ [ N ] and D̃ . Let 1 ( · ) be the indicator function taking 2In this paper , the noisy dataset refers to a dataset with noisy examples . A noisy example is either a clean example ( whose label is true ) or a corrupted example ( whose label is wrong ) . value 1 when the specified condition is satisfied and 0 otherwise . Similar to the goals in surrogate loss ( Natarajan et al. , 2013 ) , LDMI ( Xu et al. , 2019 ) and peer loss ( Liu & Guo , 2020 ) , we aim to learn a classifier f from the noisy distribution D̃ which also minimizes P ( f ( X ) 6= Y ) , ( X , Y ) ∼ D. Beyond their results , we attempt to propose a theoretically sound approach addressing a general instance-based noise regime without knowing or estimating noise rates . 2.1 CONFIDENCE REGULARIZATION . In this section , we present a new confidence regularizer ( CR ) . Our design of the CR is mainly motivated by a recently proposed robust loss function called peer loss ( Liu & Guo , 2020 ) . For each example ( xn , ỹn ) , peer loss has the following form : ` PL ( f ( xn ) , ỹn ) : = ` ( f ( xn ) , ỹn ) − ` ( f ( xn1 ) , ỹn2 ) , where ( xn1 , ỹn1 ) and ( xn2 , ỹn2 ) are two randomly sampled and paired peer examples ( with replacement ) for n. Let Xn1 and Ỹn2 be the corresponding random variables . Note Xn1 , Ỹn2 are two independent and uniform random variables being each xn′ , n′ ∈ [ N ] and ỹn′ , n′ ∈ [ N ] with probability 1N respectively : P ( Xn1 = xn′ |D̃ ) = P ( Ỹn2 = yn′ |D̃ ) = 1 N , ∀n ′ ∈ [ N ] . Let DỸ |D̃ be the distribution of Ỹn2 given dataset D̃ . Peer loss then has the following equivalent form in expectation : 1 N ∑ n∈ [ N ] EXn1 , Ỹn2 |D̃ [ ` ( f ( xn ) , ỹn ) − ` ( f ( Xn1 ) , Ỹn2 ) ] = 1 N ∑ n∈ [ N ] [ ` ( f ( xn ) , ỹn ) − ∑ n′∈ [ N ] P ( Xn1 = xn′ |D̃ ) EDỸ |D̃ [ ` ( f ( xn′ ) , Ỹ ) ] ] = 1 N ∑ n∈ [ N ] [ ` ( f ( xn ) , ỹn ) − ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] ] . This result characterizes a new loss denoted by ` CA : ` CA ( f ( xn ) , ỹn ) : = ` ( f ( xn ) , ỹn ) − ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] . ( 1 ) Though not studied rigorously by Liu & Guo ( 2020 ) , we show , under conditions3 , ` CA defined in Eqn . ( 1 ) encourages confident predictions4 from f by analyzing the gradients : Theorem 1 . For ` CA ( · ) , solutions satisfying fxn [ i ] > 0 , ∀i ∈ [ K ] are not locally optimal at ( xn , ỹn ) . See Appendix B.2 for the proof . Particularly , in binary cases , we have constraint f ( xn ) [ 0 ] + f ( xn ) [ 1 ] = 1 . Following Theorem 1 , we know minimizing ` CA ( f ( xn ) , ỹn ) w.r.t f under this constraint leads to either f ( xn ) [ 0 ] → 1 or f ( xn ) [ 1 ] → 1 , indicating confident predictions . Therefore , the addition of term −ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] helps improve the confidence of the learned classifier . Inspired by the above observation , we define the following confidence regularizer : Confidence Regularizer : ` CR ( f ( xn ) ) : = −β · ED Ỹ |D̃ [ ` ( f ( xn ) , Ỹ ) ] , where β is positive and ` ( · ) refers to the CE loss . The prior probability P ( Ỹ |D̃ ) is counted directly from the noisy dataset . In the remaining of this paper , ` ( · ) indicates the CE loss by default . Why are confident predictions important ? Intuitively , when model fits to the label noise , its predictions often become less confident , since the noise usually corrupts the signal encoded in the clean data . From this perspective , encouraging confident predictions plays against fitting to label noise . Compared to instance-independent noise , the difficulties in estimating the instance-dependent noise rates largely prevent us from applying existing techniques . In addition , as shown by Manwani & Sastry ( 2013 ) , the 0-1 loss function is more robust to instance-based noise but hard to optimize with . To a certain degree , pushing confident predictions results in a differentiable loss function that approximates the 0-1 loss , and therefore restores the robustness property . Besides , as observed by Chatterjee ( 2020 ) and Zielinski et al . ( 2020 ) , gradients from similar examples would reinforce each other . When the overall label information is dominantly informative that Tii ( X ) > Tij ( X ) , DNNs 3Detailed conditions for Theorem 1 are specified at the end of our main contents . 4Our observation can also help partially explain the robustness property of peer loss ( Liu & Guo , 2020 ) . will receive more correct information statistically . Encouraging confident predictions would discourage the memorization of the noisy examples ( makes it hard for noisy labels to reduce the confidence of predictions ) , and therefore further facilitate DNNs to learn the ( clean ) dominant information . ` CR is NOT the entropy regularization Entropy regularization ( ER ) is a popular choice for improving confidence of the trained classifiers in the literature ( Tanaka et al. , 2018 ; Yi & Wu , 2019 ) . Given a particular prediction probability p for a class , the ER term is based on the function −p ln p , while our ` CR is built on ln p. Later we show ` CR offers us favorable theoretical guarantees for training with instance-dependent label noise , while ER does not . In Appendix C.1 , we present both theoretical and experimental evidences that ` CR serves as a better regularizer compared to ER .

Summary: The authors of the paper propose a new method, the CORES (COnfidence REgularized Sample Sieve), to tackle the important problem of learning under instance dependent label noise. The proposed method, in essence, involves the use of a confidence regularization term that encourages more confident predictions and a sieving process to remove the samples with large losses. Theoretical justification and empirical experiments were conducted to demonstrate the effectiveness of the proposed method.