Learning with Feature-Dependent Label Noise: A Progressive Approach

1 INTRODUCTION . Addressing noise in training set labels is an important problem in supervised learning . Incorrect annotation of data is inevitable in large-scale data collection , due to intrinsic ambiguity of data/class and mistakes of human/automatic annotators ( Yan et al. , 2014 ; Andreas et al. , 2017 ) . Developing methods that are resilient to label noise is therefore crucial in real-life applications . Classical approaches take a rather simplistic i.i.d . assumption on the label noise , i.e. , the label corruption is independent and identically distributed and thus is feature-independent . Methods based on this assumption either explicitly estimate the noise pattern ( Reed et al. , 2014 ; Patrini et al. , 2017 ; Dan et al. , 2019 ; Xu et al. , 2019 ) or introduce extra regularizer/loss terms ( Natarajan et al. , 2013 ; Van Rooyen et al. , 2015 ; Xiao et al. , 2015 ; Zhang & Sabuncu , 2018 ; Ma et al. , 2018 ; Arazo et al. , 2019 ; Shen & Sanghavi , 2019 ) . Some results prove that the commonly used losses are naturally robust against such i.i.d . label noise ( Manwani & Sastry , 2013 ; Ghosh et al. , 2015 ; Gao et al. , 2016 ; Ghosh et al. , 2017 ; Charoenphakdee et al. , 2019 ; Hu et al. , 2020 ) . Although these methods come with theoretical guarantees , they usually do not perform as well as expected in practice due to the unrealistic i.i.d . assumption on noise . This is likely because label noise is heterogeneous and feature-dependent . A cat with an intrinsically ambiguous appearance is more likely to be mislabeled as a dog . An image with poor lighting or severe occlusion can be mislabeled , as important visual clues are imperceptible . Methods that can combat label noise of a much more general form are very much needed to address real-world challenges . To adapt to the heterogeneous label noise , state-of-the-arts ( SOTAs ) often resort to a data-recalibrating strategy . They progressively identify trustworthy data or correct data labels , and then train using these data ( Tanaka et al. , 2018 ; Wang et al. , 2018 ; Lu et al. , 2018 ; Li et al. , 2019 ) . The models gradually improve as more clean data are collected or more labels are corrected , eventually converging to models of high accuracy . These data-recalibrating methods best leverage the learning power of deep neural nets and achieve superior performance in practice . However , their underlying mechanism remains a mystery . No methods in this category can provide theoretical insights as to why the model ∗Equal contributions . can converge to an ideal one . Thus , these methods require careful hyperparameter tuning and are hard to generalize . In this paper , we propose a novel and principled method that specifically targets the heterogeneous , feature-dependent label noise . Unlike previous methods , we target a much more general family of noise , called Polynomial Margin Diminishing ( PMD ) label noise . In this noise family , we allow arbitrary noise level except for data far away from the true decision boundary . This is consistent with the real-world scenario ; data near the decision boundary are harder to distinguish and more likely to be mislabeled . Meanwhile , a datum far away from the decision boundary is a typical example of its true class and should have a reasonably bounded noise level . Assuming this new PMD noise family , we propose a theoretically-guaranteed data-recalibrating algorithm that gradually corrects labels based on the noisy classifier ’ s confidence . We start from data points with high confidence , and correct the labels of these data using the predictions of the noisy classifier . Next , the model is improved using cleaned labels . We continue alternating the label correction and model improvement until it converges . See Figure 1 for an illustration . Our main theorem shows that with a theory-informed criterion for label correction at each iteration , the improvement of the label purity is guaranteed . Thus the model is guaranteed to improve with sufficient rate through iterations and eventually becomes consistent with the Bayes optimal classifier . Beside the theoretical strength , we also demonstrate the power of our method in practice . Our method outperforms others on CIFAR-10/100 with various synthetic noise patterns . We also evaluate our method against SOTAs on three real-world datasets with unknown noise patterns . To the best of our knowledge , our method is the first data-recalibrating method that is theoretically guaranteed to converge to an ideal model . The PMD noise family encompasses a broad spectrum of heterogeneous and feature-dependent noise , and better approximates the real-world scenario . It also provides a novel theoretical setting for the study of label noise . Related works . We review works that do not assume an i.i.d . label noise . Menon et al . ( 2018 ) generalized the work of ( Ghosh et al. , 2015 ) and provided an elegant theoretical framework , showing that loss functions fulfilling certain conditions naturally resist instance-dependent noise . The method can achieve even better theoretical properties ( i.e. , Bayes-consistency ) with stronger assumption on the clean posterior probability η . In practice , this method has not been extended to deep neural networks . Cheng et al . ( 2020 ) proposed an active learning method for instance-dependent label noise . The algorithm iteratively queries clean labels from an oracle on carefully selected data . However , this approach is not applicable to settings where kosher annotations are unavailable . Another contemporary work ( Chen et al. , 2021 ) showed that the noise in real-world dataset is unlikely to be i.i.d. , and proposed to fix the noisy labels by averaging the network predictions on each instance over the whole training process . While being effective , their method lacks theoretical guarantees . Chen et al . ( 2019 ) showed by regulating the topology of a classifier ’ s decision boundary , one can improve the model ’ s robustness against label noise . Data-recalibrating methods use noisy networks ’ predictions to iteratively select/correct data and improve the models . Tanaka et al . ( 2018 ) introduced a joint training framework which simultaneously enforces the network to be consistent with its own predictions and corrects the noisy labels during training . Wang et al . ( 2018 ) identified noisy labels as outliers based on their label consistencies with surrounding data . Lu et al . ( 2018 ) used a curriculum learning strategy where the teacher net is trained on a small kosher dataset to determine if a datum is clean ; then the learnt curriculum that gives the weight to each datum is fed into the student net for the training and inference . ( Yu et al. , 2019 ; Bo et al. , 2018 ) trained two synchronized networks ; the confidence and consistency of the two networks are utilized to identify clean data . Wu et al . ( 2020 ) selected the clean data by investigating the topological structures of the training data in the learned feature space . For completeness , we also refer to other methods of similar design ( Li et al. , 2017 ; Vahdat , 2017 ; Andreas et al. , 2017 ; Ma et al. , 2018 ; Thulasidasan et al. , 2019 ; Arazo et al. , 2019 ; Shu et al. , 2019 ; Yi & Wu , 2019 ) . As for theoretical guarantees , Ren et al . ( 2018 ) proposed an algorithm that iteratively re-weights each data point by solving an optimization problem . They proved the convergence of the training , but provided no guarantees that the model converges to an ideal one . Amid et al . ( 2019b ) generalized the work of ( Amid et al. , 2019a ) and proposed a tempered matching loss . They showed that when the final softmax layer is replaced by the bi-tempered loss , the resulting classifier will be Bayes consistent . Zheng et al . ( 2020 ) proved a one-shot guarantee for their data-recalibrating method ; but the convergence of the model is not guaranteed . Our method is the first data-recalibrating method which is guaranteed to converge to a well-behaved classifier . 2 METHOD . We start by introducing the family of Poly-Margin Diminishing ( PMD ) label noise . In Section 2.2 , we present our main algorithm . Finally , we prove the correctness of our algorithm in Section 3 . Notations and preliminaries . Although the noise setting and algorithm naturally generalize to multiclass , for simplicity we focus on binary classification . Let the feature space be X . We assume the data ( x , y ) is sampled from an underlying distribution D on X × { 0 , 1 } . Define the posterior probability η ( x ) = P [ y = 1 | x ] . Let τ0,1 ( x ) = P [ ỹ = 1 | y = 0 , x ] and τ1,0 ( x ) = P [ ỹ = 0 | y = 1 , x ] be the noise functions , where ỹ denotes the corrupted label . For example , if a datum x has true label y = 0 , it has τ0,1 ( x ) chance to be corrupted to 1 . Similarly , it has τ1,0 ( x ) chance to be corrupted from 1 to 0 . Let η̃ ( x ) = P [ ỹ = 1 | x ] be the noisy posterior probability of ỹ = 1 given feature x . Let η∗ ( x ) = I { η ( x ) ≥ 12 } be the ( clean ) Bayes optimal classifier , where IA equals 1 if A is true , and 0 otherwise . Finally , let f ( x ) : X → [ 0 , 1 ] be the classifier scoring function ( the softmax output of a neural network in this paper ) . 2.1 POLY-MARGIN DIMINISHING NOISE . We first introduce the family of noise functions τ this paper will address . We introduce the concept of polynomial margin diminishing noise ( PMD noise ) , which only upper bounds the noise τ in a certain level set of η ( x ) , thus allowing τ to be arbitrarily high outside the restricted domain . This formulation not only covers the feature-independent scenario but also generalizes scenarios proposed by ( Du & Cai , 2015 ; Menon et al. , 2018 ; Cheng et al. , 2020 ) . Definition 1 ( PMD noise ) . A pair of noise functions τ0,1 ( x ) and τ1,0 ( x ) are polynomial-margin diminishing ( PMD ) , if there exist constants t0 ∈ ( 0 , 12 ) , and c1 , c2 > 0 such that : τ1,0 ( x ) ≤ c1 [ 1− η ( x ) ] 1+c2 ; ∀η ( x ) ≥ 1 2 + t0 , and τ0,1 ( x ) ≤ c1η ( x ) 1+c2 ; ∀η ( x ) ≤ 1 2 − t0 . ( 1 ) We abuse notation by referring to t0 as the “ margin ” of τ . Note that the PMD condition only requires the upper bound on τ to be polynomial and monotonically decreasing in the region where the Bayes classifier is fairly confident . For the region { x : |η ( x ) − 12 | < t0 } , we allow both τ0,1 ( x ) and τ1,0 ( x ) to be arbitrary . Figure 2 ( d ) illustrates the upper bound ( orange curve ) and a sample noise function ( blue curve ) . We also show the corrupted data according to this noise function ( black points are the clean data whereas red points are the data with corrupted labels ) . The PMD noise family is much more general than existing noise assumptions . For example , the boundary consistent noise ( BCN ) ( Du & Cai , 2015 ; Menon et al. , 2018 ) assumes a noise function that monotonically decreases as the data are moving away from the decision boundary . See Figure 2 ( c ) for an illustration . This noise is much more restrictive compared to our PMD noise which ( 1 ) only requires a monotonic upper bound , and ( 2 ) allows arbitrary noise strength in a wide buffer near the decision boundary . Figure 2 ( b ) shows a traditional feature-independent noise pattern ( Reed et al. , 2014 ; Patrini et al. , 2017 ) , which assumes τ0,1 ( x ) ( resp . τ1,0 ( x ) ) to be a constant independent of x .

Summary: Label noise is very frequently in many real world applications. However, the noise can be with different distributions. If we build the learning model under a certain distribution, it is difficult to capture the discriminative information. In this paper, without assuming that the noise is a certain distribution, the proposed method can handle the general noise, and it mainly target a new family of feature-dependent label noise, which is much more general than commonly used i.i.d. label noise and encompasses a broad spectrum of noise patterns. The experimental results show that the proposed method is promising. Meanwhile, the theoretical analysis of the proposed method is well inferred.