Confidence Scores Make Instance-dependent Label-noise Learning Possible

1 INTRODUCTION . The recent success of deep neural networks has increased the need for high-quality labeled data . However , such a labelling process can be time-consuming and costly . A compromise is to resort to weakly-supervised annotations , using crowdsourcing platforms or trained classifiers that annotate the data automatically . These weakly-supervised annotations tend to be low-quality and noisy , which negatively affects the accuracy of high-capacity models due to memorization effects ( Zhang et al. , 2017 ) . Thus , learning with noisy labels has often drawn a lot of attention . Early works on noisy labels studied random classification noise ( RCN ) for binary classification ( Angluin & Laird , 1988 ; Kearns , 1993 ) . In the RCN model , each instance has its label flipped with a fixed noise rate ρ ∈ [ 0 , 12 ) . A natural extension of RCN is class-conditional noise ( CCN ) for multiclass classification ( Stempfel & Ralaivola , 2009 ; Natarajan et al. , 2013 ; Scott et al. , 2013 ; Menon et al. , 2015 ; van Rooyen & Williamson , 2015 ; Patrini et al. , 2016 ) ( Appendix A ) . In the CCN model , each instance from class i has a fixed probability ρi , j of being assigned to class j . Thus , it is possible to encode some similarity information between classes . For example , we can expect that the image of a “ dog ” is more likely to be erroneously labelled as “ cat ” than “ boat ” . To handle the CCN model , a common method is the loss correction , which aims to correct the prediction or the loss of the classifier using an estimated noise transition matrix ( Patrini et al. , 2017 ; Sukhbaatar et al. , 2015 ; Goldberger & Ben-Reuven , 2017 ; Ma et al. , 2018 ) . Another common approach is the label correction , which aims to improve the label quality during training . For example , Reed et al . ( 2015 ) introduced a bootstrapping scheme . Similarly , Tanaka et al . ( 2018 ) proposed to update the weights of a classifier iteratively using noisy labels , and use the updated classifier to yield more high-quality pseudo-labels for the training set . Although these methods have theoretical guarantees , they are unable to cope with real-world noise , e.g. , instance-dependent noise ( IDN ) . The IDN model considers a more general noise ( Manwani & Sastry , 2013 ; Ghosh et al. , 2014 ; Menon et al. , 2016 ; Cheng et al. , 2017 ; Menon et al. , 2018 ) , where the probability that an instance is mislabeled depends on both its class and features . Intuitively , this noise is quite realistic , as poorquality or ambiguous instances are more likely to be mislabeled in real-world datasets . However , it is much more complex to formulate the IDN model , since the probability of a mislabeled instance is a function of not only the label space but also the input space that can be very high dimensional . As a result , several pioneer works have considered stronger assumptions on noise functions . However , stronger assumptions tend to restrict the utility of these works ( Table 1 ) . For instance , the boundary-consistent noise model considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classifier ( Du & Cai , 2015 ; Menon et al. , 2018 ) . However , such a model is restricted to binary and can not estimate noise functions . Cheng et al . ( 2017 ) recently studied a particular case of the IDN model , where noise functions are upper-bounded . Nonetheless , their method is limited to binary classification and has only been tested on small datasets . Instead of simplifying assumptions on noise functions , we propose to tackle the IDN model from the source , by considering confidence scores to be available for the label of each instance . We term this new setting confidence-scored instance-dependent noise ( CSIDN , Figure 1c ) . The confidence scores denote how likely an instance is to be correctly labeled . Assuming that ( i ) confidence scores are available for each instance , ( ii ) transitions probabilities to other classes are independent of the instance conditionally on the assigned label being erroneous and ( iii ) a set of anchor points is available , we derive an instance-level forward correction algorithm which can fully estimate the transition probability for each instance , and subsequently train a robust classifier with a loss-correction method similarly to Patrini et al . ( 2017 ) . It is noted that confidence scores can be easily and cheaply derived during the construction of the dataset . For example , in crowdsourcing platforms , simply counting how many annotators agree on a given instance can give a notion of how confident a label is . Besides , many real-world datasets are automatically annotated using a trained classifier , such as web-scraped datasets ( Tong Xiao et al. , 2015 ) and physiological features inferred from medical records ( Agarwal et al. , 2016 ) . In these cases , the class-probabilities of the labels assigned by the classifier can be seen as confidence scores , provided that the classifier is well calibrated ( Guo et al. , 2017 ) . To sum up , we first formulate instance-dependent noise in Section 2.1 , and expose its robustness challenge in Section 2.2 . Then , we explain our motivation to use confidence scores , and propose the confidence-scored instance-dependent noise ( CSIDN ) model in Section 2.3 . Lastly , to handle this new noise model , we present the first practical algorithm termed instance-level forward correction in Section 3 , and validate the proposed algorithm through extensive experiments in Section 4 . 2 TACKLING INSTANCE-DEPENDENT NOISE FROM THE SOURCE . In this section , we present the IDN model along with the limitations of existing approaches , and introduce the CSIDN model as a tractable instance-dependent noise model . 2.1 NOISE MODELS : FROM CLASS-CONDITIONAL TO INSTANCE-DEPENDENT NOISE . We formulate the problem of learning with noisy labels in this section . Let D be the distribution of a pair of random variables ( X , Y ) ∈ X × Y , where X ∈ Rd , Y = { 1 , 2 , . . . , K } and K is the number of classes . In the classification task with noisy labels , we hope to train a classifier while having only access to samples from a noisy distribution D̄ of random variables ( X , Ȳ ) ∈ X × Y . Given a point x sampled from X , Ȳ is derived from the random variable Y via a noise transition matrix T ( x ) = ( Ti , j ( x ) ) K i , j=1 ∈ [ 0 , 1 ] K×K : ∀1 ≤ j ≤ K , P ( Ȳ = j|X = x ) = K∑ i=1 Ti , j ( x ) P ( Y = i|X = x ) . ( 1 ) Each noise function Ti , j : X 7→ [ 0 , 1 ] is defined as Ti , j ( x ) = P ( Ȳ = j|Y = i , X = x ) . In the class-conditional noise ( CNN ) model ( Figure 1a ) , the transition matrix does not depend on the instance x and the noise is entirely characterized by theK2 constants Ti , j . However , in the instancedependent noise ( IDN ) model ( Figure 1b ) , the transition matrix depends on the actual instance . This tremendously complicates the problem , as the noise is now characterized by K2 functions over the latent space X , which can be very high dimensional ( e.g. , d ∼ 104-106 for an object recognition dataset ) . 2.2 CHALLENGES FROM INSTANCE-DEPENDENT NOISE . Limitation of existing CCN methods . Due to the complexity of the IDN model , most recent works in learning with noisy labels have focused on the CCN model ( Figure 1a ) , and the CCN model can be seen as a simplified IDN model ( Figure 1b ) free of feature information . In addition to loss correction and label correction mentioned before , another method for the CCN model is sample selection , which aims to find reliable samples during training , such as the smallloss approaches ( Jiang et al. , 2018 ; Han et al. , 2018 ) . Inspired by the memorization in deep learning ( Arpit et al. , 2017 ) , those methods first run a standard classifier on a noisy dataset , then select the small-loss samples for reliable training . However , all approaches can not handle the IDN model directly . Specifically , loss correction considers the noise model to be characterized by a fixed transition matrix , which does not include any instance-level information . Meanwhile , label correction is vulnerable to the IDN model , since the classifier will be much weaker on noisy regions and labels corrected by the current prediction would likely be erroneous . Similarly , sample selection is easily affected by the IDN model . For example , in the small-loss approaches , instance-dependent noise functions can leave partial regions of the input space clean and other regions very noisy ( e.g. , in an object recognition dataset , poor-quality pictures will tend to receive more noisy labels than high-quality ones ) . Since clean regions will tend to receive smaller losses than noisy regions , the small-loss approaches , which only trains on points with the smallest-losses , will focus on clean regions and neglect harder noisy regions . Then , since the distribution of clean regions will subsequently be different from the global distribution , this will introduce a covariate-shift ( Shimodaira , 2000 ) , which greatly degrades performances . Moreover , it is hard to use importance reweighting ( Sugiyama et al. , 2007 ) for alleviate the issue , since importance reweighting would require estimating the clean posterior probability that is intractable for the IDN model . To validate this fact , we generate a 3-class distribution of concentric circles ( cf . Figure 2a ) , with ∀ ( x , y ) ∈ R2 × { 1 , 2 , 3 } , P ( ȳ 6= y|x ) = 12 ( w·x ‖w‖‖x‖ + 1 ) with w = ( 0 , 1 ) ( cf . Figure 2b ) . We then train a network on the top R ( T ) small-loss instances at each epoch T based on the losses of the previous epoch , with R ( T ) decreasing in T as described in Han et al . ( 2018 ) . Figure 2c shows the density of the top 50 % small-loss instances selected after 10 epochs : since noisy regions are associated to higher losses , the network eventually tends to select instances from the clean region and neglect the noisy region . This leads to covariate-shift , which is associated with decreased performances ( Shimodaira , 2000 ) . Limitation of pioneer IDN methods . The main challenge of the IDN model is the wide range of possible noise functions included in its formulation . Since each Ti , j ( · ) is a function of the highdimensional input space X , it is challenging for a model to be flexible enough to fit any real-world noise function while being trainable on corrupted datasets , let alone derive theoretical results . Instead , various recent works have considered stronger assumptions on noise functions . For instance , boundary-consistent noise ( BCN ) , first introduced by ( Du & Cai , 2015 ) and generalized in Menon et al . ( 2018 ) , considers stronger noise for samples closer to the decision boundary of the Bayesian optimal classifier . This is a reasonable model for noise from human annotators , since “ harder ” instances ( i.e. , instances closer to the decision boundary ) are more likely to be corrupted . Moreover , it is simple enough to derive some theoretical guarantees , as done in Menon et al . ( 2018 ) . Additionally , an extension of the BCN model was studied in Bootkrajang & Chaijaruwanich ( 2018 ) , where the noise function is a Gaussian mixture of the distance to the Bayesian optimal boundary . However , the BCN model and its extension are restricted to binary classification , and their geometry-based assumption becomes difficult to fathom for high-dimensional input spaces . Furthermore , Cheng et al . ( 2017 ) recently studied a particular case of the IDN model , where the probabilities that the true labels of samples flip into corrupted ones have upper bounds . They proposed a method based on distilled samples , where noisy labels agree with the optimal Bayesian classifier on the clean distribution . However , their method is limited to binary classification and has only been tested on small UCI datasets . Table 1 summarizes the characteristics of those approaches .

Summary: Learning with noise labels is a hot topic now due to the reason that deep learning algorithms often require large-scale supervised training samples and labelling a large amount of data is costly. However, almost all of the existing methods assume that the label noise is instance-independent. It either depends on the clean classes or is completely random. This paper studies the instance-dependent label noise, which is more realistic and applicable, but difficulty to address. The authors target to solve this problem. A feasible solution would contribute to the community a lot.