Label Cleaning with Likelihood Ratio Test

1 INTRODUCTION . Label noise is ubiquitous in real world data . It may be caused by unintentional mistakes of manual or automatic annotators ( Yan et al. , 2014 ; Veit et al. , 2017 ) . It may also be introduced by malicious attackers ( Steinhardt et al. , 2017 ) . Noisy labels impair the performance of a model ( Smyth et al. , 1994 ; Brodley & Friedl , 1999 ) , especially a deep neural network , which tends to have strong memorization power ( Frnay & Verleysen , 2014 ; Zhang et al. , 2017 ) . Improving the robustness of a model to label noise is a crucial yet challenging task in many applications ( Mnih & Hinton , 2012 ; Wu et al. , 2018 ) . Existing methods mainly follow two directions , probabilistic reasoning and data selecting . Probabilistic methods explicitly model a noise transition matrix , namely , the probability of one label being corrupted into another ( Goldberger & Ben-Reuven , 2017 ; Patrini et al. , 2017 ) . The transition matrix is often estimated from the data , and is used to re-calibrate the training loss or to correct the prediction . Explicit estimation of the transition matrix can be problematic due to the large variation of noise patterns , e.g. , uniform noise , asymmetric noise , or mixtures . Furthermore , the transition matrix size is quadratic to the number of classes , making the estimation task prohibitive when the data has hundreds or even thousands of classes . Data-selecting methods are agnostic of the underlying noise pattern . These methods gradually collect clean data whose labels are trustworthy ( Malach & Shalev-Shwartz , 2017 ; Jiang et al. , 2018 ; Han et al. , 2018 ) . As more clean data are collected , the quality of the trained models improves . The major issue of these methods is the lack of a quantitative control of the quality of the collected clean data . Without a principled guideline , it is hard to find the correct data collection pace . An aggressive selection can unknowingly accumulate irreversible errors . On the other hand , an overly-conservative strategy can be very slow in training , or stops with insufficient clean data and mediocre models . We propose a novel method with the benefit from both the probabilistic and the data-selecting approaches . Similar to data-selecting methods , our method continuously improves the purity of the data labels by correcting the noise-corrupted ones . Meanwhile , we improve the classifier using the updated labels . Our label correction algorithm is based on statistical principles and is theoretically guaranteed to deliver a high quality label set . Instead of explicitly estimating the transition matrix , the correction algorithm only depends on the prediction of the current model , denoted as f . Using an f -based likelihood ratio test , we determine whether the current label of each data should be corrected . Our main theorem proves that the label correction algorithm will clean a majority of noisy labels with high probability . In practice , we incorporate the label correction algorithm into the training of deep neural networks . Our method iteratively updates the labels of the data while continuously training a deep neural network . To ensure the deep neural network does not overfit with noise labels that are yet to be corrected , we introduce a new retroactive loss term that regulates the model by enforcing its consistency with models in previous epochs . The rationale is that the model in an earlier training stage tends to fit the true signal rather than noise , although its overall performance is sub-optimal . Through experiments on various datasets with various noise patterns and levels , we show that our method produces robust neural network models with superior performance . To the best of our knowledge , our method is the first to correct labels with theoretical guarantees . It is has advantages over both probabilistic methods and data-selecting methods . Compared with other data-selecting methods , it has a better quantitative control of the label quality and thus is less brittle when generalizing to different datasets and different noise patterns . Also note that we are not selecting clean data . Instead , we correct labels and always use the whole training set to train . This brings an additional advantage of fully leveraging the data . Compared with other probabilistic methods , our correction algorithm assumes a rather general family of underlying noise patterns and avoids an explicit estimation of the transition matrix . 1.1 RELATED WORK . One representative strategy for handling label noise is to model and employ noise transition matrix to correct the loss . For example , Patrini et al . ( 2017 ) propose to correct the loss function with estimated noise pattern . The resulting loss is an unbiased estimator of the ground truth loss , and enables the trained model to achieve better performance . However , such an estimator relies on strong assumptions and could be inaccurate in certain scenarios . Reed et al . ( 2014 ) consider modeling the noise pattern with a hidden layer . The learning of this hidden layer is regularized with a feature reconstruction loss , yet without a guarantee that the true label distribution is learned . Another method mentioned in their work is to minimize the entropy of neural network output ; however , this method tends to predict a single class . To address this weakness , Hendrycks et al . ( 2019 ) propose to utilize a small number of kosher data to pre-train a network and estimate the noise pattern . However , such clean data may not always be available in practice . Alternatively , another direction proposes to design models that are intrinsically robust to noisy data . Crammer et al . ( 2009 ) introduce a regularized confidence weighting learning algorithm ( AROW ) , where parameters of classifiers are assumed normally distributed and the mean and covariance of this distribution is updated during training . The idea here is to preserve the weight distribution as much as possible while requiring the model to maintain predictive ability . The follow-up work ( Crammer & Lee 2010 ) proposes to improve this algorithm by herding the updating direction via specific velocity field ( NHERD ) , achieving better performance . Both of these works impose parametric constraint on parameters , which could prevent classifiers from adapting to complex dataset . Arpit et al . ( 2017 ) show that deep neural networks tend to learn meaningful patterns before they over-fit to noisy ones . Based on this observation , they propose to add Gaussian or adversarial noise to input when training with noisy labels , and empirically show that such data perturbation is able to make the resulting model more robust . Other commonly adopted techniques , such as weight decay and dropout , are also shown to be effective in increasing the robustness of trained classifier ( Arpit et al . 2017 ; Zhang et al . 2017 ) . However , the intrinsic reasons for this phenomenon still remains unclear and overfitting to noisy label is still inevitable . Apart from the above mentioned strategies , one recent work proposes to correct the corrupted labels during training . In particular , Tanaka et al . ( 2018 ) propose to jointly train the deep network and estimate the underlying true labels . While achieving improved performance , their method largely relies on the prior distribution and is difficult to deploy under cases where there is a large number of classes . In addition , there are a few other works attempting to select clean data while eliminating noisy ones during training ( Malach & Shalev-Shwartz , 2017 ; Jiang et al. , 2018 ; Han et al. , 2018 ) . These methods demonstrate promising results , but lack quantitative control of the quality of the collected clean data . Finally , beyond deep learning framework , there are several theoretic works that demonstrate the robustness of a variety of losses to label noise ( Long & Servedio 2010 ; Natarajan et al . 2013 ; Ghosh et al . 2015 ; van Rooyen et al . 2015 ) . Following the work of ( Wang & Chaudhuri 2018 ) , Gao et al . ( 2016 ) propose an algorithm that can converge to the Bayesian optimal classifier under different noisy settings . Moreover , they also provide in-depth discussion regarding the performance of k- nearest neighbor ( KNN ) classifiers . However , the problem with KNN is that it is computationally intensive and difficult to be incorporated into a learning context . Within the framework of deep learning , there are more efforts that need to be made to bridge theory and practice . 2 METHOD . Our method has two synchronized modules , the training module and the label correction module . The training module continues to learn a classifier based on the current labels . Meanwhile , the label correction module uses the prediction of the classifier to correct labels . We start with some preliminaries necessary for the exposition ( Section 2.1 ) . In Section 2.2 , we explain our correction algorithm . It uses the prediction of the classifier ( trained on noisy labels ) for a likelihood ratio test . Based on the test result , it decides whether to correct the label of a data . Theoretically , we prove that under certain assumptions of the prediction , f , the algorithm will change the labels to the correct ones , i.e. , the ones consistent with the Bayes optimal classifier ( Theorem 1 ) . In Section 2.3 , we present the overall training method . We incrementally train a deep neural network based on the corrected labels , ỹ . Meanwhile , the network ’ s prediction is used for label correction . To improve the quality of the prediction , we introduce a new loss term , called the retroactive loss . The goal is to regulate the model using models trained in earlier epoches , as they may be less overfitting with corrupted labels . For the label correction module , we focus on a binary classifier . But the algorithm and the theoretical results can easily be generalized to the multiclass setting ( Corollary 1 ) . 2.1 PRELIMINARIES . Let X be the feature space , Y = { 0 , 1 } be the label space , and D be an unknown distribution on X × Y . The joint probability can be factored as D ( x , y ) = Pr ( y|x ) Pr ( x ) . We denote by η ( x ) = Pr ( y = 1|x ) the true conditional probability . The Bayes risk of a classifier h : X → Y is R ( D , h ) = Pr ( x , y ) ∼D ( h ( x ) 6= y ) . A Bayes optimal classifier is the minimizer of the Bayes risk , i.e. , h∗ = arg minhR ( D , h ) . It can be calculated using the true conditional probability , η , h∗ ( x ) = 1 { η ( x ) ≥ 12 } ( x ) = { 1 if η ( x ) > = 12 0 otherwise . ( 1 ) We assume the true conditional probability , η , satisfies the Tsybakov condition ( also called the TNC condition ) ( Tsybakov et al. , 2004 ) . This condition , also called margin assumption , stipulates that the uncertainty of η ( region close to decision boundary at 1/2 ) and thus the Bayes optimal classifier is bounded . Definition 1 ( Tsybakov Condition ) . There exist C > 0 , λ > 0 , and t0 ∈ ( 0 , 12 ] , such that for all t ≤ t0 , Pr [ ∣∣∣∣η ( x ) − 12 ∣∣∣∣ < t ] ≤ Ctλ . The noisy label setting . Instead of samples from D , we are given a sample set with noisy labels S = { ( x , ỹ ) } where ỹ is the possibly corrupted label based on the true label y . We assume a transition probability τi→j = Pr ( ỹ = j|y = i ) , i.e. , the chance a ground truth label y is flipped from class i to class j . For simplicity , we denote τij = τi→j . The transition probabilities τ01 and τ10 are independent of the true joint distribution D and the feature x . We denote the conditional probability of the noisy labels as η̃ ( x ) = Pr ( ỹ = 1|x ) . In short , we call η̃ the noisy conditional probability . It is related linearly to the true conditional probability , η : η̃ ( x ) = ( 1− τ10 ) η ( x ) + τ01 [ 1− η ( x ) ] = ( 1− τ01 − τ10 ) η ( x ) + τ01 . ( 2 )

Summary: This paper proposes a label correction approach based on a likelihood ratio test, for robust training of deep neural networks against label noise. First, this paper introduces the LRT-Correction procedure, which is the main component of the proposed label correction approach. LRT-Correction uses the current model prediction to run a likelihood ratio test and flip labels when they are rejected. The decision is made by comparing the likelihood test results with a predefined value \Delta. Then they introduce the full algorithm, AdaCorr, where the LRT-Correction procedure serves as an inner loop for the label correction. Lastly, there are experiments done on four datasets to conclude the superior performance of the proposed AdaCorr in contrast to several existing methods.