An Energy-Based Framework for Arbitrary Label Noise Correction

1 INTRODUCTION . Machine learning algorithms depend on reliable training labels or , when applicable , sufficiently robust learning models in order to produce generalizable predictions ( Zhu & Wu , 2004 ; Frénay & Verleysen , 2014 ) . Many empirical datasets suffer from training label corruption , which can result from annotation error , human bias , or a noisy process for generating labels ( Smyth , 1996 ; Brodley & Friedl , 1999 ) . In practice , it can be costly to obtain noise-free labels ; hence , research on reducing the effects of label noise on learning has received considerable attention . Most work , however , focuses on label noise that is independent of the input features ( e.g. , random label noise or class conditional random noise ) . The introduction of feature dependency significantly complicates mathematical analysis ( Natarajan et al. , 2013 ; Liu & Tao , 2014 ; Northcutt et al. , 2017 ) . Although work in this domain has yielded impressive results and improved generalization ( Rebbapragada & Brodley , 2007a ; Natarajan et al. , 2013 ; Patrini et al. , 2017 ; Rolnick et al. , 2017 ; Ren et al. , 2018 ) , the simplicity of these noise assumptions fails to capture crucial mislabelling processes that arise in practice . In this work , we propose a semi-supervised framework for correcting the effects of feature dependent label noise on supervised learning algorithms . Label noise processes are categorized into three types , as illustrated in Figure 1 . Type I refers to a noise model where any label in the training data is incorrect with probability γ . In the case of type II noise , the probability of label corruption is conditioned on the class , such that we have different noise rate γ0 and γ1 for each class . Finally , type III noise models describe how label noise depends explicitly on input features . Feature dependent label noise or equivalently type III noise is a more realistic type of label noise that is ubiquitous in empirical datasets ( Lachenbruch , 1974 ; Schafer & Graham , 2002 ; Frénay & Verleysen , 2014 ) . As an example of type III noise , consider the diagnostic labels of Alzheimer ’ s disease . In this setting , the probability of label noise depends on both age and sex ( i.e. , younger male patients are harder to diagnose ) ( Khachaturian , 1985 ; Murray et al. , 2016 ) . Type III noise also includes the prevalence of unreliable labels for training instances in low density regions of feature space ( Denoeux , 1995 ; 1997 ; 2000 ) or near classification boundaries ( Lachenbruch , 1966 ; 1974 ; Chhikara & McKeon , 1984 ; Cohen , 1997 ; Beigman & Klebanov , 2009 ; Beigman Klebanov & Beigman , 2009 ; Kolcz & Cormack , 2009 ) . In this paper , we present a practical framework for correcting label noise that extends beyond type I and type II noise . We train an energy-based generative model on a small subset of the training data with reliable labels , either manually annotated or automatically learned . This model is used to identify and correct mislabelled examples based on the whether the assigned energy of a given point is compatible with its original label . We empirically demonstrate our framework ’ s ability to handle input dependent label noise on both simulated and real datasets . 2 RELATED WORK . The full body of work on the problem of label noise is too extensive to review here ; however , we direct interested readers to the detailed review by Frénay & Verleysen ( 2014 ) . In this section , we highlight key contributions in the label noise literature . We then briefly discuss existing theoretical work . 2.1 LABEL NOISE CORRECTION . Existing work in label noise correction is designed for Type I and Type II noise and falls into three categories : relabeling , learning procedures and loss functions . We expand on each area below . Relabeling : Earlier approaches to label noise focus on relabeling the noisy training set . Brodley & Friedl ( 1999 ) use the output of an ensemble of classifiers to identify mislabeled training examples . Sun et al . ( 2007 ) identify mislabeled instances based on the entropy of class probabilities outputted by a Bayesian classifier . These approaches , while robust to simpler label noise models , do not account for Type III noise . Learning Procedures : Methods modifying the learning procedure include the perceptron algorithm with margin ( PAM ) Frénay & Verleysen ( 2014 ) . Crammer & Lee ( 2010 ) use a velocity-based learning procedure to learn the weight vector distribution , termed gaussian herding ( NHERD ) . Sukhbaatar et al . ( 2014 ) introduce a noise layer to a neural network architecture to learn the noise function , while Rebbapragada & Brodley ( 2007b ) weight each example by class confidence in the training procedure . These methods commonly suffer from overfitting to the noise and again , primarily focus on Type I and Type II noise . Loss Functions : Long & Servedio ( 2010 ) have shown classification algorithms that optimize a convex potential over a linear class are not robust to random label noise . This has led to work which aims to modify convex loss functions in order to make them noise-tolerant in the presence of type I and type II noise . Ghosh et al . ( 2015 ) derived sufficient conditions for classification losses that render them robust to random noise ; namely , the components of a loss ( where each component is defined over a given class ) must sum to a constant value . Rooyen et al . ( 2015 ) proposed a convex loss that avoids the negative result of Long & Servedio ( 2010 ) for type I noise by virtue of being negatively unbounded . Natarajan et al . ( 2013 ) have shown that risk minimization over the corrupted data is consistent with risk minimization over clean data provided that the standard convex loss ( e.g. , binary cross entropy ) is modified appropriately to produce an unbiased loss estimator ( ULE ) . Each of these loss functions is presented for and validated on Type I and Type II noise . 2.2 EXISTING THEORETICAL GUARANTEES . ( Bylander , 1994 ; Blum et al. , 1998 ; Blum & Mitchell , 1998 ) offer guarantees for hypothesis generalization in the face of Type I and Type II noise . Angluin & Laird ( 1988 ) ; Bylander ( 1997 ; 1998 ) ; Servedio ( 1999 ) present guarantees for type III noise , where the probability of error depends on the distance to the margin . We introduce a gradient-based interpretation of label noise to these existing results . 3 BINARY CLASSIFICATION IN THE PRESENCE OF LABEL NOISE . 3.1 PROBLEM SETUP AND NOTATION . Let P denote the true distribution from which n i.i.d . training examples ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xn , yn ) have been drawn , where ( xi , yi ) ∈ Rd × { 0 , 1 } . The clean and corrupted training datasets are denoted as T = { ( xi , yi ) for i = 1 , 2 , . . . , n } & T̃ = { ( xi , ỹi ) for i = 1 , 2 , . . . , n } , ( 1 ) respectively , where due to some label noise process yi → ỹi . Hence , we have access to the noisy data T̃ during training instead of the clean data T . It can be assumed that the corrupted samples ( xi , ỹi ) are drawn from a noisy distribution P̃ . In supervised binary classification , we aim to learn a discriminator f : Rd → R that minimizes the risk with respect to a given loss function1 . Formally , we want to minimize the empirical risk is R̂ [ ` , f , T ] = 1m ∑m i=1 ` ( f ( xi , θ ) , yi ) , where the m , ` , θ denote the size of the mini-batch size , loss function , and the learnable model parameters respectively . Gradient based learning algorithms compute ∇θ ( R̂ [ ` , f , T ] ) and update model parameters . For example , as in mini-batch gradient descent , θt+1 = θt − η∇θ ( R̂ [ ` , f , T ] ) , ( 2 ) where η is the learning rate . In practice , we may not have access to the clean data T . Instead we must learn the discriminator using the noisy data T̃ . Thus , the second term in Eq . 2 becomes ∇θ ( R̂ [ ` , f , T̃ ] ) , which denotes the risk with respect to the noisy training data . 3.2 LABEL NOISE EFFECT ON RISK MINIMIZATION . We can describe the effect of label noise on learning by expressing the noisy gradient∇θ ( R̂ [ ` , f , T̃ ] ) in terms of the true gradient ∇θ ( R̂ [ ` , f , T ] ) and a noise term . In the case of type I noise where any label yi can be flipped to 1− yi with probability γ ∈ [ 0 , 1/2 ] , we have E ( x , ỹ ) ∼P̃ [ ∇θR̂ [ ` , T̃ ] ] = ( 1− 2γ ) Etrue + 2γ Erand , ( 3 ) where Etrue ≡ E ( x , ỹ ) ∼P̃ [ ∇θR̂ [ ` , T ] ] is the true ( i.e . noise-free ) gradient , and Erand is the completely noisy term obtained by replacing all targets yi with a random value of either 0 or 1 . For type II noise , we have an equation analogous to Eq . 3 , where γ is replaced by the average γ̄ = ( γ0 + γ1 ) /2 noise rate and we have an additional term reflecting the noise imbalance between each class when γ0 6= γ1 . For more information on this and derivations of these equations , see Appendix A . From Eq . 3 we deduce that for small γ the true gradients dominate the dynamics of searching θ space for optimal parameters . As γ grows , the true gradients are scaled down by the factor ( 1−2γ ) , which means that the contribution of Etrue diminishes and the random perturbation term Erand starts to dominate as γ grows . A similar conclusion can be reached for type II noise ( replacing γ with γ̄ ) where we have an additional term modifying the true gradients based on the class imbalance ( see A.2 ) . The effect of Erand can be negated by increasing the amount of training data proportionately with γ̄ as discussed by Rolnick et al . ( 2017 ) : i.e . as the probability of a random error occurring increases , 1In practice the loss function ` is usually classification-calibrated to minimize the 0-1 loss given that the sample size is sufficiently large ( Bartlett et al. , 2006 ) . the model needs more data to make progress in the direction of the true gradient . We can neutralize the effects ( i.e . suppress Erand ) of random noise ( type I ) or class conditional random noise ( type II ) by increasing the number of training examples and by utilizing a robust loss function when possible ( Natarajan et al. , 2013 ) . In contrast , Type III noise is substantially more difficult to address because the available training data T̃ can have optimal discriminators that are significantly different from the true discriminator corresponding to the clean data T . Separating Type III noise into the true term and the perturbation term necessitates restrictive assumptions on the noise model ( See Appendix A . In Figure 2 , we show this by visualizing SGD over the true training set ( gold lines ) and the noise training set ( blue lines ) . As shown in Figure 2 , increasing the size of the training set is effective in address Type II noise . This strategy is not only ineffective for addressing type III noise , but they can also lead to our model learning the biased discriminator ; thus , hindering any possibility of generalization .

Summary: This submission proposes an energy-based method to correct mislabelled examples. Intuitively, the authors claim that contradictions between energy and noisy labels can be used to identify label noise. To make the idea reliable, the authors propose to compute the energy by using learned (commonly shared) features. Experiment results look good. The presentation is also clear. My concerns are as follows: