Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates

1 INTRODUCTION . The quality of supervised learning models depends on the training data { ( xn , yn ) } Nn=1 . In practice , label noise can arise due to a host of reasons . For instance , the observed labels ỹns may represent human observations of a ground truth label . In this case , human annotators may observe the label imperfectly due to differing degrees of expertise or measurement error , see e.g. , medical examples such as labeling MRI images from patients . Many prior approaches to this problem in the machine learning literature aim to develop algorithms to learn models that are robust to label noise ( Bylander , 1994 ; Cesa-Bianchi et al. , 1999 ; 2011 ; Ben-David et al . ; Scott et al. , 2013 ; Natarajan et al. , 2013 ; Scott , 2015 ) . Typical approaches require a priori knowledge of noise rates , i.e. , a set of parameters that control the severity of label noise . Working with unknown noise rates is difficult in practice : Often , one must estimate the noise rates from data , which may require additional data collection ( Natarajan et al. , 2013 ; Scott , 2015 ; Van Rooyen et al. , 2015 ) ( e.g. , be a redundant set of noisy labels for each sample point , or a set of ground truth labels for tuning these parameters ) and may introduce estimation error that can affect the final model in less predictable ways . Our main goal is to provide an alternative that does not require the specification of the noise rates , nor an additional estimation step for the noises . This target solution might help when the practitioner does not have access to reliable estimates of the noise rates ( e.g. , when the training data has limited size for the estimation tasks , or when the training data is already collected in a form that makes the estimation hard to perform ) . In this paper , we introduce a new family of loss functions , peer loss functions , to empirical risk minimization ( ERM ) , for a broad class of learning with noisy labels problems . Peer loss functions operate under different noise rates without requiring either a priori knowledge of the embedded noise rates , or an estimation procedure . This family of loss functions builds on approaches developed in the peer prediction literature ( Miller et al. , 2005 ; Dasgupta & Ghosh , 2013 ; Shnayder et al. , 2016 ) , which studies how to elicit information from self-interested agents without verification . Typical approaches in the peer prediction literature design scoring functions to score each reported data using another noisy reference answer , without accessing ground truth information . We borrow this idea and the associated scoring functions via making a connection through treating each classifier ’ s prediction as an agent ’ s private information to be elicited and evaluated , and the noisy label as an imperfect reference from a “ noisy label agent ” . The peer loss takes a form of evaluating classifiers ’ prediction using noisy labels on both the targeted samples and a particular form of constructed “ peer ” samples . The evaluation on the constructed peer sample encodes implicitly the information about the noises as well as the underlying true labels , which helps us offset the effects of label noises . The peer sample evaluation returns us a favorable property that expected risk of peer loss turns to be an affine transformation of the true risk of the classifier defined on the clean distribution . In other words , peer loss is invariant to label noises when optimizing with it . This effect helps us get rid of the estimation of noise rates . The main contributions of this work are : 1 . We propose a new family of loss functions that can easily adapt to existing ERM framework that i ) is robust to asymmetric label noises with formal theoretical guarantees and ii ) requires no prior knowledge or estimation of the noise rates ( no need for specifying noise rates ) . We believe having the second feature above is a non-trivial progress , and it features a promising solution to deploy in an unknown noisy training environment . 2 . We present formal results showing that performing ERM with a peer loss function can recover an optimal , or a near optimal classifier f∗ as if performing ERM on the clean data ( Theorem 2 , 3 , 4 ) . We also provide analysis for peer loss functions ’ risk guarantees ( Theorem 5 and 7 ) . 3 . We present extensive experimental results to validate the usefulness of peer loss ( Section 5 and Appendix ) . This result is encouraging as it is able to remove the long-standing requirement of learning error rates of noises ( or estimating transition matrices as used in many relevant papers ) before many of the existing methods can be applied . We also provide preliminary results on how peer loss generalizes to multi-class classification problems . 4 . We will contribute to the community by publishing our codes and implementations . 1.1 RELATED WORK . Learning from Noisy Labels Our work fits within a stream of research on learning with noisy labels . A large stream of research on this topic works with the random classification noise ( RCN ) model , where observed labels are flipped independently with probability e ∈ [ 0 , 12 ] ( Bylander , 1994 ; Cesa-Bianchi et al. , 1999 ; 2011 ; Ben-David et al. ) . Recently , learning with asymmetric noisy data ( or also referred as class-conditional random classification noise ( CCN ) ) for binary classification problems has been rigorously studied in ( Stempfel & Ralaivola , 2009 ; Scott et al. , 2013 ; Natarajan et al. , 2013 ; Scott , 2015 ; Van Rooyen et al. , 2015 ; Menon et al. , 2015 ) . For a more thorough survey of classical results on learning with noisy data , please refer to ( Frénay & Verleysen , 2014 ) . Symmetric loss For RCN , where the noise parameters are symmetric , there exists works that show symmetric loss functions ( Manwani & Sastry , 2013 ; Ghosh et al. , 2015 ; 2017 ; Van Rooyen et al. , 2015 ) are robust to the underlying noises , without specifying the noise rates . It was also shown that under certain conditions , the proposed loss functions are able to handle asymmetric noises . Our focus departs from this line of works , and we will exclusively focus on asymmetric noise setting , and study the possibility of an approach that can ignore the knowledge of noise rates . Follow-up works ( Du Plessis et al. , 2013 ; van Rooyen et al. , 2015 ; Menon et al. , 2015 ; Charoenphakdee et al. , 2019 ) have looked into leveraging symmetric conditions and 0-1 loss with asymmetric noises , and with more evaluation metrics , such as balanced error rate and AUROC . In particular , experimental evidences are reported in ( Charoenphakdee et al. , 2019 ) on the importance of symmetricity when learning with noisy labels . More recent works More recent developments include an importance re-weighting algorithm ( Liu & Tao , 2016 ) , a noisy deep neural network learning setting ( Sukhbaatar & Fergus , 2014 ; Han et al. , 2018 ; Song et al. , 2019 ) , and learning from massive noisy data for image classification ( Xiao et al. , 2015 ) , robust cross entropy loss for neural network ( Zhang & Sabuncu , 2018 ) , loss correction ( Pa- trini et al. , 2017 ) , among many others . Loss or sample correction has also been studied in the context of learning with unlabeled data with weak supervisions ( Lu et al. , 2018 ) . Most of above works either lacks theoretical guarantee of the proposed method against asymmetric noise rates ( ( Sukhbaatar & Fergus , 2014 ; Zhang & Sabuncu , 2018 ) ) , or requires estimating the noise rate ( or transition matrix between noisy and true labels , ( Liu & Tao , 2016 ; Xiao et al. , 2015 ; Patrini et al. , 2017 ; Lu et al. , 2018 ) ) . A good number of the recent works can be viewed as derivatives or extention of the unbiased surrogate loss function idea introduced in ( Natarajan et al. , 2013 ) , therefore they would naturally require the knowledge of the noise rates or transition matrix . We do provide thorough comparisons between peer loss and the unbiased surrogate loss methods . Mostly relevant to us is a recent work ( Xu et al. , 2019 ) that proposes an information theoretical loss ( an idea adapted from an earlier theoretical contribution ( Kong & Schoenebeck , 2018 ) ) that is also robust to asymmetric noises rate . We aimed for a simple-to-optimize loss function that can easily adapt to existing ERM solutions . ( Xu et al. , 2019 ) involves estimating a joint distribution matrix between classifiers and noisy labels , and then invokes computing a certain information theoretical measure based on this matrix . Therefore , its sample complexity requirement and the sensitivity to noises in this estimation are not entirely clear to us ( not provided in the paper either ) . We do provide calibration guarantees and generalization bounds . We provide conditions when the loss functions are convex . In general , we do think computationally peer loss functions are easy to optimize with , in comparing to information theoretical measures . Experiments comparing with ( Xu et al. , 2019 ) are also given in Section 5 . Peer Prediction Our work also builds on the literature for peer prediction ( Prelec , 2004 ; Miller et al. , 2005 ; Witkowski & Parkes , 2012 ; Radanovic & Faltings , 2013 ; Witkowski et al. , 2013 ; Dasgupta & Ghosh , 2013 ; Shnayder et al. , 2016 ; Liu & Chen , 2017 ) . ( Miller et al. , 2005 ) established that strictly proper scoring rule ( Gneiting & Raftery , 2007 ) could be adopted to elicit truthful reports from self-interested agents . Follow-up works that have been done to relax the assumptions imposed ( Witkowski & Parkes , 2012 ; Radanovic & Faltings , 2013 ; Witkowski et al. , 2013 ; Radanovic et al. , 2016 ; Liu & Chen , 2017 ) . Most relevant to us is ( Dasgupta & Ghosh , 2013 ; Shnayder et al. , 2016 ) where a correlated agreement ( CA ) type of mechanism was proposed . CA evaluates a report ’ s correlations with another reference agent - its specific form inspired our peer loss . 2 PRELIMINARIES . Notations and preliminaries : For positive integer n , denote by [ n ] : = { 1 , 2 , ... , n } . Suppose ( X , Y ) ∈ X × Y are drawn from a joint distribution D , with their marginal distributions denoted as PX , PY respectively . We assume X ⊆ Rd , and Y = { −1 , +1 } , that is we consider a binary classification problem . Denote by p : = P ( Y = +1 ) ∈ ( 0 , 1 ) . There are N training samples ( x1 , y1 ) , ... , ( xN , yN ) drawn i.i.d . from D. Instead of observing yns , the learner can only collect a noisy set of training labels ỹns , generated according to yns and a certain error rate model , that is we observe a dataset { ( xn , ỹn ) } Nn=1 . We assume a uniform error model for all the training samples we collect , in that errors in ỹns follow the same error rate model : denoting the random variable for noisy labels as Ỹ and we denote e+1 : = P ( Ỹ = −1|Y = +1 ) , e−1 : = P ( Ỹ = +1|Y = −1 ) such that 0 ≤ e+1 + e−1 < 1. e−1 + e+1 < 1 is not unlike the condition imposed in the existing learning literature ( Natarajan et al. , 2013 ) , and it simply implies that the noisy labels are positively correlating with the true labels ( informative about the true labels ) . Label noises are conditional independent from the features , that is the error rate is uniform across xns : P ( Ỹ = y′|Y = y ) = P ( Ỹ = y′|X , Y = y ) , ∀y , y′ ∈ { −1 , +1 } . Denote the distribution of the noisy data ( X , Ỹ ) as D̃ . f : X → R is a real-valued decision function , and its risk w.r.t . the 0-1 loss is defined as E ( X , Y ) ∼D [ 1 ( f ( X ) , Y ) ] . The Bayes optimal classifier f∗ is the one that minimizes the 0-1 risk : f∗ = argminf E ( X , Y ) ∼D [ 1 ( f ( X ) , Y ) ] . Denote this optimal risk as R∗ . Instead of minimizing the above 0-1 risk , the learner often uses a surrogate loss function ` : R× { −1 , +1 } → R+ , and find a f ∈ F that minimizes the following error : E ( X , Y ) ∼D [ ` ( f ( X ) , Y ) ] . Denote the following measures : RD ( f ) = E ( X , Y ) ∼D [ 1 ( f ( X ) , Y ) ] , R ` , D ( f ) = E ( X , Y ) ∼D [ ` ( f ( X ) , Y ) ] . When there is no confusion , we will also short-hand E ( X , Y ) ∼D [ ` ( f ( X ) , Y ) ] as ED [ ` ( f ( X ) , Y ) ] . Using D to denote a dataset collected from distribution D ( correspondingly D̃ : = { ( xn , ỹn ) } Nn=1 for D̃ ) , the empirical risk measure for f is defined as R̂ ` , D ( f ) = 1|D| ∑ ( x , y ) ∈D ` ( f ( x ) , y ) .

Summary: This paper studies the problem of learning classifiers from noisy data without specifying the noise rates. Inspired by the literature of peer prediction, the authors propose peer loss. First, a scoring function is introduced, minimizing which we can elicit the Bayes optimal classifier f*. Then the authors use the setting of CA to induces a scoring matrix, and then the peer loss. Moreover, this paper explores the theoretical properties of peer loss when p=0.5. In particular, the authors propose \alpha weighted peer loss to provide strong theoretical guarantees of the proposed ERM framework. The calibration and generalization abilities are also discussed in section 4.3. Finally, empirical studies show that the propose peer loss indeed remedies the difficulty of determining the noise rates in noisy label learning.