An information-theoretic framework for learning models of instance-independent label noise

1 INTRODUCTION . Real-world datasets are inherently noisy . Although there are numerous existing methods for learning classifiers in the presence of label noise ( e.g . Han et al . ( 2018 ) ; Hendrycks et al . ( 2018 ) ; Natarajan et al . ( 2013 ) ; Tanaka et al . ( 2018 ) ) , there is still a gap between empirical success and theoretical understanding of conditions required for these methods to work . For instance-independent label noise , all methods with theoretical performance guarantees require a good estimation of the noise transition matrix as a key indispensable step ( Cheng et al. , 2017 ; Jindal et al. , 2016 ; Patrini et al. , 2017 ; Thekumparampil et al. , 2018 ; Xia et al. , 2019 ) . Recall that for any dataset D with label noise , we can associate to it a noise transition matrix QD , whose entries are conditional probabilities p ( y|z ) that a randomly selected instance of D has the given label y , under the condition that its correct label is z . Many algorithms for estimating QD either require that a small clean subset Dclean of D is provided ( Liu & Tao , 2015 ; Scott , 2015 ) , or assume that the noise model is a mixture model ( Ramaswamy et al. , 2016 ; Yu et al. , 2018 ) , where at least some anchor points are known for every component . Here , “ anchor points ” refer to datapoints belonging to exactly one component of the mixture model almost surely ( cf . Vandermeulen et al . ( 2019 ) ) , while “ clean ” refers to instances with correct labels . Recently , it was shown that the knowledge of anchor points or Dclean is not required for estimating QD . The proposed approach , known as T-Revision ( Xia et al. , 2019 ) , learns a classifier from D and simultaneously identifies anchor-like instances in D , which are used iteratively to estimate QD , which in turn is used to improve the classifier . Hence for T-Revision , a good estimation for QD is inextricably tied to learning a classifier with high classification accuracy . In this paper , we propose a framework for estimating QD solely from D , without requiring anchor points , a clean subset , or even anchor-like instances . In particular , we show that high classification accuracy is not required for a good estimation of QD . Our framework is able to robustly estimate QD at all noise levels , even in extreme scenarios where anchor points are removed from D , or where D is imbalanced . Our key starting point is that Shannon entropy and other related information-theoretic concepts can be defined analogously for datasets with label noise . Suppose we have a discriminator Φ that takes any dataset D′ as its input , and gives a binary output that predicts whether D′ has maximum entropy . Given D , a dataset with label noise , we shall synthesize multiple new datasets D̂ by inserting additional label noise into D , using different noise levels for different label classes . Intuitively , the more label noise that D initially has , the lower the minimum amount of additional label noise we need to insert into D to reach near-maximum entropy . We show that among those datasets D̂ that are predicted by Φ to have maximum entropy , their associated levels of additional label noise can be used to compute a single estimate for QD . Our estimator is statistically consistent : We prove that by repeating this method , the average of the estimates would converge to the true QD . As a concrete realization of this idea , we shall construct Φ using the notion of Local Intrinsic Dimensionality ( LID ) ( Houle , 2013 ; 2017a ; b ) . Intuitively , the LID computed at a feature vector v is an approximation of the dimension of a smooth manifold containing v that would “ best ” fit the distribution D in the vicinity of v. LID plays a fundamental role in an important 2018 breakthrough in noise detection ( Ma et al. , 2018c ) , wherein it was empirically shown that sequences of LID scores could be used to distinguish clean datasets from datasets with label noise . Roughly speaking , the training data for Φ consists of LID sequences that correspond to multiple datasets synthesized from D. In particular , we show that Φ can be trained without needing any clean data . Since we are optimizing the predictive accuracy of Φ , rather than optimizing the classification accuracy for D , we also do not require state-of-the-art architectures . For example , in our experiments on the CIFAR-10 dataset ( Krizhevsky et al. , 2009 ) , we found that LID sequences generated by training on shallow “ vanilla ” convolutional neural networks ( CNNs ) , were sufficient for training Φ . Our contributions are summarized as follows : • We introduce an information-theoretic-based framework for estimating the noise transition matrix of any datasetD with instance-independent label noise . We do not make any assumptions on the structure of the noise transition matrix . • We prove that our noise transition matrix estimator is consistent . This is the first-ever estimator that is proven to be consistent without needing to optimize classification accuracy . Notably , our consistency proof does not require anchor points , a clean subset , or any anchor-like instances . • We construct an LID-based discriminator Φ and show experimentally that training a shallow CNN to generate LID sequences is sufficient for obtaining high predictive accuracy for Φ . Using our LID-based discriminator Φ , our proposed estimator outperforms the state-of-the-art methods , especially in the case when anchor-like instances are removed from D. • Given access to a clean subset Dclean , we show that our method can be used to further improve existing competitive estimation methods . 2 PROPOSED INFORMATION-THEORETIC FRAMEWORK . Our framework hinges on a simple yet crucial observation : Datasets with different label noise levels have different entropies . Although the entropy of any given dataset D is ( initially ) unknown to us , we do know , crucially , that a complete uniformly random relabeling of D would yield a new dataset with maximum entropy ( which we call “ baseline datasets ” ) , and we can easily generate multiple such datasets . We could also use partial relabelings to generate a spectrum of new datasets whose entropies range from the entropy of D , to the maximum possible entropy . We call them “ α-increment datasets ” , where α is a parameter that we control . The minimum value αmin for α , such that an α-increment dataset reaches maximum entropy , would depend on the original entropy of D. See Fig . 1 for a visualization of the spectrum of entropies for α-increment datasets and baseline datasets . Our main idea is to train a discriminator Φ that recognizes datasets with maximum entropy , and then use Φ to determine this minimum value αmin . Once this value is estimated , we are then able to estimate QD . Specific realizations of our framework correspond to specific designs for Φ . An illustration of our framework using LID-based discriminators is given in Fig . 2 ; details on LID-based discriminators can be found in Section 3 , and will be further elaborated in the appendix . Throughout this paper , given any discrete random variablesX , Y , we shall write pX ( x ) and pX|Y ( x|y ) to mean Pr ( X = x ) and Pr ( X = x|Y = y ) respectively . We assume that the reader is familiar with the basics of information theory ; see Cover & Thomas ( 2012 ) for an excellent introduction . 2.1 ENTROPY OF DATASETS WITH LABEL NOISE . Given D a dataset with instance-independent label noise ( DILN ) , let A be its set of all label classes , and let Y ( resp . Z ) be the given ( resp . correct ) label of a randomly selected instance X of D.1 For convenience , we say that D is a DILN with noise model ( Y |Z ; A ) . The noise transition matrix of D is a matrix QD whose ( i , j ) -th entry is qDi , j : = pY |Z ( j , i ) . We shall define the entropy of D by H ( D ) : = − ∑ i∈A pZ ( i ) ∑ j∈A qDi , j log q D i , j . Notice that H ( D ) is precisely the conditional entropy of Y given Z . ( We use the convention that 0 log 0 = 0 . ) Hence , it is easy to prove that 0 ≤ H ( D ) ≤ log |A| . In particular , H ( D ) = 0 if and only if every pair of instances of D in the same class have the same given labels . Note also that D has maximum entropy log |A| if and only if every entry of QD equals 1|A| ( i.e . the given labels of D are completely noisy ) . Thus , H ( D ) could be interpreted as a measure of the label noise level of D. 1Every datapoint of D is a pair ( x , y ) , where x is an instance , and y is its given label , which may differ from the correct label z associated to x . Note that Z is a function of X , and Y is a random function of Z . By instance-independent label noise , we mean that Pr ( Y = y|Z = z , X = x ) = Pr ( Y = y|Z = z ) . A more detailed treatment of DILNs can be found in Appendix A . In particular , a DILN includes its noise model . A derived DILN of D shall mean a DILN D′ with noise model ( Y ′|Z ; A ) for some Y ′ independent of Z , such that both D , D′ have the same underlying set of instances , given in the same sequential order . For example , D′ could be “ derived ” from D by inserting additional instance-independent label noise , in which case D′ can be interpreted as a partial relabeling of D. For convenience , we say that D′ is a Y ′-derived DILN of D . 2.2 SYNTHESIS OF NEW DATASETS FROM D. Let D be a DILN with noise model ( Y |Z ; A ) . Without loss of generality , assume A = { 1 , . . . , k } , assume that D has N instances , and write QD = [ qi , j ] 1≤i , j≤k . The correct labels for the instances are fixed and unknown to us , hence all entries of QD are fixed constants with unknown values . Our goal is to estimate QD . As alluded to earlier , we shall be synthesizing two types of datasets from D. The first type is what we call a baseline dataset , described as follows : Let D be a random dataset obtained from D by replacing the given label of each instance of D by a label chosen uniformly at random from A . Hence D is a random DILN with expected entropy log k ( i.e . maximum entropy ) , which we shall denote by Dmax : = E [ D ] . The noise transition matrix QD = [ Q′i , j ] 1≤i , j≤k of D is a random matrix whose entries Q′i , j = 1 k ( 1 + E ′ i , j ) are random variables , where each “ error ” E ′ i , j is a random variable with mean 0 . Any observed D is called a baseline DILN of D. The second type is what we call an α-increment dataset , where α = ( α1 , . . . , αk ) is a vector whose entries satisfy 0 ≤ αi ≤ 1 for all i . Let Dα be obtained from D as follows : For each 1 ≤ i ≤ k , select uniformly at random αi × 100 % of the instances with given label i , and reassign each selected given label to one of the remaining k−1 classes , chosen uniformly at random . HenceDα is a random DILN , and its noise transition matrix QDα = [ Q ′′ i , j ] 1≤i , j≤k is a random matrix whose entries Q′′i , j = qi , j ( 1− αj ) + ∑ 1≤t≤k t6=j qi , tαt 1 k−1 ( 1 + E ′′ i , j ) ( for all 1 ≤ i , j ≤ k ) ( 1 ) are random variables , where each “ error ” E′′i , j is a random variable with mean 0 . Any observed Dα is called an α-increment DILN of D .

Summary: The paper aims to develop an information-theoretic framework for learning the underlying model from data with instance-independent label noises. It first formalizes the problem and relevant definitions with information measures, such as conditional entropy and conditional KL divergence, then argues that if the underlying model is well-separated, the proposed algorithm will be consistent in learning the noise-labeling matrix. At the heart of the algorithm is a discriminator being able to measure the labelings' noise level. A class of discriminators based on 'local intrinsic dimension (LID)' is then provided and evaluated empirically. The resulting algorithm performs well on the tested instances compared to existing ones.