An information-theoretic framework for learning models of instance-independent label noise

1 INTRODUCTION . Real-world datasets are inherently noisy . Although there are numerous existing methods for learning classifiers in the presence of label noise ( e.g . Han et al . ( 2018 ) ; Hendrycks et al . ( 2018 ) ; Natarajan et al . ( 2013 ) ; Tanaka et al . ( 2018 ) ) , there is still a gap between empirical success and theoretical understanding of conditions required for these methods to work . For instance-independent label noise , all methods with theoretical performance guarantees require a good estimation of the noise transition matrix as a key indispensable step ( Cheng et al. , 2017 ; Jindal et al. , 2016 ; Patrini et al. , 2017 ; Thekumparampil et al. , 2018 ; Xia et al. , 2019 ) . Recall that for any dataset D with label noise , we can associate to it a noise transition matrix QD , whose entries are conditional probabilities p ( y|z ) that a randomly selected instance of D has the given label y , under the condition that its correct label is z . Many algorithms for estimating QD either require that a small clean subset Dclean of D is provided ( Liu & Tao , 2015 ; Scott , 2015 ) , or assume that the noise model is a mixture model ( Ramaswamy et al. , 2016 ; Yu et al. , 2018 ) , where at least some anchor points are known for every component . Here , “ anchor points ” refer to datapoints belonging to exactly one component of the mixture model almost surely ( cf . Vandermeulen et al . ( 2019 ) ) , while “ clean ” refers to instances with correct labels . Recently , it was shown that the knowledge of anchor points or Dclean is not required for estimating QD . The proposed approach , known as T-Revision ( Xia et al. , 2019 ) , learns a classifier from D and simultaneously identifies anchor-like instances in D , which are used iteratively to estimate QD , which in turn is used to improve the classifier . Hence for T-Revision , a good estimation for QD is inextricably tied to learning a classifier with high classification accuracy . In this paper , we propose a framework for estimating QD solely from D , without requiring anchor points , a clean subset , or even anchor-like instances . In particular , we show that high classification accuracy is not required for a good estimation of QD . Our framework is able to robustly estimate QD at all noise levels , even in extreme scenarios where anchor points are removed from D , or where D is imbalanced . Our key starting point is that Shannon entropy and other related information-theoretic concepts can be defined analogously for datasets with label noise . Suppose we have a discriminator Φ that takes any dataset D′ as its input , and gives a binary output that predicts whether D′ has maximum entropy . Given D , a dataset with label noise , we shall synthesize multiple new datasets D̂ by inserting additional label noise into D , using different noise levels for different label classes . Intuitively , the more label noise that D initially has , the lower the minimum amount of additional label noise we need to insert into D to reach near-maximum entropy . We show that among those datasets D̂ that are predicted by Φ to have maximum entropy , their associated levels of additional label noise can be used to compute a single estimate for QD . Our estimator is statistically consistent : We prove that by repeating this method , the average of the estimates would converge to the true QD . As a concrete realization of this idea , we shall construct Φ using the notion of Local Intrinsic Dimensionality ( LID ) ( Houle , 2013 ; 2017a ; b ) . Intuitively , the LID computed at a feature vector v is an approximation of the dimension of a smooth manifold containing v that would “ best ” fit the distribution D in the vicinity of v. LID plays a fundamental role in an important 2018 breakthrough in noise detection ( Ma et al. , 2018c ) , wherein it was empirically shown that sequences of LID scores could be used to distinguish clean datasets from datasets with label noise . Roughly speaking , the training data for Φ consists of LID sequences that correspond to multiple datasets synthesized from D. In particular , we show that Φ can be trained without needing any clean data . Since we are optimizing the predictive accuracy of Φ , rather than optimizing the classification accuracy for D , we also do not require state-of-the-art architectures . For example , in our experiments on the CIFAR-10 dataset ( Krizhevsky et al. , 2009 ) , we found that LID sequences generated by training on shallow “ vanilla ” convolutional neural networks ( CNNs ) , were sufficient for training Φ . Our contributions are summarized as follows : • We introduce an information-theoretic-based framework for estimating the noise transition matrix of any datasetD with instance-independent label noise . We do not make any assumptions on the structure of the noise transition matrix . • We prove that our noise transition matrix estimator is consistent . This is the first-ever estimator that is proven to be consistent without needing to optimize classification accuracy . Notably , our consistency proof does not require anchor points , a clean subset , or any anchor-like instances . • We construct an LID-based discriminator Φ and show experimentally that training a shallow CNN to generate LID sequences is sufficient for obtaining high predictive accuracy for Φ . Using our LID-based discriminator Φ , our proposed estimator outperforms the state-of-the-art methods , especially in the case when anchor-like instances are removed from D. • Given access to a clean subset Dclean , we show that our method can be used to further improve existing competitive estimation methods . 2 PROPOSED INFORMATION-THEORETIC FRAMEWORK . Our framework hinges on a simple yet crucial observation : Datasets with different label noise levels have different entropies . Although the entropy of any given dataset D is ( initially ) unknown to us , we do know , crucially , that a complete uniformly random relabeling of D would yield a new dataset with maximum entropy ( which we call “ baseline datasets ” ) , and we can easily generate multiple such datasets . We could also use partial relabelings to generate a spectrum of new datasets whose entropies range from the entropy of D , to the maximum possible entropy . We call them “ α-increment datasets ” , where α is a parameter that we control . The minimum value αmin for α , such that an α-increment dataset reaches maximum entropy , would depend on the original entropy of D. See Fig . 1 for a visualization of the spectrum of entropies for α-increment datasets and baseline datasets . Our main idea is to train a discriminator Φ that recognizes datasets with maximum entropy , and then use Φ to determine this minimum value αmin . Once this value is estimated , we are then able to estimate QD . Specific realizations of our framework correspond to specific designs for Φ . An illustration of our framework using LID-based discriminators is given in Fig . 2 ; details on LID-based discriminators can be found in Section 3 , and will be further elaborated in the appendix . Throughout this paper , given any discrete random variablesX , Y , we shall write pX ( x ) and pX|Y ( x|y ) to mean Pr ( X = x ) and Pr ( X = x|Y = y ) respectively . We assume that the reader is familiar with the basics of information theory ; see Cover & Thomas ( 2012 ) for an excellent introduction . 2.1 ENTROPY OF DATASETS WITH LABEL NOISE . Given D a dataset with instance-independent label noise ( DILN ) , let A be its set of all label classes , and let Y ( resp . Z ) be the given ( resp . correct ) label of a randomly selected instance X of D.1 For convenience , we say that D is a DILN with noise model ( Y |Z ; A ) . The noise transition matrix of D is a matrix QD whose ( i , j ) -th entry is qDi , j : = pY |Z ( j , i ) . We shall define the entropy of D by H ( D ) : = − ∑ i∈A pZ ( i ) ∑ j∈A qDi , j log q D i , j . Notice that H ( D ) is precisely the conditional entropy of Y given Z . ( We use the convention that 0 log 0 = 0 . ) Hence , it is easy to prove that 0 ≤ H ( D ) ≤ log |A| . In particular , H ( D ) = 0 if and only if every pair of instances of D in the same class have the same given labels . Note also that D has maximum entropy log |A| if and only if every entry of QD equals 1|A| ( i.e . the given labels of D are completely noisy ) . Thus , H ( D ) could be interpreted as a measure of the label noise level of D. 1Every datapoint of D is a pair ( x , y ) , where x is an instance , and y is its given label , which may differ from the correct label z associated to x . Note that Z is a function of X , and Y is a random function of Z . By instance-independent label noise , we mean that Pr ( Y = y|Z = z , X = x ) = Pr ( Y = y|Z = z ) . A more detailed treatment of DILNs can be found in Appendix A . In particular , a DILN includes its noise model . A derived DILN of D shall mean a DILN D′ with noise model ( Y ′|Z ; A ) for some Y ′ independent of Z , such that both D , D′ have the same underlying set of instances , given in the same sequential order . For example , D′ could be “ derived ” from D by inserting additional instance-independent label noise , in which case D′ can be interpreted as a partial relabeling of D. For convenience , we say that D′ is a Y ′-derived DILN of D . 2.2 SYNTHESIS OF NEW DATASETS FROM D. Let D be a DILN with noise model ( Y |Z ; A ) . Without loss of generality , assume A = { 1 , . . . , k } , assume that D has N instances , and write QD = [ qi , j ] 1≤i , j≤k . The correct labels for the instances are fixed and unknown to us , hence all entries of QD are fixed constants with unknown values . Our goal is to estimate QD . As alluded to earlier , we shall be synthesizing two types of datasets from D. The first type is what we call a baseline dataset , described as follows : Let D be a random dataset obtained from D by replacing the given label of each instance of D by a label chosen uniformly at random from A . Hence D is a random DILN with expected entropy log k ( i.e . maximum entropy ) , which we shall denote by Dmax : = E [ D ] . The noise transition matrix QD = [ Q′i , j ] 1≤i , j≤k of D is a random matrix whose entries Q′i , j = 1 k ( 1 + E ′ i , j ) are random variables , where each “ error ” E ′ i , j is a random variable with mean 0 . Any observed D is called a baseline DILN of D. The second type is what we call an α-increment dataset , where α = ( α1 , . . . , αk ) is a vector whose entries satisfy 0 ≤ αi ≤ 1 for all i . Let Dα be obtained from D as follows : For each 1 ≤ i ≤ k , select uniformly at random αi × 100 % of the instances with given label i , and reassign each selected given label to one of the remaining k−1 classes , chosen uniformly at random . HenceDα is a random DILN , and its noise transition matrix QDα = [ Q ′′ i , j ] 1≤i , j≤k is a random matrix whose entries Q′′i , j = qi , j ( 1− αj ) + ∑ 1≤t≤k t6=j qi , tαt 1 k−1 ( 1 + E ′′ i , j ) ( for all 1 ≤ i , j ≤ k ) ( 1 ) are random variables , where each “ error ” E′′i , j is a random variable with mean 0 . Any observed Dα is called an α-increment DILN of D .

Summary: The paper considers the problem of estimating instance-independent label noise. More formally, it is assumed that the true labels for any data point are modified based on a noise transition matrix, and the goal is to estimate this noise transition matrix. The paper proposed an information-theoretic approach for this task, the key idea behind which is to estimate if a particular dataset has maximum entropy with respect to the labels. This estimation problem is solved using a recent discovery that the training dynamics of a neural network can be used to infer the presence of label noise.