DIME: AN INFORMATION-THEORETIC DIFFICULTY MEASURE FOR AI DATASETS

1 INTRODUCTION . Empirical machine learning research relies heavily on comparing performance of algorithms on a few standard benchmark datasets . Moreover , researchers frequently introduce new datasets that they believe to be more challenging than existing benchmarks . However , we lack objective measures of dataset difficulty that are independent of the choices made about algorithm- and model-design . Moreover , it is also hard to compare algorithmic progress across data modalities , such as language and vision . So , for instance , it is difficult to compare the relative progress made on a sentiment analysis benchmark such as the Stanford Sentiment Treebank ( SST ) ( Socher et al. , 2013 ) and an image classification benchmark , like CIFAR-10 ( Krizhevsky , 2009 ) . With these challenges in mind , we propose a model-agnostic and modality-agnostic measure for comparing how difficult it is to perform supervised learning on a given dataset . Intuitively , assuming that dataset examples are sampled i.i.d . from a static true distribution , we argue that the difficulty of a dataset can be decoupled into two relatively independent sources : ( a ) approximation complexity , the number of samples required to approximate the true distribution up to certain accuracy , and ( b ) distributional complexity , the intrinsic difficulty involved in modeling the statistical relationship between the labels and features . We focus our analysis on the second source of the intrinsic difficulty in supervised learning , where both features and labels are available . To provide a model-agnostic measure , we turn to the information-theoretic approach . Indeed , there already exist lower bounds on the lowest possible errors given the distribution of the data . If both the samples and labels are discrete , Fano ’ s inequality suggests the probability of 0-1 error is bounded by terms related to the conditional entropy H ( Y |X ) , where X is the random variable representing the features and Y is the label . When both the features X and label Y are continuous , results on differential entropy also suggest the expected L2 error is lower-bounded . However , in most of the supervised learning datasets where the features are continuous and the labels are discrete , it is unknown how the lowest possible error can be controlled regardless of models . In this paper , we show that even for the hybrid case where labels are discrete and features are continuous , with some additional assumptions , Fano ’ s inequality still holds . Moreover , we show that the lowest possible probability Pe of the 0-1 error for a given data distribution is lower bounded by terms related to a hybrid conditional entropy H ( Y |X ) . We further design an estimator for the lower bound of Pe based on our generalized Fano ’ s inequality . The estimator uses neural networks to approximate the KL divergence based on Donsker-Varadhan representation , which is then used to estimate the hybrid conditional entropy H ( Y |X ) as well as the lower bound of Pe . We emphasize that even though our lower bound is model-agnostic , the proposed estimator is based on a neural network . However , we empirically show that , for most image and natural language datasets , a multilayer perceptron-based estimator produces a measure that effectively captures the difficulty of the data and aligns well with the performance of state-of-the-art models . Related Work : Although conditional entropy and mutual information estimation have been extensively studied , research has focused on purely discrete or continuous data . Nair et al . ( 2006 ) were among the first to study the theoretical properties for the case of mixture of discrete and continuous variables . Ross ( 2014 ) , Gao et al . ( 2017 ) and Beknazaryan et al . ( 2019 ) proposed approaches for estimating mutual information for the mixture case based on density ratio estimation ( e.g. , binning , kNN or kernel methods ) , which is unsatisfactory for high dimensional data such as image and text . We use neural network estimation ( Belghazi et al. , 2018 ) to avoid these issues . More importantly , we are the first to connect the hybrid conditional entropy with the lowest classification error and are able to use it as a difficulty measure for datasets . 2 DESIGNING A DATASET DIFFICULTY MEASURE . For supervised learning across data modalities such as images and text , data samples can usually be viewed as feature-label pairs ( x , y ) where x ∈ X ⊂ Rdx , and y ∈ Y . We focus on classification problems where the labels y are discrete , i.e. , Y ⊂ Z+ . We denote the joint distribution of the feature-label pairs as PXY . The marginal distributions of the features and labels are denoted as PX and PY respectively . We make the following widely adopted assumption from learning theory literature about how samples are generated : Assumption 1 . The feature-label pairs ( x , y ) in the datasets , both training and testing , are sampled i.i.d . from a static distribution ( x , y ) ∼ PXY . Intuitively , there are many possible indicators for the potential difficulty of a dataset : the number of features , the number of classes , the number of samples , the distinguishability of samples across classes , as well as the difference between the data distributions of the training set and the testing set . However , none of these indicators alone can fully describe the relative difficulty of a dataset . From Assumption 1 , if the samples ( x , y ) are sampled i.i.d . from PXY , where Y is discrete , a natural measure that characterizes the difficulty of the data distribution is the best probability of the 0-1 error that can be achieved by any estimator . Definition 1 ( Model-Agnostic Error ) . Pe = inff Px , y∼PXY [ f ( x ) 6= y ] The measure 1 is straightforward , but unfortunately it is hard to compute since it involves evaluations against all possible estimators . However , with mild assumptions , it can be lower bounded by terms related to the conditional entropy , which is much easier to evaluate . 2.1 DISCRETE FEATURES . In the case where the features x ∈ X are discrete , according to Fano ’ s theorem , Pe is lower bounded : Fano ’ s inequality . If both X and Y are discrete random variables , then Pe ≥ H ( Y |X ) −1log |Y| , where |Y| is the cardinality of the label set Y , and H ( Y |X ) is the conditional entropy . However , even though data can be represented using discrete integers , treating the features as discrete random variables leads to the following difficulties : 1 . The cardinality of the feature space becomes extremely large if discrete features are used . For image data , since each pixel is represented as an integer , the ( raw ) feature dimension would become the number of pixels in the image , which can be extremely large . Similarly , for language data , when the sequence length is long , the feature dimension becomes large very quickly . 2 . Given the large feature space , finding a matching set of features between training and testing data from the limited number of training and test samples would be unlikely . As a result , probability mass estimation on each discrete value would be impractical , since we may only see at most one sample for each discrete configuration . 2.2 CONTINUOUS FEATURES . As opposed to treating the features x as discrete random variables , if we view them as i.i.d . samples from a continuous distribution with probability density px , we can estimate the conditional entropy H ( Y |X ) under some smoothness assumptions and also infer the model-agnostic error Pe . However , classical Fano ’ s inequality only holds for discrete random variables . For the case with the continuous features and discrete labels , it has not been shown how Pe can be controlled . In this paper , we prove a generalized version of Fano ’ s inequality that holds for the continuous-featurediscrete-label scenario . Formally , for the continuous-X-discrete-Y case : Definition 2 . Hybrid Conditional Entropy H ( Y |X ) : = EX − |Y|∑ y=1 P ( Y = y|X ) logP ( Y = y|X ) , ( 1 ) where P ( Y = y|X ) : = E [ 1 ( Y = y ) |X ] . This definition of H ( Y |X ) is consistent with the classical definition , in the sense that both of them give the intuition that how much information or uncertainty is left for Y given X . Next , to connect Pe and the hybrid H ( Y |X ) , we introduce an assumption on the function f Definition 3 . Smooth Discretization Property : The function f : X → Y satisfies the smooth discretization property if for every y ∈ Y , almost every x ∈ X ( x a.e . in X ) , f ( x ) = y ⇐⇒ ∃δ > 0 s.t . ∀x̃ ∈ Bδ ( x ) , f ( x̃ ) = y , where Bδ ( x ) : = { x̃ ∈ Ω : ‖x̃− x‖2 < δ } is a δ−neighborhood of x in X . This assumption on the classifier function f is not unnatural considering f maps a continuous variable to a discrete variable . Without this assumption , it would be extremely hard to quantify the population error probability P ( f ( X ) 6= Y ) since the behavior of f may be erratic . Furthermore , in the real data setting , this assumption is always satisfied for every classifier f , as we can always construct a small enough neighborhood of each data point such that they are disjoint and assume f is a constant in each neighborhood . In this sense the assumption on f is pretty minimal . Now we are ready to extend Fano ’ s inequality : Theorem 1 . Fano ’ s Inequality for Continuous Features : Let Pe be the minimum error probability , i.e. , Pe = inf f P ( f ( X ) 6= Y ) where f is any estimator of Y based on the observation X that satisfies the smooth discretization property . Then we have H ( Pe ) + Pe log ( m− 1 ) ≥ H ( Y |X ) , ( 2 ) where H ( Pe ) : = −Pe logPe − ( 1− Pe ) log ( 1− Pe ) . Proof . See appendix B . 3 ESTIMATING THE LOWER BOUND . It is natural to consider Pe defined in Theorem 1 as a measure of dataset difficulty . Unfortunately , direct estimation of Pe is impractical since one has to evaluate the estimation error against all possible estimators . However , theorem 1 provides an alternative towards estimating a lower bound on Pe through estimating the hybrid conditional entropy H ( Y |X ) defined in Equation ( 1 ) . 3.1 CONDITIONAL ENTROPY ESTIMATION . In real applications , direct calculation of hybrid conditional entropy H ( Y |X ) according to Definition 2 is impossible since P ( Y = y|X ) is unknown . However , similar to the conditional entropy for discrete random variables , the hybrid conditional entropy H ( Y |X ) can also be written as H ( Y |X ) = H ( Y ) − |Y|∑ y=1 P ( Y = y ) KL ( X|Y = y||X ) . ( 3 ) Please refer to Appendix A for a detailed proof of Equation ( 3 ) . We also define the hybrid mutual information I ( X ; Y ) , which is compatible with H ( Y |X ) : I ( X ; Y ) = |Y|∑ y=1 P ( Y = y ) KL ( X|Y = y||X ) ( 4 ) In some benchmark datasets with balanced classes ( e.g. , CIFAR-10 and MNIST ) , H ( Y |X ) = log |Y| − 1|Y| ∑|Y| y=1 KL ( X|Y = y||X ) . This indicates that if a dataset has more classes and the features for different classes are closer to each other on average , then H ( Y |X ) would be larger .

Summary: The paper proposes a measure of difficulty for datasets. Prior work in this space has often utilized certain indicators like the overlap of samples across different classes etc. [A] While this work defines a model-agnostic error as the measure of difficulty, which should encompass all possible indicators of error. Then, the paper provides a lower bound on this error which can be estimated using neural network [B]