DIME: AN INFORMATION-THEORETIC DIFFICULTY MEASURE FOR AI DATASETS

1 INTRODUCTION . Empirical machine learning research relies heavily on comparing performance of algorithms on a few standard benchmark datasets . Moreover , researchers frequently introduce new datasets that they believe to be more challenging than existing benchmarks . However , we lack objective measures of dataset difficulty that are independent of the choices made about algorithm- and model-design . Moreover , it is also hard to compare algorithmic progress across data modalities , such as language and vision . So , for instance , it is difficult to compare the relative progress made on a sentiment analysis benchmark such as the Stanford Sentiment Treebank ( SST ) ( Socher et al. , 2013 ) and an image classification benchmark , like CIFAR-10 ( Krizhevsky , 2009 ) . With these challenges in mind , we propose a model-agnostic and modality-agnostic measure for comparing how difficult it is to perform supervised learning on a given dataset . Intuitively , assuming that dataset examples are sampled i.i.d . from a static true distribution , we argue that the difficulty of a dataset can be decoupled into two relatively independent sources : ( a ) approximation complexity , the number of samples required to approximate the true distribution up to certain accuracy , and ( b ) distributional complexity , the intrinsic difficulty involved in modeling the statistical relationship between the labels and features . We focus our analysis on the second source of the intrinsic difficulty in supervised learning , where both features and labels are available . To provide a model-agnostic measure , we turn to the information-theoretic approach . Indeed , there already exist lower bounds on the lowest possible errors given the distribution of the data . If both the samples and labels are discrete , Fano ’ s inequality suggests the probability of 0-1 error is bounded by terms related to the conditional entropy H ( Y |X ) , where X is the random variable representing the features and Y is the label . When both the features X and label Y are continuous , results on differential entropy also suggest the expected L2 error is lower-bounded . However , in most of the supervised learning datasets where the features are continuous and the labels are discrete , it is unknown how the lowest possible error can be controlled regardless of models . In this paper , we show that even for the hybrid case where labels are discrete and features are continuous , with some additional assumptions , Fano ’ s inequality still holds . Moreover , we show that the lowest possible probability Pe of the 0-1 error for a given data distribution is lower bounded by terms related to a hybrid conditional entropy H ( Y |X ) . We further design an estimator for the lower bound of Pe based on our generalized Fano ’ s inequality . The estimator uses neural networks to approximate the KL divergence based on Donsker-Varadhan representation , which is then used to estimate the hybrid conditional entropy H ( Y |X ) as well as the lower bound of Pe . We emphasize that even though our lower bound is model-agnostic , the proposed estimator is based on a neural network . However , we empirically show that , for most image and natural language datasets , a multilayer perceptron-based estimator produces a measure that effectively captures the difficulty of the data and aligns well with the performance of state-of-the-art models . Related Work : Although conditional entropy and mutual information estimation have been extensively studied , research has focused on purely discrete or continuous data . Nair et al . ( 2006 ) were among the first to study the theoretical properties for the case of mixture of discrete and continuous variables . Ross ( 2014 ) , Gao et al . ( 2017 ) and Beknazaryan et al . ( 2019 ) proposed approaches for estimating mutual information for the mixture case based on density ratio estimation ( e.g. , binning , kNN or kernel methods ) , which is unsatisfactory for high dimensional data such as image and text . We use neural network estimation ( Belghazi et al. , 2018 ) to avoid these issues . More importantly , we are the first to connect the hybrid conditional entropy with the lowest classification error and are able to use it as a difficulty measure for datasets . 2 DESIGNING A DATASET DIFFICULTY MEASURE . For supervised learning across data modalities such as images and text , data samples can usually be viewed as feature-label pairs ( x , y ) where x ∈ X ⊂ Rdx , and y ∈ Y . We focus on classification problems where the labels y are discrete , i.e. , Y ⊂ Z+ . We denote the joint distribution of the feature-label pairs as PXY . The marginal distributions of the features and labels are denoted as PX and PY respectively . We make the following widely adopted assumption from learning theory literature about how samples are generated : Assumption 1 . The feature-label pairs ( x , y ) in the datasets , both training and testing , are sampled i.i.d . from a static distribution ( x , y ) ∼ PXY . Intuitively , there are many possible indicators for the potential difficulty of a dataset : the number of features , the number of classes , the number of samples , the distinguishability of samples across classes , as well as the difference between the data distributions of the training set and the testing set . However , none of these indicators alone can fully describe the relative difficulty of a dataset . From Assumption 1 , if the samples ( x , y ) are sampled i.i.d . from PXY , where Y is discrete , a natural measure that characterizes the difficulty of the data distribution is the best probability of the 0-1 error that can be achieved by any estimator . Definition 1 ( Model-Agnostic Error ) . Pe = inff Px , y∼PXY [ f ( x ) 6= y ] The measure 1 is straightforward , but unfortunately it is hard to compute since it involves evaluations against all possible estimators . However , with mild assumptions , it can be lower bounded by terms related to the conditional entropy , which is much easier to evaluate . 2.1 DISCRETE FEATURES . In the case where the features x ∈ X are discrete , according to Fano ’ s theorem , Pe is lower bounded : Fano ’ s inequality . If both X and Y are discrete random variables , then Pe ≥ H ( Y |X ) −1log |Y| , where |Y| is the cardinality of the label set Y , and H ( Y |X ) is the conditional entropy . However , even though data can be represented using discrete integers , treating the features as discrete random variables leads to the following difficulties : 1 . The cardinality of the feature space becomes extremely large if discrete features are used . For image data , since each pixel is represented as an integer , the ( raw ) feature dimension would become the number of pixels in the image , which can be extremely large . Similarly , for language data , when the sequence length is long , the feature dimension becomes large very quickly . 2 . Given the large feature space , finding a matching set of features between training and testing data from the limited number of training and test samples would be unlikely . As a result , probability mass estimation on each discrete value would be impractical , since we may only see at most one sample for each discrete configuration . 2.2 CONTINUOUS FEATURES . As opposed to treating the features x as discrete random variables , if we view them as i.i.d . samples from a continuous distribution with probability density px , we can estimate the conditional entropy H ( Y |X ) under some smoothness assumptions and also infer the model-agnostic error Pe . However , classical Fano ’ s inequality only holds for discrete random variables . For the case with the continuous features and discrete labels , it has not been shown how Pe can be controlled . In this paper , we prove a generalized version of Fano ’ s inequality that holds for the continuous-featurediscrete-label scenario . Formally , for the continuous-X-discrete-Y case : Definition 2 . Hybrid Conditional Entropy H ( Y |X ) : = EX − |Y|∑ y=1 P ( Y = y|X ) logP ( Y = y|X ) , ( 1 ) where P ( Y = y|X ) : = E [ 1 ( Y = y ) |X ] . This definition of H ( Y |X ) is consistent with the classical definition , in the sense that both of them give the intuition that how much information or uncertainty is left for Y given X . Next , to connect Pe and the hybrid H ( Y |X ) , we introduce an assumption on the function f Definition 3 . Smooth Discretization Property : The function f : X → Y satisfies the smooth discretization property if for every y ∈ Y , almost every x ∈ X ( x a.e . in X ) , f ( x ) = y ⇐⇒ ∃δ > 0 s.t . ∀x̃ ∈ Bδ ( x ) , f ( x̃ ) = y , where Bδ ( x ) : = { x̃ ∈ Ω : ‖x̃− x‖2 < δ } is a δ−neighborhood of x in X . This assumption on the classifier function f is not unnatural considering f maps a continuous variable to a discrete variable . Without this assumption , it would be extremely hard to quantify the population error probability P ( f ( X ) 6= Y ) since the behavior of f may be erratic . Furthermore , in the real data setting , this assumption is always satisfied for every classifier f , as we can always construct a small enough neighborhood of each data point such that they are disjoint and assume f is a constant in each neighborhood . In this sense the assumption on f is pretty minimal . Now we are ready to extend Fano ’ s inequality : Theorem 1 . Fano ’ s Inequality for Continuous Features : Let Pe be the minimum error probability , i.e. , Pe = inf f P ( f ( X ) 6= Y ) where f is any estimator of Y based on the observation X that satisfies the smooth discretization property . Then we have H ( Pe ) + Pe log ( m− 1 ) ≥ H ( Y |X ) , ( 2 ) where H ( Pe ) : = −Pe logPe − ( 1− Pe ) log ( 1− Pe ) . Proof . See appendix B . 3 ESTIMATING THE LOWER BOUND . It is natural to consider Pe defined in Theorem 1 as a measure of dataset difficulty . Unfortunately , direct estimation of Pe is impractical since one has to evaluate the estimation error against all possible estimators . However , theorem 1 provides an alternative towards estimating a lower bound on Pe through estimating the hybrid conditional entropy H ( Y |X ) defined in Equation ( 1 ) . 3.1 CONDITIONAL ENTROPY ESTIMATION . In real applications , direct calculation of hybrid conditional entropy H ( Y |X ) according to Definition 2 is impossible since P ( Y = y|X ) is unknown . However , similar to the conditional entropy for discrete random variables , the hybrid conditional entropy H ( Y |X ) can also be written as H ( Y |X ) = H ( Y ) − |Y|∑ y=1 P ( Y = y ) KL ( X|Y = y||X ) . ( 3 ) Please refer to Appendix A for a detailed proof of Equation ( 3 ) . We also define the hybrid mutual information I ( X ; Y ) , which is compatible with H ( Y |X ) : I ( X ; Y ) = |Y|∑ y=1 P ( Y = y ) KL ( X|Y = y||X ) ( 4 ) In some benchmark datasets with balanced classes ( e.g. , CIFAR-10 and MNIST ) , H ( Y |X ) = log |Y| − 1|Y| ∑|Y| y=1 KL ( X|Y = y||X ) . This indicates that if a dataset has more classes and the features for different classes are closer to each other on average , then H ( Y |X ) would be larger .

Summary: So this paper is interesting. It's sort of pursuing a similar path as recent works that use neural networks to evaluate (e.g., inception score, FID), notably those that optimize some lower bound of an information measure (e.g., MINE). In this case, the setting is "datasets", and the thing they are trying to quantify is the difficulty of the dataset as expressed by a lower bound to the lowest possible probability of the 0-1 error, which they should is related to the conditional entropy of the underlying input / label random variables (which makes sense). This direction seems very useful, and using neural network optimization to attempt to crack defining "dataset complexity" or "difficulty" seems a worthwhile venture.