The Intrinsic Dimension of Images and Its Impact on Learning

1 INTRODUCTION . The idea that real-world data distributions can be described by very few variables underpins machine learning research from manifold learning to dimension reduction ( Besold & Spokoiny , 2019 ; Fodor , 2002 ) . The number of variables needed to describe a data distribution is known as its intrinsic dimension ( ID ) . In applications , such as crystallography , computer graphics , and ecology , practitioners depend on data having low intrinsic dimension ( Valle & Oganov , 2010 ; Desbrun et al. , 2002 ; Laughlin , 2014 ) . The utility of representations which are low-dimensional has motivated a variety of deep learning techniques including autoencoders and regularization methods ( Hinton & Salakhutdinov , 2006 ; Vincent et al. , 2010 ; Gonzalez & Balajewicz , 2018 ; Zhu et al. , 2018 ) . It is also known that dimensionality plays a strong role in learning function approximations and non-linear class boundaries . The exponential cost of learning in high dimensions is easily captured by the trivial case of sampling a function on a cube ; in d dimensions , sampling only the cube vertices would require 2d measurements . Similar behaviors emerge in learning theory . It is known that learning a manifold requires a number of samples that grows exponentially with the manifold ’ s intrinsic dimension ( Narayanan & Mitter , 2010 ) . Similarly , the number of samples needed to learn a well-conditioned decision boundary between two classes is an exponential function of the intrinsic dimension of the manifold on which the classes lie ( Narayanan & Niyogi , 2009 ) . Furthermore , these learning bounds have no dependence on the ambient dimension in which manifold-structured datasets live . In light of the exponentially large sample complexity of learning high-dimensional functions , the ability of neural networks to learn from image data is remarkable . Networks learn complex decision boundaries from small amounts of image data ( often just a few hundred or thousand samples per class ) . At the same time , generative adversarial networks ( GANs ) are able to learn image “ manifolds ” from merely a few thousand samples . The seemingly low number of samples needed to learn these manifolds strongly suggests that image datasets have extremely low-dimensional structure . Despite the established role of low dimensional data in deep learning , little is known about the intrinsic dimension of popular datasets and the impact of dimensionality on the performance of neural networks . Computational methods for estimating intrinsic dimension enable these measurements . We adopt tools from the dimension estimation literature to shed light on dimensionality in settings of interest to the deep learning community . Our contributions can be summarized as follows : • We verify the reliability of intrinsic dimension estimation on high-dimensional data using generative adversarial networks ( GANs ) , a setting in which we can a priori upper-bound the intrinsic dimension of generated data by the dimension of the latent noise vector . • We measure the dimensionality of popular datasets such as MNIST , CIFAR-10 , and ImageNet . In our experiments , we find that natural image datasets whose images contain thousands of pixels can , in fact , be described by orders of magnitude fewer variables . For example , we estimate that ImageNet , despite containing 224 × 224 × 3 = 150528 pixels per image , only has intrinsic dimension between 26 and 43 ; see Figure 1 . • We train classifiers on data , synthetic and real , of various intrinsic dimension and find that this variable correlates closely with the number of samples needed for learning . On the other hand , we find that extrinsic dimension , the dimension of the ambient space in which data is embedded , has little impact on generalization . Together , these results put experimental weight behind the hypothesis that the unintuitively low dimensionality of natural images is being exploited by deep networks , and suggest that a characterization of this structure is an essential building block for a successful theory of deep learning . 2 RELATED WORK . While the hypothesis that natural images lie on or near a low-dimensional manifold is controversial , Goodfellow et al . ( 2016 ) argue that the low-dimensional manifold assumption is at least approximately correct for images , supported by two observations . First , natural images are locally connected , with each image surrounded by other highly similar images reachable through image transformations ( e.g. , contrast , brightness ) . Second , natural images seem to lie on a low-dimensional structure , as the probability distribution of images is highly concentrated ; uniformly sampled pixels can hardly assemble a meaningful image . It is widely believed that the combination of natural scenes and sensor properties yields very sparse and concentrated image distributions , as has been supported by several empirical studies on image patches ( Lee et al. , 2003 ; Donoho & Grimes , 2005 ; Carlsson et al. , 2008 ) . This observation motivated work on efficient coding ( Olshausen & Field , 1996 ) and served as a prior in computer vision ( Peyré , 2009 ) . Further , rigorous experiments have been conducted clearly supporting the low-dimensional manifold hypothesis for many image datasets ( Ruderman , 1994 ; Schölkopf et al. , 1998 ; Roweis & Saul , 2000 ; Tenenbaum et al. , 2000 ; Brand , 2003 ) ; see also ( Fefferman et al. , 2016 ) for principled algorithms on verifying the manifold hypothesis . The generalization literature seeks to understand why some models generalize better from training data to test data than others . One line of work suggests that the loss landscape geometry explains why neural networks generalize well ( Huang et al. , 2019 ) . Other generalization work predicts that data with low dimension , along with other properties which do not include extrinsic dimension , characterize the generalization difficulty of classification problems ( Narayanan & Niyogi , 2009 ) . In the context of deep learning , Gong et al . ( 2019 ) found that neural network features are lowdimensional . Ansuini et al . ( 2019 ) further found that the intrinsic dimension of features decreases in late layers of neural networks and observed interesting trends in the dimension of features in early layers . In contrast to Gong et al . ( 2019 ) and Ansuini et al . ( 2019 ) , who find that the intrinsic dimension of internal representations is inversely correlated with high performance , we study the dimensionality of data and its impact on performance , and we make a similar finding . Zhu et al . ( 2018 ) proposed a regularizer derived from the intrinsic dimension of images augmented with their corresponding feature vectors . Another line of work in deep learning has found that neural networks rely heavily on textures which are low-dimensional ( Geirhos et al. , 2018 ; Brendel & Bethge , 2019 ) . Similarly , some have suggested that natural images can be represented as mixtures of textures which lie on a low-dimensional manifold ( Vacher & Coen-Cagli , 2019 ; Vacher et al. , 2020 ) . 3 INTRINSIC DIMENSION ESTIMATION . Given a set of sample points P ⊂ RN , it is common to assume that P lies on or near a lowdimensional manifoldM ⊆ RN of intrinsic dimension dim ( M ) = d N . As a measure of the degrees of freedom in a dataset , as well as the information content , there is great interest in estimating the intrinsic dimension d. In the remainder of this section , we briefly describe the dimension estimation method we use in this paper ; for further information , see ( Kim et al. , 2019 ) and references therein . One of the main approaches to intrinsic dimension estimation is to examine a neighborhood around each point in the dataset , and compute the Euclidean distance to the kth nearest neighbor . Assuming that density is constant within small neighborhoods , the Maximum Likelihood Estimation ( MLE ) of Levina & Bickel ( 2005 ) uses a Poisson process to model the number of points found by random sampling within a given radius around each sample point . By relating the rate of this process to the surface area of the sphere , the likelihood equations yield an estimate of the ID at a given point x as : m̂k ( x ) = 1 k − 1 k−1∑ j=1 log Tk ( x ) Tj ( x ) −1 , ( 1 ) where Tj ( x ) is the Euclidean ( ` 2 ) distance from x to its jth nearest neighbor . Levina & Bickel ( 2005 ) propose to average the local estimates at each point to obtain a global estimate m̄k = 1 n ∑n i=1 m̂k ( xi ) . MacKay & Ghahramani ( 2005 ) suggestion a correction based on averaging of inverses m̄k = [ 1 n n∑ i=1 m̂k ( xi ) −1 ] −1 = 1 n ( k − 1 ) n∑ i=1 k−1∑ j=1 log Tk ( xi ) Tj ( xi ) −1 , ( 2 ) where n is the number of samples . We use Equation ( 2 ) as our MLE estimator throughout this paper . Since the geometry of natural images is complex and unknown , we face two challenges when verifying the accuracy of MLE on natural image datasets . First , we need to choose a proper value of k. As shown by MacKay & Ghahramani ( 2005 ) , the positive bias of the corrected estimator Equation ( 2 ) increases as k increases , but the variance decreases . In order to navigate this bias-variance tradeoff , we try various values of k in Section 4 . Second , in addition to the aforementioned local uniformity assumption , MLE assumes that data arises as a sequence of i.i.d . random variables which can be written as a continuous and sufficiently smooth function of a random variable with smooth density , which may or may not be true for natural image datasets . While the truth of these assumptions is unknown on natural images , we verify the accuracy of our MLE estimates in a controlled setting in the following section . We briefly discuss other notable techniques for dimensionality estimation . GeoMLE ( Gomtsyan et al. , 2019 ) attempts to account for non-uniformity of density and nonlinearity of manifold using a polynomial regression of standard MLE based on distances to nearest neighbors in different sized neighborhoods . However , GeoMLE chooses to approximate averages of m̂k ( xi ) , instead of averaging its reciprocal like Equation ( 2 ) , resulting in a potentially wrong maximum likelihood estimator . As a result , we find its estimation deviates significantly from expected dimensionalities . TwoNN ( Facco et al. , 2017 ) is based on the ratio of the distances to the first and second nearest neighbors . Finally , the approach of ( Granata & Carnevale , 2016 ) considers the distribution of geodesic distances over the data manifold , approximated by distances through kNN graphs , compared to the distribution of distances over hyperspheres of varying dimension . Unlike MLE , our preliminary experiments suggest that these techniques do not provide reasonable estimates for some natural and synthetic images which are key to this work ; see Appendix A.5 for further discussion . 4 VALIDATING DIMENSION ESTIMATION WITH SYNTHETIC DATA . Dimensionality estimates are often applied on “ simple ” manifolds or toy datasets where the dimensionality is known , and so the accuracy of the methods can be validated . Image manifolds , by contrast , are highly complex , may contain many symmetries and modes , and are of unknown dimension . In principle , there is no reason why MLE-based dimensionality estimates can not be applied to image datasets . However , because we lack knowledge of the exact dimensionality of image datasets , we can not directly verify that MLE-based dimensionality estimates scale up to the complexity of image structures . There is an inherent uncertainty in estimating the ID of a given dataset . First , we can not be sure if the dataset actually resembles a sampling of points on or near a manifold . Second , there are typically no guarantees that the sampling satisfies the conditions assumed by the ID estimators we are using . Towards a principled application of ID estimates in contexts of practical relevance to deep learning , we begin by validating that MLE methods can generate accurate dimensionality estimates for complex image data . We do this by generating synthetic image datasets using generative models for which the intrinsic dimensionality can be upper-bounded a priori . We believe such validations are essential to put recent findings in perspective ( Gong et al. , 2019 ; Ansuini et al. , 2019 ) . GAN Images We use the BigGAN variant with 128 latent entries and outputs of size 128×128×3 trained on the ImageNet dataset ( Deng et al. , 2009 ) . Using this GAN , we generate datasets with a varying number of images , where we fix most entries of the latent vectors to zero leaving only d̄ free entries to be chosen at random . As we increase the number of free entries , we expect the intrinsic dimension to increase with d̄ as an upper bound ; see Section A.1 for further discussion . In particular , we create several synthetic datasets of varying intrinsic dimensionality using the ImageNet class , basenji , and check if the estimates match our expectation . As seen in Figure 2 , we observe increasing diversity with increasing intrinsic dimension . In Figure 3 , we show convergence of the MLE estimate on basenji data with dimension bounded above by d̄ = 10 . We observe that the estimates can be sensitive to the choice of k as discussed in prior work ; see Appendix A.2 for additional GAN classes . Scaling to large datasets . We develop a practical approach for estimating the ID of large datasets such as ImageNet . In this approach , we randomly select a fraction α of the dataset as anchors . Then , we evaluate the MLE estimate using only the anchor points , where nearest-neighbors are computed over the entire dataset . Note that , when anchors are chosen randomly , this acceleration has no impact on the expected value of the result . See Appendix A.3 for an evaluation of this approach .

Summary: The authors report a novel application of GANs to validate the maximum likelihood estimator (MLE) of the intrinsic dimension (ID) of image data sets. Then they use the MLE ID estimator to characterize the intrinsic dimension of several commonly used computer vision data sets, and link the data set ID to the generalizability of trained classifiers.  They provide additional experiments that support the notion that it is intrinsic dimension, and not extrinsic dimension (i.e. # of pixels), that governs the performance of a binary classifier on these data sets. Also, they verify that dimension plays a large role in learning on natural data.