Quality Evaluation of GANs Using Cross Local Intrinsic Dimensionality

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) are powerful models for data generation , composed of two neural networks , known as the ‘ generator ’ and the ‘ discriminator ’ . The generator maps random noise vectors to locations in the data domain in an attempt to approximate the distribution of the input data . The discriminator accepts a data sample and returns a decision as to whether or not the sample is from the input or was artificially generated . While the discriminator is trained to distinguish input samples from generated ones , the generator ’ s objective is to deceive the discriminator by producing data that can not be distinguished from input data . The two networks are jointly trained to optimize an objective function resembling a two-player minimax game . GANs were first formulated by Goodfellow et al . ( 2014 ) , and have been applied to tasks such as image generation ( Denton et al. , 2015 ; Radford et al. , 2016 ) and image inpainting ( Pathak et al. , 2016 ) . Despite their elegant theoretical formulation ( Goodfellow et al. , 2014 ) , training of GANs can be difficult in practice due to instability issues such as vanishing gradients and mode collapse . The vanishing gradient problem occurs whenever gradients become too small to allow sufficient progress towards an optimization goal within the allotted number of training iterations . The latter occurs when the generator produces samples for only a limited number of data modes , without covering the full distribution of the input data . Deployment of GANs is further complicated by the difficulty of evaluating the quality of their output . Researchers often rely on visual inspection of generated samples , which is both time-consuming and subjective . A quantitative quality metric is clearly desirable , and several such methods do exist ( Goodfellow et al. , 2014 ; Salimans et al. , 2016 ; Odena et al. , 2017 ; Che et al. , 2017 ; Heusel et al. , 2017 ; Karras et al. , 2018 ; Lucic et al. , 2017 ; Lopez-Paz & Oquab , 2017 ; Shmelkov et al. , 2018 ) . However , past research has identified various limitations of some existing metrics ( Theis et al. , 2016 ; Barratt & Sharma , 2018 ) , and effective evaluation of GAN models is still an open issue . In this paper , we show how the data space learned by a GAN model can be understood in terms of the Local Intrinsic Dimensionality ( LID ) model of distance distributions ( Houle , 2013 ) . LID assesses the number of latent variables ( the intrinsic dimensionality ) needed to characterize the distribution of distances to a reference point x — or equivalently , the discriminability of a distance measure in the vicinity of x . For the GAN discriminator , input samples are more likely to be discriminable from generated samples if the learning process maps them into a space where the distance measure becomes more discriminable — or equivalently , one where the local intrinsic dimensionality is relatively low . For the GAN generator , a generated sample is more likely to be accepted as realistic if the learning process maps it into a local submanifold whose dimensionality matches that of its neighbors among the input data samples ( after the mapping to the learned space ) . Our objective here is to develop this intuition into a technique for assessing the quality of the GAN learning process . The main contributions of the paper are as follows : • We propose CrossLID , a cross estimation technique based on LID that is capable of assessing the alignment of the data embedding learned by the GAN generator with that learned by the GAN discriminator . • We show how CrossLID can be employed to avoid mode collapse during GAN training , to identify classes for which some or all modes are not well-covered by the learning process . We also show how this knowledge can then be used to bias the GAN discriminator via an oversampling strategy so as to improve its performance on such classes . • Experimentation showing that our CrossLID measure is well correlated with GAN sample quality , and comparison to two state-of-the-art evaluation measures . 2 EVALUATION METRICS FOR GAN MODELS . GAN-based learning is an extensively researched area . Here , we briefly review the topic most relevant to our work , evaluation metrics for GAN models . Past research has employed several different metrics , including log-likelihood measures ( Goodfellow et al. , 2014 ) , the Inception score ( Salimans et al. , 2016 ) , the MODE score ( Che et al. , 2017 ) , Kernel MMD ( Gretton et al. , 2006 ) , the MS-SSIM index ( Odena et al. , 2017 ) , the Fréchet Inception Distance ( Heusel et al. , 2017 ) , the sliced Wasserstein distance ( Karras et al. , 2018 ) , and Classifier Two-Sample Tests ( Lopez-Paz & Oquab , 2017 ) . In our study , we focus on the two most widely used metrics for image data , the Inception score ( IS ) and the Fréchet Inception Distance ( FID ) , as well as a recently proposed measure , the Geometry Score ( GS ) ( Khrulkov & Oseledets , 2018 ) . The Inception score uses an associated Inception classifier ( Szegedy et al. , 2016 ) to extract output class probabilities for each image , and then computes the Kullback-Leibler ( KL ) divergence of these probabilities with respect to the marginal probabilities of all classes : IS = exp ( Ex∼pGDKL ( p ( y|x ) ||p ( y ) ) , ( 1 ) where x ∼ pG implies a sample x drawn from the generator outputs , p ( y|x ) is the probability of class y being assigned to x by the Inception classifier , p ( y ) = ∫ x ( p ( y|x ) dx is the marginal class distribution , and DKL is the KL divergence . IS measures two aspects of a generative model : 1 ) the images generated should be both clear and highly distinguishable by the classifier , as indicated by low entropy of p ( y|x ) when marginalized over y , and 2 ) all classes should have good representation over the set of generated images , which can be indicated by high entropy of p ( y|x ) when marginalized over x . However , a recent study has shown that IS is susceptible to variations in the Inception network weights when trained on different platforms ( Barratt & Sharma , 2018 ) . The Fréchet Inception Distance ( FID ) passes both input and generated images to an Inception classifier , and extracts activations from an intermediate pooling layer . Activations are assumed to follow multidimensional Gaussians parameterized by their means and covariances . The FID is defined as FID = ( ||µI − µG|| ) 22 + Tr ( ΣI + ΣG − 2 ( ( ΣIΣG ) 1 2 ) ) , ( 2 ) where ( µI , ΣI ) and ( µG , ΣG ) represent the mean and covariance of activations for input and generated data samples , respectively . Compared to IS , FID has been shown to more consistent with human judgment and more robust to noise ; however , it also requires an external Inception classifier for its calculation ( Heusel et al. , 2017 ) . A recently proposed metric , the Geometry Score ( GS ) ( Khrulkov & Oseledets , 2018 ) assesses the conformity between manifolds of input and generated data , in terms of the persistence of certain topological properties in a manifold approximation process . The topological relationships are extracted in terms of the counts of 1-dimensional loops in a graph structure built up from proximity relationships as a distance threshold is increased . Although it may be indirectly sensitive to variations in the dimensionality of the manifolds , the Geometry Score ( GS ) explicitly rewards only matches in terms of the specific topology of these loop structures in approximations of the manifolds . However , due to its strictly topological nature , GS is insensitive to differences in relative embedding distances or orientations within the manifold — this issue is acknowledged by the authors , who advise that GS would be best suited for use in conjunction with other metrics ( Khrulkov & Oseledets , 2018 ) . 3 LOCAL INTRINSIC DIMENSIONALITY . In its most general sense , the Local Intrinsic Dimensionality ( LID ) is a characteristic of smooth functions that vanish at zero . The LID value can be regarded as the degree of the polynomial with the best fit to the function , taken over an infinitesimally small domain that includes the origin . Definition of LID : To formalize LID , let F be a function that is positive and continuously differentiable over some open interval containing r > 0 . The LID of F at r is defined as : LIDF ( r ) : = r F ′ ( r ) F ( r ) = lim →0+ log ( F ( ( 1 + ) r ) /F ( r ) ) log ( 1 + ) = lim →0+ F ( ( 1 + ) r ) − F ( r ) F ( r ) , ( 3 ) wherever the limits exists . The local intrinsic dimensionality of F is then : LID∗F = lim r→0+ LIDF ( r ) . ( 4 ) In our context , and as originally proposed in ( Houle , 2013 ) , we are interested in functions that are the distributions of distances induced by some global distribution of data points : for each data sample generated with respect to the global distribution , its distance to a predetermined reference point determines a sample from the local distance distribution . The LID model has the interesting property that the definition can be motivated in two different ways . The first limit stated in the definition follows from a modeling of the growth of probability measure in a small expanding neighborhood of the origin : as the radius r increases , the amount of data encountered can be expected to grow proportionally to the r to the power of the intrinsic dimension . Although the LID model is oblivious of the representational dimension of the data domain , in the setting of a uniform distribution with a manifold of dimensionm , if F is the distribution of distances to a reference point in the relative interior of the manifold , then LID∗F = m. The second limit expresses the ( in ) discriminability of F when interpreted as a distance measure evaluated at distance r ( with low values of LIDF ( r ) indicating higher discriminability ) . As implied by Eq . 3 , the LID framework is extremely convenient in that the local intrinsic dimensionality and the discriminability of distance measures are shown to be equivalent and interchangeable concepts . For more information on the formal definition of LID and its properties , see ( Houle , 2017a ; b ) . Estimating LID : LID is a generalization of pre-existing expansion-based measures which implicitly use neighborhood set sizes as a proxy for probability measure . These earlier models include the expansion dimension ( Karger & Ruhl , 2002 ) and its variants ( Houle et al. , 2012a ) , and the minimum neighbor distance ( MinD ) ( Rozza et al. , 2012 ) , all of which have been shown to be crude estimators of LID ( Amsaleg et al. , 2018 ) . Although the popular estimator due to Levina & Bickel ( 2005 ) can be regarded as a smoothed version of LID , its derivation depends on the assumption that the observed data can be treated as a homogeneous Poisson process . However , the only assumptions made by the LID model is that the underlying ( distribution ) function be continuously differentiable . For this work , we estimate LID using the Maximum Likelihood Estimator ( MLE ) as proposed in ( Amsaleg et al. , 2018 ) , due to its ease of implementation and its superior convergence properties relative to the other estimators studied there . Given a set of data points X , and a distinguished data sample x , the MLE estimator of LID is : LID ( x ; X ) = − ( 1 k k∑ i=1 ln ri ( x ; X ) rmax ( x ; X ) ) −1 = ( ln rmax ( x ; X ) − 1 k k∑ i=1 ln ri ( x ; X ) ) −1 . ( 5 ) where k is the neighborhood size , ri ( x ; X ) is the distance from x to its i-th nearest neighbor in X \ { x } , and rmax ( x ; X ) denotes the maximum distance within the neighborhood ( which by convention can be rk ( x ; X ) ) . Due to the deep equivalence between the LID model and the statistical theory of extreme values ( EVT ) shown in ( Houle , 2017a ; Amsaleg et al. , 2018 ) , the first of the two equivalent formulations in Eq . 5 coincides with the well-known Hill estimator of scale derived from EVT ( Hill , 1975 ) . As can be seen from the second formulation , the reciprocal of the MLE estimator assesses the discriminability within the k-NN set of x as the difference between the maximum and mean of log-distance values . Note that in these estimators , no explicit knowledge of the underlying function F is needed - this information is implicit in the distribution of neighbor distances themselves . Estimation of LID characteristics has seen practical applications for assessing the complexity of search queries in approximate similarity search ( Houle et al. , 2012b ; Houle & Nett , 2015 ; Casanova et al. , 2017 ) , as a measure of the outlierness of data ( Houle et al. , 2018 ) and in the detection of adversarial samples for deep neural networks ( Ma et al. , 2018a ) . LID has also been used to detect and prevent overfitting by DNN classifiers on datasets with noisy labels ( Ma et al. , 2018b ) . LID can characterize the intrinsic dimensionality of the data submanifold in the vicinity of a distinguished point x . The LID ( x ; X ) values of all data samples x from a dataset X can thus be averaged to characterize the overall intrinsic dimensionality of the manifold within which X resides . In Romano et al . ( 2016 ) ; Houle ( 2017b ) , it was shown that this type of average is in fact an estimator of the correlation dimension over the sample domain ( or manifold ) . Henceforth , whenever the context set X is understood , we will use the simplified notation LID ( x ) to refer to LID ( x ; X ) , and to denote the average of these estimates over all x ∈ X by LID ( X ) .

Summary: Statistics based on KNN distances are ubiquitous in machine learning. In this paper the authors propose to apply the existing LID metric to GANs. The metric can be decomposed as follows: (1) Given a point x in X, compute the k-nearest neighbors KNN(x, X) and let those distances be R1, R2, …, Rk. Now, rewrite LID(x, X) = [max_over_i (log Ri) - mean_over_i (log Ri)] to uncover that the distribution of (log-)distances is summarized as a function of the max distance and the mean distance. (2) To extend the metric to two sets, A and B, define CrossLID(A; B) = E_(x in A) [LID(x, B)]. To see why CrossLID is useful, let X be the observed data and G the generated data. First consider CrossLID(A, B) where A=B=X which determines a lower-bound which is essentially the average (over elements of A) LID statistic determined by the underlying KNN graph of X. Now, keep A=X, and progressively change B to G (say by replacing some points from X with some points from G). This will induce a change of the distance statistics of some points from A, which will be detected on the individual LID scores of those points, and will hence be propagated to CrossLID. As a result, LID close to the baseline LID detects both sample quality issues as well as mode dropping/collapse issues. In practice, instead of computing this measure in the pixel space, one can compute it in the feature space of some feature extractor, or in some cases directly in the learned feature space of the generator. Finally, given some labeling of the points, one can keep track of the CrossLID statistic for each mode and use this during training to oversample modes for which the gap between the expected CrossLID and computed one is large.