Improving Sample-based Evaluation for Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have shown outstanding abilities on many computer vision tasks including generating domain-specific images ( Goodfellow et al. , 2014a ) , style transfer ( Zhu et al. , 2017 ) , super resolution ( Ledig et al. , 2017 ) , etc . The basic idea of GANs is to hold a two-player game between generator and discriminator , where the discriminator aims to distinguish between real and fake samples while the generator tries to generate samples as real as possible to fool the discriminator . Researchers ( Radford et al. , 2016 ; Arjovsky et al. , 2017 ; Berthelot et al. , 2017 ; Mao et al. , 2017 ) have been continuously exploring better GAN architectures . However , developing a widely-accepted GAN evaluation framework remains to be a challenging topic ( Theis et al. , 2016 ) . Due to a lack of GAN benchmark results , newly proposed GAN variants are validated on different evaluation frameworks and therefore incomparable . Because human judgements are inherently limited by manpower resource , good quantitative evaluation frameworks are of very high importance to guide future research on designing , selecting , and interpreting GAN models . There have been varieties of efforts on designing sample-based evaluation for GANs on its ability of generating domain-specific images . The goal is to measure the distance between the generated samples and the real in the dataset . Most existing methods utilized the ImageNet ( Russakovsky et al. , 2015 ) inception model to map images onto the feature space . The most widely used criteria is probably the Inception Score ( Salimans et al. , 2016 ) , which measures the distance via Kullback-Leiber Divergence ( KLD ) . However , it is probability based and is unable to report overfitting . Recently , Frechet Inception Distance ( FID ) was proposed ( Heusel et al. , 2017 ) on improving Inception Score . It directly measures Frechet Distance on the feature space with the Gaussian assumption . It has been proved that FID is far better than Inception Score ( Huang et al. , 2018 ; Im et al. , 2018 ; Lucic et al. , 2017 ) . However , we argue that assuming normality on the whole feature distribution may lose class information on labeled datasets . In this work , we propose an improved quantitative sample-based evaluating criteria . We improve conventional evaluation methods on two levels : the feature representation and the evaluation metric . Unlike most existing methods including the Inception Score ( Salimans et al. , 2016 ) and FID ( Heusel et al. , 2017 ) , our framework uses a specialized encoder trained on the dataset to get domain-specific representation . We argue that applying the ImageNet model to either labeled or unlabeled datasets is ineffective . Moreover , we propose Class-Aware Frechet Distance ( CAFD ) in our framework to measure the distribution distance of each class ( mode ) respectively on the feature space to include class information . Instead of the single Gaussian assumption , we employ a Gaussian mixture model ( GMM ) to better fit the feature distribution . We also include KL divergence ( KLD ) between mode distribution of real data and generated samples into the framework to help detect mode dropping . Experiments and analysis on both the feature level and the image level were conducted to demonstrate the improved effectiveness of our proposed framework . To our best knowledge , we are the first ( Borji , 2018 ) to provide counter examples where FID is inconsistent with human judgements ( See Figs . 1 and 2 ) . It is shown in the experiments that our framework is able to overcome the shortness of existing methods . 2 RELATED WORK . Evaluation Methods . Several GAN evaluation methods have been proposed by researchers . While model-based methods including Parzen window estimation and the annealed importance sampling ( AIS ) ( Wu et al. , 2017 ) require either density estimation or observation on the inner structure of the decoder , model-agnostic methods ( Heusel et al. , 2017 ; Salimans et al. , 2016 ; Im et al. , 2018 ; Che et al. , 2017 ; Dziugaite et al. , 2015 ; Lopez-Paz & Oquab , 2017 ; Li et al. , 2015 ) are more popular in the GAN community . These methods are sample based . Most of them map images onto the feature space via an ImageNet pretrained model and measure the similarity of the distribution between the dataset and the generated data . Maximum mean discrepancy ( MMD ) was proposed by ( Dziugaite et al. , 2015 ; Li et al. , 2015 ) and it has been further used in classifier two-sample tests ( Lopez-Paz & Oquab , 2017 ) , where statistical hypothesis testing is used to assess whether two sample sets are from the same distribution . Inception Score ( Salimans et al. , 2016 ) , along with its improved version Mode Score ( Che et al. , 2017 ) , was the most widely used metric in the last two years . FID ( Heusel et al. , 2017 ) was proposed on improving the Inception Score . Recently , several interesting methods were also proposed including classification accuracy ( Shmelkov et al. , 2018 ) , precisionrecall measuring ( Sajjadi et al. , 2018 ) and skill rating ( Olsson et al. , 2018 ) . These metrics give complementary perspectives towards sample-based methods . Studies on Existing Frameworks . It is common ( Barratt & Sharma , 2018 ) in the literature to see algorithms which use existing metrics to optimize early stopping , hyperparameter tuning , and even model architecture . Thus , comparison and analysis on previous evaluation methods have been attracting more and more attention recently ( Theis et al. , 2016 ; Huang et al. , 2018 ; Im et al. , 2018 ; Lucic et al. , 2017 ) . While Inception Score was the most popular metric in the last two years , it was believed to be misleading in recent literature ( Heusel et al. , 2017 ; Huang et al. , 2018 ; Lucic et al. , 2017 ; Borji , 2018 ; Barratt & Sharma , 2018 ) . Applying the ImageNet model to encode features in Inception Score is ineffective ( Theis et al. , 2016 ; Barratt & Sharma , 2018 ; Rosca et al. , 2017 ) . The recently proposed FID has been proved to be far better than Inception Score ( Heusel et al. , 2017 ; Huang et al. , 2018 ; Im et al. , 2018 ) . And its robustness was experimentally demonstrated recently in a technical report ( Lucic et al. , 2017 ) . However , we argue that FID still has problems and provide counter examples where FID gives inconsistent results with human judgements . Moreover , we propose an improved version of evaluation which overcomes its shortness . 3 PROBLEMS ON FID . The evaluation problem can be formulated as modeling the distance between two distributions Pr and Pg , where Pr denotes the distribution of real samples in the dataset and Pg denotes the distributions of new samples generated by GAN models . The main difficulties for GANs on generating domain-specific images can be summarized into three types below . • Lack of generating ability . Either the generator can not generate useful samples or the GAN training can not diverge . • Mode collapse . Different modes collapse to a new mixed mode in the generated samples . ( e.g . An animal resembling both a horse and a deer . ) • Mode dropping . Only part of the modes in the dataset are generated while some modes are implicitly ignored . ( e.g . The handwritten 5 can hardly be generated by GAN trained on MNIST . ) Therefore , a good evaluation framework should be consistent to human judgements , penalize on mode collapse and mode dropping . Most of the conventional methods utilized an ImageNet pretrained inception model to map images onto the feature space . Inception Score , which was originally formulated as Eq . ( 1 ) , ignored information in the dataset completely . Thus , its original formulation was considered to be relatively misleading . IS = exp ( Ex [ KL ( p ( y|x ) ||p ( y ) ) ] ) ( 1 ) The Mode Score was proposed ( Che et al. , 2017 ) to overcome this shortness . Its formulation is shown in Eq . ( 2 ) . By including the prior distribution of the ground truth labels , Mode Score improved Inception Score ( Che et al. , 2017 ) on reporting mode dropping . MS = exp ( Ex [ KL ( p ( y|x ) ||p ( y∗ ) ) ] −KL ( p ( y∗ ) ||p ( y ) ) ) ( 2 ) FID ( Heusel et al. , 2017 ) , which was formulated in Eq . ( 3 ) , was proposed on improving Inception Score ( Salimans et al. , 2016 ) . FID ( Pr , Pg ) = ||µr − µg||+ Tr ( Cr + Cg − 2 ( CrCg ) 1 2 ) ( 3 ) ( µg , Cg ) , ( µr , Cr ) are the first-order and second-order statistics for generated samples and real data respectively . Unlike the previous two metrics which are probability-based , FID directly measures Frechet distance on the feature space . It uses an ImageNet model for encoding features and assumes normality on the whole feature distribution . FID was believed to be better than Inception Score ( Huang et al. , 2018 ; Im et al. , 2018 ; Lucic et al. , 2017 ) . However , we argue that FID still has two major problems ( See Section 3.1 and 3.2 ) . 3.1 INEFFECTIVE REPRESENTATION . As both Inception Score ( Salimans et al. , 2016 ) and Mode Score ( Che et al. , 2017 ) is probabilitybased , applying the ImageNet pretrained model on non-ImageNet dataset is relatively meaningless . This misuse of representation on Inception Score was mentioned previously ( Rosca et al. , 2017 ) . However , we argue that applying the ImageNet model to map the generated images to the feature space in FID can also be misleading . While both of the ( Barratt & Sharma , 2018 ; Rosca et al. , 2017 ) mentioned that applying the ImageNet pretrained model to the probability-based metric Inception Score ( Salimans et al. , 2016 ) is inadequate , the trend for applying it to feature-based metric such as FID ( Heusel et al. , 2017 ) is widely followed . ( Barratt & Sharma , 2018 ) pointed out that because classes are unmatched , the p ( y|x ) and p ( y∗ ) in the formulation of Inception Score are meaningless . However , we argue that applying the ImageNet model for mapping the generated images to the feature space in FID can also be misleading for the two reasons below . First , On labeled datasets with multiple classes , the class labels unmatch those in ImageNet . For example , the class ‘ Bird ’ in CIFAR-10 ( Krizhevsky & Hinton , 2009 ) is divided into several sophisticated category labels in ImageNet . This will make the CNN representations trained on ImageNet is either meaningless or over-complicated . Specifically , some features distinguishing the “ acoustic guitar ” from “ electric guitar ” are hardly useful on CIFAR-10 while fine-grained features distinguishing “ African hunting dog ” from “ Cape hunting dog ” ( which all belong to the category “ dog ” in CIFAR-10 ) are not needed as well . On unlabeled datasets with images from a single class such as CelebA ( Liu et al. , 2015 ) , applying the ImageNet inception model is also inappropriate . The categories of ImageNet labels are so sophisticated that the trained model needs to encode diverse features on various objects . However , this will get encoded features limited to a relatively low-dimensional subspace lack of fine-grained information . For example , the ImageNet models can hardly distinguish different faces . In Section 5.1 , we designed experiments on both the feature level and the image level to demonstrate the effects of using different representations .

Summary: The authors study the task of sample-based quantitative evaluation applied to GANs. The authors suggest multiple modifications to existing evaluation pipelines: (1) Instead of embedding the samples in the InceptionNet feature space, train a domain-specific encoder. If labeled data is available, add a cross-entropy loss to the encoder training objective so that the class can be predicted. (2) Instead of fitting a single Gaussian in the feature space, fit a GMM instead. This should allow for a more fine-grained “class-aware” distance between the (empirical) distributions.