Towards GAN Benchmarks Which Require Generalization

1 INTRODUCTION . In machine learning , it is often difficult to directly measure progress towards our goals ( e.g . “ classify images correctly in the real world ” , “ generate valid translations of text ” ) . To enable progress despite this difficulty , it is useful to define a standardized benchmark task which is easy to evaluate and serves as a proxy for some final task . This enables much stronger claims of improvement ( albeit towards a somewhat artificial task ) by reducing the risk of inadequate baselines or evaluation mistakes , with the hope that progress on the benchmark will yield discoveries and methods which are useful towards the final task . This approach requires that a benchmark task satisfy two properties : 1 . It should define a straightforward and objective evaluation procedure , such that strong claims of improvement can be made . Any off-limits methods of obtaining a high score ( e.g . abusing a test set ) should be clearly defined . 2 . Improved performance on the benchmark should require insights which are likely to be helpful towards the final task . The benchmark should , by construction , reflect at least some of the kinds of difficulty inherent in the final task . Together these imply that a benchmark should at least be nontrivial : we should not know a priori how to obtain an arbitrarily high score , except perhaps by clearly-off-limits methods . Crucial to a useful benchmark is an evaluation metric which measures what we care about for the final task and which satisfies the requirements outlined above . This paper deals with unconditional generation of natural images , which has been the goal of much recent work in generative modeling ( e.g . Radford et al. , 2015 ; Karras et al. , 2017 ) . Our ideas are general , but we hope to improve evaluation practice in models like Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) . Generative modeling has many possible final tasks ( Theis et al. , 2015 ) . Of these , unconditional image generation is perhaps not very useful directly , but the insights and methods discovered in its pursuit have proven useful to other tasks like domain adaptation ( Shrivastava et al. , 2017 ) , disentangled representation learning ( Chen et al. , 2016 ) , and imitation learning ( Ho & Ermon , 2016 ) . Some notion of generalization is crucial to why unconditional generation is difficult ( and hence interesting ) . Otherwise , simply memorizing the training data exactly would yield perfect “ generations ” and the task would be meaningless . One might argue that some work—particularly recent GAN research—merely aims to improve convergence properties of the learning algorithm , and so generalization isn ’ t a big concern . However , generalization is an important part of why GANs themselves are interesting . A GAN with better convergence properties but no ability to generalize is arguably not a very interesting GAN ; we believe our definition of the task , and our benchmarks , should reflect this . Because our task is to generate samples , and often our models only permit sampling , recent work ( e.g . Huang et al. , 2016 ; Gulrajani et al. , 2017 ; Tolstikhin et al. , 2017 ) has adopted benchmarks based on evaluation metrics ( Salimans et al. , 2016 ; Heusel et al. , 2017 ) which measure the perceptual quality and diversity of samples . However these particular metrics are , by construction , trivially “ won ” by a model which memorizes the training set . In other words , they mostly ignore any notion of generalization , which is central to why the task is difficult to begin with . This idea is illustrated in Figure 1 . In this sense they give rise to “ trivial ” benchmark tasks which can lead to less convincing claims of improvement , and ultimately less progress towards useful methods . While “ nontrivial ” benchmark tasks based on downstream applications can be used , the goals of these tasks are at best indirectly related to , and at worst opposite from , sample generation ( e.g . for semi-supervised learning ( Dai et al. , 2017 ) ) . This paper considers evaluation metrics for generative models which give rise to nontrivial benchmarks , and which are aligned with the final task of generating novel , perceptually realistic and diverse data . We stress the difference between evaluating models and defining benchmarks . The former assumes that models have been chosen ahead of time , independently of the metric . The latter assumes that models will be developed with the metric in mind , and so seeks to avoid falsely high scores resulting from exploiting undesirable solutions to the benchmark ( e.g . memorizing the training set ) . Our goal is to define benchmarks , and many of our decisions follow from this . Our contributions are as follows : • We establish a framework for sample-based evaluation that permits a meaningful notion of generalization . We clarify a necessary condition for the ability to measure generalization : That the evaluation requires a large sample from the model . • We investigate using neural network divergences ( Arora et al. , 2017 ) as evaluation metrics which have attractive properties for this application . We survey past work exploring the use of neural network divergences for evaluation . • We study an example neural network divergence called “ CNN divergence ” ( DCNN ) experimentally . We demonstrate that it can detect and penalize memorization and that it measures diversity relatively more than other evaluation functions . 2 BACKGROUND . Throughout the paper we cast evaluation metrics as statistical divergences specifying a notion of dissimilarity between distributions . The goal of generative modeling , then , is minimizing some divergence , and the choice of divergence reflects the properties of our final task . Let X denote a set of possible observations ( a common choice for image modeling is R32×32×3 ) , and P ( X ) the set of probability distributions on X . The goal of generative modeling is : given a data distribution p ∈ P ( X ) ( which we can only access through a finite sample ) and a set of distributions Q ⊆ P ( X ) ( usually a parametric model family ) , find q∗ ∈ Q which is closest to p according to some definition of closeness . A notion of closeness between distributions is characterized by a statistical divergence , which is a function D : P ( X ) × P ( X ) → R≥0 ∪ { +∞ } satisfying D ( p , q ) = 0 iff p = q.1 Usually we can not exactly evaluate D ( p , q ) , and so for training and evaluation purposes we must compute an estimate using finite samples from p and q . We denote such an estimate of D as D̂ . With some abuse of notation , we denote by p̂ either a finite sample from p or the empirical distribution corresponding to that sample , and likewise for q̂ and q . 2.1 EVALUATION METRICS FOR MODELS OF NATURAL IMAGES . Various evaluation criteria have been proposed for evaluating a generative model of natural images solely based on a finite sample from the model . Perhaps the most commonly applied metrics are the “ Inception Score ” ( IS ) ( Salimans et al. , 2016 ) and “ Fréchet Inception Distance ” ( FID ) ( Heusel et al. , 2017 ) . These metrics , described in detail in Appendix B and Appendix C respectively , compute statistics on features produced by an Inception network ( Szegedy et al. , 2016 ) trained on ImageNet ( Deng et al. , 2009 ) . They are motivated by the intuition that statistics computed on samples from the model q should match those of real data . Apart from the IS and FID , test set log-likelihood is a popular metric for evaluation , but is not directly computable for models from which we can only draw samples . While Wu et al . ( 2016b ) proposed a method of estimating log-likelihoods for “ decoder-based ” generative models , it requires additional assumptions to be made and was shown to be a poor estimate in some cases ( Grover et al. , 2017 ) . Log-likelihood has also been shown to not correlate well with downstream needs such as sample quality ( Theis et al. , 2015 ) . 3 GENERALIZATION IN SAMPLE-BASED EVALUATION . In this section we establish a framework for sample-based evaluation that permits a meaningful notion of generalization . We state a property of divergences which are estimable from a finite sample ( that they ’ re somewhat insensitive to diversity ) , propose a baseline for non-trivial performance ( the model should outperform training set memorization ) , explain that beating the baseline might only be achievable under certain kinds of divergences ( the divergence should consider a large model sample ) , and finally explain that minimizing such a divergence requires paying attention to generalization . 3.1 EASILY-ESTIMATED DIVERGENCES CAN ’ T MEASURE DIVERSITY . Because we can only sample from our models , and we have finite compute , we are restricted to divergences which can be estimated using only a finite sample from the model . We begin with an obvious but important note : any divergence estimated from a sample can be “ fooled ” by a model which memorizes a finite sample of not much greater size . Specifically : Remark 1 . If D ( p , q ) is a divergence which can always be estimated with error less than using only m points sampled from q2 , qgood is a model distribution , and qbad is an empirical version of qgood with m2 points , then |D ( p , qgood ) −D ( p , qbad ) | < 2 . In this case , we say that D is ( to some extent ) insensitive to “ diversity ” , which we define as any differences between a distribution and its empirical counterpart . The reasoning is that a sample of m points ( with replacement ) from qbad is quite likely to also be a valid sample from qgood ( because it ’ s unlikely that a point in qbad gets sampled twice ) , making it impossible to reliably tell whether the sample came from qgood or qbad . So , either the estimates must sometimes have substantial error or the difference between the true divergences must also be small . This comes almost directly from Gretton et al . ( 2012 ) ’ s Example 1 and is similar in spirit to Arora et al . ( 2017 ) ’ s Corollary 3.2 . This version requires that D can be estimated with bounded error for a finite sample size , which is slightly too restrictive . Nonetheless it clarifies the intuition : the sample size required to estimate a divergence upper-bounds the divergence ’ s sensitivity to diversity . 1In this work we permit D ( p , q ) = 0 for some q 6= p , since these cases aren ’ t relevant if we assume our model q is never good enough to exactly attain D ( p , q ) = 0 . 2By this , we mean that there exists a function D̂ satisfying |D̂ ( p , q̂m ) −D ( p , q ) | < for all p , q . 3.2 MODELS SHOULD OUTPERFORM TRAINING SET MEMORIZATION . From the previous section , we know that even trivially memorizing the training set ( that is , choosing q = p̂train ) is likely to minimize D ( p , q ) ( to an extent depending on D and the size of p̂train ) because p̂train is a finite sample from p. Such memorization , however , is presumably not a useful solution in terms of our final task . Therefore we propose that in order for a model to be considered useful , it should at least outperform that trivial baseline . Specifically : Definition 1 . A model q outperforms training set memorization under a divergence D if D ( p , q ) < D ( p , p̂train ) . Cornish et al . ( 2018 ) propose essentially the same criterion . The next section explains how to choose D such that this goal is achievable .

Summary: The paper looks at the problem of benchmarking models that unconditionally generate images. In particular they focus on GAN models and discuss the Inception Score (IS) and Fréchet Inception Distance (FID) metrics. The authors argue that a good benchmark should not have a trivial solution (e.g. memorising the dataset) and find that a necessary condition for such a metric is a large number of samples. They also find that for IS and FID , a GAN is outperformed by a model that memorises the dataset, while a method based on neural network divergences (NND) does not show the same behaviour. NND works by training a discriminative model to discriminate between samples of the generative model and samples from a held out test set. The poorer the discriminative model performs, the better the generative model is.