A GOODNESS OF FIT MEASURE FOR GENERATIVE NETWORKS

1 INTRODUCTION AND RELATED WORK . Generative adversarial networks ( Goodfellow et al. , 2014 ) are a specific type of generative model that has shown impressive performance lately . The main idea is that there are two networks that compete against each other : a generator network that generates images and a discriminator network that tries to distinguish between real and fake images . These models are useful because they can generate very realistic images that are not in the training set . Throughout the rest of the paper , we will use GANs as a specific class of models to study , however , the goodness of fit measure discussed in Section 3.2 and its applications can be extended to other generative networks such as Variational Autoencoders . Some GANs that appear to be successful in practice can not actually reproduce the training set , as we will see . Other generative models , such as Generative Latent Optimization ( GLO ) ( Bojanowski et al. , 2017 ) and Implicit Maximum Likelihood Estimation ( IMLE ) ( Li and Malik , 2018 ) attempt to memorize the training data as part of the learning algorithm . These methods are not as successful as GANs in producing realistic images . We believe that the reason for this difference in performance is due to a lack of overparameterization in GANs , GLO , and IMLE . Our solution starts with measuring how well a generative model can generate the training data . We explain in Section 3.2 that our goodness of fit measure F ( G ) is zero if we are able to perfectly generate the training data . If we can not generate the training data , then F ( G ) represents how far away we are from generating our training set in an average of total least square sense . We use this goodness of fit measure to evaluate different models and training settings as well as study the evolution of the approximation error through training . Ideally , we would like to overparameterize GANs in order to increase their capacity and reduce F ( G ) . Recently , it was shown that overparameterization in classifiers and autoencoders leads to better performance ( Radhakrishnan et al. , 2019 ; Belkin et al. , 2018 ) . Another reason to overparameterize is that we observed , while calculating F , that our models actually use the the full potential of the latent distribution on z to generate different images . In other words , suppose that we train a GAN with z ∼ N ( 0 , I ) , then we observe that the optimal z ’ s corresponding to the closest generated images from the training set are also distributed as N ( 0 , I ) . That is , once trained , the latent distribution fixed a priori becomes the optimal one minimizing the approximation error . Despite the above findings , increasing the complexity of the generator becomes very difficult due to the training algorithms for GANs requiring careful hyper-parameter settings for convergence ( Rege and Monteleoni , 2019 ) . As such , we explore two alternatives that do not impact the training stability . First , increasing the dimension of the latent space which is currently set to 100 across models and dataset . We demonstrate that this solution indeed allows to reduce the approximation error . Second , we consider a mixture of GAN setting . That is , we train K different GANs on subsets of the data of size approximately NK for total data size N . Note that training a mixture of GANs has been done in practice Hoang et al . ( 2017 ) , we thus will quantify the approximation error reduction that can be obtained with this solution . Hence , we see that if our original GAN has P parameters , we now have KP parameters total . Also , each one is trained on N/K data . Hence , we see that our ” effective ” parameters for this mixture of GANs is P ′ N ′ = K 2 P N . Hence we can get an effective overparameterization of 100 fold if we divide our data into K = 10 subsets . We first demonstrate how a single GAN trained on a smaller dataset has a smaller approximation error . In particular we also find that how the dataset is subsampled , random or from a clustering based partitioning , matters for performances . We then build on those finding to train the mixture of GANs on a K-means based partitioned dataset and demonstrate important reduction in the approximation error . We summarize our contributions that apply to arbitrary generative models s.a. GANs , VAE as follows : • We provide a novel goodness of fit measure for generative networks and how it defines necessary conditions for generative networks to be optimal ( Sec . 3.1 ) . We also relate the metric to mode collapse and provide implementation details on computing it efficiently ( Sec . 3.2 ) . Finally we demonstrate how our metric compares to standard GAN metric s.a. the Frechet score ( Sec . 3.3 ) . • We demonstrate how our goodness of fit metric allows to gain novel insights into GANs . We show that DCGAN and WGAN do not memorize and have very different behavior w.r.t . overfitting Sec . 4.1 . In particular DCGAN is able to match with WGAN performances if early stopping is performed . We then show the impact architecture and residual connection ( Sec . 4.2 ) . Finally , we study the latent space distribution and in particular demonstrate how the optimal latent space distribution which minimizes the approximation error of a trained GAN matches the distribution used for training , highlighting how current approximation errors are due to underparametrized GANs ( Sec . 4.3 ) . • We provide two solutions to reduce the approximation error without altering training stability Sec . 5 . First , we propose to increasing the latent space dimension in Sec . 5.1 . Then we study how dataset subsampling also helps reducing the approximation error Sec . 5.2 which motives the use of a mixture of GANs Sec . 5.3 . 2 BACKGROUND . In this section we briefly overview Generative Adversarial Networks ( GANs ) and generative latent optimization ( GLO ) ( Bojanowski et al. , 2017 ) , which is another generative model . Finally describe how one can optimize the generator network latent space to obtain a desired generated sample . We remind that all our development applies to arbitrary black/white box generative networks even though we only focus on GANs in this paper . Generative Adversarial Networks . Generative adversarial networks ( GANs ) are generative neural networks that use an adversarial loss ; the adversarial loss is typically another neural network . In other words , a GAN consists of two neural networks that compete against each other . The generator network G : R ` → Rp generates p-dimensional images from an ` -dimensional latent space . The discriminator D : Rp → ( 0 , 1 ) is a classifier which is trained to distinguish between the training set and generated images . The training loss for a batch size of NB for the discriminator is given by LD = − 1 NB NB∑ i=1 log ( D ( xi ) ) − 1 NB NB∑ j=1 log ( 1−D ( G ( zj ) ) ) , where xi and G ( zi ) is a real image and a generated image for each i ∈ { 1 , 2 , . . . , NB } , respectively . The generator loss is given by LD = 1 NB NB∑ j=1 log ( 1−D ( G ( zj ) ) ) . Notice that the loss for the generator does not explicitly use the training data . Instead , the training data is used indirectly through the training of the discriminator . In this paper , we discuss two popular GANs : DCGAN ( Radford et al. , 2015 ) which uses the loss above and WGAN ( Arjovsky et al. , 2017 ) which uses a slightly different learning algorithm . Generative latent optimization . In contrast to GANs , GLO is a generative network that does not use an adversarial loss . Instead , GLO attempts to memorize the training data by using this loss : LG = 1 NN NB∑ i=1 min z∈R ` L ( G ( z ) , xi ) where L is a loss function . In the original paper , the authors use different loss functions to demonstrate how the model differs . Latent space optimization . The generator is a mapping from a latent space R ` into an image space Rp . In the GLO paper one aims at finding a specific z ∈ R ` such that the generated sample G ( z ) is close to a target output . In particular , one picks a target as being a randomly generated GAN image from some target vector z∗ . That is , the target is guaranteed to lie in the span of G. Following this , one aims at finding the z vector that led to the target G ( z∗ ) by ẑ = argmin z∈R ` ‖G ( z ) −G ( z∗ ) ‖22 . ( 1 ) Since the above optimization problem in non-convex , there is not theoretic guarantee of finding a global minima , in general . However , empirically it was shown that the above optimization problem is solved 100 % of the time in practice ( Lipton and Tripathi , 2017 ) . We now propose to leverage the above to develop our goodness of fit measure . 3 GOODNESS OF FIT METRIC . In this section we first motivate and define our metric ( Sec . 3.1 ) and provide its approximation ( Sec . 3.2 ) . We demonstrate that our measure being minimized is a necessary and sufficient condition to detect mode collapse . Finally , we study the different with current GAN measures that are the Inception Score and the Frechet Inception Score ( Sec . 3.3 ) . 3.1 METRIC , OPTIMUM GENERATIVE NETWORK , AND MODE COLLAPSE . The generator G is a continuous mapping for any type of layer used as current layers in deep learning are all continuous . As such , the following defines the image of the generator : Imag ( G ) = { G ( z ) : z ∈ R ` } , ( 2 ) with ` the dimension of the latent space . Then the approximation of the true data manifold denoted as X with G can be measure by ” how far ” is the span of G. Since the two quantities to compare are sets one solution is the following standard Total Least Square metric defines as d ( Imag ( G ) , X ) = ∫ min z ‖G ( z ) − x‖dx ( 3 ) which in practice with a finite dataset becomes our proposed measure : F ( G , X ) = 1 N N∑ n=1 min z ‖G ( z ) − xn‖2 = 1 N N∑ n=1 m ( xn ; G ) . ( 4 ) As a result , F is an empirical average least square distance between reference points , the observed input , and Imag ( G ) . Turning the above argument into a probabilistic setting , we obtain the following motivation . A generative model or network is learned to approximate some target distribution . This target distribution is itself observed via some samples , the given observations . It is common to use the likelihood as a measure of fitness for such models , and it is defined as L ( X ) = ∏N n=1 p ( xn ) , with p ( xn ) some distribution density . A necessary condition to allow maximization of the likelihood is that the samples lie in the support of the distribution , that is p ( xn ) > 0 ⇐⇒ xn ∈ Support ( p ) , as Support ( p ) = { x ∈ X : p ( x ) > 0 } . In the case of a generative network , the distribution p ( x ) is not easily available . However its support is directly accessible as we have Proposition 1 . Support ( p ) = Imag ( G ) . As a result , from a probabilistic fitting point of view , ensuring that all the samples lie in the span of the generator is a necessary condition that must be fulfilled , which otherwise would prevent maximization of the likelihood , leading to the following result . Theorem 1 . The optimal generative model in term of distribution approximation must have F ( G , X ) = 0 . The above setting also allows the following direct and intuitive result that any sample x in X such that m ( x ; G ) = 0 is a sample that can be generated , potentially with very small probability , by a generative network . We conveniently employed the L2 in our metric and we will use this throughout the rest of the paper . However , notice that if X ⊂ Imag ( G ) , the choice of the distance function is not important because d ( x , y ) = 0 implies that x = y for any distance metric d. Relation to mode collapse . We now relate the value of m ( xn ; G ) for samples from the training set to mode collapse . We also refer to the case m ( xn ; G ) = 0 as a memorized sample and in general we refer as memorization F ( X , G ) = 0 . The lack of memorization in current generative networks is evident when generators experience mode collapse . Mode collapse translates into the absence of approximation of the generator to some parts of the target distribution . Suppose that the data comes from a distribution PX . Then , mode collapse happens if PX ( x ) > 0 but minz ||G ( z ) − x|| > 0 . In practice , we do not have PX , but we do have the empirical distribution P̂X , so that mode collapse will occur if we do not memorize the training data . A generative model must memorize the training data in order to avoid mode collapse . We thus obtain the following result . Proposition 2 . A necessary condition to avoid mode collapse for a generative network G is to memorize , that is , F ( X , G ) = 0 . We now demonstrate how one can compute our metric and in particular m ( xn ; G ) efficiently .

Summary: This work proposed a new goodness of fit measure for generative network evaluations, which is based on how well the network can generate the training data. The measure is zero if the network could perfectly recover the training data, and would represent how far it is from generating the training set in the average manner of the total least square sense, where the one-to-one mapping between the generated data and the training sample is constructed through latent space optimization. Using the proposed measure, the authors showed an interesting trend present in the DCGAN training and the impact of the residual connection. The authors might want to add some discussion in Section 4.2 regarding why the residual connection is detrimental for covering the support.  Increasing the model complexity through larger latent space dimension and learning mixtures is proposed as solutions to improve the measure as well.