Evaluating GANs via Duality

1 INTRODUCTION . In the past few years , generative models have become extremely popular in the machine learning community . This is largely due to the recent advances in the field of deep learning , which allowed deep neural generators to produce remarkable results for various tasks , including for example image generation ( Radford et al. , 2015 ) . Two notable approaches in this area are variational auto-encoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , and generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) . In this paper , we focus on GANs , which are especially attractive as they circumvent the notoriously hard optimization of the data likelihood and instead use an adversarial game approach for training a generator . Let us denote the data distribution by pdata ( x ) and the model distribution by pu ( x ) . A probabilistic discriminator is denoted by Dv : x 7→ [ 0 ; 1 ] and a generator by Gu : z 7→ x . The GAN objective is : min u max v M ( u , v ) = 1 2 Ex∼pdata logDv ( x ) + 1 2 Ez∼pz log ( 1−Dv ( Gu ( z ) ) ) . ( 1 ) Each of the two players ( generator/discriminator ) tries to optimize their own objective , which is exactly balanced by the loss of the other player , thus yielding a two-player zero-sum minimax game . Standard GAN approaches aim at finding a pure Nash Equilibrium by using traditional gradientbased techniques to minimize each players cost in an alternating fashion . However , the minimax objective of GANs makes the optimization process challenging . One of the central open issues is the non-convergence problem , which in practice leads to oscillations between different kinds of generated samples ( Metz et al. , 2016 ) . Many different techniques have been proposed to address the shortcomings of the original technique introduced by Goodfellow et al . ( 2014 ) . Given the plethora of GAN-like approaches as well as different training methods , the community is now facing the problem of determining which approaches produce ” better ” generative models . This is in itself a challenging question as there is no clear evaluation criterion . In a recent large-scale empirical study , Lucic et al . ( 2018 ) showed that using various evaluation measures , many state-ofthe-art models are able to reach similar scores with enough tuning . In a recent survey , Borji ( 2018 ) discussed the merits of various evaluation metrics , pointing out that there is no clear consensus regarding which metric is the most appropriate to evaluate the quality of a generative model . While many metrics achieve reasonable discriminability ( i.e . ability to distinguish generated samples from real ones ) , they also tend to have a high computational complexity . Many existing metrics are also specific to datasets of natural images , but we are also starting to notice the success of GANs in other fields such as in cosmology ( Rodriguez et al. , 2018 ) or in the medical domain ( Schlegl et al. , 2017 ) . Another problem related to the evaluation of samples produced by a GAN is the lack of evaluation procedures to detect convergence . While various approaches have analyzed the convergence properties of alternating gradient-based techniques , most of them require the objective function to be convex-concave ( Goodfellow et al. , 2014 ; Nowozin et al. , 2016 ) with the exception of Grnarova et al . ( 2018 ) who prove convergence for semi-concave zero-sum games . In general , alternating gradient descent can fail to converge ( Salimans et al. , 2016 ) . While this non-convergence behavior can in practice be visually recognized for some low-dimensional examples ( such as a 2D mixture of Gaussian ) , this is in general more difficult in high-dimensional spaces due to the lack of a convergence metric . This has been pointed out before ( see e.g . Mescheder et al . ( 2018 ) and ( Fedus et al. , 2018 ) who examine convergence on 2D problems and problems for which the true data distribution is known . One of the common challenges that practitioners are facing is when to stop training . In particular , it is well known that the curves of the discriminator and generator losses oscillate ( see Figures 12 and 13 ) and are non-informative as to whether the model is improving or not ( Arjovsky et al. , 2017 ) . This is especially troublesome when a GAN is trained on non-image data in which case one might not be able to use visual inspection or FID/Inception score as a proxy . In this paper , our main contribution is to leverage existing ideas from game theory to propose a novel and computationally efficient convergence metric for GANs . The principle we follow is based on the theory of duality that was developed after Von Neumann ( 1928 ) derived the minimax theorem for zero-sum games . From duality , it follows that the minimax and maximin 1 are dual to each other and strong duality holds for feasible problems . This duality also implicitly gives us a way to evaluate the deviation between minimax and maximin . As an example , let ’ s consider a two-player game between the discriminator D and the generator G. The duality gap measures the difference between the values of the game under two scenarios : when D first commits to a strategy , and then G gets to respond ; and when G commits first , and then D gets to respond . In this paper , we advocate the use of the duality gap as a measure of convergence for GANs and demonstrate its effectiveness for various settings commonly considered in the literature . As a second contribution , we also demonstrate how to use the minimax loss value in order to detect mode collapse and measure sample quality . Unlike FID or the Inception score that require labelled data or a domain dependent classifier , our metric is domain independent and does not require labels . Our experiments demonstrate that the duality gap and minimax loss are valuable tools to measure convergence and correlate well with existing evaluation metrics . 2 RELATED WORK . Despite the impressive empirical performance achieved by the latest GAN models ( Karras et al. , 2017 ) , they are still subject to many unanswered questions . Among them is the issue of a fair evaluation procedure . Since the log-likelihood of the data is a common objective function to train a generative model , it would appear to be a sensible metric for GANs . However , its computation is often intractable and Theis et al . ( 2015 ) also demonstrate that it has severe limitations as it might yield low visual quality samples despite of a high likelihood . Perhaps the most popular evaluation metric for GANs is the inception score introduced by Salimans et al . ( 2016 ) that measures both diversity of the generated samples and discriminability . While diversity is measured as the entropy of the output distribution , the discriminability aspect requires a pretrained neural network to assign high scores to images close to training images . Various modifications of the inception score have been suggested . Gurumurthy et al . ( 2017 ) adds a term in the cost function to take into account diversity within samples in a particular class . Heusel et al . ( 2017 ) uses features from a hidden layer of the Inception Net , which are modelled as two multivariate Gaussians for the generated and true data . They then suggest using the Frechet Inception Distance ( FID ) between these two Gaussians to assess the quality of the samples . The authors demonstrate improvements over the inception score , especially more robustness to noise , as well as being able to detect intra-class mode dropping . Furthermore , they also demonstrate high degrees of consistency with human judgments . Although this measure appears to have nice properties from an empirical point of view , the Gaussian assumption might not hold in practice . Furthermore , FID requires labelled data in order to train a classifier . Without labels , transfer learning is possible to datasets under limited conditions ( i.e . the source and target distributions should not be too dissimilar ) . 1In game theory , the maximin value of a player is the highest value that the player is ensured to receive without knowing the actions of the other player ( s ) while the minimax value of a player is the smallest value that the other player ( s ) can force the player to receive , without knowing the player ’ s actions . Another popular metric introduced by Gretton et al . ( 2012 ) is the Maximum Mean Discrepancy ( MMD ) which measures the dissimilarity between pdata and pu using independently drawn samples . It is a specific instance of an integral probability metric . A potential hurdle for MMD is that its computational complexity is quadratic in the sample size , although some linear approximation do exist ( Gretton et al. , 2012 ) . Finally , it also depends on the choice of the kernel . There are many other metrics that have been suggested over the past few years . We refer the reader to Borji ( 2018 ) for a detailed survey . Perhaps the two approaches that are the most relevant to our approach are inspired by skill rating systems in games and were both introduced in Olsson et al . ( 2018 ) . The first method named tournament win rate considers a single model playing against past and future versions of itself . This measure is designed to monitor the progress of a single model as it learns during training . The second method named skill rating measures the aptitude of two different fully trained models . In some way , skill rating and tournament win rate can be seen as an approximation of the worst minimax value we advocate in this paper , where instead of doing a full-on optimization in order to find the best adversary for the fixed generator , the search space is limited to discriminators that are snapshots from training , or discriminators trained with different seeds . We would like to conclude our discussion of related work by pointing out to the vast literature on duality used in the optimization community as a convergence criterion for min-max saddle point problem , see e.g . Nemirovski et al . ( 2009 ) ; Komodakis & Pesquet ( 2015 ) . Some recent work by Chen et al . ( 2018 ) also used duality in order to derive a Lagrangian objective to train GANs . Although we also make use of duality , there are some major differences that are worth pointing out . Unlike prior work , our contribution does not relate to optimising GANs , but instead in showing that the duality gap can be empirically used as a proxy to track convergence . We also demonstrate it can be computed at low cost . 3 DUALITY GAP AS A NATURAL PERFORMANCE MEASURE . Standard learning tasks are often described as ( stochastic ) optimization problems ; this applies to common Deep Learning scenarios as well as to classical tasks such as logistic and linear regression . This formulation gives rise to a natural performance measure , namely the test loss2 . In contrast , GANs are formulated as ( stochastic ) zero-sum games . Unfortunately , this fundamentally different formulation does not allow us to use the same performance metric . In this section , we describe a performance measure for GANs , which naturally arises from a game theoretic perspective . We start this section with a brief overview of zero-sum games , including a description of the Duality gap metric and some of its properties . Then we illustrate the usefulness of this metric by analyzing the idealized setting where both G and D have unbounded capacity ( this setting was previously discussed in Goodfellow et al . ( 2014 ) ) . A zero-sum game is defined by two players P1 and P2 who choose a decision from their respective decision sets K1 and K2 . A game objective M : K1 × K2 7→ R , sets the utilities of the players . Concretely , upon choosing a pure strategy ( u , v ) ∈ K1 × K2 the utility of P1 is −M ( u , v ) , while the utility of P2 is M ( u , v ) . The goal of either P1/P2 is to maximize their worst case utilities ; thus , min u∈K1 max v∈K2 M ( u , v ) ( Goal of P1 ) , & max v∈K2 min u∈K1 M ( u , v ) ( Goal of P2 ) ( 2 ) The above formulation raises the question of whether there exists a solution ( u∗ , v∗ ) to which both players may jointly converge . The latter only occurs if there exists ( u∗ , v∗ ) such that neither P1 nor P2 may increase their utility by unilateral deviation . Such a solution is called a pure equilibrium , and is formally defined as follows , max v∈K2 M ( u∗ , v ) = min u∈K1 M ( u , v∗ ) ( Pure Equilibrium ) . While a pure equilibrium does not always exist , the seminal work of Nash et al . ( 1950 ) shows that an extended notion of equilibrium always does . Specifically , there always exists a distribution D1 over elements of K1 , and a distribution D2 over elements of K2 , such that the following holds , max v∈K2 Eu∼D1M ( u , v ) = min u∈K1 Ev∼D2M ( u , v ) ( Mixed Nash Equilibrium ) . 2For classification tasks using the zero-one test error is also very natural . Nevertheless , in regression tasks the test loss is often the only reasonable performance measure . Such a solution is called a Mixed Nash Equilibrium ( MNE ) . This notion of equilibrium gives rise to the following natural performance measure of a given pure/mixed strategy . Definition 1 ( Duality Gap ) . Let D1 and D2 be fixed distributions over elements from K1 and K2 respectively . Then the duality gap of ( D1 , D2 ) is defined as follows , DualGap : = max v∈K2 Eu∼D1M ( u , v ) − min u∈K1 Ev∼D2M ( u , v ) . ( 3 ) Particularly , for a given pure strategy ( u , v ) ∈ K1 ×K2 we define , DualGap : = max v∈K2 M ( u , v ) − min u∈K1 M ( u , v ) . ( 4 ) A well-known , straightforward property of the duality gap is that it is always non-negative . Moreover , by definition , the gap is exactly zero in ( mixed ) Nash Equilibrium solutions . This property is very appealing from a practical point of view , since it means that the duality gap gives us an immediate handle for measuring convergence . Next we illustrate the usefulness of the duality gap metric by analyzing the ideal case where both the generator and discriminator have unbounded capacity . The latter basically means that the generator can represent any distribution , and the discriminator can represent any decision rule . The next proposition shows that in this case , as long as G is not equal to the true distribution then the duality gap is always positive . In particular we show that duality gap is at least as large as the JensenShannon divergence between true and fake distributions ( which is always non-negative ) . We also show that if G outputs the true distribution , then there exists a discriminator such that the duality gap is zero . Proposition 1 . Consider the GAN game objective appearing in Equation ( 1 ) , and assume that the generator and discriminator networks have unbounded capacity . Also , let ( G , D ) be a fixed solution . Then the duality gap of ( G , D ) is larger than the Jensen-Shannon divergence between the true distribution and the fake distribution generated by G. Moreover , if G outputs the true distribution , then there exists a discriminator D such that the Duality gap of ( G , D ) is zero .

Summary: In this paper, the authors proposed the duality gap as the criterion for evaluating the training of GAN. To justify the proposed criterion, the authors designed empirical experiments on both synthetic and real-world datasets to demonstrate the ability of the duality gap for detecting divergence, mode collapse, sample equality, as well as the generalization to other application domains besides image generation. Comparing with the existing criteria, e.g., FID and INC, the duality gap shows better ability and computational efficiency.