CP-GAN: Towards a Better Global Landscape of GANs

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have been one of the most popular methods for generating data . The original GAN minimizes the loss φ ( pg , pdata ) = maxD hD ( pg , pdata ) , where hD is a binary classification loss that depends on the discriminator D. To justify the loss , Goodfellow et al . ( 2014 ) prove two theoretical results : first , for a given pg , the outer function φ ( pg , pdata ) is the Jenson-Shannon ( JS ) distance ( minus a constant ) ; second , the outer function is convex in pg , so a gradient descent method on pg converges to the global minimum . In order to further understand the training behavior of GANs , there has been a surge of interest in min-max optimization or games . One major challenge is the question about which problems to analyze . The first candidates are the two-person matrix game minψ maxθ ψTAθ or general convex-concave problems . But the GAN formulation is not a convex-concave problem . As stated by Daskalakis & Panageas ( 2018 ) , our knowledge of min-max optimization in non convex-concave settings is “ very limited. ” Therefore , existing results often only analyze local stability or local convergence ( Daskalakis et al. , 2017 ; Daskalakis & Panageas , 2018 ; Azizian et al. , 2019 ; Gidel et al. , 2018 ; Mazumdar et al. , 2019 ; Yazıcı et al. , 2018 ; Jin et al. , 2019 ; Sanjabi et al. , 2018 ) . Local analysis can tell us the behavior of the algorithms near the desired point , but how do we know that the algorithm will arrive there and not at some highly sub-optimal points ? Answering this question requires a global landscape analysis or a study of global training dynamics . There are some attempts on a global analysis of GANs . Mescheder et al . ( 2018 ) and Feizi et al . ( 2017 ) analyzed the simplest setting : the true data distribution is a single point and a single Gaussian distribution respectively . In these cases , the training dynamics of some GANs can converge to the globally optimal solution . Thus at least in the simplest single-mode setting , there is evidence that the global landscape of ( some ) GANs is compelling . However , it remains unclear whether those GANs have similarly nice properties in the multi-modal setting . Again , the difficulty here is that the practical GAN formulation is not convex-concave , even for simple discriminators/generators . These works and other related ones are reviewed in more detail in Appendix B . Our contributions . In this work , we make a step towards understanding the global behavior of GAN training , under multi-modal settings . We focus on analyzing GANs for learning a multi-point distribution . This is a finite-sample version of the problem analyzed by Goodfellow et al . ( 2014 ) , and a multi-mode extension of the Dirac-GAN problem analyzed by Mescheder et al . ( 2018 ) . We approach the problem from two perspectives . First , we consider the min-max optimization problem and analyze the landscape of the outer problem assuming powerful discriminators . This perspective was used by Goodfellow et al . ( 2014 ) . In contrast , as optimization variables we use the samples Y instead of the probability density pg . Second , we consider the training dynamics of the game formulation assuming linear discriminators . Our contributions are summarized as follows . • For the original GAN , we prove that the outer-minimization problem has exponentially many sub-optimal strict local minima . Each strict local minima corresponds to a modecollapse situation . • For both the original GAN and WGAN-GP , we prove that with linear discriminators , the training dynamics can not escape from mode-collapse . • We propose a new GAN formulation called CP-GAN ( CoupleGAN ) that enjoys nice global properties . From the first perspective , we prove that the outer-minimization problem of CPGAN has no bad strict local minima , improving upon the original GAN . From the second perspective , we prove that with linear discriminators , the training dynamics of CP-GAN can escape from mode-collapse ; in addition , it has a global Lyapunov function that permits to prove global convergence . • Our simulation results show that the new GAN performs better than WGAN-GP and the original GAN in standard datasets , in terms of FID scores . Our modification to the loss function has some similarity with the Wasserstein distance ( coupling data points ) , but it turns out it works better than WGAN-GP on all datasets we tested . We remark that CPGAN is orthogonal to other techniques such as spectral normalization Miyato et al . ( 2018 ) , thus we only compare with vanilla methods such as WGAN-GP . Finally , we remark that CP-GAN is just one example of GAN problems with nice global properties , and we hope our analysis can shed light on the design of other tractable GAN formulations . Outline . The rest of the paper is structured as follows . We present the CP-GAN loss in Section 2 , and discuss our analysis framework in Section 3 . In Section 4 , we analyze the outer-optimization problem of JS-GAN and the training dynamics of JS-GAN and W-GAN . In Section 5 , we analyze the outer-optimization problem and the training dynamics of CP-GAN . We evaluate its performance on synthetic and standard datasets in Section 6 . All proofs are in the Appendix . 2 NEW LOSS : CP-GAN . In this section , we first review the formulation of GANs briefly . Next , we present the new formulation of CP-GAN . The theoretical advantage of the new formulation will be discussed later . 2.1 GENERATIVE ADVERSARIAL NETWORKS . We first review the formulation of the original GAN proposed by Goodfellow et al . ( 2014 ) . Given samples from a true data distribution pdata , we want to generate a new distribution pg to mimic pdata . To judge how far away the generated distribution pg is from pdata , a discriminator ( a.k.a . critic ) computes a loss value that measures their gap . This discriminator D addresses a binary classification problem , which should yield 1 for the true data point and 0 for the generated data point . The goal of the generator is to generate pg to fool the discriminator so that the loss value is minimized . Formally , the problem for a GAN is given by min pg max D Ex∼pdata , y∼pg log ( D ( x ) ) + log ( 1−D ( y ) ) . ( 1 ) A common choice for the discriminator is D ( u ) = 11+exp ( −f ( u ) ) , where f is a function . Given this choice , the formulation given in Eq . ( 1 ) becomes min pg max f Ex∼pdata , y∼pg log 1 1 + exp ( −f ( x ) ) + log 1 1 + exp ( f ( y ) ) . We will use a generator Gθ ( z ) parameterized by θ to produce samples from a distribution pg , where z is drawn from a given distribution pz . This yields the following program : min G max f Ex∼pdata , z∼pz log 1 1 + exp ( −f ( x ) ) + log 1 1 + exp ( f ( G ( z ) ) ) . ( 2 ) To resolve the vanishing gradient issue , Goodfellow et al . ( 2014 ) proposed to use a game formulation ( often referred to as non-saturating GAN ) : max f Ex∼pdata , z∼pz log 1 1 + exp ( −f ( x ) ) + log 1 1 + exp ( f ( G ( z ) ) ) , ( 3a ) min G Ex∼pdata , z∼pz log ( 1 + exp ( −f ( G ( z ) ) ) ) . ( 3b ) 2.2 COUPLING GAN . As mentioned before , the program is given in Eq . ( 3 ) has shortcomings , which we propose to address via the following formulation : min ψ max θ Ex∼pdata , z∼pz [ log 1 1 + exp ( fθ ( Gψ ( z ) ) − fθ ( x ) ) ] . ( 4a ) Similar to the − logD trick used in the original GAN training ( a.k.a . non-saturating GAN ) , in practice we always use the non-saturating version for training : D problem : min θ Ex∼pdata , z∼pz [ log ( 1 + exp ( fθ ( Gψ ( z ) ) − fθ ( x ) ) ) ] , ( 5a ) G problem : min ψ Ex∼pdata , z∼pz [ log ( 1 + exp ( fθ ( x ) − fθ ( Gψ ( z ) ) ) ) ] . ( 5b ) Here fθ is a deep net parameterized by θ , and Gψ is a generator net parameterized by ψ . How to interpret the new loss ? Our original intuition is to use “ personalized criterion ” . In JSGAN , the discriminator wants to solve a binary classification problem by finding a single decision boundary that separates true data and generated data . For instance , if true data are x1 = 1 and x2 = 2 , and generated data are y1 = 0.5 and y2 = 0.5 , then the boundary is drawn at , say , 0.8 . Then the discriminator only has the motivation to approach x2 = 1 , since this would cross the bar of the current discriminator . An analogy is that if the requirement for the students is “ get 60 points ” , then they will try to get 60 points and then rest ( surely , there will be follow-up raise , but the raise can not be too fast , since most students can not catch ) . This causes x2 to be missed by the generator , since it is “ too good to learn ” . In contrast , in CP-GAN , the discriminator provides two boundaries at 0.8 and 1.5 , and y1 will try to cross 0.8 to approach x1 = 1 , and y2 will try to cross 1.5 to approach x2 = 2 . Thus it can better learn the true distribution . An analogy is that if the requirements are set differently for different students , like one “ 60 point ” and one “ 90 point ” , then they will learn to achieve the two goals . Thus CP-GAN is adopting “ personalized criterion ” for the generated data . 3 MULTI-DIRACGAN : MULTI-MODE GENERATION . Mescheder ( 2018 ) quoted Rahimi : “ simple experiments , simple theorems are the building blocks that help us understand more complicated systems. ” Then they defined a simple model called DiracGAN , which uses a linear discriminator and a powerful generator to recover a single-point distribution . We define a much more general model than the DiracGAN . We refer it as Multi-DiracGAN . Definition 3.1 The Multi-DiracGAN consists of a generator distribution Y = ( y1 , . . . , yn ) ∈ Rd×n and a discriminator f ( x ) . The true data distribution is given by a n-point distribution X = ( x1 , . . . , xn ) ∈ Rd×n , where xi ’ s are distinct . One motivation of the Multi-DiracGAN is to consider an “ empirical ” version of the problem . Yet another motivation is the construction of a simple model of multi-mode distributions : for instance , when n = 2 , we have a two-point distribution which is the simplest two-mode distribution . We are not aware of an existing global analysis that can be applied to this simple 2-point model . See Appendix A for more discussions . When analyzing an optimization formulation , it is natural to proceed in the following steps : ( 1 ) Sanity check , i.e. , study whether the globally optimal solutions are desired ; ( 2 ) Landscape analysis , i.e. , check whether there are undesirable local minima ; ( 3 ) Convergence analysis , i.e. , check whether the proposed algorithm converges to a local minimum or stationary point . We will first perform a sanity check and landscape analysis ( for powerful discriminators ) , and then study the dynamics ( for linear discriminators ) .

Summary: In this paper, the authors introduce a new training loss for GANs.  This loss allows the outer optimization problem to have no spurious local minima, under an appropriate finite sample analysis.  In contrast, the authors establish that there are exponentially many spurious local minima under the conventional GAN training loss.  Under a linear discriminator model, the authors show that a standard GAN can not escape from collapsed modes in a finite sample analysis, whereas the new trining loss allows for such an escape (due to the presence of a Lyapunov functional with favorable properties).  The authors use this new training loss to train GANS on MNIST, CIFAR10, CelebA, and LSUN datasets, and observe mild improvements in Inception Scores and Frechet Inception Distances of the resulting generated images.