Finding Mixed Nash Equilibria of Generative Adversarial Networks

We reconsider the training objective of Generative Adversarial Networks ( GANs ) from the mixed Nash Equilibria ( NE ) perspective . Inspired by the classical prox methods , we develop a novel algorithmic framework for GANs via an infinite-dimensional two-player game and prove rigorous convergence rates to the mixed NE . We then propose a principled procedure to reduce our novel prox methods to simple sampling routines , leading to practically efficient algorithms . Finally , we provide experimental evidence that our approach outperforms methods that seek pure strategy equilibria , such as SGD , Adam , and RMSProp , both in speed and quality . 1 Introduction . The Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) has become one of the most powerful paradigms in learning real-world distributions , especially for image-related data . It has been successfully applied to a host of applications such as image translation ( Isola et al. , 2017 ; Kim et al. , 2017 ; Zhu et al. , 2017 ) , super-resolution imaging ( Wang et al. , 2015 ) , pose editing ( Pumarola et al. , 2018b ) , and facial animation ( Pumarola et al. , 2018a ) . Despite of the many accomplishments , the major hurdle blocking the full impact of GAN is its notoriously difficult training phase . In the language of game theory , GAN seeks for a pure strategy equilibrium , which is well-known to be ill-posed in many scenarios ( Dasgupta & Maskin , 1986 ) . Indeed , it is known that a pure strategy equilibrium might not exist ( Arora et al. , 2017 ) , might be degenerate ( Sønderby et al. , 2017 ) , or can not be reliably reached by existing algorithms ( Mescheder et al. , 2017 ) . Empirically , it has also been observed that common algorithms , such as SGD or Adam ( Kingma & Ba , 2015 ) , lead to unstable training . While much efforts have been devoted into understanding the training dynamics of GANs ( Balduzzi et al. , 2018 ; Gemp & Mahadevan , 2018 ; Gidel et al. , 2018a ; b ; Liang & Stokes , 2018 ) , a provably convergent algorithm for general GANs , even under reasonably strong assumptions , is still lacking . In this paper , we address the above problems with the following contributions : 1 . We propose to study the mixed Nash Equilibrium ( NE ) of GANs : Instead of searching for an optimal pure strategy which might not even exist , we optimize over the set of probability distributions over pure strategies of the networks . The existence of a solution to such problems was long established amongst the earliest game theory work ( Glicksberg , 1952 ) , leading to well-posed optimization problems . 2 . We demonstrate that the prox methods of ( Nemirovsky & Yudin , 1983 ; Nemirovski , 2004 ) , which are fundamental building blocks for solving two-player games with finitely many strategies , can be extended to continuously many strategies , and hence applicable to training GANs . We provide an elementary proof for their convergence rates to learning the mixed NE . 3 . We construct a principled procedure to reduce our novel prox methods to certain sampling tasks that were empirically proven easy by recent work ( Chaudhari et al. , 2017 ; 2018 ; Dziugaite & Roy , 2018 ) . We further establish heuristic guidelines to greatly scale down the memory and computational costs , resulting in simple algorithms whose per-iteration complexity is almost as cheap as SGD . 4 . We experimentally show that our algorithms consistently achieve better or comparable performance than popular baselines such as SGD , Adam , and RMSProp ( Tieleman & Hinton , 2012 ) . Related Work : While the literature on training GANs is vast , to our knowledge , there exist only few papers on the mixed NE perspective . The notion of mixed NE is already present in ( Goodfellow et al. , 2014 ) , but is stated only as an existential result . The authors of ( Arora et al. , 2017 ) advocate the mixed strategies , but do not provide a provably convergent algorithm . ( Oliehoek et al. , 2018 ) also considers mixed NE , but only with finitely many parameters . The work ( Grnarova et al. , 2018 ) proposes a provably convergent algorithm for finding the mixed NE of GANs under the unrealistic assumption that the discriminator is a single-layered neural network . In contrast , our results are applicable to arbitrary architectures , including popular ones ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) . Due to its fundamental role in game theory , many prox methods have been applied to study the training of GANs ( Daskalakis et al. , 2018 ; Gidel et al. , 2018a ; Mertikopoulos et al. , 2018 ) . However , these works focus on the classical pure strategy equilibria and are hence distinct from our problem formulation . In particular , they give rise to drastically different algorithms from ours and do not provide convergence rates for GANs . In terms of analysis techniques , our framework is closely related to ( Balandat et al. , 2016 ) , but with several important distinctions . First , the analysis of ( Balandat et al. , 2016 ) is based on dual averaging ( Nesterov , 2009 ) , while we consider Mirror Descent and also the more sophisticated Mirror-Prox ( see Section 3 ) . Second , unlike our work , ( Balandat et al. , 2016 ) do not provide any convergence rate for learning mixed NE of two-player games . Finally , ( Balandat et al. , 2016 ) is only of theoretical interest with no practical algorithm . Notation : Throughout the paper , we use z to denote a generic variable and Z ⊆ Rd its domain . We denote the set of all Borel probability measures on Z by M ( Z ) , and the set of all functions on Z by F ( Z ) .1 We write dµ = ρdz to mean that the density function of µ ∈ M ( Z ) with respect to the Lebesgue measure is ρ . All integrals without specifying the measure are understood to be with respect to Lebesgue . For any objective of the form minxmaxy F ( x , y ) , we say that ( xT , yT ) is an O ( T−1/2 ) -NE if maxx , y { F ( xT , y ) − F ( x , yT ) } = O ( T−1/2 ) . Similarly we can define O ( T−1 ) -NE . The symbol ‖·‖L∞ denotes the L∞-norm of functions , and ‖·‖TV denotes the total variation norm of probability measures . 2 Problem Formulation . We review standard results in game theory in Section 2.1 , whose proof can be found in ( Bubeck , 2013a ; b ; c ) . Section 2.2 relates training of GANs to the two-player game in Section 2.1 , thereby suggesting to generalize the prox methods to infinite dimension . 2.1 Preliminary : Prox Methods for Finite Games . Consider the classical formulation of a two-player game with finitely many strategies : min p∈∆m max q∈∆n 〈q , a〉 − 〈q , Ap〉 , ( 1 ) where A is a payoff matrix , a is a vector , and ∆d : = { z ∈ Rd | ∑d i=1 zi = 1 } is the proba- bility simplex , representing the mixed strategies ( i.e. , probability distributions ) over d pure strategies . A pair ( pNE , qNE ) achieving the min-max value in ( 1 ) is called a mixed NE . Assume that the matrix A is too expensive to evaluate whereas the ( stochastic ) gradients of ( 1 ) are easy to obtain . Under such settings , a celebrated algorithm , the so-called entropic Mirror Descent ( entropic MD ) , learns an O ( T−1/2 ) -NE : Let φ ( z ) : = ∑d i=1 zi log zi be the entropy function and φ ? ( y ) : = log ∑d i=1 e yi = supz∈∆d { 〈z , y〉 − φ ( z ) } be its Fenchel dual . 1Strictly speaking , our derivation requires mild regularity ( see Appendix A.1 ) assumptions on the probability measure and function classes , which are met by most practical applications . For a learning rate η and an arbitrary vector b ∈ Rd , define the MD iterates as z′ = MDη ( z , b ) ≡ z′ = ∇φ ? ( ∇φ ( z ) − ηb ) ≡ z′i = zie −ηbi∑d i=1 zie −ηbi , ∀1 ≤ i ≤ d. ( 2 ) The equivalence of the last two formulas in ( 2 ) can be readily checked . Denote by p̄T : = 1 T ∑T t=1 pt and q̄T : = 1 T ∑T t=1 qt the ergodic average of two sequences { pt } Tt=1 and { qt } Tt=1 . Then , with a properly chosen step-size η , we have { pt+1 = MDη ( pt , −A > qt ) qt+1 = MDη ( qt , −a+Apt ) ⇒ ( p̄T , q̄T ) is an O ( T− 1/2 ) -NE . Moreover , a slightly more complicated algorithm , called the entropic Mirror-Prox ( entropy MP ) ( Nemirovski , 2004 ) , achieves faster rate than the entropic MD : { pt = MDη ( p̃t , −A > q̃t ) , p̃t+1 = MDη ( p̃t , −A > qt ) qt = MDη ( q̃t , −a+Ap̃t ) , q̃t+1 = MDη ( q̃t , −a+Apt ) ⇒ ( p̄T , q̄T ) is an O ( T−1 ) -NE . If , instead of deterministic gradients , one uses unbiased stochastic gradients for entropic MD and MP , then both algorithms achieve O ( T−1/2 ) -NE in expectation . 2.2 Mixed Strategy Formulation for Generative Adversarial Networks . For illustration , let us focus on the Wasserstein GAN ( Arjovsky et al. , 2017 ) , and we perform a common bilinearization trick that dates back at least to the early game theory literature ( Glicksberg , 1952 ) , and is also well-known in optimal transport theory ( Villani , 2003 ) . The training objective of Wasserstein GAN is min θ∈Θ max w∈W EX∼Preal [ fw ( X ) ] − EX∼Pθ [ fw ( X ) ] , ( 3 ) where Θ is the set of parameters for the generator and W the set of parameters for the discriminator f , typically both taken to be neural nets . As mentioned in the introduction , such an optimization problem can be ill-posed , which is also supported by empirical evidence . The high-level idea of our approach is , instead of solving ( 3 ) directly , we focus on the mixed strategy formulation of ( 3 ) . In other words , we consider the set of all probability distributions over Θ andW , and we search for the optimal distribution that solves the following program : min ν∈M ( Θ ) max µ∈M ( W ) Ew∼µEX∼Preal [ fw ( X ) ] − Ew∼µEθ∼νEX∼Pθ [ fw ( X ) ] . ( 4 ) Define the function g : W → R by g ( w ) : = EX∼Preal [ fw ( X ) ] and the operator G : M ( Θ ) → F ( W ) as ( Gν ) ( w ) : = Eθ∼ν , X∼Pθ [ fw ( X ) ] . Denoting 〈µ , h〉 : = Eµh2 for any probability measure µ and function h , we may rewrite ( 4 ) as min ν∈M ( Θ ) max µ∈M ( W ) 〈µ , g〉 − 〈µ , Gν〉 . ( 5 ) Furthermore , the Fréchet derivative ( the analogue of gradient in infinite dimension ) of ( 5 ) with respect to µ is simply g − Gν , and the derivative of ( 5 ) with respect to ν is −G†µ , where G† : M ( W ) → F ( Θ ) is the adjoint operator of G defined via the relation ∀µ ∈M ( W ) , ν ∈M ( Θ ) , 〈µ , Gν〉 = 〈 ν , G†µ 〉 . ( 6 ) One can easily check that ( G†µ ) ( θ ) : = EX∼Pθ , w∼µ [ fw ( X ) ] achieves the equality in ( 6 ) . To summarize , the mixed strategy formulation of Wasserstein GAN is ( 5 ) , whose derivatives can be expressed in terms of g and G. We now make the crucial observation that ( 5 ) is the infinite-dimensional analogue of ( 1 ) : The distributions over finite strategies are replaced with probability measures over a continuous parameter set , the vector a is replaced with a function g , the matrix A is replaced with a linear operator3 G , and the gradients are replaced with Fréchet derivatives . Based on Section 2.1 , it is then natural to ask : 2It should be noted that 〈µ , h〉 is NOT an inner product , and rather is the dual pairing in Banach spaces ( Halmos , 2013 ) . 3The linearity of G trivially follows from the linearity of expectation . Can the entropic Mirror Descent and Mirror-Prox be extended to infinite dimension to solve ( 5 ) ? Can we retain the convergence rates ? We provide an affirmative answer to both questions in the next section . Remark . The derivation in Section 2.2 can be applied to any GAN objective .

Summary: This paper extends the mirror-descent and mirror-prox algorithms to infinite dimensional Banach spaces so that they can be applied to solve the mixed Nash equilibrium of the popular generative adversarial networks. The main technical results appear to be formal but straightforward extensions of existing techniques in finite dimensional spaces. A sample-based practical algorithm is proposed so that the infinite dimensional algorithms can still be computed. Experiments are a bit disappointing as the authors only used visual appeal as an evaluation criterion. (I understand why the authors chose to do so but as an algorithmic paper, resorting to an evaluation based on visual appeal is almost always unsatisfactory.)