Finding Mixed Nash Equilibria of Generative Adversarial Networks

We reconsider the training objective of Generative Adversarial Networks ( GANs ) from the mixed Nash Equilibria ( NE ) perspective . Inspired by the classical prox methods , we develop a novel algorithmic framework for GANs via an infinite-dimensional two-player game and prove rigorous convergence rates to the mixed NE . We then propose a principled procedure to reduce our novel prox methods to simple sampling routines , leading to practically efficient algorithms . Finally , we provide experimental evidence that our approach outperforms methods that seek pure strategy equilibria , such as SGD , Adam , and RMSProp , both in speed and quality . 1 Introduction . The Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) has become one of the most powerful paradigms in learning real-world distributions , especially for image-related data . It has been successfully applied to a host of applications such as image translation ( Isola et al. , 2017 ; Kim et al. , 2017 ; Zhu et al. , 2017 ) , super-resolution imaging ( Wang et al. , 2015 ) , pose editing ( Pumarola et al. , 2018b ) , and facial animation ( Pumarola et al. , 2018a ) . Despite of the many accomplishments , the major hurdle blocking the full impact of GAN is its notoriously difficult training phase . In the language of game theory , GAN seeks for a pure strategy equilibrium , which is well-known to be ill-posed in many scenarios ( Dasgupta & Maskin , 1986 ) . Indeed , it is known that a pure strategy equilibrium might not exist ( Arora et al. , 2017 ) , might be degenerate ( Sønderby et al. , 2017 ) , or can not be reliably reached by existing algorithms ( Mescheder et al. , 2017 ) . Empirically , it has also been observed that common algorithms , such as SGD or Adam ( Kingma & Ba , 2015 ) , lead to unstable training . While much efforts have been devoted into understanding the training dynamics of GANs ( Balduzzi et al. , 2018 ; Gemp & Mahadevan , 2018 ; Gidel et al. , 2018a ; b ; Liang & Stokes , 2018 ) , a provably convergent algorithm for general GANs , even under reasonably strong assumptions , is still lacking . In this paper , we address the above problems with the following contributions : 1 . We propose to study the mixed Nash Equilibrium ( NE ) of GANs : Instead of searching for an optimal pure strategy which might not even exist , we optimize over the set of probability distributions over pure strategies of the networks . The existence of a solution to such problems was long established amongst the earliest game theory work ( Glicksberg , 1952 ) , leading to well-posed optimization problems . 2 . We demonstrate that the prox methods of ( Nemirovsky & Yudin , 1983 ; Nemirovski , 2004 ) , which are fundamental building blocks for solving two-player games with finitely many strategies , can be extended to continuously many strategies , and hence applicable to training GANs . We provide an elementary proof for their convergence rates to learning the mixed NE . 3 . We construct a principled procedure to reduce our novel prox methods to certain sampling tasks that were empirically proven easy by recent work ( Chaudhari et al. , 2017 ; 2018 ; Dziugaite & Roy , 2018 ) . We further establish heuristic guidelines to greatly scale down the memory and computational costs , resulting in simple algorithms whose per-iteration complexity is almost as cheap as SGD . 4 . We experimentally show that our algorithms consistently achieve better or comparable performance than popular baselines such as SGD , Adam , and RMSProp ( Tieleman & Hinton , 2012 ) . Related Work : While the literature on training GANs is vast , to our knowledge , there exist only few papers on the mixed NE perspective . The notion of mixed NE is already present in ( Goodfellow et al. , 2014 ) , but is stated only as an existential result . The authors of ( Arora et al. , 2017 ) advocate the mixed strategies , but do not provide a provably convergent algorithm . ( Oliehoek et al. , 2018 ) also considers mixed NE , but only with finitely many parameters . The work ( Grnarova et al. , 2018 ) proposes a provably convergent algorithm for finding the mixed NE of GANs under the unrealistic assumption that the discriminator is a single-layered neural network . In contrast , our results are applicable to arbitrary architectures , including popular ones ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) . Due to its fundamental role in game theory , many prox methods have been applied to study the training of GANs ( Daskalakis et al. , 2018 ; Gidel et al. , 2018a ; Mertikopoulos et al. , 2018 ) . However , these works focus on the classical pure strategy equilibria and are hence distinct from our problem formulation . In particular , they give rise to drastically different algorithms from ours and do not provide convergence rates for GANs . In terms of analysis techniques , our framework is closely related to ( Balandat et al. , 2016 ) , but with several important distinctions . First , the analysis of ( Balandat et al. , 2016 ) is based on dual averaging ( Nesterov , 2009 ) , while we consider Mirror Descent and also the more sophisticated Mirror-Prox ( see Section 3 ) . Second , unlike our work , ( Balandat et al. , 2016 ) do not provide any convergence rate for learning mixed NE of two-player games . Finally , ( Balandat et al. , 2016 ) is only of theoretical interest with no practical algorithm . Notation : Throughout the paper , we use z to denote a generic variable and Z ⊆ Rd its domain . We denote the set of all Borel probability measures on Z by M ( Z ) , and the set of all functions on Z by F ( Z ) .1 We write dµ = ρdz to mean that the density function of µ ∈ M ( Z ) with respect to the Lebesgue measure is ρ . All integrals without specifying the measure are understood to be with respect to Lebesgue . For any objective of the form minxmaxy F ( x , y ) , we say that ( xT , yT ) is an O ( T−1/2 ) -NE if maxx , y { F ( xT , y ) − F ( x , yT ) } = O ( T−1/2 ) . Similarly we can define O ( T−1 ) -NE . The symbol ‖·‖L∞ denotes the L∞-norm of functions , and ‖·‖TV denotes the total variation norm of probability measures . 2 Problem Formulation . We review standard results in game theory in Section 2.1 , whose proof can be found in ( Bubeck , 2013a ; b ; c ) . Section 2.2 relates training of GANs to the two-player game in Section 2.1 , thereby suggesting to generalize the prox methods to infinite dimension . 2.1 Preliminary : Prox Methods for Finite Games . Consider the classical formulation of a two-player game with finitely many strategies : min p∈∆m max q∈∆n 〈q , a〉 − 〈q , Ap〉 , ( 1 ) where A is a payoff matrix , a is a vector , and ∆d : = { z ∈ Rd | ∑d i=1 zi = 1 } is the proba- bility simplex , representing the mixed strategies ( i.e. , probability distributions ) over d pure strategies . A pair ( pNE , qNE ) achieving the min-max value in ( 1 ) is called a mixed NE . Assume that the matrix A is too expensive to evaluate whereas the ( stochastic ) gradients of ( 1 ) are easy to obtain . Under such settings , a celebrated algorithm , the so-called entropic Mirror Descent ( entropic MD ) , learns an O ( T−1/2 ) -NE : Let φ ( z ) : = ∑d i=1 zi log zi be the entropy function and φ ? ( y ) : = log ∑d i=1 e yi = supz∈∆d { 〈z , y〉 − φ ( z ) } be its Fenchel dual . 1Strictly speaking , our derivation requires mild regularity ( see Appendix A.1 ) assumptions on the probability measure and function classes , which are met by most practical applications . For a learning rate η and an arbitrary vector b ∈ Rd , define the MD iterates as z′ = MDη ( z , b ) ≡ z′ = ∇φ ? ( ∇φ ( z ) − ηb ) ≡ z′i = zie −ηbi∑d i=1 zie −ηbi , ∀1 ≤ i ≤ d. ( 2 ) The equivalence of the last two formulas in ( 2 ) can be readily checked . Denote by p̄T : = 1 T ∑T t=1 pt and q̄T : = 1 T ∑T t=1 qt the ergodic average of two sequences { pt } Tt=1 and { qt } Tt=1 . Then , with a properly chosen step-size η , we have { pt+1 = MDη ( pt , −A > qt ) qt+1 = MDη ( qt , −a+Apt ) ⇒ ( p̄T , q̄T ) is an O ( T− 1/2 ) -NE . Moreover , a slightly more complicated algorithm , called the entropic Mirror-Prox ( entropy MP ) ( Nemirovski , 2004 ) , achieves faster rate than the entropic MD : { pt = MDη ( p̃t , −A > q̃t ) , p̃t+1 = MDη ( p̃t , −A > qt ) qt = MDη ( q̃t , −a+Ap̃t ) , q̃t+1 = MDη ( q̃t , −a+Apt ) ⇒ ( p̄T , q̄T ) is an O ( T−1 ) -NE . If , instead of deterministic gradients , one uses unbiased stochastic gradients for entropic MD and MP , then both algorithms achieve O ( T−1/2 ) -NE in expectation . 2.2 Mixed Strategy Formulation for Generative Adversarial Networks . For illustration , let us focus on the Wasserstein GAN ( Arjovsky et al. , 2017 ) , and we perform a common bilinearization trick that dates back at least to the early game theory literature ( Glicksberg , 1952 ) , and is also well-known in optimal transport theory ( Villani , 2003 ) . The training objective of Wasserstein GAN is min θ∈Θ max w∈W EX∼Preal [ fw ( X ) ] − EX∼Pθ [ fw ( X ) ] , ( 3 ) where Θ is the set of parameters for the generator and W the set of parameters for the discriminator f , typically both taken to be neural nets . As mentioned in the introduction , such an optimization problem can be ill-posed , which is also supported by empirical evidence . The high-level idea of our approach is , instead of solving ( 3 ) directly , we focus on the mixed strategy formulation of ( 3 ) . In other words , we consider the set of all probability distributions over Θ andW , and we search for the optimal distribution that solves the following program : min ν∈M ( Θ ) max µ∈M ( W ) Ew∼µEX∼Preal [ fw ( X ) ] − Ew∼µEθ∼νEX∼Pθ [ fw ( X ) ] . ( 4 ) Define the function g : W → R by g ( w ) : = EX∼Preal [ fw ( X ) ] and the operator G : M ( Θ ) → F ( W ) as ( Gν ) ( w ) : = Eθ∼ν , X∼Pθ [ fw ( X ) ] . Denoting 〈µ , h〉 : = Eµh2 for any probability measure µ and function h , we may rewrite ( 4 ) as min ν∈M ( Θ ) max µ∈M ( W ) 〈µ , g〉 − 〈µ , Gν〉 . ( 5 ) Furthermore , the Fréchet derivative ( the analogue of gradient in infinite dimension ) of ( 5 ) with respect to µ is simply g − Gν , and the derivative of ( 5 ) with respect to ν is −G†µ , where G† : M ( W ) → F ( Θ ) is the adjoint operator of G defined via the relation ∀µ ∈M ( W ) , ν ∈M ( Θ ) , 〈µ , Gν〉 = 〈 ν , G†µ 〉 . ( 6 ) One can easily check that ( G†µ ) ( θ ) : = EX∼Pθ , w∼µ [ fw ( X ) ] achieves the equality in ( 6 ) . To summarize , the mixed strategy formulation of Wasserstein GAN is ( 5 ) , whose derivatives can be expressed in terms of g and G. We now make the crucial observation that ( 5 ) is the infinite-dimensional analogue of ( 1 ) : The distributions over finite strategies are replaced with probability measures over a continuous parameter set , the vector a is replaced with a function g , the matrix A is replaced with a linear operator3 G , and the gradients are replaced with Fréchet derivatives . Based on Section 2.1 , it is then natural to ask : 2It should be noted that 〈µ , h〉 is NOT an inner product , and rather is the dual pairing in Banach spaces ( Halmos , 2013 ) . 3The linearity of G trivially follows from the linearity of expectation . Can the entropic Mirror Descent and Mirror-Prox be extended to infinite dimension to solve ( 5 ) ? Can we retain the convergence rates ? We provide an affirmative answer to both questions in the next section . Remark . The derivation in Section 2.2 can be applied to any GAN objective .

Summary: This paper proposes to consider the mixed equilibrium objective function for GANS. The authors generalize the mirror descent/mirror prox to handle continuous games. The technical challenge is to write those algorithms in infinite dimensional spaces. This reviewer finds this however to be a mere technicality, and there seems to be no conceptual obstruction. In fact other paper have already written this, see for example ``Mirror Descent Learning in Continuous Games" by Zhou et al. at CDC 2017 (I'm sure there are other references too).