Generalized Natural Gradient Flows in Hidden Convex-Concave Games and GANs

1 INTRODUCTION . Min-max optimization has found extensive applications in modern Machine Learning ( ML ) and Deep Learning . Popular application settings include Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , adversarial training ( Madry et al. , 2018 ) , and multi-agent reinforcement learning ( Silver et al. , 2017 ) . In all of these cases , a pair of networks is typically trained towards finding an approximate equilibrium of a highly non-convex non-concave problem . However , such settings go beyond classical and well-known results in game theory for which equilibria only exist in more restrictive settings , i.e. , convex-concave games ( Sion et al. , 1958 ) . Unfortunately , there is no parallel guarantee if the payoff is not convex-concave , and all these applications are indeed based upon non-convex non-concave games . Even worse , many negative results occur when dealing with such payoffs : global or local minimax may not exist ( Jin et al. , 2020 ) , and even if they exist , there is , in general , no “ reasonable algorithm ” 1 that can globally converge to any meaningful notion of local optimum ( Letcher , 2021 ; Hsieh et al. , 2020 ) . In order to overcome these negative results , we analyze a specific restricted class of non-convex non-concave games called Hidden Convex-Concave ( HCC ) games ( Flokas et al. , 2020 ; Gidel et al. , 2021 ; Flokas et al. , 2021 ) that include many Machine Learning applications such as GANs ( Goodfellow et al. , 2014 ) , Adversarial Example Games ( AEG ) ( Bose et al. , 2020 ) , or Minimax Estimation of Conditional Moment ( Dikkala et al. , 2020 ) . Our contributions . In the setting of HCC games , our analysis tackles unique challenges not captured l standard in the convex-concave games setting . As a way to overcome those challenges , we propose a new type of dynamics , dubbed Natural Hidden Gradient dynamics ( NHG ) , and we prove its convergence to stationary points of the HCC game . Critically , our convergence results are global , i.e. , we do not make any assumptions about initial conditions , e.g. , that the initial points belong to some local neighborhood . This novel algorithm is , in fact , inspired by the generalization of gradient flows in different Banach spaces than the standard Euclidean one . Arguably , the most well-known non-Euclidean gradient is the natural gradient flow induced by the Fisher information matrix , which enjoys deep connections to the Replicator Dynamics ( RD ) . Thus , a natural question emerges : Can we have similar convergence guarantees for RD ? In that regard , we provide a complete picture of the 1see ( Letcher , 2021 , Definition 5 ) for a definition of “ reasonable algorithm ” resulting dynamics in a setting of team competition via invariant function analysis . Specifically , the flow of the dynamics is shown to be reduced to a competition between two generalized gradients , one for each team . We show that the behavior of the dynamics in this setting is constrained by a maximal number of invariant functions , which , combined with the actual geometry of the available strategies , shows that the system can either converge or cycle . These results showcase , both , the importance of adapting the “ algorithmic geometry ” to the application and the data at hand , as well as the unexplored effects of feasibility constraints on the complexity of well-known game dynamics . 2 RELATED WORK . Despite the popularity of deep learning applications involving non-convex non-concave games , our understanding of their optimization dynamics and the nature of their solution concepts is still preliminary . However , this space has already witnessed a few important early works that focus on identifying new solution concepts . These solution concepts—which are also broadly applicable in general min-max games—include ( local/differential ) Nash equilibria ( Adolphs et al. , 2019 ; Mazumdar & Ratliff , 2019 ) , ( local/differential ) Stackelberg equilibria ( Fiez et al. , 2020 ; Wang et al. , 2020 ) , local minmax ( Daskalakis & Panageas , 2018 ) , local robust points ( Zhang et al. , 2020 ) , and approximate minimax theorems ( Jin et al. , 2019 ; Gidel et al. , 2021 ) . Numerous solutions concepts such as cycles ( Vlatakis-Gkaragkounis et al. , 2019 ) , chaotic behavior ( Cheung & Piliouras , 2019 ; Cheung & Tao , 2021 ) , and computational issues ( Daskalakis et al. , 2021 ) indicate that solving min-max games , in general , might involve challenging and complex behavior . Many algorithms have been proposed to solve restricted classes of non-convex non-concave games such as Polyak-Łojasiewicz games ( Nouiehed et al. , 2019 ; Yang et al. , 2020 ) , nonconvex-concave games ( Lin et al. , 2020 ; Ostrovskii et al. , 2021 ; Yang et al. , 2020 ; Kong & Monteiro , 2021 ) , as well as classes of games inspired by variational inequalities ( Mertikopoulos et al. , 2019 ; Diakonikolas et al. , 2021 ; Lee & Kim , 2021 ) . However , even though they are significant advances in the understanding of non-convex non-concave game dynamics , such classes of games may not encompass games where the players are parameterized neural networks such as GANs . While convergence in GANs has been a topic of exploration ( Kodali et al. , 2017 ; Heusel et al. , 2017 ; Mescheder et al. , 2018 ; Gemp & Mahadevan , 2018 ; Li et al. , 2018 ; Hsieh et al. , 2019 ; Cao & Guo , 2020 ) , its hidden convex-concave structure has not been exploited before , and there were no theoretical global convergence guarantees in a general setting up to this date . In particular , Hsieh et al . ( 2019 ) use a lifting trick and proceed in solving a relaxation of the problem in the distribution space , while in our work we work entirely in the parameter space . With this respect , we consider Flokas et al . ( 2021 ) , and Gemp & Mahadevan ( 2018 ) as the closest related works . On the one hand , in Gemp & Mahadevan ( 2018 ) , the authors propose crossing-the-curl , a second-order technique , and provide global convergence guarantees for the Wasserstein Linear Quadratic GAN ( W-LQGAN ) . While the W-LQGANs is a class of non-convex non-concave GANs it remains far from the GAN formulation used in practice where the discriminator and the generator are neural networks . On the other hand , the idea of Hidden Convex-Concave ( HCC ) games was first proposed by Flokas et al . ( 2021 ) and Gidel et al . ( 2021 ) . While Flokas et al . ( 2021 ) study the Gradient Descent-Ascent ( GDA ) dynamics in the HCC setting , their work relies on hard-to-verify safety conditions . Our global convergence results without hard-to-test assumptions on initialization are first of their kind , to the best of knowledge . 3 PRELIMINARIES . 3.1 HIDDEN CONVEX-CONCAVE GAMES . Many game-theoretic applications in machine learning often involve a specific structure where the models ’ payoff is a convex-concave function ( e.g. , minimizing Jensen-Shannon-Divergence when training GANs ) . In order to model this specific structure , we propose the following definition of Hidden Convex-Concave ( HCC ) games where intuitively , the game ’ s payoff is convex-concave while the actions are parametrized by non-convex mappings . Definition 1 ( Hidden Convex-Concave game ) . A Hidden Convex-Concave game comprises a collection of payoff ( Lx , x′ ) x , x′∈Rd , a distribution p , and two parametrized mappings , F : RM×Rd → R , and G : RN × Rd′ → R , such that the minimax game of interest is min θ∈RM max φ∈RN Ψ ( θ , φ ) where Ψ ( θ , φ ) = L ( Fθ , Gφ ) : = E ( x , x′ ) ∼p [ Lx , x′ ( Fθ ( x ) , Gφ ( x′ ) ) ] . ( 1 ) In this setting , while we do not expect Ψ ( θ , φ ) to be convex-concave , we assume the function L : F×G→ R to be convex-concave where F and G are , respectively , convex subsets of { F : Rd → R } and { G : Rp → R } .2 We extended Flokas et al . ( 2021 ) ’ s definition to now be able to include most minimax machine learning applications such as GANs or AEG . Example 1 ( Hidden Matching Pennies ( HMP ) games ) . Let us consider p , q ∈ [ 0 , 1 ] the probabilities of picking HEADS for the first and the second player , respectively , in a Matching Pennies game with payoff matrix A ∈ R2×2 , where Ai , j = 1 if i = j ; −1 , otherwise . Then the payoff of this game is defined via Lx , x′ ( p , q ) : = { ( 1− 2p ) ( 1− 2q ) , if ( p , q ) ∈ [ 0 , 1 ] 2 ( Player 1 matches the pennies ) 0 , otherwise . ( 2 ) Now , let us consider any mappings F : RM×Rd → [ 0 , 1 ] andG : RN×Rd → [ 0 , 1 ] , such that their output does not depend on the d-dimensional input , i.e. , Fθ ( x ) = f ( θ ) , and Gφ ( x ) = g ( φ ) , ∀x ∈ Rd . The payoff Ψ ( θ , φ ) = L ( Fθ , Gφ ) : = E [ Lx , x′ ( Fθ ( x ) , Gφ ( x′ ) ) ] = ( 1 − 2f ( θ ) ) ( 1 − 2g ( φ ) ) defines a HCC game . In this example , the two agents play the typical bilinear game of Matching Pennies . However , they do not act on it directly ( i.e. , choose randomized actions to apply , e.g. , the probability of playing HEADS ) , instead they choose the input parameters θ , and φ , which are fed into functions f , and g , respectively , whose outputs define the probability of playing HEADS for each agent . Example 2 ( GANs ) . A Generative Adversarial Network ( GAN ) is a minimax game where the first player , i.e. , generator , aims at learning a distribution pθ similar to a reference data distribution pdata . In practice , the reference data distribution is taken to be the empirical data distribution . Conversely , the second player , usually called the discriminator or critic , Dφ , tries to distinguish the distributions of pθ and pdata . The payoff , Ψ , of this game is defined as : Ψ ( θ , φ ) = Ex∼pdata [ logDφ ( x ) ] + Ex∼pθ [ log ( 1−Dφ ( x ) ) ] . ( 3 ) Assuming that pdata and pθ have a density with respect to Lebesgue measure , 3 and that pdata has full support , we can consider the distribution p such that p ( x , x′ ) = pdata ( x ) if x = x′ ; 0 , otherwise , and set Lx , x ( p′ , D ) : = logD+ p ′ pdata ( x ) log ( 1−D ) , and Lx , x′ ( p′ , D ) = 0 , ∀x 6= x′ . Thus Lx , x′ is convex-concave for any x ∈ Rd . We have that Ψ ( θ , φ ) = E ( x , x′ ) ∼p [ Lx , x′ ( pθ ( x ) , Dφ ( x′ ) ) ] , which is a HCC game . A GAN formulates the generative modeling task as finding a Nash equilibrium of a minimax game . The generator of a GAN is defined as a function that aims to produce realistic data samples by transforming samples drawn from a fixed noise distribution , e.g. , N ( 0 , Id ) . Here , we notice that the GAN payoff is convex ( actually linear ) with respect to the density of the generated distribution . An alternative formulation of a GAN is a Wasserstein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) . It turns out that this GAN is also a HCC game . Example 3 ( WGANs ) . A Wasserstein GAN is a constrained minimax game , where the second player Dφ is a 1-Lipchitz function and where the payoff is Ψ ( θ , φ ) = Ex∼pdata [ Dφ ( x ) ] − Ex∼pθ [ Dφ ( x ) ] . ( 4 ) Similarly , as in Example 2 , the WGAN payoff can be shown to be a HCC game . As the last class of examples of HCC games we present the Adversarial Example Games ( AEG ) ( Bose et al. , 2020 ) . 2One can always assume F and G to be convex sets by considering their convex hulls . 3We make this assumption for simplicity . We can consider Radon–Nikodym derivatives for the general case . Example 4 ( AEG ) . An Adversarial Example Game is a minimax game between a generatorGθ and a classifier f . Given samples ( x , y ) ∼ pdata , the generatorGθ aims at finding adversarial examples x′ such that ‖x − x′‖∞ ≤ and that x′ is not classified as y by f , thus generating a distribution pθ . Overall , the payoff of this game is Ψ ( θ , φ ) = −E ( x′ , y ) ∼pθ [ ` ( fφ ( x ′ ) , y ) ] for ( x′ , y ) ∼ pθ ⇐⇒ x′ = Gθ ( x ) , ( x , y ) ∼ pdata , ( 5 ) and where ` is the cross-entropy loss and Gθ is such that ‖x−Gθ ( x ) ‖ ≤ , ∀x . By using a similar construction as in Example 2 , we can show that ( 5 ) is a HCC game with respect to pθ and fφ . Throughout this work , we make the assumption that the minimax problem induced by L admits a solution : Assumption 1 . The HCC game defined by ( 1 ) admits a Nash equilibrium , ( F ∗ , G∗ ) , i.e. , min F∈F max G∈G L ( F , G ) = max G∈G min F∈F L ( F , G ) = L ( F ∗ , G∗ ) . ( 6 ) Such an assumption is relatively mild since , by Sion et al . ( 1958 ) , it holds when the set F is compact ( since , by definition , it is convex ) . Here we do not assume that this solution is achievable , i.e. , we do not assume that there exists θ∗ and φ∗ such that ( Fθ∗ , Gφ∗ ) = ( F ∗ , G∗ ) . Such sufficient conditions for the existence of a Nash equilibrium of L : F × G → R are discussed in detail in Gidel et al . ( 2021 , Prop . 1 ) , e.g. , Assumption 1 holds if the parameters θ and φ are bounded . From a high-level perspective , this assumption is analogous to the existential assumption of a optimizer in non-convex optimization . In that sense , we use the equilibrium ( F ∗ , G∗ ) as a target to build a Lyapunov function for a natural gradient flow .

Summary: This paper studied a non-convex non-concave Zero-Sum game setting called Hidden Convex-Concave game (HCC), as shown in Def. 1, which includes several examples, such as Hidden Matching Pennies games, Generative Adversarial Networks (GANs) / WGANs, and Adversarial Example Game (AEG). The observation is that although the min-max objective is non-convex non-concave w.r.t. the parameters (e.g., weights of neural networks), it could be convex concave of the outputs/strategies of the players $F$ and $G$.  Under Assumption 1 of there exists a Nash equilibrium $(F^*, G^*)$, the authors studies/proposed an algorithm called Natural Hidden Gradient flow (NHG), which is a preconditioned gradient flow update, such that $F$ and $G$ are doing normal Euclidean gradient descent because of the HCC game is convex concave w.r.t. $F$ and $G$.  The authors first showed that for a one-dimensional and finite-sum of convex-concave objectives, the distance to optimum is a Lyapunov function of NHG, and thus NHG converges to the Nash equilibrium.  The authors then connected NHG to replicator dynamics (RD) in literature, under a certain metric of geometry. They showed that in a 2-Team Hidden Matching Pennies game, there exist several invariant functions, which explains why RD manifest cycling orbits.  Finally, the authors conducted simulations to verify the proposed NHG method, showing that it is more stable than Gradient Descent-Ascent (GDA).