Solving Min-Max Optimization with Hidden Structure via Gradient Descent Ascent

1 Introduction . Traditionally , our understanding of convex-concave games revolves around von Neumann ’ s celebrated minimax theorem , which implies the existence of saddle point solutions with a uniquely defined value . These solutions are called von Nemann solutions and guarantee to each agent their corresponding value regardless of opponent play . Although many learning algorithms are known to be able to compute such saddle points [ 13 ] , recently there has there has been a fervor of activity in proving stronger results such as faster regret minimization rates or analysis of the day-to-day behavior [ 46 , 17 , 7 , 1 , 66 , 19 , 2 , 45 , 5 , 25 , 70 , 29 , 6 , 48 , 30 , 56 ] . This interest has been largely triggered by the impressive successes of AI architectures inspired by min-max games such as Generative Adversarial Networks ( GANS ) [ 26 ] , adversarial training [ 40 ] and reinforcement learning self-play in games [ 63 ] . Critically , however , all these applications are based upon non-convex non-concave games , our understanding of which is still nascent . Nevertheless , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . some important early work in the area has focused on identifying new solution concepts that are widely applicable in general min-max games , such as ( local/differential ) Nash equilibrium [ 3 , 41 ] , local minmax [ 18 ] , local minimax [ 31 ] , ( local/differential ) Stackleberg equilibrium [ 24 ] , local robust point [ 69 ] . The plethora of solutions concepts is perhaps suggestive that “ solving '' general min-max games unequivocally may be too ambitious a task . Attraction to spurious fixed points [ 18 ] , cycles [ 65 ] , robustly chaotic behavior [ 15 , 16 ] and computational hardness issues [ 20 ] all suggest that general min-max games might inherently involve messy , unpredictable and complex behavior . Are there rich classes of non-convex non-concave games with an effectively unique game theoretic solution that is selected by standard optimization dynamics ( e.g . gradient descent ) ? Our class of games . We will define a general class of min-max optimization problems , where each agent selects its own vectors of parameters which are then processed separately by smooth functions . Each agent receives their respective payoff after entering the outputs of the processed decision vectors as inputs to a standard convex-concave game . Formally , there exist functions F : RN → X ⊂ Rn and G : RM → Y ⊂ Rm and a continuous convex-concave function L : X × Y → R , such that the min-max game is min θ∈RN max φ∈RM L ( F ( θ ) , G ( φ ) ) . ( Hidden Convex-Concave ( HCC ) ) We call this class of min-max problems Hidden Convex-Concave Games . It generalizes the recently defined hidden bilinear games of [ 65 ] . Our solution concept . Out of all the local Nash equilibria of HCC games , there exists a special subclass , the vectors ( θ∗ , φ∗ ) that implement the von Neumann solution of the convex-concave game . This solution has a strong and intuitive game theoretic justification . Indeed , it is stable even if the agents could perform arbitrary deviations directly on the output spaces X , Y . These parameter combinations ( θ∗ , φ∗ ) “ solve '' the “ hidden ” convex-concave L and thus we call them von Neumann solutions . Naturally , HCCs will typically have numerous local saddle/Nash equilibria/fixed points that do not satisfy this property . Instead , they correspond to stationary points of the F , G where their output is stuck , e.g. , due to an unfortunate initialization . At these points the agents may be receiving payoffs which can be arbitrarily smaller/larger than the game theoretic value of game L. Fortunately , we show that Gradient Descent Ascent ( GDA ) strongly favors von Neumann solutions over generic fixed points . Our results . In this work , we study the behavior of continuous GDA dynamics for the class of HCC games where each coordinate of F , G is controlled by disjoint sets of variables . In a nutshell , we show that GDA trajectories stabilize around or converge to the corresponding von Neumann solutions of the hidden game . Despite restricting our attention to a subset of HCC games , our analysis has to overcome unique hurdles not shared by standard convex concave games . Challenges of HCC games . In convex-concave games , deriving the stability of the von Neumann solutions relies on the Euclidean distance from the equilibrium being a Lyapunov function . In contrast , in HCC games where optimization happens in the parameter space of θ , φ , the non-linear nature of F , G distorts the convex-concave landscape in the output space . Thus , the Euclidean distance will not be in general a Lyapunov function . Moreover , the existence of any Lyapunov function for the trajectories in the output space of F , G does not translate to a well-defined function in the parameter space ( unless F , G are trivial , invertible maps ) . Worse yet , even if L has a unique solution in the output space , this solution could be implemented by multiple equilibria in the parameter space and thus each of them can not be individually globally attracting . Clearly any transfer of stability or convergence properties from the output to the parameter space needs to be initialization dependent . It is worth mentioning that similar challenges like transfering results from the output to the input space was also faced in the simpler class of hidden bilinear games . However , [ 65 ] to sidestep this issue assume the restricitve requirement of F , G to be invertible operators . Our results go beyond this simplified case requiring new proof techniques . Specifically , we show how to combine the powerful technologies of the the Center-Stable Manifold Theorem , typically used to argue convergence to equilibria in non-convex optimization settings [ 34 , 52 , 54 , 53 , 35 ] , along with a novel Lyapunov function argument to prove that almost all initial conditions converge to the our game theoretic solution . Lyapunov Stability . Our first step is to construct an initialization-dependent Lyapunov function that accounts for the curvature induced by the operators F and G ( Lemma 2 ) . Leveraging a potentially infinite number of initialization-dependent Lyapunov functions in Theorem 5 we prove that under mild assumptions the outputs of F , G stabilize around the von Neumann solution of L. Convergence . Mirroring convex concave games , we require strict convexity or concavity of L to provide convergence guarantees to von Neumann solutions ( Theorem 6 ) . Barring initializations where von Neumann solutions are not reachable due to the limitations imposed by F and G , the set of von Neumann solutions are globally asymptotically stable ( Corollary 1 ) . Even in non-strict HCC games , we can add regularization terms to make L strictly convex concave . Small amounts of regularization allows for convergence without significantly perturbing the von Neumann solution ( Theorem 7 ) while increasing regularization enables exponentially faster convergence rates ( Theorem 8 ) . Similar to the aforementioned theoretical work , our model of HCC games provides a formal and theoretical tractable testbed for evaluating the performance of different training methods in GAN inspired architectures . As a concrete example , [ 36 ] recently proved the success of WGAN training for learning the parameters of non-linearly transformed Gaussian distributions , where for simplicity they replaced the typical Lipschitz constraint of the discriminator function with a quadratic regularizer . Interestingly , we can elucidate on why regularized learning is actually necessary by establishing a formal connection to HCC games . On top of other such ML applications , our game theoretic framework can furthermore capture and generalize evolutionary game theoretic models . [ 57 ] analyze a model of evolutionary competition between two species ( host-parasite ) . The outcome of this competition depends on their respective phenotypes ( informally their properties , e.g. , agility , camouflage , etc. ) . These phenotypes are encoded via functions that map input vectors ( here genotype/DNA sequences ) to phenotypes . While [ 57 ] proved that learning in these games does not converge to equilibria and typically cycles for almost all initial conditions , we can explicitly construct initial conditions that do not satisfy our definition of safety and end up converging to artificial fixed points . Safety conditions aside , we show that a slight variation of the evolutionary/learning algorithm suffices to resolve the cycling issues and for the dynamics to equilibrate to the von Neumann solution . Hence , we provide the first instance of team zero-sum games [ 62 ] , a notoriously hard generalization of zero-sum games with a large duality gap , that is solvable by decentralized dynamics . Organization . In Section 2 we provide some preliminary notation , the definition of our model and some useful technical lemmas . Section 3 is devoted to the presentation of our the main results . Section 4 discusses applications of our framework to specific GAN formulations . Section 5 concludes our work with a discussion of future directions and challenges . We defer the full proofs of our results as well as further discussion on applications to the Appendix . 2 Preliminaries . 2.1 Notation . Vectors are denoted in boldface x , y unless otherwise indicated are considered as column vectors . We use ‖·‖ to denote the ` 2−norm . For a function f : Rd → R we use ∇f to denote its gradient . For functions of two vector arguments , f ( x , y ) : Rd1 × Rd2 → R , we use ∇xf , ∇yf to denote its partial gradient . For the time derivative we will use the dot accent abbreviation , i.e. , ẋ = ddt [ x ( t ) ] . A function f will belong to Cr if it is r times continuously differentiable . Additionally , f ◦ g = f ( g ( · ) ) denotes the composition of f , g. Finally , the term “ sigmoid ” function refers to σ : R→ R such that σ ( x ) = ( 1 + e−x ) −1 . 2.2 Hidden Convex Concave Games . θ11 θ12 · · · θ1n1 θ1 f1 ( θ1 ) ... θN1 θN2 · · · θNnN θN fN ( θN ) F ( θ ) L ( F ( θ ) , G ( φ ) ) φ11 φ12 · · · φ1n1 φ1 g1 ( φ1 ) ... φM1 φM2 · · · φMnM φM gM ( φM ) G ( φ ) θ̇i = −∇θiL ( F ( θ ) , G ( φ ) ) φ̇j = ∇φjL ( F ( θ ) , G ( φ ) ) Figure : Hidden Seperable Zero-Sum Game Model & Optimization Dynamics We will begin our discussion by defining the notion of convex concave functions as well as strictly convex concave functions . Note that our definition of strictly convex concave functions is a superset of strictly convex strictly concave functions that are usually studied in the literature . Definition 1 . L : Rn × Rm → R is convex concave if for every y ∈ Rn L ( · , y ) is convex and for every x ∈ Rm L ( x , · ) is concave . Function L will be called strictly convex concave if it is convex concave and for every x×y ∈ Rn×Rm either L ( · , y ) is strictly convex or L ( x , · ) is strictly concave . At the center of our definition of HCC games is a convex concave utility function L. Additionally , each player of the game is equipped with a set of operator functions . The minimization player is equipped with n functions fi : Rni → R while the maximization player is equipped with m functions gj : Rmj → R. We will assume in the rest of our discussion that fi , gj , L are all C2 functions . The inputs θi ∈ Rni and φj ∈ Rmj are grouped in two vectors θ = [ θ1 · · · θn ] > F ( θ ) = [ f1 ( θ1 ) · · · fn ( θn ) ] > φ = [ φ1 · · · φm ] > G ( φ ) = [ g1 ( φ1 ) · · · gm ( φm ) ] > We are ready to define the hidden convex concave game ( θ∗ , φ∗ ) = arg min θ∈RN arg max φ∈RM L ( F ( θ ) , G ( φ ) ) . where N = ∑n i=1 ni and M = ∑m j=1mj . Given a convex concave function L , all stationary points of L are ( global ) Nash equilibria of the min-max game . We will call the set of all equilibria of L , von Neumann solutions of L and denote them by Solution ( L ) . Unfortunately , Solution ( L ) can be empty for games defined over the entire Rn × Rm . For games defined over convex compact sets , the existence of at least one solution is guaranteed by von Neumann ’ s minimax theorem . Our definition of HCC games can capture games on restricted domains by choosing appropriately bounded functions fi and gj . In the following sections , we will just assume that Solution ( L ) is not empty . We note that our results hold for both bounded and unbounded fi and gj . We are now ready to write down the equations of the GDA dynamics for a HCC game : θ̇i = −∇θiL ( F ( θ ) , G ( φ ) ) =−∇θifi ( θi ) ∂L ∂fi ( F ( θ ) , G ( φ ) ) φ̇j = ∇φjL ( F ( θ ) , G ( φ ) ) =∇φjgj ( φj ) ∂L ∂gj ( F ( θ ) , G ( φ ) ) ( 1 )

Summary: This paper extends "hidden bilinear" games [1]  to "hidden convex-concave (HCC)" games, and studies a special case of HCC games (denoted as HCC* from below) where output of each dimension of a nonlinear function depends on a seperate group of parameters .  For HCC* games, the authors utilize Lyapunov-type arguments and show that GDA dynamics stabilizes around the output-space Nash equilibrium (Theorem 3), whose parameter counterparts are named as "von Neumann solutions".  With an additional assumption that the game has one-sided strictness property, the authors further show that GDA converges to a "von Neumann solution" (Theorem 6), by combining Lasalle's principle and the fact that "safe initializations" reside in ROA of such solutions.  Additionally, the authors show that adding a regularization term can accelerate the convergence of GDA towards the perturbed equilibrium in the output space (Theorem 8).  Finally, the authors exemplify GANs and evolutionary games, and argue that such games can be nicely formulated as HCC games.