Solving Min-Max Optimization with Hidden Structure via Gradient Descent Ascent

1 Introduction . Traditionally , our understanding of convex-concave games revolves around von Neumann ’ s celebrated minimax theorem , which implies the existence of saddle point solutions with a uniquely defined value . These solutions are called von Nemann solutions and guarantee to each agent their corresponding value regardless of opponent play . Although many learning algorithms are known to be able to compute such saddle points [ 13 ] , recently there has there has been a fervor of activity in proving stronger results such as faster regret minimization rates or analysis of the day-to-day behavior [ 46 , 17 , 7 , 1 , 66 , 19 , 2 , 45 , 5 , 25 , 70 , 29 , 6 , 48 , 30 , 56 ] . This interest has been largely triggered by the impressive successes of AI architectures inspired by min-max games such as Generative Adversarial Networks ( GANS ) [ 26 ] , adversarial training [ 40 ] and reinforcement learning self-play in games [ 63 ] . Critically , however , all these applications are based upon non-convex non-concave games , our understanding of which is still nascent . Nevertheless , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . some important early work in the area has focused on identifying new solution concepts that are widely applicable in general min-max games , such as ( local/differential ) Nash equilibrium [ 3 , 41 ] , local minmax [ 18 ] , local minimax [ 31 ] , ( local/differential ) Stackleberg equilibrium [ 24 ] , local robust point [ 69 ] . The plethora of solutions concepts is perhaps suggestive that “ solving '' general min-max games unequivocally may be too ambitious a task . Attraction to spurious fixed points [ 18 ] , cycles [ 65 ] , robustly chaotic behavior [ 15 , 16 ] and computational hardness issues [ 20 ] all suggest that general min-max games might inherently involve messy , unpredictable and complex behavior . Are there rich classes of non-convex non-concave games with an effectively unique game theoretic solution that is selected by standard optimization dynamics ( e.g . gradient descent ) ? Our class of games . We will define a general class of min-max optimization problems , where each agent selects its own vectors of parameters which are then processed separately by smooth functions . Each agent receives their respective payoff after entering the outputs of the processed decision vectors as inputs to a standard convex-concave game . Formally , there exist functions F : RN → X ⊂ Rn and G : RM → Y ⊂ Rm and a continuous convex-concave function L : X × Y → R , such that the min-max game is min θ∈RN max φ∈RM L ( F ( θ ) , G ( φ ) ) . ( Hidden Convex-Concave ( HCC ) ) We call this class of min-max problems Hidden Convex-Concave Games . It generalizes the recently defined hidden bilinear games of [ 65 ] . Our solution concept . Out of all the local Nash equilibria of HCC games , there exists a special subclass , the vectors ( θ∗ , φ∗ ) that implement the von Neumann solution of the convex-concave game . This solution has a strong and intuitive game theoretic justification . Indeed , it is stable even if the agents could perform arbitrary deviations directly on the output spaces X , Y . These parameter combinations ( θ∗ , φ∗ ) “ solve '' the “ hidden ” convex-concave L and thus we call them von Neumann solutions . Naturally , HCCs will typically have numerous local saddle/Nash equilibria/fixed points that do not satisfy this property . Instead , they correspond to stationary points of the F , G where their output is stuck , e.g. , due to an unfortunate initialization . At these points the agents may be receiving payoffs which can be arbitrarily smaller/larger than the game theoretic value of game L. Fortunately , we show that Gradient Descent Ascent ( GDA ) strongly favors von Neumann solutions over generic fixed points . Our results . In this work , we study the behavior of continuous GDA dynamics for the class of HCC games where each coordinate of F , G is controlled by disjoint sets of variables . In a nutshell , we show that GDA trajectories stabilize around or converge to the corresponding von Neumann solutions of the hidden game . Despite restricting our attention to a subset of HCC games , our analysis has to overcome unique hurdles not shared by standard convex concave games . Challenges of HCC games . In convex-concave games , deriving the stability of the von Neumann solutions relies on the Euclidean distance from the equilibrium being a Lyapunov function . In contrast , in HCC games where optimization happens in the parameter space of θ , φ , the non-linear nature of F , G distorts the convex-concave landscape in the output space . Thus , the Euclidean distance will not be in general a Lyapunov function . Moreover , the existence of any Lyapunov function for the trajectories in the output space of F , G does not translate to a well-defined function in the parameter space ( unless F , G are trivial , invertible maps ) . Worse yet , even if L has a unique solution in the output space , this solution could be implemented by multiple equilibria in the parameter space and thus each of them can not be individually globally attracting . Clearly any transfer of stability or convergence properties from the output to the parameter space needs to be initialization dependent . It is worth mentioning that similar challenges like transfering results from the output to the input space was also faced in the simpler class of hidden bilinear games . However , [ 65 ] to sidestep this issue assume the restricitve requirement of F , G to be invertible operators . Our results go beyond this simplified case requiring new proof techniques . Specifically , we show how to combine the powerful technologies of the the Center-Stable Manifold Theorem , typically used to argue convergence to equilibria in non-convex optimization settings [ 34 , 52 , 54 , 53 , 35 ] , along with a novel Lyapunov function argument to prove that almost all initial conditions converge to the our game theoretic solution . Lyapunov Stability . Our first step is to construct an initialization-dependent Lyapunov function that accounts for the curvature induced by the operators F and G ( Lemma 2 ) . Leveraging a potentially infinite number of initialization-dependent Lyapunov functions in Theorem 5 we prove that under mild assumptions the outputs of F , G stabilize around the von Neumann solution of L. Convergence . Mirroring convex concave games , we require strict convexity or concavity of L to provide convergence guarantees to von Neumann solutions ( Theorem 6 ) . Barring initializations where von Neumann solutions are not reachable due to the limitations imposed by F and G , the set of von Neumann solutions are globally asymptotically stable ( Corollary 1 ) . Even in non-strict HCC games , we can add regularization terms to make L strictly convex concave . Small amounts of regularization allows for convergence without significantly perturbing the von Neumann solution ( Theorem 7 ) while increasing regularization enables exponentially faster convergence rates ( Theorem 8 ) . Similar to the aforementioned theoretical work , our model of HCC games provides a formal and theoretical tractable testbed for evaluating the performance of different training methods in GAN inspired architectures . As a concrete example , [ 36 ] recently proved the success of WGAN training for learning the parameters of non-linearly transformed Gaussian distributions , where for simplicity they replaced the typical Lipschitz constraint of the discriminator function with a quadratic regularizer . Interestingly , we can elucidate on why regularized learning is actually necessary by establishing a formal connection to HCC games . On top of other such ML applications , our game theoretic framework can furthermore capture and generalize evolutionary game theoretic models . [ 57 ] analyze a model of evolutionary competition between two species ( host-parasite ) . The outcome of this competition depends on their respective phenotypes ( informally their properties , e.g. , agility , camouflage , etc. ) . These phenotypes are encoded via functions that map input vectors ( here genotype/DNA sequences ) to phenotypes . While [ 57 ] proved that learning in these games does not converge to equilibria and typically cycles for almost all initial conditions , we can explicitly construct initial conditions that do not satisfy our definition of safety and end up converging to artificial fixed points . Safety conditions aside , we show that a slight variation of the evolutionary/learning algorithm suffices to resolve the cycling issues and for the dynamics to equilibrate to the von Neumann solution . Hence , we provide the first instance of team zero-sum games [ 62 ] , a notoriously hard generalization of zero-sum games with a large duality gap , that is solvable by decentralized dynamics . Organization . In Section 2 we provide some preliminary notation , the definition of our model and some useful technical lemmas . Section 3 is devoted to the presentation of our the main results . Section 4 discusses applications of our framework to specific GAN formulations . Section 5 concludes our work with a discussion of future directions and challenges . We defer the full proofs of our results as well as further discussion on applications to the Appendix . 2 Preliminaries . 2.1 Notation . Vectors are denoted in boldface x , y unless otherwise indicated are considered as column vectors . We use ‖·‖ to denote the ` 2−norm . For a function f : Rd → R we use ∇f to denote its gradient . For functions of two vector arguments , f ( x , y ) : Rd1 × Rd2 → R , we use ∇xf , ∇yf to denote its partial gradient . For the time derivative we will use the dot accent abbreviation , i.e. , ẋ = ddt [ x ( t ) ] . A function f will belong to Cr if it is r times continuously differentiable . Additionally , f ◦ g = f ( g ( · ) ) denotes the composition of f , g. Finally , the term “ sigmoid ” function refers to σ : R→ R such that σ ( x ) = ( 1 + e−x ) −1 . 2.2 Hidden Convex Concave Games . θ11 θ12 · · · θ1n1 θ1 f1 ( θ1 ) ... θN1 θN2 · · · θNnN θN fN ( θN ) F ( θ ) L ( F ( θ ) , G ( φ ) ) φ11 φ12 · · · φ1n1 φ1 g1 ( φ1 ) ... φM1 φM2 · · · φMnM φM gM ( φM ) G ( φ ) θ̇i = −∇θiL ( F ( θ ) , G ( φ ) ) φ̇j = ∇φjL ( F ( θ ) , G ( φ ) ) Figure : Hidden Seperable Zero-Sum Game Model & Optimization Dynamics We will begin our discussion by defining the notion of convex concave functions as well as strictly convex concave functions . Note that our definition of strictly convex concave functions is a superset of strictly convex strictly concave functions that are usually studied in the literature . Definition 1 . L : Rn × Rm → R is convex concave if for every y ∈ Rn L ( · , y ) is convex and for every x ∈ Rm L ( x , · ) is concave . Function L will be called strictly convex concave if it is convex concave and for every x×y ∈ Rn×Rm either L ( · , y ) is strictly convex or L ( x , · ) is strictly concave . At the center of our definition of HCC games is a convex concave utility function L. Additionally , each player of the game is equipped with a set of operator functions . The minimization player is equipped with n functions fi : Rni → R while the maximization player is equipped with m functions gj : Rmj → R. We will assume in the rest of our discussion that fi , gj , L are all C2 functions . The inputs θi ∈ Rni and φj ∈ Rmj are grouped in two vectors θ = [ θ1 · · · θn ] > F ( θ ) = [ f1 ( θ1 ) · · · fn ( θn ) ] > φ = [ φ1 · · · φm ] > G ( φ ) = [ g1 ( φ1 ) · · · gm ( φm ) ] > We are ready to define the hidden convex concave game ( θ∗ , φ∗ ) = arg min θ∈RN arg max φ∈RM L ( F ( θ ) , G ( φ ) ) . where N = ∑n i=1 ni and M = ∑m j=1mj . Given a convex concave function L , all stationary points of L are ( global ) Nash equilibria of the min-max game . We will call the set of all equilibria of L , von Neumann solutions of L and denote them by Solution ( L ) . Unfortunately , Solution ( L ) can be empty for games defined over the entire Rn × Rm . For games defined over convex compact sets , the existence of at least one solution is guaranteed by von Neumann ’ s minimax theorem . Our definition of HCC games can capture games on restricted domains by choosing appropriately bounded functions fi and gj . In the following sections , we will just assume that Solution ( L ) is not empty . We note that our results hold for both bounded and unbounded fi and gj . We are now ready to write down the equations of the GDA dynamics for a HCC game : θ̇i = −∇θiL ( F ( θ ) , G ( φ ) ) =−∇θifi ( θi ) ∂L ∂fi ( F ( θ ) , G ( φ ) ) φ̇j = ∇φjL ( F ( θ ) , G ( φ ) ) =∇φjgj ( φj ) ∂L ∂gj ( F ( θ ) , G ( φ ) ) ( 1 )

Summary: In this paper, the authors study a two-player zero-sum convex-concave game whose inputs are parameterized by nonlinear functions. This results in the overall game being nonconvex-nonconcave in the parameter space. This setting is similar to that observed in GANs where the game is convex-concave in function space of discriminator and generator but is nonconvex-nonconcave in the space of parameters. The authors call such games as Hidden Convex-Concave (HCC) games.