Global Convergence  to Local Minmax Equilibrium in Classes of Nonconvex Zero-Sum Games

1 Introduction . We study continuous action zero-sum games of the form min x∈X max y∈Y f ( x , y ) where f ∈ C2 ( X × Y , R ) and X = Rd1 and Y = Rd2 denote the individual action spaces and d = d1 + d2 . In particular , we focus on unconstrained , continuous strategy space zero-sum games in which f ( · , y ) is potentially nonconvex in x ∈ X and f ( x , · ) satisfies the Polyak-Łojasiewicz ( PŁ ) condition [ 47 ] or is strongly-concave ( SC ) in y ∈ Y . We refer to these classes of games as nonconvex-PŁ and nonconvex-SC zero-sum games , respectively . This general formulation has a broad spectrum of applications such as fair classification [ 45 ] , distributionally robust optimization [ 44 , 48 , 53 ] , and adversarial training [ 38 ] . Consequently , there has been a surge of interest in recent years toward developing methods for solving these problems efficiently . So far , existing work on gradient-based learning in nonconvex-PŁ/SC zero-sum games has exclusively focused on providing global convergence rates to approximate stationary points with no attention given to the characterization in terms of game-theoretic equilibrium concepts [ 33 , 34 , 36 , 45 , 48 , 62 ] . In contrast , a common theme in the study of general nonconvex-nonconcave zero-sum games is to assess the types of stationary points an algorithm locally converges toward in terms of their higher 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . order structure [ 10 , 15 , 16 , 23 , 40 , 41 , 58 , 64 ] . The purpose of this analysis is generally to determine whether commonly deployed algorithms can guarantee local convergence to only game-theoretic equilibria or to design algorithms that achieve this objective . The goal of this paper is to close the gap between the two problem classes and determine whether gradient-based learning algorithms in nonconvex-PŁ/SC zero-sum games can be shown to globally converge to only game-theoretically meaningful equilibria ( local minmax/Stackelberg or Nash ) . We focus our attention on the canonical gradient descent-ascent learning dynamics with timescale separation between players . In this algorithm , timescale separation is manifested in the different ( yet constant ) learning rates of the maximizing and minimizing players . The description of this system , which we refer to as τ -GDA , where the ratio of learning rates denoted by τ > 0 parameterizes the timescale separation between players , is provided in Algorithms 1–2 for deterministic and stochastic settings respectively . Simply put , τ -GDA ( Algorithm 1 ) corresponds to each player following their individual gradient in a noiseless setting , while stochastic τ -GDA ( Algorithm 2 ) describes each player following their individual stochastic gradient . The τ -GDA update only requires first-order gradients and thus is a computationally efficient method for machine learning problems formulated as games . 1.1 Contributions . We show that τ -GDA has global convergence guarantees in nonconvex-PŁ/SC zero-sum games to the natural game-theoretic solution concept for this problem class of strict local minmax equilibria.1 The specific contributions of this work are now summarized . 1 ) In Theorem 1 , we prove the only critical points that are locally stable with respect to the τ -GDA continuous-time limiting system are strict local minmax equilibrium in nonconvex-PŁ/SC zerosum games . A key implication of this is that any critical point which is not a strict local minmax equilibrium is a saddle point of the continuous-time limiting system . 2 ) In Theorem 2 , we combine Theorem 1 with a potential function construction ( Lemma 1 ) and an asymptotic saddle avoidance result ( Lemma 2 ) to prove that the τ -GDA update in deterministic settings ( Algorithm 1 ) globally asymptotically converges to strict local minmax equilibria almost surely in the class of nonconvex-PŁ/SC zero-sum games . 3 ) In Corollary 1 , using the potential function from Lemma 1 , we show that τ -GDA in deterministic settings reaches an ε-critical point in Õ ( ε−2 ) steps in nonconvex-PŁ/SC zero-sum games . Moreover , specific to nonconvex-SC zero-sum games , Lemma 3 shows there exists learning rates for τ -GDA such that the cost function of the game itself can be made a potential . Corollary 2 then uses the potential function from Lemma 3 to show that τ -GDA reaches an ε-critical point in Õ ( ε−2 ) and Õ ( ε−6 ) steps in deterministic and stochastic problems , respectively . Prior to moving on , we briefly comment on and provide context for each contribution of this paper . As we discuss later on in Remark 1 , beyond its importance toward proving Theorem 2 , Theorem 1 is potentially of independent interest given the implications it also has for the local stability of τ -GDA around critical points in the more general setting of nonconvex-nonconvave zero-sum games . Furthermore , to our knowledge , Theorem 2 provides the broadest existing global convergence guarantee for gradient-based algorithms to game-theoretically meaningful equilibria in zero-sum continuous games . Finally , while there exists known convergence rates for gradient-based learning algorithms to ε-critical points in nonconvex-PŁ/SC zero-sum games , we are unaware of such a result in nonconvex-PŁ zero-sum games for τ -GDA , and the fact that we derive a rate in nonconvex-SC zero-sum games using the cost function itself as a potential function has practical implications given that it can easily be monitored when running the algorithm to evaluate progress toward a solution . 1.2 Practical Motivation . The study of nonconvex-PŁ/SC zero-sum games has often been motivated by machine learning problems formulated as games . We remark that given the problem formulations , it is natural from both game-theoretic and machine learning perspectives to seek notions of minmax equilibria . Indeed , notions of stationarity are not guaranteed to reflect a meaningful solution to the underlying problem . Consequently , it is critical to give convergence guarantees to minmax equilibrium as we pursue in 1Strict local minmax equilibria characterized by gradient-based sufficient conditions are also known as differential Stackelberg equilibria in the literature [ 15 , 16 ] and we use the terms interchangeably in this work . this work . We now provide examples from the literature of machine learning applications that are relevant to the class of games studied in this paper . Note that the following application problems are illustrative in nature and do not immediately fall into the classes of games we study . However , as is elaborated on shortly , each of the problems can be transformed into unconstrained nonconvex-PŁ/SC zero-sum games while retaining the optimization objectives after simple , standard modifications . Example 1 . In fair classification [ 45 ] and learning from multiple distributions , the objective is to minimize the maximum loss over multiple categories . An example formulation is the problem minw∈Rd maxi∈ { 1 , ... , K } ` i ( w ) where ` i ( w ) represents the loss on category i with w denoting neural network parameters . A reformulation of this problem [ 45 ] with T is the simplex in RK is given by the zero-sum game minw∈Rd maxt∈T ∑N i=1 ti ` i ( w ) . ( 1 ) Example 2 . To train a neural network classifier robust against adversarial attacks , a common approach is to formulate training as a robust minmax optimization problem of the form minw∈Rd ∑N i=1 maxδi : |δi|∞≤ ` ( xi + δi , yi , w ) , where ` ( xi + δi , yi , w ) represents the loss on sample xi perturbed by δi with budget and label yi as a function of the parameter weights w [ 38 ] . A reformulation of this problem [ 45 ] is given by minw∈Rd ∑N i=1 maxt∈T ∑K j=1 tj ` ( x̂ij , yi , w ) ( 2 ) where x̂ij is the result of a targeted attack on the sample xi seeking to change the output of the network to label j and T is the simplex in RK . Example 3 . Distributionally robust optimization often results in a zero-sum game of the form minw∈Rd maxt∈T ∑N i=1 ti ` i ( w ) − r ( t ) ( 3 ) where ` i ( x ) is the loss of a model w on the i–th data point , T is the simplex in RN , and r ( t ) is carefully selected convex regularizer [ 44 , 48 , 53 ] . The machine learning problem formulations in ( 1 ) – ( 3 ) from Examples 1–3 represent nonconvexconcave zero-sum games in which the strategy space of the minimizing player is unconstrained and the strategy space of the maximizing player is constrained to the simplex . These problems are naturally adapted to the unconstrained nonconvex-PŁ/SC zero-sum game setting considered in this paper by removing the constraint on the strategy space of the maximizing player and including a suitable PŁ/SC regularization penalty on the choice variable of the maximizing player . 1.3 Organization . The rest of the paper is organized as follows . Section 2 details related work and Section 3 presents game-theoretic , mathematical , and algorithmic preliminaries for our results . Section 4 is devoted to studying the local stability properties around critical points of the continuous-time limiting system for the τ -GDA learning dynamics . In Section 5 , we present our study of the convergence properties of τ -GDA in nonconvex-PŁ/SC zero-sum games . We conclude with a discussion in Section 6 . Finally , the supplementary material ( appendix ) contains the proofs of theoretical results . 2 Related Work . We now cover the most relevant related work with further discussion provided in Appendix A. Nonconvex-Nonconcave Zero-Sum Games . A common theme in analyzing gradient descent-ascent with or without timescale separation in nonconvex-nonconcave zero-sum games has been to assess the local stability around critical points of the continuous-time limiting system and draw connections to local Nash and Stackelberg equilibrium notions characterized by gradient-based sufficient conditions ( see Definition 4 ) [ 10 , 15 , 16 , 23 , 40 , 42 , 43 , 63 ] . Importantly , it has been shown that unless the timescale separation is chosen very carefully , the stable critical points of gradient descent-ascent may not be game-theoretically meaningful and game-theoretically meaningful critical points may not be stable [ 15 , 23 ] . We obtain stronger stability characterizations ( see Theorem 1 ) for τ -GDA in nonconvex-PŁ/SC zero-sum games using the structure imposed on the game cost function . The result of Theorem 1 also has novel implications for the local stability of τ -GDA around critical points in the more general setting of nonconvex-nonconvave zero-sum games ( see Remark 1 ) . Moreover , it is fundamental toward proving the global almost sure asymptotic convergence guarantee for τ -GDA in deterministic settings to strict local minmax equilibria in nonconvex-PŁ/SC zero-sum games ( Theorem 2 ) . Further discussion of this topic is given in Section 4 . Nonconvex-PŁ and Nonconvex-SC Zero-Sum Games . Table 1 provides a comprehensive comparison between our convergence results and existing convergence results for gradient descent variants in nonconvex optimization and nonconvex-PŁ/SC zero-sum games . We leave discussion of research on these classes of games with other algorithmic methods to Appendix A . The key distinction between this paper and past work is that instead of assessing convergence in terms of only reaching an approximate stationary point of the dynamics or a surrogate function , we obtain convergence guarantees in regards to differential Stackelberg equilibria ( equivalently strict local minmax equilibria ) . Despite this being a much stricter and meaningful notion of solving the problem , Theorem 2 shows that global asymptotic convergence guarantees for τ -GDA in deterministic settings to this type of solution remain obtainable in nonconvex-PŁ/SC zero-sum games . This is the main result of the paper . While we do not prove a rate of convergence to differential Stackelberg equilibria or an approximate notion , Corollary 1 shows τ -GDA in deterministic settings with parameter choices satisfying the conditions of Theorem 2 reaches an ε-critical point in Õ ( ε−2 ) steps in nonconvex-PŁ/SC zero-sum games . We are unaware of an existing rate for τ -GDA in deterministic settings in nonconvex-PŁ zero-sum games . Finally , as a complementary result , Lemma 3 shows there exists learning rates for τ -GDA such that the cost function of the game itself can be made a potential in nonconvex-SC zero-sum games , which is a desirable property given that this potential function can be monitored while running the algorithm . Corollary 2 then uses this potential function to show that τ -GDA reaches an ε-critical point in Õ ( ε−2 ) and Õ ( ε−6 ) steps in deterministic and stochastic nonconvex-SC zero-sum games , respectively . Escaping Saddle Points . Saddle avoidance results for variants of gradient descent in nonconvex optimization are asymptotic or finite-time . The former states that almost surely the algorithm does not converge to saddles points [ 28 , 29 ] , while the latter gives rates of escape to conclude convergence to approximate local minimum [ 17 , 20 , 24 ] . A primary assumption in the aforementioned works is what is known as the strict saddle property , which ensures directions of escape exists from a saddle point . The methods for showing gradient descent escapes saddle points almost surely in nonconvex optimization have more recently been extended to gradient descent-ascent in the setting of continuous games under an analogous strict saddle assumption [ 10 , 40 ] . A key component of proving the global asymptotic convergence guarantee to strict local minmax for τ -GDA turns out to be ensuring that the update escapes saddle points of the continuous-time limiting system almost surely . To show this property holds for τ -GDA in nonconvex-PŁ/SC zero-sum games , we are able to invoke existing saddle avoidance results for gradient-based learning in continuous games [ 40 ] .

Summary: This paper studies gradient descent ascent learning dynamics with timescale separation in unconstrained continuous action zero-sum games. The minimizing player faces a nonconvex optimization problem, and the maximizing player optimizes a Polyak-Łojasiewicz (PŁ) or strongly-concave (SC) objective. Like [15], this paper assesses convergence to Stackelberg equilibria instead of only notions of stationary. A differential Stackelberg equilibrium corresponds to a joint strategy at which the minimizing player is at a local optimum to its choice variable along the best response curve of the maximizing player, and the maximizing player is at a local optimum concerning its choice variable. In pursuit of this goal, the authors prove that the only locally stable points of the continuous-time limiting system correspond to strict local Stackelberg equilibria in nonconvex-PŁ and nonconvex-SC games. Specifically, this paper exploits timescale separation to construct a potential function of the nonconvex-PŁ game that gives a global asymptotic almost-sure convergence guarantee to the set of strict local Stackelberg equilibria with the stability and asymptotic saddle avoidance. Moreover, this paper shows that nonconvex-SC games do gradient descent ascent provably converge in an almost sure asymptotic sense and a finite time escape time from saddle points.  Overall, this paper gives some theoretical contributions, with the caveat for the clarifications of the assumptions and the shortage of empirical studies.