Global Convergence  to Local Minmax Equilibrium in Classes of Nonconvex Zero-Sum Games

1 Introduction . We study continuous action zero-sum games of the form min x∈X max y∈Y f ( x , y ) where f ∈ C2 ( X × Y , R ) and X = Rd1 and Y = Rd2 denote the individual action spaces and d = d1 + d2 . In particular , we focus on unconstrained , continuous strategy space zero-sum games in which f ( · , y ) is potentially nonconvex in x ∈ X and f ( x , · ) satisfies the Polyak-Łojasiewicz ( PŁ ) condition [ 47 ] or is strongly-concave ( SC ) in y ∈ Y . We refer to these classes of games as nonconvex-PŁ and nonconvex-SC zero-sum games , respectively . This general formulation has a broad spectrum of applications such as fair classification [ 45 ] , distributionally robust optimization [ 44 , 48 , 53 ] , and adversarial training [ 38 ] . Consequently , there has been a surge of interest in recent years toward developing methods for solving these problems efficiently . So far , existing work on gradient-based learning in nonconvex-PŁ/SC zero-sum games has exclusively focused on providing global convergence rates to approximate stationary points with no attention given to the characterization in terms of game-theoretic equilibrium concepts [ 33 , 34 , 36 , 45 , 48 , 62 ] . In contrast , a common theme in the study of general nonconvex-nonconcave zero-sum games is to assess the types of stationary points an algorithm locally converges toward in terms of their higher 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . order structure [ 10 , 15 , 16 , 23 , 40 , 41 , 58 , 64 ] . The purpose of this analysis is generally to determine whether commonly deployed algorithms can guarantee local convergence to only game-theoretic equilibria or to design algorithms that achieve this objective . The goal of this paper is to close the gap between the two problem classes and determine whether gradient-based learning algorithms in nonconvex-PŁ/SC zero-sum games can be shown to globally converge to only game-theoretically meaningful equilibria ( local minmax/Stackelberg or Nash ) . We focus our attention on the canonical gradient descent-ascent learning dynamics with timescale separation between players . In this algorithm , timescale separation is manifested in the different ( yet constant ) learning rates of the maximizing and minimizing players . The description of this system , which we refer to as τ -GDA , where the ratio of learning rates denoted by τ > 0 parameterizes the timescale separation between players , is provided in Algorithms 1–2 for deterministic and stochastic settings respectively . Simply put , τ -GDA ( Algorithm 1 ) corresponds to each player following their individual gradient in a noiseless setting , while stochastic τ -GDA ( Algorithm 2 ) describes each player following their individual stochastic gradient . The τ -GDA update only requires first-order gradients and thus is a computationally efficient method for machine learning problems formulated as games . 1.1 Contributions . We show that τ -GDA has global convergence guarantees in nonconvex-PŁ/SC zero-sum games to the natural game-theoretic solution concept for this problem class of strict local minmax equilibria.1 The specific contributions of this work are now summarized . 1 ) In Theorem 1 , we prove the only critical points that are locally stable with respect to the τ -GDA continuous-time limiting system are strict local minmax equilibrium in nonconvex-PŁ/SC zerosum games . A key implication of this is that any critical point which is not a strict local minmax equilibrium is a saddle point of the continuous-time limiting system . 2 ) In Theorem 2 , we combine Theorem 1 with a potential function construction ( Lemma 1 ) and an asymptotic saddle avoidance result ( Lemma 2 ) to prove that the τ -GDA update in deterministic settings ( Algorithm 1 ) globally asymptotically converges to strict local minmax equilibria almost surely in the class of nonconvex-PŁ/SC zero-sum games . 3 ) In Corollary 1 , using the potential function from Lemma 1 , we show that τ -GDA in deterministic settings reaches an ε-critical point in Õ ( ε−2 ) steps in nonconvex-PŁ/SC zero-sum games . Moreover , specific to nonconvex-SC zero-sum games , Lemma 3 shows there exists learning rates for τ -GDA such that the cost function of the game itself can be made a potential . Corollary 2 then uses the potential function from Lemma 3 to show that τ -GDA reaches an ε-critical point in Õ ( ε−2 ) and Õ ( ε−6 ) steps in deterministic and stochastic problems , respectively . Prior to moving on , we briefly comment on and provide context for each contribution of this paper . As we discuss later on in Remark 1 , beyond its importance toward proving Theorem 2 , Theorem 1 is potentially of independent interest given the implications it also has for the local stability of τ -GDA around critical points in the more general setting of nonconvex-nonconvave zero-sum games . Furthermore , to our knowledge , Theorem 2 provides the broadest existing global convergence guarantee for gradient-based algorithms to game-theoretically meaningful equilibria in zero-sum continuous games . Finally , while there exists known convergence rates for gradient-based learning algorithms to ε-critical points in nonconvex-PŁ/SC zero-sum games , we are unaware of such a result in nonconvex-PŁ zero-sum games for τ -GDA , and the fact that we derive a rate in nonconvex-SC zero-sum games using the cost function itself as a potential function has practical implications given that it can easily be monitored when running the algorithm to evaluate progress toward a solution . 1.2 Practical Motivation . The study of nonconvex-PŁ/SC zero-sum games has often been motivated by machine learning problems formulated as games . We remark that given the problem formulations , it is natural from both game-theoretic and machine learning perspectives to seek notions of minmax equilibria . Indeed , notions of stationarity are not guaranteed to reflect a meaningful solution to the underlying problem . Consequently , it is critical to give convergence guarantees to minmax equilibrium as we pursue in 1Strict local minmax equilibria characterized by gradient-based sufficient conditions are also known as differential Stackelberg equilibria in the literature [ 15 , 16 ] and we use the terms interchangeably in this work . this work . We now provide examples from the literature of machine learning applications that are relevant to the class of games studied in this paper . Note that the following application problems are illustrative in nature and do not immediately fall into the classes of games we study . However , as is elaborated on shortly , each of the problems can be transformed into unconstrained nonconvex-PŁ/SC zero-sum games while retaining the optimization objectives after simple , standard modifications . Example 1 . In fair classification [ 45 ] and learning from multiple distributions , the objective is to minimize the maximum loss over multiple categories . An example formulation is the problem minw∈Rd maxi∈ { 1 , ... , K } ` i ( w ) where ` i ( w ) represents the loss on category i with w denoting neural network parameters . A reformulation of this problem [ 45 ] with T is the simplex in RK is given by the zero-sum game minw∈Rd maxt∈T ∑N i=1 ti ` i ( w ) . ( 1 ) Example 2 . To train a neural network classifier robust against adversarial attacks , a common approach is to formulate training as a robust minmax optimization problem of the form minw∈Rd ∑N i=1 maxδi : |δi|∞≤ ` ( xi + δi , yi , w ) , where ` ( xi + δi , yi , w ) represents the loss on sample xi perturbed by δi with budget and label yi as a function of the parameter weights w [ 38 ] . A reformulation of this problem [ 45 ] is given by minw∈Rd ∑N i=1 maxt∈T ∑K j=1 tj ` ( x̂ij , yi , w ) ( 2 ) where x̂ij is the result of a targeted attack on the sample xi seeking to change the output of the network to label j and T is the simplex in RK . Example 3 . Distributionally robust optimization often results in a zero-sum game of the form minw∈Rd maxt∈T ∑N i=1 ti ` i ( w ) − r ( t ) ( 3 ) where ` i ( x ) is the loss of a model w on the i–th data point , T is the simplex in RN , and r ( t ) is carefully selected convex regularizer [ 44 , 48 , 53 ] . The machine learning problem formulations in ( 1 ) – ( 3 ) from Examples 1–3 represent nonconvexconcave zero-sum games in which the strategy space of the minimizing player is unconstrained and the strategy space of the maximizing player is constrained to the simplex . These problems are naturally adapted to the unconstrained nonconvex-PŁ/SC zero-sum game setting considered in this paper by removing the constraint on the strategy space of the maximizing player and including a suitable PŁ/SC regularization penalty on the choice variable of the maximizing player . 1.3 Organization . The rest of the paper is organized as follows . Section 2 details related work and Section 3 presents game-theoretic , mathematical , and algorithmic preliminaries for our results . Section 4 is devoted to studying the local stability properties around critical points of the continuous-time limiting system for the τ -GDA learning dynamics . In Section 5 , we present our study of the convergence properties of τ -GDA in nonconvex-PŁ/SC zero-sum games . We conclude with a discussion in Section 6 . Finally , the supplementary material ( appendix ) contains the proofs of theoretical results . 2 Related Work . We now cover the most relevant related work with further discussion provided in Appendix A. Nonconvex-Nonconcave Zero-Sum Games . A common theme in analyzing gradient descent-ascent with or without timescale separation in nonconvex-nonconcave zero-sum games has been to assess the local stability around critical points of the continuous-time limiting system and draw connections to local Nash and Stackelberg equilibrium notions characterized by gradient-based sufficient conditions ( see Definition 4 ) [ 10 , 15 , 16 , 23 , 40 , 42 , 43 , 63 ] . Importantly , it has been shown that unless the timescale separation is chosen very carefully , the stable critical points of gradient descent-ascent may not be game-theoretically meaningful and game-theoretically meaningful critical points may not be stable [ 15 , 23 ] . We obtain stronger stability characterizations ( see Theorem 1 ) for τ -GDA in nonconvex-PŁ/SC zero-sum games using the structure imposed on the game cost function . The result of Theorem 1 also has novel implications for the local stability of τ -GDA around critical points in the more general setting of nonconvex-nonconvave zero-sum games ( see Remark 1 ) . Moreover , it is fundamental toward proving the global almost sure asymptotic convergence guarantee for τ -GDA in deterministic settings to strict local minmax equilibria in nonconvex-PŁ/SC zero-sum games ( Theorem 2 ) . Further discussion of this topic is given in Section 4 . Nonconvex-PŁ and Nonconvex-SC Zero-Sum Games . Table 1 provides a comprehensive comparison between our convergence results and existing convergence results for gradient descent variants in nonconvex optimization and nonconvex-PŁ/SC zero-sum games . We leave discussion of research on these classes of games with other algorithmic methods to Appendix A . The key distinction between this paper and past work is that instead of assessing convergence in terms of only reaching an approximate stationary point of the dynamics or a surrogate function , we obtain convergence guarantees in regards to differential Stackelberg equilibria ( equivalently strict local minmax equilibria ) . Despite this being a much stricter and meaningful notion of solving the problem , Theorem 2 shows that global asymptotic convergence guarantees for τ -GDA in deterministic settings to this type of solution remain obtainable in nonconvex-PŁ/SC zero-sum games . This is the main result of the paper . While we do not prove a rate of convergence to differential Stackelberg equilibria or an approximate notion , Corollary 1 shows τ -GDA in deterministic settings with parameter choices satisfying the conditions of Theorem 2 reaches an ε-critical point in Õ ( ε−2 ) steps in nonconvex-PŁ/SC zero-sum games . We are unaware of an existing rate for τ -GDA in deterministic settings in nonconvex-PŁ zero-sum games . Finally , as a complementary result , Lemma 3 shows there exists learning rates for τ -GDA such that the cost function of the game itself can be made a potential in nonconvex-SC zero-sum games , which is a desirable property given that this potential function can be monitored while running the algorithm . Corollary 2 then uses this potential function to show that τ -GDA reaches an ε-critical point in Õ ( ε−2 ) and Õ ( ε−6 ) steps in deterministic and stochastic nonconvex-SC zero-sum games , respectively . Escaping Saddle Points . Saddle avoidance results for variants of gradient descent in nonconvex optimization are asymptotic or finite-time . The former states that almost surely the algorithm does not converge to saddles points [ 28 , 29 ] , while the latter gives rates of escape to conclude convergence to approximate local minimum [ 17 , 20 , 24 ] . A primary assumption in the aforementioned works is what is known as the strict saddle property , which ensures directions of escape exists from a saddle point . The methods for showing gradient descent escapes saddle points almost surely in nonconvex optimization have more recently been extended to gradient descent-ascent in the setting of continuous games under an analogous strict saddle assumption [ 10 , 40 ] . A key component of proving the global asymptotic convergence guarantee to strict local minmax for τ -GDA turns out to be ensuring that the update escapes saddle points of the continuous-time limiting system almost surely . To show this property holds for τ -GDA in nonconvex-PŁ/SC zero-sum games , we are able to invoke existing saddle avoidance results for gradient-based learning in continuous games [ 40 ] .

Summary: The authors focus on the analysis of gradient descent ascent dynamics with timescale separation to solve unconstrained min-max optimization for landscapes that are non-convex on the min player (say x) and strongly concave or satisfy the PL condition for the max player. They focus on the stability analysis and convergence rates of GDA (and its perturbed variant) to the local minimax points (defined in Jin et al paper "What is locally..."). The definition of local min-max points is that the max player plays local best response and the min player chooses a local minimum for the function \max_{y \in B} f(.,y) where B is a ball of small radius. The authors show that local min-max points are stable/ attracting for GDA dynamics; the learning rate of the max player must be naturally larger than that of the min player in GDA (in some sense y updates more often). They also show convergence in \tilde{O}(1/\eps^2) of GDA to an \esp- local min-max in the case of non-convex stronly concave landscapes and non-convex, PL and \tilde{O}(1/\eps^4) of SGDA for the case of non-convex strongly concave landscapes.