Title: Local Convergence of Gradient Methods for Min-Max Games: Partial Curvature Generically Suffices
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gradient methods

min-max optimization

spectral analysis

last-iterate convergence
Abstract: We study the convergence to local Nash equilibria of gradient methods for two-player zero-sum differentiable games.
It is well-known that, in the continuous-time setting, such dynamics converge locally when $S \succ 0$ and may diverge when $S=0$, where $S\succeq 0$ is the symmetric part of the Jacobian at equilibrium that accounts for the "potential" component of the game. We show that these dynamics also converge as soon as $S$ is nonzero (*partial curvature*) and the eigenvectors of the antisymmetric part $A$ are in general position with respect to the kernel of $S$.
We then study the convergence rate when $S \ll A$ and prove that it typically depends on the *average* of the eigenvalues of $S$, instead of the minimum as an analogy with minimization problems would suggest.
To illustrate our results, we consider the problem of computing mixed Nash equilibria of continuous games. We show that, thanks to partial curvature, conic particle methods -- which optimize over both weights and supports of the mixed strategies -- generically converge faster than fixed-support methods.
For min-max games, it is thus beneficial to add degrees of freedom "with curvature": this can be interpreted as yet another benefit of over-parameterization.

Local Convergence of Gradient Methods for Min-Max Games: Partial Curvature Generically Suffices

Abstract

We study the convergence to local Nash equilibria of gradient methods for twoplayer zero-sum differentiable games. It is well-known that such dynamics converge locally when S ≻ 0 and may diverge when S = 0 , where S ⪰ 0 is the symmetric part of the Jacobian at equilibrium that accounts for the 'potential' component of the game. We show that these dynamics also converge as soon as S is nonzero ( partial curvature ) and the eigenvectors of the antisymmetric part A are in general position with respect to the kernel of S . We then study the convergence rates when S ≪ A and prove that they typically depend on the average of the eigenvalues of S , instead of the minimum as an analogy with minimization problems would suggest. To illustrate our results, we consider the problem of computing mixed Nash equilibria of continuous games. We show that, thanks to partial curvature, conic particle methods - which optimize over both weights and supports of the mixed strategies - generically converge faster than fixed-support methods. For min-max games, it is thus beneficial to add degrees of freedom 'with curvature': this can be interpreted as yet another benefit of over-parameterization.

1 Introduction

Min-max optimization is notoriously subtler than minimization, even in convex-concave settings. While many of the proof techniques for minimization have a natural equivalent in the min-max world, some common intuitions fail to transfer. The picture is clear for strongly convex-strongly concave (SC-SC) min-max games: all the classical gradient methods converge exponentially (for small enough step-sizes) with worst-case convergence rates dependent on the strong convexity and strong concavity parameters µ x , µ y . But, most famously perhaps, for bilinear min-max games the last iterate of simultaneous Gradient Descent-Ascent (GDA) diverges, while the Proximal Point (PP) method converges, and alternating GDA and the continuous-time limit of all of these algorithms Gradient Flow (GF) - exhibit a cycling behavior.

Based on those two extreme cases, it still seems that part of our intuition from minimization, where the last-iterate convergence rate is indeed determined by the strong convexity parameter, is preserved. In this paper, we argue that this intuition is in fact overly pessimistic, and that gradient methods behave in general more favorably in the min-max setting than in the minimization setting.

Let us first look at a basic example. Consider a bilinear min-max game regularized by a quadratic term only in one scalar variable :

[x, y]

[x,y5]

α=0.01

α=0.1

α=0.4

10-1

rln(observed)

pM

10-2

10-3

10-4

10-5

10-6

10-3

10-2

10-1

100

101

α

α=10-3

10-2

2

+

10-1

1

100

(V)C

0

-1

-2

10-4

10-3

10-2

10-1

100

101

R(A)

Here there is no strong convexity in either player (so µ x = µ y = 0 ), yet as observed on Fig. 1a, Fig. 1b, when P ∈ R d × d is random with independent standard normal entries, GDA (with a small step-size) typically converges at an exponential rate that scales linearly with α . 1

As we will see, this phenomenon is a consequence of the existing theory for general smooth min-max games

with a local Nash equilibrium (NE), or local saddle point, z ∗ = ( x ∗ , y ∗ ) . It is well-known from the dynamical systems literature that GF converges locally to z ∗ if

the Jacobian of the skewed gradient field at z ∗ , and where Sp( · ) denotes the spectrum, i.e., the set of eigenvalues. For our starting basic example, this quantity 2 can be visualized in Fig. 1c. Moreover, ˜ µ M also characterizes the convergence behavior of gradient methods in the leading order in the step-size η : they all converge to z ∗ with the rate η ˜ µ M + O ( η 2 ) , see the Appendix for a review. 3

While the condition ˜ µ M > 0 is very general and tight, it is not obvious how to control or interpret it. Our purpose with this paper is to explore what this condition entails and to emphasize some of its surprising consequences.

The rest of the paper is organized as follows. In Sec. 2 we show that, generically, ˜ µ M > 0 as long as the problem has partial curvature , i.e. the diagonal blocks of M are non-zero. In Sec. 3, we study more precisely the case of games where the 'interaction' component dominates the 'potential' component, i.e., when ∇ 2 xx f ( z ∗ ) , ∇ 2 yy f ( z ∗ ) ≪ ∇ 2 xy f ( z ∗ ) . For such games, in a certain random setting, we show that ˜ µ M scales as the average of the eigenvalues of the potential part, instead of the minimum as an analogy with minimization problems would suggest. In Sec. 4 we consider the computation of mixed Nash equilibria of continuous games using particle methods, a setting where the convergence under partial curvature has a striking consequence. By optimizing over both the weights and supports of the mixed strategies, one obtains dynamics that generically converge faster than fixed-support methods, even when the latter are using the optimal supports.

For ease of exposition, in most of the paper we focus on the local convergence of GF. We discuss the convergence of discrete time algorithms in Sec. 3.2.

1.1 Related work

Throughout this paper, by 'convergence' we mean convergence of the last iterate to a (local) NE, while a different line of work also considers convergence of the averaged iterate [Nem04], and another considers convergence to any critical point [ALW21]. Also to prevent possible confusion, let us mention a related but distinct line of work [DH19; Doa22] that considers using different step-sizes to update the x and the y variables. The resulting two-timescale dynamics can be analyzed globally assuming a min-max analog of the Polyak-Łojasiewicz inequality, using quite different considerations than the ones developed in this paper.

Analysis of GDA and PP for SC-SC, bilinear or convex-SC games. The fact that PP converges for SC-SC and bilinear min-max games is well-established since at least [Roc76]; for a modern reformulation, see e.g. [MOP20]. The convergence of simultaneous GDA for SC-SC games and its divergence for bilinear games is also a classical fact, see e.g. [LS19; Lu22]. The cycling behavior of alternating GDA for bilinear games is proved in [BGP20, Theorem 4].

As shown more recently in [NK17, Appendix G] and [ZWLG22, Theorem 6], convexity in x , strong concavity in y and non-degeneracy of the interaction component are in fact sufficient to ensure ˜ µ M > 0 , and so local convergence of GDA. The setting of the former work corresponds to ∇ 2 xx f ( z ∗ ) = 0 , -∇ 2 yy f ( z ∗ ) ≻ 0 and ∇ 2 xy f ( z ∗ ) full-row-rank, extended in the latter work to ∇ 2 xx f ( z ∗ ) ⪰ 0 . Those two works also provide bounds on ˜ µ M for those specific cases in terms of the least eigenvalues of -∇ 2 yy f ( z ∗ ) and of ∇ 2 xy f ( z ∗ ) ∇ 2 xy f ( z ∗ ) ⊤ .

Spectral analysis-based convergence analyses. Our approach to analyzing min-max algorithms is to directly study the properties of the update operator (e.g., of T ( z ) = z -( ∇ x f -∇ y f ) ( z ) for simultaneous GDA). Specifically, by a classical result on discrete-time dynamical systems, the local exponential convergence is characterized by the spectral radius of that operator's Jacobian. This approach is used for example in [Gid+19] to analyze the local convergence behavior of alternating GDA with negative momentum, and in [AMLG20] to derive tight convergence bounds - dependent explicitly on η and Sp( M ) - for simultaneous GDA and multi-step Extra-Gradient.

Average-case analysis of the (local) convergence rate. In [PS20; DPS21], the authors analyze the convergence of gradient methods for affine operator problems - i.e., for finding z such that M ( z -z ∗ ) = 0 - when M is a random normal matrix with a known spectral distribution. They derive average-case optimal methods for this setting. Our analysis in Sec. 3 also has an average-case flavor, but our random model is different: we assume that the symmetric S and antisymmetric parts A of M are independent and that S ≪ A . In particular we do not require M to be normal (i.e., S and A do not commute a priori, which is the typical case in min-max optimization).

Hypocoercivity. In the context of partial differential equations (PDEs), the phenomenon that a linear PDE ∂ t u t = -Lu t may exhibit linear convergence to 0 even when the generator L is not coercive, is called hypocoercivity and is studied in detail in [Vil09]. This is precisely the infinite-dimensional analog of the phenomenon studied in the present paper: still denoting by S, A the symmetric resp. antisymmetric parts of M , coercivity of L corresponds to S ≻ 0 , while hypocoercivity corresponds to

˜ µ M > 0 . Specifically, [Vil09] shows how to construct Lyapunov functions to establish hypocoercivity and convergence rates of certain PDEs, by exploiting properties of the iterated commutators of S and A . By contrast our focus is on the finite-dimensional case, where it is easier and more natural to directly study the spectrum of M .

1.2 Notation

For any matrix T ∈ R d × d or C d × d , denote by Sp( T ) ⊂ C its spectrum, i.e., the set of its eigenvalues, and by ρ ( T ) = max λ ∈ Sp( T ) | λ | its spectral radius. Recall that the spectral radius is distinct from the operator norm, which is the largest singular value, although they coincide for Hermitian matrices. Denote eigenspaces as E λ ( T ) = { z ∈ C d ; Tz = λz } and let Eigvecs( T ) = ⋃ λ ∈ Sp( T ) E λ ( T ) the set of all (complex) eigenvectors. ∥·∥ denotes Euclidean or Hermitian norm. For a collection of square matrices (resp. scalars) ( C k ) k , Diag(( C k )) denotes the (block-)diagonal matrix with blocks (resp. coefficients) ( C k ) k .

2 Characterization of local convergence

Consider the general smooth min-max game of Equation (1.1) and assume M is invertible. We decompose M into its symmetric and antisymmetric parts as

By the second-order optimality condition in the definition of NE, Q ⪰ 0 and R ⪰ 0 .

Following [Let+19], S and A can be thought of intuitively as the 'potential' resp. 'interaction' (or Hamiltonian) components of the two-player zero-sum game (1.1). Indeed, consider the quadratic

essentially sufficient to understand the local behavior of GF for (1.1) around z ∗ . Then we can interpret the terms in Q and R as quadratic potentials to be optimized independently by each player, with the bilinear term in P capturing all of the interaction between the players.

Partial curvature generically suffices. Let us recall that ˜ µ M = min λ ∈ Sp( M ) ℜ ( λ ) governs the local convergence of GF around the local NE z ∗ . It is a general fact that this quantity is nonnegative (the proof is included below). In the following theorem, we give necessary and sufficient conditions for it to be positive, in terms of geometric conditions on Q,R and P .

Theorem 2.1. The following conditions are equivalent:

( i ) ˜ µ M > 0 .

( ii ) Eigvecs( A ) ∩ Ker S = { 0 } .

( iii ) For any eigenvector x of PP ⊤ (i.e., left-singular vector of P ), x ̸∈ Ker Q or P ⊤ x ̸∈ Ker R .

( iv ) For any eigenvector y of P ⊤ P (i.e., right-singular vector of P ), Py ̸∈ Ker Q or y ̸∈ Ker R .

̸

As a consequence of Thm. 2.1, the condition ˜ µ M > 0 holds generically in the following sense: for any fixed S = 0 , the set of matrices P such that ˜ µ M > 0 is dense and open in R n × m . In particular, this property holds with probability 1 if P is drawn from an absolutely continuous distribution and is independent from Q and R , as in the experiment of Fig. 1.

Proof. Let λ ∈ Sp( M ) and z ∈ C n + m non-zero such that Mz = ( S + A ) z = λz . Since S and A are real and symmetric resp. antisymmetric,

So by taking the real part in z ⊤ ( S + A ) z = λ ∥ z ∥ 2 ,

since S = Diag( Q,R ) ⪰ 0 . This shows that ˜ µ M ≥ 0 .

Now let us show the equivalence of the conditions.

( ii ) = ⇒ ( i ) : By contraposition, suppose there exists λ ∈ Sp( M ) with ℜ ( λ ) = 0 , and let z non-zero such that Mz = λz . Then z ⊤ Sz = ℜ ( λ ) ∥ z ∥ 2 = 0 , so z ∈ Ker S . So Mz = Az = λz , and z ∈ ( E λ ( A ) ∩ Ker S ) \ { 0 } .

( i ) = ⇒ ( ii ) : By contraposition, suppose there exists λ ∈ Sp( A ) and a non-zero z ∈ E λ ( A ) ∩ Ker S . Since A is antisymmetric, then ℜ ( λ ) = 0 . On the other hand, Mz = ( S + A ) z = Az = λz , i.e., λ ∈ Sp( M ) . So ˜ µ M ≤ ℜ ( λ ) = 0 .

( i ) = ⇒ ( iii ) , ( iv ) : By contraposition, suppose there exists an eigenvector x of PP ⊤ such that x ∈ Ker Q and P ⊤ x ∈ Ker R , and denote σ ∈ R such that PP ⊤ x = σ 2 x (since PP ⊤ ⪰ 0 ). Then

and so -iσ ∈ Sp( M ) . This shows ( i ) = ⇒ ( iii ) , and ( i ) = ⇒ ( iv ) follows analogously.

( iii ) , ( iv ) = ⇒ ( ii ) : By contraposition, suppose there exists λ = iσ ∈ Sp( A ) and a non-zero z = ( x, y ) ∈ E λ ( A ) ∩ Ker S . Expanding the blocks in Az = λz ,

̸

Moreover, this implies that x = 0 ⇐⇒ y = 0 , and since z = 0 , then both x = 0 and y = 0 . So x is an eigenvector of PP ⊤ and, since Sz = 0 , x ∈ Ker Q and P ⊤ x = -iσy ∈ Ker R , which contradicts ( iii ) . Likewise, y is an eigenvector of P ⊤ P and y ∈ Ker R and Py = iσx ∈ Ker Q , which contradicts ( iv ) .

̸

Geometric interpretation using real vectors. We draw the attention of the reader to the fact that ( ii ) involves complex eigenvectors. For a rephrasing in terms of real objects, note that if

span( ℜ ( z ) , ℑ ( z )) is a 'rotation plane' of the GF for the bilinear game min x max y x ⊤ Py , in the sense that the projection of GF on F iσ ( A ) is a circular motion with constant speed σ . Condition ( ii ) expresses that for each such F iσ ( A ) , there exists an eigenspace E µ ( S ) of S (for a µ > 0 ) that is not orthogonal to it. This causes the GF for min x max y 1 2 x ⊤ Qx -1 2 y ⊤ Ry + x ⊤ Py projected on F iσ ( A ) to spiral down to 0 instead of cycling around it.

One may naturally wonder whether a notion of non-orthogonality between the potential ( S ) and interaction components ( A ) can be used to bound the convergence quantitatively; this is developed in the next section in the particular case S ≪ A .

3 Convergence rate when interaction dominates ( S ≪ A )

Let us now discuss the case of games with a small symmetric part, that is, whose Jacobian at optimum is M α = A + αS for some symmetric S = [ Q 0 0 ⊤ R ] , antisymmetric A = [ 0 P -P ⊤ 0 ] and some small α > 0 . In this section we assume n = m (the general case is technically more challenging as Prop. 3.1 requires A to have distinct eigenvalues, which requires | n -m | ≤ 1 ).

3.1 Convergence rate of Gradient Flow

As discussed previously, the normalized local exponential convergence rate r/η of gradient methods in the asymptotic regime η → 0 - or equivalently, the convergence rate of GF - for a game with Jacobian at optimum M α is equal to ˜ µ M α . We can estimate this quantity using the standard formula for the asymptotic expansion of the eigenvalues of a perturbed matrix, which takes an interesting form in our context.

̸

Proposition 3.1. Suppose that P is full-rank and has distinct singular values, and let P = U Σ V ⊤ = ∑ n j =1 σ j u j v ⊤ j be its singular value decomposition. Then it holds

This expansion explains in particular why the normalized convergence rate r/η ∼ ˜ µ M α is approximately proportional to α in Fig. 1. Interestingly, the error term is O ( α 3 ) , which suggests that the approximation can be reasonably accurate even for quite large values of α , as illustrated in Fig. 1c.

Proof. Here M 0 = A has distinct eigenvalues { isσ j , s ∈ {-1 , 1 } , 1 ≤ j ≤ n } , with unit-norm eigenvectors A ( -isu j / √ 2 v j / √ 2 ) = isσ j ( -isu j / √ 2 v j / √ 2 ) . By the calculation of the eigenvalue derivatives from [Tao08], we obtain the following expansion for Sp( M α ) :

̸

Now the zeroth- and second-order terms are all in i R , hence the announced expansion for ˜ µ M α .

Estimate of the leading term under a probabilistic model. Assuming the singular vectors ( u 1 , ..., u n ) , ( v 1 , ..., v n ) of P are distributed uniformly at random - which is the case for example if P has i.i.d. Gaussian entries by rotational invariance 4 -, the leading term in the expansion of ˜ µ M α can be estimated in expectation as follows. The proof is placed in the Appendix, where we also include a high-probability version of the estimate.

Proposition 3.2. Suppose Q,R are fixed and U, V are independently distributed uniformly on the set of n × n orthonormal matrices. Then

where ∥·∥ F denotes the Frobenius norm. In particular, provided that Tr( S ) ∥ S ∥ F ≥ 2 √ log n (1 + c ) for some fixed c > 0 , we have E [ min 1 ≤ j ≤ n u ⊤ j Qu j + v ⊤ j Rv j ] ≍ Tr( S ) n as n →∞ .

Note that Tr( S ) ∥ S ∥ F , which always lies in the interval [1 , √ 2 n ] , is a measure of the effective sparsity of the spectrum of S (larger meaning less sparse), so the condition Tr( S ) ∥ S ∥ F ≥ 2 √ log n (1 + c ) means the spectrum of S is well spread-out. So the proposition shows that the exponential convergence rate depends on the average of the eigenvalues of S , when αS ≪ A and the spectrum of S is well spread-out. This fact should be contrasted with the case of minimization, where the convergence rate scales as the minimum eigenvalue of the Hessian.

Interestingly, when the spectrum of S is sparse, the typical behavior of the leading term in the expansion of ˜ µ M α , min j u ⊤ j Qu j + v ⊤ j Rv j , is quite different. In this case, that quantity depends on the geometric mean of the non-zero eigenvalues of S , rather than the arithmetic mean, as formalized in the following proposition. The proof is placed in the Appendix, along with a high-probability version of the estimate.

Proposition 3.3. Suppose Q,R are fixed and U, V are independently distributed uniformly on the set of n × n orthonormal matrices. Let s 1 ≥ ... ≥ s r > 0 = s r +1 = ... = s 2 n the eigenvalues of S .

Then

In particular, E [ min j ≤ n u ⊤ j Qu j + v ⊤ j Rv j ] ≳ n -2 r -1 when n →∞ and r and s ∈ R r are fixed.

Numerically, the quantity min j ≤ n u ⊤ j Qu j + v ⊤ j Rv j indeed scales as n -2 r -1 under the conditions of the proposition, suggesting that our lower estimate could be tight.

3.2 Convergence rate of discrete-time algorithms

Prop. 3.1 gave an expansion of the normalized convergence rate r/η of gradient methods (for a game with Jacobian at optimum M α ), non-asymptotically in α and in the asymptotic limit η → 0 . In this subsection, we give expansions of the convergence rate r that are non-asymptotic in α and η .

The algorithms we will consider can be written in the form z k +1 = T ( z k ) with the update operator T dependent only on ∇ f and on step-size η , and satisfying z ∗ = T ( z ∗ ) . It is well-known that local convergence of such methods is determined by ρ ( ∇ T ( z ∗ )) , where ∇ T is the Jacobian of T and ρ ( · ) denotes spectral radius. Namely, if ρ ( ∇ T ( z ∗ )) < 1 then the iterates converge locally with ∥ ∥ z k -z ∗ ∥ ∥ = O ( ( ρ ( ∇ T ( z ∗ )) + ε ) k ) , with ε > 0 an arbitrarily small constant [Ber97, Proposition 4.4.1].

In Table 1 we give an expansion for ρ ( ∇ T ( z ∗ )) for three classical gradient methods: simultaneous GDA (Sim-GDA), alternating GDA (Alt-GDA), and Extra-Gradient (EG). Let us clarify immediately that throughout this paper, statements made about 'GDA' without further specification apply to both

Sim-GDA and Alt-GDA. In the second column we denoted by g ( z ) = ( ∇ x f -∇ y f ) ( z ) the skewed gradient field of the game, and in the third column we wrote M instead of M α for concision. In the fourth column, ± iσ j denotes the eigenvalues of A assumed distinct and w j are the associated eigenvectors - equivalently, the singular value decomposition of P is P = ∑ n j =1 σ j u j v ⊤ j and for each j ≤ n , w ⊤ j Sw j = w ⊤ j + n Sw j + n = 1 2 ( u ⊤ j Qu j + v ⊤ j Rv j ) . We refer to the Appendix for the derivation of this table and for explicit bounds on the ' O ( · ) ' terms.

Informally, as one can directly see from the fourth column, Sim-GDA requires a very small step-size for the first term in η to overcome the terms + η 2 σ 2 j , while for EG those terms actually appear with a favorable sign, and Alt-GDA neither benefits nor suffers from those terms. We also see that Alt-GDA is quite faithful to GF, in that their normalized convergence rates coincide up to terms of order η 3 + α 4 . All of these insights are in line with common intuition in the min-max optimization literature [BGP20; Lu22], as well as with our numerical experiments for the next section, Fig. 2.

A symmetrized formulation of Alt-GDA. In order to derive the rate for Alt-GDA, we used the following symmetrized formulation of it: we let ( x 0 , y 1 / 2 ) ∈ R d × R d and

That is, x gets updated with the gradient rule at integer time-steps, y at half-integer time-steps, and we define x k , y k at non-updating time-steps as the average of the preceding and following updating time-steps. Assuming η ≤ ∥ ∥ ∇ 2 xx f ∥ ∥ -1 ∞ ∧ ∥ ∥ ∇ 2 yy f ∥ ∥ -1 ∞ , we show in the Appendix that z k +1 is indeed

entirely determined by z k = ( x k , y k ) for each k ∈ N , and that the associated update operator T satisfies

For comparison, the usual formulation of Alt-GDA considers as the iterates (˜ x k , ˜ y k ) = ( x k , y k + 1 / 2 ) . We emphasize that (˜ x k , ˜ y k ) and ( x k , y k ) have the same convergence rate if they converge exponentially, as one can check directly from the definition.

4 Illustration: sparse mixed Nash equilibria of continuous games

In this section we apply the above considerations to a particular class of min-max problems, which is of its own interest in game theory. Namely we consider the classical problem of finding the mixed Nash equilibria (MNE) of two-player zero-sum games, that is, given strategy spaces X , Y and a payoff function f : X × Y → R , solving the min-max problem

Here P ( X ) denotes the space of probability measures - representing mixed strategies - over X . Let us focus on cases where X and Y are continuous sets, say, X = Y = T 1 the one-dimensional Euclidean torus, 5 and f : T 1 × T 1 → R is smooth. The above min-max problem is then infinitedimensional, and algorithms to solve it explicitly must be based on reparameterized formulations. More specifically, suppose that the MNE ( µ ∗ , ν ∗ ) is unique and 'sparse', i.e., has finite supports: supp( µ ∗ ) = { x ∗ I , 1 ≤ I ≤ N } , supp( ν ∗ ) = { y ∗ J , 1 ≤ J ≤ M } (this is the case for example if f is a sum of separable functions [SOP08]).

In this setting there are two natural reparameterizations and associated algorithms:

If the optimal support points { x ∗ I } I , { y ∗ J } J are known, then we may reparameterize by µ = ∑ N I =1 a I δ x ∗ I , ν = ∑ M J =1 b J δ y ∗ J and optimize over the a I , b J . The problem reduces to a constrained bilinear game

where ∆ N denotes the standard simplex. A classical approach is then to apply Mirror Prox (MP) with entropy link function [Nem04] (MP is the Bregman-geometry analog of EG).

If only the number of optimal support points is known, then we may reparameterize by µ = ∑ N I =1 a I δ x I , ν = ∑ M J =1 b J δ y J and optimize over both the weights ( a I , b J ) and the support points ( x I , y J ). The problem reduces to

Apossible approach is to iteratively update (simultaneously) the a, b using MP steps with step-size η and the x, y using EG steps with step-size γη , for some parameter γ > 0 . This algorithm is called Conic Particle Mirror Prox in [WC22], but for concision we will refer to it simply as 'EG' in this section. Note that the main challenge in that reference is to deal with the case where N and M are unknown, but here we assume they are known for the sake of simplicity.

In Fig. 2, we show the dependency on η and γ of the local convergence rate of MP and EG, as well as that of the analogs of Sim-GDA and Alt-GDA and GF for the problem min ( a,x ) max ( b,y ) F 2 ( a, x, b, y ) in order to illustrate the insights from Sec. 3.2. We used a randomly generated payoff function f , and the convergence is measured by the iterates' ℓ 2 distance to the solution; see the Appendix for details.

(a) Convergence rate

r

vs.

γ

(fixed

η

= 10

-

2

)

(b) Convergence rate vs. η (fixed γ = 10 - 2 )

10-3

10-1

Convergence rate

10-2

gence

10-4

Converge

10-3

Sim-GDA

Sim-GDA

EG

10-4

EG

Alt-GDA

Alt-GDA

MP (y =0)

10-5

GF (theory)

10-5

GF (theory)

10-3

10-2

10-1

10-2

10-1

100

n

MP (the algorithm based on the reparameterization F 1 ) converges to the MNE for any small enough η with an exponential rate at least proportional to η 2 [WLZL21], and this scaling is tight numerically (green dots in Fig. 2b). Its explicit variant (Mirror Descent-Ascent) and its continuous-time flow are known to diverge for any η [BP18] [MPP18].

EG is known to converge locally to the MNE for any small enough η (and any γ > 0 ) with an exponential rate at least proportional to η 2 , despite the non-convexity of F 2 , under some nondegeneracy assumptions [WC22, Sec. 3.1]. However numerically the convergence rate of EG typically scales as η , not η 2 (orange dots in Fig. 2b) - a fact which we will explain below. The explicit variant of EG and its continuous-time flow have not previously been analyzed; the discussion below will give a characterization of when they converge locally.

Note that, at least for the particular f and γ used for Fig. 2b, EG converges locally faster than MP for the same η (in number of iterations, the per-iteration costs differing only by a constant factor), even though the former does not use the knowledge of the { x ∗ I } I , { y ∗ J } ! In other words, even when the optimal support points are known, it is beneficial to use the overparameterized formulation F 2 where we also vary the support points.

Overparameterization induces partial curvature. Let us inspect the 'Jacobians' at optimum for the two dynamics, MP vs. EG with parameter γ . Due to the simplex constraints and the non-Euclidean nature of the updates for a, b , the relevant matrices are M MP and M γ defined below, in the sense that the exponential convergence rate of each algorithm is ˜ µ M η + O ( η 2 ) (see the Appendix). Namely, omitting half of the antisymmetric off-diagonals for readability,

where D a = Diag( √ a ∗ ) (square roots being taken component-wise) and Π a ∈ R ( N -1) × N is any matrix such that Π a Π ⊤ a = I N -1 and Π ⊤ a Π a = I N -√ a ∗ √ a ∗ ⊤ , and likewise for D b and Π b ∈ R ( M -1) × M ; and for EG, where [ ∂ x P ] IJ = ∂ x f ( x ∗ I , y ∗ J ) , and likewise for ∂ y P , ∂ 2 xx P , ∂ 2 yy P , ∂ 2 xy P .

For MP, it is clear that the equivalent conditions of Thm. 2.1 are violated for any payoff function f , and so ˜ µ M MP = 0 . For EG, depending on f and γ , they may or may not be violated. For all of the random payoff functions we considered in our experiments, we observed that ˜ µ M γ > 0 , suggesting

that the conditions hold generically. They are violated for certain f 's however, as shown in the Appendix, so that the scaling in η 2 of the convergence rate proved in [WC22] is tight in the worst case.

More precisely as we show in the next proposition, assuming that the blocks of M γ are in general position, we expect ˜ µ M γ to scale as γ 2 , which is indeed observed in the numerical experiment reported in Fig. 2a. The proof, placed in the Appendix, relies on the same tools as Prop. 3.1, that is, on the asymptotic expansions of the eigenvalues of perturbed matrices.

Proposition 4.1. Let S 2 symmetric and A 0 , A 1 , A 2 antisymmetric real matrices of the form

5 Conclusion

We have investigated the local convergence of gradient methods for min-max games and found that they converge generically under partial curvature. In more specific settings, we have obtained quantitative estimates of the local convergence rate which exhibit the average of the eigenvalues of S as the driving quantity for typical problems. For the computation of mixed Nash equilibria of continuous games, this leads to a behavior of conic particle gradient methods that is more favorable than that described by the worst-case bounds.

More generally, our analysis leads to the following insights: (i) worst-case bounds might be looser in min-max optimization (compared to minimization) as they fail to capture the interplay between interaction and potential parts; (ii) the addition of new degrees of freedom with curvature typically accelerates the local convergence, as we illustrated in Sec. 4.

We note that the phenomenon described in this paper is fundamentally a consequence of the fact that the skewed gradient field's Jacobian at optimum has a positive-semidefinite symmetric part. This property is satisfied at local Nash equilibria of min-max games, i.e., of two-player zero-sum differentiable games, but is not true for general differentiable games.

References

[ALW21] Jacob Abernethy, Kevin A Lai, and Andre Wibisono. 'Last-iterate convergence rates for min-max optimization: Convergence of hamiltonian gradient descent and consensus optimization'. In: Algorithmic Learning Theory . PMLR. 2021, pp. 3-47.

[AW20] Ehsan Amid and Manfred KK Warmuth. 'Reparameterizing mirror descent as gradient descent'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 84308439.

[AMLG20] Waïss Azizian, Ioannis Mitliagkas, Simon Lacoste-Julien, and Gauthier Gidel. 'A tight and unified analysis of gradient-based methods for a whole spectrum of differentiable games'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2020, pp. 2863-2873.

[BGP20] James P Bailey, Gauthier Gidel, and Georgios Piliouras. 'Finite regret and cycles with fixed step-size via alternating gradient descent-ascent'. In: Conference on Learning Theory . PMLR. 2020, pp. 391-407.

[BP18] James P Bailey and Georgios Piliouras. 'Multiplicative weights update in zero-sum games'. In: Proceedings of the 2018 ACM Conference on Economics and Computation . 2018, pp. 321-338.

[Ber97] Dimitri P Bertsekas. 'Nonlinear programming'. In: Journal of the Operational Research Society 48.3 (1997), pp. 334-334.

[Doa22] Thinh Doan. 'Convergence rates of two-time-scale gradient descent-ascent dynamics for solving nonconvex min-max problems'. In: Learning for Dynamics and Control Conference . PMLR. 2022, pp. 192-206.

[DPS21] Carles Domingo-Enrich, Fabian Pedregosa, and Damien Scieur. 'Average-case Acceleration for Bilinear Games and Normal Matrices'. In: International Conference on Learning Representations . 2021.

[DH19] Simon S Du and Wei Hu. 'Linear convergence of the primal-dual gradient method for convex-concave saddle point problems without strong convexity'. In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR. 2019, pp. 196205.

[Gid+19] Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas. 'Negative momentum for improved game dynamics'. In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR. 2019, pp. 1802-1811.

[GLWM22] Benjamin Grimmer, Haihao Lu, Pratik Worah, and Vahab Mirrokni. 'Limiting behaviors of nonconvex-nonconcave minimax optimization via continuous-time systems'. In: International Conference on Algorithmic Learning Theory . PMLR. 2022, pp. 465-487.

[GWS21] Suriya Gunasekar, Blake Woodworth, and Nathan Srebro. 'Mirrorless mirror descent: A natural derivation of mirror descent'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2021, pp. 2305-2313.

[LM00] Beatrice Laurent and Pascal Massart. 'Adaptive estimation of a quadratic functional by model selection'. In: Annals of Statistics (2000), pp. 1302-1338.

[Let+19] Alistair Letcher, David Balduzzi, Sébastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. 'Differentiable game mechanics'. In: The Journal of Machine Learning Research 20.1 (2019), pp. 3032-3071.

[LS19] Tengyuan Liang and James Stokes. 'Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks'. In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR. 2019, pp. 907-915.

[Lu22] Haihao Lu. 'An O ( s r ) -resolution ODE framework for understanding discrete-time algorithms and applications to the linear convergence of minimax problems'. In: Mathematical Programming 194.1 (2022), pp. 1061-1112.

[MPP18] Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. 'Cycles in adversarial regularized learning'. In: Proceedings of the twenty-ninth annual ACMSIAM symposium on discrete algorithms . SIAM. 2018, pp. 2703-2717.

[MOP20] Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. 'A unified analysis of extragradient and optimistic gradient methods for saddle point problems: Proximal point approach'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2020, pp. 1497-1507.

[NK17] Vaishnavh Nagarajan and J Zico Kolter. 'Gradient descent GAN optimization is locally stable'. In: Advances in neural information processing systems 30 (2017).

[Nem04] Arkadi Nemirovski. 'Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems'. In: SIAM Journal on Optimization 15.1 (2004), pp. 229-251.

[PS20] Fabian Pedregosa and Damien Scieur. 'Acceleration through spectral density estimation'. In: International Conference on Machine Learning . PMLR. 2020, pp. 75537562.

[Per13] Lawrence Perko. Differential equations and dynamical systems . Vol. 7. Springer Science & Business Media, 2013.

[PP+08] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. 'The matrix cookbook'. In: Technical University of Denmark 7.15 (2008), p. 510.

[Roc76] R Tyrrell Rockafellar. 'Monotone operators and the proximal point algorithm'. In: SIAM journal on control and optimization 14.5 (1976), pp. 877-898.

[SOP08] Noah D Stein, Asuman Ozdaglar, and Pablo A Parrilo. 'Separable and low-rank continuous games'. In: International Journal of Game Theory 37.4 (2008), pp. 475504.

[Tao08] Terence Tao. When are eigenvalues stable . Oct. 28, 2008. URL: https : / / terrytao.wordpress.com/2008/10/28/when-are-eigenvaluesstable .

Peer reviews:
comments: SUMMARY:
This paper studies the convergence of Gradient based methods to local Nash equilibria. The properties of the “potential” and the “interaction” parts of the game are analyzed, and conditions when this leads to convergence of gradient methods are studied.

STRENGTHS:
This paper provides insights into the convergence of minimax algorithms, and highlights important differences from the minimization setting.

WEAKNESSES:
The writing and presentation of the results can be improved. 

 - Is the y-axis in Figure 1(a) the rate of convergence. What is the significance of including the observed and the predicted $\tilde{\mu}_M$? 
Can a plot of the the actual iterated of the algorithm for any one particular realization of $P$ also be included? 
[Like how GDA spirals out for a simple bilinear game, how does the trajectory change for this new game? Does the spiraling towards the solution happen in a skewed manner? 
Maybe a plot which shows the trajectories (for one particular P) of GDA diverging for the bilinear game, but converging when the new term is added could help the reader visualize how these algorithms converge
]

 - The proofs on page 4 has a few terms which are overflowing to the left.

 - The result showing the dependence on the average of the eigenvalues is surprising. How crucial is this result on the choice of the matrices $U$ and $V$ being distributed uniformly on the set of $n \times n$ orthonormal matrices? Proposition 3.2 and 3.3 cover the cases when the matrices $S$ is not sparse and sparse, but can a complete characterization on the way $P$ is drawn also be done?

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

comments: SUMMARY:
This paper focuses on min-max games with partial curvature, i.e., the symmetric part S of the Jacobian is p.s.d. and nonzero, and specifies the necessary and sufficient conditions for the convergence of gradient flow. The authors show that when the interaction term dominates, the convergence rate could depend on the average of the eigenvalues of S.
They also use the problem of computing mixed Nash equilibria of continuous games to illustrate the results.

STRENGTHS:
1. The paper is well-organized.
2. The results are novel and interesting.
3. The analysis is detailed and solid.

WEAKNESSES:
1. Propositions 4.2 and 4.3 are based on a very restrictive setting (see Questions).
2. Some expressions are not easy to follow.

QUESTIONS:
Propositions 3.2 and 3.3 require that U and V are uniformly distributed on $O_n$. After taking the expectation over all these problem instances, the averaged convergence rate depends on the average of the eigenvalues of S.
However, in practice, we are faced with a specific instance and such an expectation step might be invalid. When the antisymmetric part of the Jacobian is given in advance, is the convergence rate still related to the average of the eigenvalues of S?

LIMITATIONS:
This paper only studies local convergence.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the convergence of gradient descent-type algorithms for saddle-point problems of the form $\\displaystyle\\min_{x \\in \\mathbb{R}^n } \\displaystyle\\min_{x \\in \\mathbb{R}^m} f(x,y)$. Let $M$ denote the Hessian of $f$ at a local saddle point: convergence is governed by the minimum real part of all eigenvalues of $M,$ denoted $\\mu_M$. We have $\\mu_M \\ge 0,$ and this inequality is strict for almost all $M\\in \\mathbb{R}^{n\\times m}$, as the authors verify in their Theorem 2.1. This observation motivates the use of regularization/overparametrization for such problem involving quadratic/higher order terms.\n\nDecomposing $M$ into symmetric and antisymmetric parts, $M=S + \\alpha A$, with $\\alpha$ small, and write \n\\begin{equation}\nA = \\left(\\begin{array}{cc} \n0 & P \\\\\\\\\n-P^T & 0\n \\end{array}\\right),\n\\quad \nS = \\begin{bmatrix} Q & 0 \\\\\\\\\n0 & R \\end{bmatrix}.\n\\end{equation}\n\nFor $m=n,$ the Taylor expansion formula of Proposition 3.1 shows that $\\mu_M$ is well-approximated by the expression $\\displaystyle\\min_{1\\le j \\le n} \\left( u_j^T Q u_j  + v_j^T R v_j \\right)$, where the $u_i, v_i$ are the left and right singular vectors of $P = U \\Sigma V^T.$ Assuming $U,V$ are drawn uniformly from $O(n),$ the authors show that this expression (in expectation and with concentration) is asymptotically the average of the spectrum of $S,$ under the assumption that the spectrum is \"spread out\" in the sense $\\operatorname{tr} (S) / \\lVert S \\rVert_F$ is bounded by $\\sqrt{\\log n} $ times some constant greater than 2. As a counterpoint, the authors  consider the \"sparse\" case where $S$ has a fixed number $r$ of nonzero eigenvalues, and derive corresponding asymptotics and concentration bounds. Some experiments illustrating the theory for classical gradient methods are given.\n\nSTRENGTHS:\nThe paper is well-written overall. I am happy with the mathematical details, having read the main paper and the first two sections of the supplementary material and found no errors.\n\nWEAKNESSES:\nThe main limitation I see is one that the authors have already suggested in the conclusion: namely, it is not clear how the insights developed in this paper could extend to $N$-player games for $N>2$, since the assumption that $S$ is positive definite is needed, and the \"interactions\" among more than two players seem to be more subtle to define.\n\nI also found the experimental section to be rather confusing. It really feels like an afterthought, and far more technical than necessary. What is the connection with the previous sections? If the point is just to illustrate that extra parameters can accelerate convergence, then be clear about that and move the many unnecessary technical details to the appendix. It is rather surprising that your \"randomly-drawn\" $f$ has the property that Theorem 2.1 fails, no? Is it by design that the family of such functions will not have this property? I also think more attention could be drawn to the fact that $M_{\\text{MP}}$ and $M_{\\gamma}$, although not the actual Hessian of the payoff function $f,$ play the same role in the dynamics as $M$ would for vanilla gradient flow. Please make this more clear.\n\nQUESTIONS:\nline 156: \"In this section we assume $m=n$ for simplicity.\" I assume this is WLOG, since $Q$ and $R$ are arbitrary in your results, and we can just add variables to maintain $A\\gg  S$?\n\nline 208: \"four\" -> \"three\"?\n\nline 225: \"x gets updated\" more precisely, updated with the gradient rule rather than by averaging.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research investigates the local convergence properties of gradient dynamics in two-player zero-sum differentiable games towards Nash equilibria. Existing knowledge suggests that such dynamics converge locally when the symmetric part of the Jacobian at equilibrium, denoted by S, is positive definite (S ≻ 0), and divergence is likely when S equals zero (S = 0). The symmetric part S accounts for the potential function of the game.\n\nThe authors advance this understanding by demonstrating that gradient dynamics can also exhibit local convergence as soon as S is non-zero but fails to be strictly positive definite, which is referred to as partial curvature. This convergence is shown to occur provided the eigenvectors of the antisymmetric component of the Jacobian, denoted as A, occupy a general position relative to the nullspace of S. This result elucidates conditions under which convergence is guaranteed, thereby broadening the scenarios where gradient dynamics can be effectively employed in such games.\n\nThe paper further explores the rates of convergence in the case where the antisymmetric part dominates the symmetric part, represented mathematically as S ≪ A. It is proven that the convergence rates typically depend on the arithmetic mean of the eigenvalues of S, in contrast to the minimization problems analogy that suggests the dependence of rates on the smallest eigenvalue. This counterintuitive finding contributes to a more nuanced understanding of the behavior of gradient dynamics in these games.\n\nTo illustrate the theoretical findings, the problem of computing mixed Nash equilibria in continuous games is considered. The authors reveal that due to the effect of partial curvature, conic particle methods, which concurrently optimize over the weights and supports of mixed strategies, converge generically faster than their fixed-support counterparts. In the context of min-max games, this implies a strategic benefit in adding degrees of freedom exhibiting curvature, presenting yet another advantage of over-parameterization. This practical manifestation of their theoretical insights underscores the significant role of over-parameterization in enhancing the efficiency of Nash equilibria computations in such games.\n\nSTRENGTHS:\nThe paper impressively contributes a significant degree of originality by extending the traditional notions of regularization and partial curvature in a fresh manner to elucidate the convergence of Gradient Descent Ascent (GDA) to a Nash Equilibrium in continuous differentiable games. The authors’ approach not only offers an intriguing new perspective but also opens up a novel theoretical pathway that future work may explore further. \n\nQuality-wise, the paper exhibits a meticulous level of mathematical rigor in establishing the convergence conditions and rates. The well-structured proofs and clear theoretical arguments bolster the reliability of the results, reinforcing the significance of the paper’s main findings. \n\nIn terms of clarity, the authors deserve commendation for maintaining a high standard of exposition. They have done an excellent job in the paper's organization and presentation of complex concepts in a manner that is accessible and engaging. The lucid language, combined with logically organized sections, facilitates an intuitive understanding of the problem setup, the methodological developments, and the theoretical results.\n\nRegarding significance, the findings of this paper hold substantial implications for the broader research community focused on game theory and optimization. By unveiling the factors that determine the local convergence of GDA in two-player zero-sum differentiable games, the paper paves the way for the development of more efficient and reliable methods for finding Nash equilibria. Moreover, the practical relevance of these findings is underscored by the authors' demonstration of the benefits of over-parameterization in mixed Nash equilibria computations. \n\nOverall, the paper stands out for its innovative approach, rigorous methodology, clear exposition, and high impact on the field of continuous differentiable games. It makes a valuable addition to the literature by advancing our understanding of the roles of partial curvature and regularization in the context of gradient dynamics convergence to Nash equilibria.\n\nWEAKNESSES:\nCertainly, here is the markdown version:\n\n1. **Clarity on Mirror Prox (MP), Exponential Gradient (EG) Methods, and Overparametrization:** The paper could enhance its clarity by offering more detailed explanations and motivations for the application of Mirror Prox and Exponential Gradient methods, particularly in the context of overparametrization. Clear elaboration on how overparametrization interacts with these methods and contributes to convergence could reinforce the paper's argument. \n\n2. **Applicability to Non-Square Games:** It would be beneficial for the authors to clarify if their results extend to non-square games, i.e., when the dimensions of the two players' strategies are different ($n \\neq m$). Currently, the paper does not explicitly address this case, which leaves a gap in the comprehension of the scope of the presented results.\n\n3. **Practical Examples of Slight Curvature Setting:** The inclusion of practical examples or real-world scenarios where the slight curvature condition holds would significantly enhance the paper's impact. Such examples can illustrate the practical relevance of the theoretical findings and provide concrete contexts in which the paper's insights can be applied.\n\n4. **Average Eigenvalues and Convex-Concave Settings:** The paper's discussion on the role of average eigenvalues of the symmetric part of the Jacobian matrix $S$ in determining convergence rates might appear unclear to some readers. It would be advantageous if the authors could elaborate on whether these results are most relevant in non-convex non-concave settings, or in convex-concave settings that are not strongly convex-concave. This would help readers better understand the implications of the paper's results in different game settings.\n\nQUESTIONS:\nPlease answer my concerns at Weaknesses' section\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-18"}