Finding and Only Finding Differential Nash Equilibria by Both Pretending to be a Follower
Authors: Xuchan Bao
Guodong Zhang
Venue: Accepted by TMLR ()

Abstract: Finding Nash equilibria in two-player differentiable games is a classical problem in game theory with important relevance in machine learning. We propose double Follow-the-Ridge (double-FTR), an algorithm that locally converges to and only to differential Nash equilibria in general-sum two-player differentiable games. To our knowledge, double-FTR is the first algorithm with such guarantees for general-sum games. Furthermore, we show that by varying its preconditioner, double-FTR leads to a broader family of algorithms with the same convergence guarantee. In addition, double-FTR avoids oscillation near equilibria due to the real-eigenvalues of its Jacobian at fixed points. Empirically, we validate the double-FTR algorithm on a range of simple zero-sum and general sum games, as well as simple Generative Adversarial Network (GAN) tasks.

Published in Transactions on Machine Learning Research (07/2023)
Finding and Only Finding Differential Nash Equilibria by Both Pretending to be a Follower
Xuchan Bao
jennyhoo@es.troottoell
Deportment of Computer Science University of Toronto, Vector Institute
Guodong Zhang
gizhang@@s.toron wwwowcoosstooooooooooooooooooo.. edu
Department Computer Science University of Toronto, Vector Institute
Reviewed on OpenReview: oppnreviee. net/ forum? id= igdWKnK5RZ
Abstract
Finding Nash equilibria in two-player differentiable games is classical problem in game theory with important relevance in machine learning. We propose double Follow-the-Rides (double-FTR), an algorithm that locally converges to and only to differential Nash equilibria general-sum two-player differentiable games. To our knowledge, double-FTR is the first algorithm with such guarantees for general-sum games. Furthermore, we show that by varying its preconditioner, double-FTR leads to broader family of algorithms with the same convergence guarantee. Tn adn,tion, double-FTTH avoids oscillation near equilibria due to the real-cigenvalues of its Jacobian at fixed points. Empirically, we validate the double-FTRR algorithm on range of simple zero-sum and general sum games, as well as simple Generative Adversarial Network (GAN) tasks.
1 Introduction
Much of the recent success in deep learning can be attributed to the effectiveness of gradient sundient-based optimization. It is well-known that for minimization problem, with appropriate choice of learning rates, gradient descent enjoys convergence guarantce to local minima (Lee et al., 2016 2019). Based on this foundational result, an array of accelerated and higher-order methods have since been proposed and widely applied training neural networks (Duchi et al. 2011 Kingma & Ba. 2014 Reddi et al. 2018 Zhang al. 2019b)..
However, once leave the realm of minimization problems and consider the multi-agent setting, the optimization landscape becomes much more complicated. Multi-agent. optimization problems arise in diverse fields such as robotics, economics and machine learning [Foerster et al.] 2016 Von Neumann & Morgenstern
2007 Goodfellow et al. 2014 Ben-Tal & Nemirovski] 2002 Gemp et al. 2020 Anil et al. 2021
A classical abstraction that is especially relevant for machine learning two-player differentiable games, where the objective gaie would be each agent applying gradient descent minimize their own objective function. However, in stark contrast with gradient descent in solving minimization problems, this muddient-descent may converge to spurions critical points that are not Nash equilibria, and in the general-sum game case, Nash equilibria might not even be stable critical points for this algorithm (Mazundar et al. 2020b
find global local Nash equilibria. The equivalent gradicnt descent such a
algorithm
These negative results have driven a surge of recent interest in developing other gradient based algorithms for finding Nash equilibria in differentiable games. Among them Mazumdar al. (2019), who proposed an update algorithm whose attracting critical points are only local Nash equilibria in the special case of zero-sum games. However, to the best of our knowledge, such guarantees have not been extended to general-sum games.

---

Published in Transactions on Machine Learning Research (07/2023)
We propose double Follow-the-Ridge (double-FTR), gradient-basee algorithm for general-sum differentiable games that locally converges to and only to differential Nash equilibria! Double-FTR is closely related to the Follow-the-Ridge (FTR) algorithm for Stackelberg games (Wang et al., 2019), which converges to and only to local Stackelberg equilibria (Ficz et al., 2019). Double-FTR can be viewed as its counterpart for sirnultaneous games, where each player adopts the "follower" strategy FTR.
The rest of this paper is organized as follows. In Section 2 we give background on two-player differentiable games and equilibrium concepts. We also explain the issues with using gradient-drscent-stye algorithms such games. In Section 8 present the double-FTR. algorithm and prove its local convergence to and only to differential Nash equilibria. We also identify a more general class of algorithms that share these properties. We discuss recent works directly relevant to double-FTRR in Section | and other related work in Section [5] In Section 6 we show empirical evidence of double-FTTP FTR's convergence to and only to differential Nash equilibria.
2 Background
2.1 Two-player differentiable games and equilibrium concepts
In a general-sum two-player differentiable game, player 1 aims to minimize f Ru+m + R with respect x I", whereas player 2 aims to maximize g Rn+m R with respect to y E R Following the notation in Mazumdar et al. (2019), we denote such a game {(S, -9), R=+m). We also make the following assumption the tvice-dillerentiabbliy of f and g. Assumption 1. a E IR" RT", ʃ and y are twice-diiferentiabbl,, and the second derivatives are continuous. Also, V and V Eyg are invertible.
For two rational, non-cooperative players, their optimal outcome is to achieve local Nash equilibrium Ratliff
et al. 2013
Definition 2.1 (Local Nash equilibrium). A point (æ",y") is local Nash equilibrium of -(f,-9), Ru+m) if there exists open sets Sn C CR. Sy C R such that x" Sz, y² E Sy, and
f(x" (a"",")) y²) S (xx,y"), g( y²) g(=", y), Vo Sa, Vy Sy-
For twiee-differentiable f and g, local Nash equilibrium at (x" v²) implics Ratliff et al. 2013
Proposition 2):
(First-order condition)
Vaf(w" y²) and Vng(w*, y²) = 0, v2 fa= yy) = and V2 fa yy) =
(Sccond-order necessary condition)
A closely related notion of equilibrium is the differential Nash equilibrium (DNE) Ratliff et al. 2013). which satisfies a second-order sufficient condition for local Nash equilibrium. Definition 2.2 (Differential Nash equilibriun). "y") is a differential Nash equilibrium of {US, g), R R=+m if the following two conditions hold:
Val(m", y²) = 0 and V """("""") 0,
(a=" (,""""")")) and V2 Vgng("" "0m("="""") .y") x O.
The conditions of DNE are slightly stronger than that of local Nash equilibria in that the second-order conditions are definite instcad of semi-definite. In this paper, we focus on DNE, as they make up almost all local Nash equilibria in the mathematical sense, and are well-suited for the analysis of second-order a algorithms.
I Slightly stronger than the local Nash equilibria, discussed in Section 2.1
Note that local Nash equilibrium guaranteed exist in conconvex-nonconccve games Jin al. 2020 Proposition
6), although the (non- Jexistexite local NE out the scope this paper.
2

---

Published in Transactions on Machine Learning Research (07/2023)
Discussion on Assumption We assume twico-differnntiabliyy of f and g since concerned with finding DNE. As for the assumption that V and V are invertible, we need it for deriving our main algorithm in its original form. However, as discussed in Section 3.2 with practical implementation of our main algorithm, the assumption on invertibility can be relaxed.
2.2
Issues with gradient-based algorithms
A natural strategy for agents to search for DNE in differentiable game is to use gradient-bascd algorithms. The simplest gradient-based algorithm the gradient descent-ascent. (GDA) (Ryu Boyd 2016 Zhang ct al. 2021b) (Algorithm 1 or its variants (Zhang et al. 2021a; Korpelevich 1976 Mokhtari et al. 2020
Algorithm Gradient descent-ascent (GDA)
Require: Number of iterations T, learning rale 7
1: for t = 1...., do
2: 1++1 = Øt V vf yt)
3: y1+1 = Yt + y V ( yt
end for
Let and 7 be the learning rate, a gradient-based update algorithm be written
Z++1 = 2t - -zW())))
(1)
The Jacobian of (z)) defined as (z) = (w(z) In the case of GDA, we have:
(cDA(2)= = Vaf(x, 3) 129 v)
2e V2 y
V Vay
JGDA
Using the Jacobian matrix, we characterize the stability around fixed points of equation ! Definition 2.3 ((Strictly) stable fixed point), z" is a stable fixed point of the discrete-time dynamical system in equation if
w(x*) 0 and p - p(I~>)(z²)) 1
where p(-) denotes the spectral radius of a matrix. If we additionally have p(I ([==)((z")) < then 2" is a strictly stable fixed point.
Strictly stable fixed points are important for analysis, as they are locally asymptotically convergent (Galor 2007), iee. there exists an open set S₂ such that z" S2 and lim 2, Vzn Sz.
A closely related concept is the locally asymptotically stable equilibrium (LASE) for the continuous-timg time system & = (2). (Ratliff et al. 2013).
Definition 2.4 (Locally asymptotically stable cquilibrium (LASE)). z" is a locally asymptotically stable equilibrium of the continuous-ting Lime dynamics = -=)
w(2*) = and Re(A) > 0 for Aλ spco(J(z")),
where Re(.) denotes the real part of a complex number, and spec(-) returns the spectrum (i.e. the set of eigenvalues) of matrix.
Note that in the limit + 0, strictly stable fixed points of GDA are equivalent to LASE of & = WGDA (z).
In this paper, we prove convergence results in discrete-time (using Definition 2.3 but we often provide intuition using contimoous-time concepts such as LASE
Unfortunately, GDA is not guarantced to converge to DNE, nor are DNE necessarily (strictly) stable fixed points of the GDA dynamics. The relationship between the LASE of -WGDA and DNE is shown in the Venn diagrams in Figure 1 Below, give few examples to help build intuition.
3

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
LASE
DNE
LASE
DNE
(a) General-sum
(b) Zero-sum
Figure 1: Venn diagrams showing the relationship between the set of locally asymptotically stable equilibria (LASE) of the GDA flow and set differential Nash equilibria (DNE) two-player differentiable games. Note that for general-sum games, there exist DNE that are unstable for GDA flow.
Zero-sum games Even in the special case of zero-sum games (g 9= GDA dynamics can still have stable fixed points that not DNE (Daskalakis & Panageas 2018 Mazumdar al.
2020bb. In Figure 2 we
demonstrate the failure modes of GDA in zero-sum games. In Figure 2a GDA converges to spurious
strictly stable fixed point which is not DNE (corresponding to the pink areas in Figure D In 2b GDA fails to converge to the unique DNE (Hsich et al. 2020 Instead, it goes into a limit cycle, due to the strong rotation introduced by large complex parts in its Jacobian cigenvalues. We stress that these pathologics are not limited to GDA, but common for many other first-order algorithms (Wang et al. 2019
Ron Nash LASE
DNE
CDA
GDA
OGOA
GDA
EG
doutl: FTR
doubie-FF
ini:
hit
-12.
-5.0
-23
0.0
5.0 100
-100
-33
-500
-25
25
7.5
(a) [(x y) g(., v) = 2x + 5ay + x²
(b) f(a, = g(x,)) = Ty + c(x² y
Figure Two examples GDA failure modes in finding DNE in zero-sum games. (a) GDA converges the spurious strictly stable fixed point (), which is not DNE. Other first-order methods such the optimistic GDA (OGDA) and extragradient (EG) converge to the spurious fixed point as well. (b) Instead the unique DNE at (0, 0), GDA converges to limit cycle both when initialized "inside" (green) and "outside" (blue). We use 0.0001, 7 0.01.
General-sum games In general-sum games, apart from non-DNE spurious fixed points for GDA, DNE might not even be a stable fixed point of GDA (the cyan area in Figure [.. For example, let n m 1, f(I, y) = x² + 2xy and g(ø, y) -3ry 0.5y². It is easy to confirm that (0,0) is DNE, However,
2 2]
JODA
spec(JaDA) = (4,-1).
3 1
This means that for the GDA dynamics, (0 is saddle point rather than LASE.
4

---

Published in Transactions on Machine Learning Research (07 (07/2023)
3 Double Follow-the-Ridee
We propose double Follow-the-Ridge (doublc-FTR), an update rule for general-sum differential games that locally converges to and only to differential Nash equilibria. The double-FTR update is shown in Algorithm 2 (the arguments Dt, yt of f and g are dropped to avoid notational clutter).
Algorithm Double Follow-the-Rine
Require: Learning rate The and fy: number iterations , T do #++1 0t 1/ V f "/ (V2 wz
for 1....,
-1 V
Ty..
4g V
y1+1 yt B/ y + 7w V (n V
4: end for
Let
ac The and c
The we can express Algorithm T
vectorized form (equation 2 To simplify
y
the notation, we drop the subscript for f and g.
1++1 2: - (Z)TR(Z)), (PTR(Z)) [- - (yg
22 am
f)
f nyg
V
(2)
2 yar
cV
3.1 Local convergence of double-FTR
In this section, give main theoretical result. First, introduce additional assumption. Assumption 2. At fixed points of equation (2) has full rank.
Assumption 2 ensures that in doublc-FTR, the additional terms in the update do not exactly cancel out the GDA terms, which in turn ensures that fixed points with double-FTR are also fixed points with double-GDA. Note similar assumption is introduced in Mazundar et al. (2099) Theorem 4.
Our main thcorctical result stated below.
Thcorem 1. Under Assumptions 7 and & and with an appropriate choice of learning rate ", z" is a strictly slable fixed point of the double-FTR updale (equation 2) if and only if il a differential Nash equilibrium of the game (()=-)))Ruttm Furthermore, fixed points of equation all eigenvalues of the Jacobion
R=+m).
JFTR Owpra are real.
05
Intuitively, the first part of the theorem classifies the strictly stable fixed points of double-FTR, which leads to local convergence to and only to DNE (Corollary []. The second part ensures that there is rotation caused by complex eigenvalues in the neighbourhood of the DNFs. This is beneficial, as eigenvalues with large imaginary parts can often cause instability (such as the oscillation when training GANs) (Mescheder
et al. 2017 Balduzzi et al. 2018). We defer the proof of Theorem | to Appendix A
Corollary (Local convergence). Let z" be a DNE of the game {(ʃ, -g), Ra+m). Under Assumptions
and 2 and with an appropriate choice learning rate 7, there exists an open set Sz C B where x" E Szy such that when following equation 2 zu E Sz lim -*
Proof. The proof follows naturally by combining Theorem I with the local convergence strictly stable fixed points (Galor (2007), Proposition 1.9).
To the best of our knowledge, double FTR is the first algorithm with such local convergence result for general-sum games.
3.2 General preconditioners
In the following remark, we show that double-FTR can be generalized include whole family of algorithms.
5

---

Published in Transactions on Machine Learning Research (07/2023)
Remark 1. Theorem 1 aaplices to more general version of the double FTR algorithm. In particular, we can generalize equation to allow broader class of "preconditioners":
1++1 (#)TR(Z)), (z)
P& 0 0 -Py
(oDA(Ee)
Vaf -cV 49
(3)
where Pæ; Pg are continuous functions of n. y respectively, which satisly P. > Py - 64A O.
V 20/ >
and
Equation corresponds to the special case P = (V directly applies the case of general preconditioners in Remark
(V 9y The proof for Theorem
Remark I provides intuition the convergence propertics of doublc-FTR. Without the preconditioner Pa and Py, double-FTR reduces to Hamillonian gradient descent (Mescheder et al. 2017 Balduzzi et al. 2018
Loizou et al. 2020 Abernethy et al., 2021), which has spurious non-Nash equilibria. It is the introduction of the preconditioner that enables strictly stable fixed points to satisfy the second-order condition DNE.
Remark 1 also sheds light on how to derive a more practical algorithm.. Naively implementing Algorithm 2 might cause instability when of and yng are near singular. practice, we use (V /V +
instead of (V) in Algorithm| 2 (where a small
the damping parameter). Note that this also allows us to drop the assumption on the invertibility of V 2 and V yyg in Assumption
3.3 N-player games
Algorithm 2 naturally extends to n-player games, The algorithm and theoretical results for n-player games are exactly analogous to the two-player setting. The n-player Follow-the-Ridge algorithm and its convergence properties are shown in Appendix B
4 Connection with other algorithms
Mazumdar et (2019) proposed local symplectic surgery (LSS) - gradient-based algorithm whose LASE are exactly DNE in two-player zero-sum games. L.S avoids oscillatory behaviour DNE, similar to double-FTR. Compared to LSS, double-FTR appears to have simpler form and enables broader family of algorithms with such local convergence result in general-sum games.
The Follow-the-Ride (FTR) algorithm (Wang et al. 2019) is closely related to our proposed double-FTR., FTR was proposed for two-player soquential games and is guarantced to converge to and only to local minimax for zero-sum and Stackelberg equilibria for general-sum sequential games. FTR applies a gradient correction term on the follower in a sequential game, so that the agents approximately follow ridge in the landscape of the objective function. The double-FTR be viewed as counterpart of FTR for simultancous games. The update rule of double-FTR resembles that of FTR, with the gradient modification term applied on both players.
Another related algorithm is the Hamiltonian gradient descent (HGD) (Meschoder et al. 2017 Balduzzi et al. 2018 Loizou et al. 2020 Abernothy et al., 2021). HGD performs gradient- on the Hamiltonian, or the squared norm the gradient. HGD is guaranteed to converge, as is essentially a minimization problem. However, in general may have spurious non-Nash equilibria points. Interestingly, our double-FTR can be viewed a preconditioned HGD.
5 Related work
Mazumdar et al. (0020b) introduced a general framework for competitive gradient-based learning. They characterized DNE in terms of the critical points of the gradient algorithms. They showed the lack of convergence of the gradient algorithm in games, which motivated the development of the double-FTR, algorithm.

---

Published in Transactions on Machine Learning Research (07/2023)
Much work has focused on improving the dynamics in finding stable fixed points, which is crucial in applications such GANs, where oscillation caused by eigenvalues with zero real parts and large imaginary parts in the gradient Jacobian can lead to training instability. Mescheder et al. (2017) proposes Consensus Optimization, which encourages agrecment between the two players by introducing a regularization term in the objectives of both players. The regularization term results in a more negalive real-part for the eigenvalues of the gradient Jacobian, therefore reduces oscillation and allows larger learning rates. [Balduzi et al. (2018); Gemp & Mahadevan (2118) proposes Symplectic Gradient Adjustment (SGA), which decomposes the gradient Jacobian into symmetric (potential) and asymmetric (Hamiltonian) parts and adds gradient adjustment term for rapid convergence to stable fixed points. [Schafer & Anandkumar (2019) proposes Competitive Gradient Descent (CGD), whose update is given by the Nash equilibrium of a regularized bilinear approximation of the original game. Compared to other methods, CGD has the ad advantage of not needing to adapt slep size when the interaction strength changes between players. Many other methods have been proposed with different strategics for predicting other agents' moves, such as Learning with Opponent Learning Awareness (LOLA) [Foerster el. al., 2016) and optimistic gradient descent-ascent (OGDA) (Popov) 1980 Rakhlin & Sridharan 2013 Daskalakis et al. 2018 Mertikopoulls et al.) 2018). However, none of these existing methods address the problem of spurious (i.e. non-Nash) stable fixed points.
A separate but related line of research is on bilevel optimization. Bilevel optimization involves nested objectives: the upper-level objective depends on the solution of the lower-level problem. Tt has wide applications such as hyperparameter optimization and GAN training (Franceschi et al. 2018 Goodfellow ct al., 2014). One approach to bilevel optimization is based on implicit differentiation (Ochs et al. 2015
Pedregosa 2016 Lorraine et al.) (020)). Implicit differentiation is closely related to the original Follow-the- Ridge algorithm (Wang et al.) (019)), which tackles bilevel optimization from a game-thooretic perspective.
Compared to simultaneous games (this paper), bilevel optimization has different notions of equilibrium, such
as the local minimax (Jin et al. 2020 and Stackelberg equilibrium (Von Stackelberg 2010).
6 Experiments
We conduct simple experiments to demonstrate the implications of our theoretical results. First, through a 2-D toy example, we show that double-FTR converges to DNE and successfully avoids spurious non-DNE fixed points, as predicted by Theorem [ Then, we demonstrate that in a general-sum linear quadratic game, double-FTR is able to converge to DNE that naive policy gradient avoids. Lastly, we show that double-FTR can be scaled up and applied to train Generative Adversarial Networks (Goodfellow al.] 2014). Unlike GDA which suffers from severe mode collapse, double-FTR recovers all the modes and learns distribution that closely matches the target.
6.1 2-D toy example
We consider the zero-sum game (J-f),me R with the following 2-D function (same as in Mazumdar et al. (201
f(x,)) v) - -0.01c++v') ((0.6)2 + w)² + (0.5g² + z))
This function has several strictly stable fixed points for the GDA dynamics, among which some are DNE and some are not. As shown in Figure 3 while GDA may converge to fixed points that are not DNE, double-FTR avoids such spurious fixed points. Also, in the neighbourhood of the DNE, GDA exhibits oscillatory behaviour due to complex eigenvalues of the Jacobian matrix. In contrast, the double-FTR does not have oscillatory behaviour near the DNE. For reference, we also show the trajectories of the Local Symplectic Surgery (LSS). In this experiment, LSS has similar convergence properties it avoids spurious fixed points and does not have oscillatory behaviour near the DNE.
Obscrving the optimization trajectory, we note that double-FTR can approach a non-Nash spurious fixed point, then steer away without converging it. In this paper, we do not discuss the convergence rate of double-FTR, because the dynamics of multi-player game is much more complicated than that of
7

---

Published in Transactions on Machine Learning Research (07/202) 2022)
20
DNEE
Non-Nash
mitt
9
BoubAl-FTR
10 LSS
10
-10
Figure 3: Left: evolution of GDA and double-FTR in the 2-D toy example from multiple initial points. Middle: zoom-in near differential DNE point. Right: zoom-in near a Non-Nash LASE for the GDA algorithm.
minimization problem, making the convergence rate a less meaningful metric. Instead, we focus on the local convergence itself and the characterization of the fixed points.
6.2 General-sum linear quadratic game
The linear quadratic (LQ) game a classic problem multi-agent learning. It is an extension the famous linear quadratic regulator (IQQR) problem of optimal control to the multi-agent setting. Just as TQR being a simple yet important benchmark problem for studying properties of reinforcement learning algorithms, the LQ game provides valuable insights to multi-agent RL algorithms (Fazel et al., 2018 Zhang al. 2019a)..
Consider the discrete-time linear dynamical system, where e Rd= is the state, and two players provide control inputs E gdd and v E gdd respectively.
2o+1 Az; Buu: + B 20 ~ p(zo)
Each player adopts a linear stete-feedback policy: un - v2 = =- K, where the parameters Ku E Rd..dd, K, E Rd,xdd are to be determined by optimization. In a general-sum LQ game, each player aims to find their corresponding policy parameters K that minimizes their individual quadratic loss function f (shown in equation fo (K,,, K&) defined analogously using Q, and R.).
20
fu(K, K.) E Ezexp(z))
Σ
z Qu= u, R.
(Qu > O, R. 0)
Despite their simplicity, LQ games are challenging to optimize, because even though the loss functions are quadratic the states and actions, they are not convex with respect the player parameters K,, and K,. One straightforward algorithm choice is GDA, which refer to as the naive policy gradient method, as the gradients are computed using the policy gradient algorithm (Williams) 1992). Importantly, Mazumdar et all (2020a) show that in general sum LQ games, naive policy gradient almost surely avoids some Nash equilibria.
We demonstrate in general-sum LQ games, double-FTR is able to find DNE that are avoided by naive policy gradient. We use a setting mentioned in Mazumdar et al. (2200a). where dN = 2, du = d, = 1,
R, R, = 0.01, and
[0.511
0.064
1
0 1
Q. = 00.01 0
1 0
A
B, = [
B, =
= Q
0.533
0.993
0
0
0.147
The initial state 2o is set to
1]
1.1]
with equal probability.
8

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
3
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.375
0.50
0.400
Kv_2
Ku 1
0.45
Kv_1 0.06
Figure 4: Evolution of the loss landscape of a general-sum linear quadratic game when optimized by double-FTR. We visualize two 2D slices (K,,1, K.,2) and (K,,1, K2,2) and the loss functions fu and fo respectively. Yellow represents higher value and purple represents lower value. As seen on the top levels of the illustration, the loss landscape is "howl-shaped" convergence, confirming that double-FTR solution satisfies the second-order conditions for DNE.
sdos Set.
1.0
1.0
0.8
0.8
0.6
0.6
0.6
06
0.2
0.2
0.2
6.13
0.13
0.025
0.34
0.99
0.54
0.000
6.15
KJ_2
1.00
0.15
Ki_2 -0.25
0.00
kv_2
1.0
slice and
Slice atong
Sicee along
slice along and
1.0
1.0
10
1.0
1.0
0.8
0.8
0.4
0.4
0.4
0.4
0.39
0.39
00.025
6.40
kv.1
0.99
0.40
Kv.1
-0.25
0.0
Kv.1
0.5
Kv.1
0.41
1.02
0.41
0.25
2.5
2.5
Slice 31 along K., and K.
Slice along Ku. and Ki.
Slice 3: along K., and Ke.
Slice 4: along K.. and Kir
(a) FTU
(b) Naive policy gradient
Figure 5: For the general-sum linear quadratic game, visualize how the Jacobian evolves during training. At each step, we visualize (z =x.² JaDA, 21), the quadratic function defined by the current JCDA, centered at the current weight values ze. Due to the difficulty of directly visualizing a 4D function, we plot different slices of the quadratic function instead. Each contour plot shows an axis-aligned 2D slice, with the remaining 2 dimensions fixed at their corresponding values at zt. Yellow represents higher value and purple represents lower value. both (a) and (b), the weights are initialized near the same DNE that is an unstable fixed point for GDA (naive policy gradient) dynamics, which is shown as saddle points some slices. (a): using double-FTR, the weights converge the DNE, At convergence, the saddle points of (z 2.) JGDA. - 2.) still remain. (b): the gradient method avoids this unstable DNE, and converges stable, farther away fixed point instead.
Figure f and shows an instance where the double-FTR is able to converge to DNE, but naive policy gradient fails to. For both algs we use the same initial policy parameters K,, and K,, Figure g visualizes the loss landscape for (K,, K.) and fo(K,, K.) when optimized by double-FTRR It confirms that the solution double-FTR converges to is indeed a DNE (the socond-order condition Definition 2.2 Figure 5aa visualizes the local vector field Jacobian (i.e. JEDA) and shows that the Jacobian contains negative eigenvalues, which makes it saddle point for the naive policy gradient method. Indeed, naive policy gradient.
9

---

Published in Transactions on Machine Learning Research (07/2023)
(shown in Figure 5b) avoids this DNE. Instead, it eventually finds another DNE that is stable fixed point for naive policy gradient.
6.3 Generative Adversarial Networks
The Generative Adversarial Network (GAN) (Goodellow et al. 2014] is popular deep learning application ion for two-player games. The goal is to find the DNE where the generator perfectly matches the target distribution, and the discriminator is completely fooled by the generator.
In this experiment, we use the GAN framework to learn mixture of Gaussians (MoG). We use the original saturating loss function. Both the generator and the discriminator are multi-layer perceptrons with 2 hidden layers and 64 hidden units in each layer. With neural networks, directly implementing the Hessian would be computationally inefficient infeasible. Instead,
conjugate gradient to approximate vector products with the Hessian inverse. Details of the experiments can be found in Appendix C
As shown in Figure and 7 we apply GDA and double-FTR to learn MoG in 1D and 2D. In both cases, GDA gets stuck at a spurious equilibrium and suffers from mode collapse. In contrast, double-FTT recovers the modes, and the generated distribution closely matches the target.
Target Density
Generator Density (GDA)
Generator Density (double-FTR)
0.20
0.25
0.20
0.35
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.05
0.00
0.00
0.00
-2
Figure 6: Mixture of Gaussians in 1D. Left: ground-truth. Middle: generator distribution learned by GDA. Right: generator distribution learned by double-FTR.
Target
Dienation
Ieration 1003
Iteration 5000
Iteration 10000
Danation 30202
Berabi 50020
Turget
Ieration
Beration 1003
Iter 5000
Iteration 10000
Baration 30202
Beral on 30020
Figure 7: Mixure of Gaussians in 2D. Top: GDA suffers from mode collapse. Bottom: the generator distribution learned by double-F'TR recovers all the modes.
7 Conclusion
We propose double Follow-the-Ridge (double-FTR), gradient-based algorithm for finding differential Nash equilibria in differentiable games. We prove that under certain regularity assumptions, double-F'TT locally converges to and only to differential Nash equilibria in the general-sum games, and avoids oscillation in the neighbourhood of fixed points. Furthermore, we remark that by varying the preconditioner, double-FTE leads a broader family of algorithms that share the same convergence guarantee. Finally, we empirically verify the effectiveness of double-FTR in finding and only finding differential Nash equilibria across a broad range of problems.
10

---

Published in Transactions on Machine Learning Research (07 (022))
Acknowledgments
The authors thank Roger Grosse and Jakob Focrster for their helpful discussions and valuable foedback.
References
Jacob Abernethy, Kevin A Lai, and Andre Wibisono. Last-iterate convergence rates for min-max optimization: Convergence of hamiltonian gradient descent and consensus optimization. In Algorittemic Learning Theory, pp. 3-47. PMLR, 2021.
Cem Anil, Guodong Zhang, Yuhuai Wu, and Roger Grosse. Learning to give checkable answers with prover-verifier games. arXiv preprint arXiv:2108. 2021.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games. In International Conference on Machine Learning, pp. 354 363.
PMLR, 2018.
Aharon Ben-Tal and Arkadi Nemirovski. Robust optimization noethoottto-methhoooooy and applications. Mathematical programming, 92(3):453-483 480, 2002.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Nocula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github. com goom//oool//aa
Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. Advances in neural information processing systems, 31, 2018.
Constantinos Daskalakis, Andrew lyyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. In International Conference on Learwing Representations, 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the lincar quadratic regulator. International Conference on Machine Learning, pp. 1467-1476. PMLR, 2018.
Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stackelberg games. arXiv preprint 19Xio:1906.0127,, 2019.
Jakob Foerster, loannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. Tn International Conference on Machine Learning, pp- 1568 1577. PMLR, 2018
Oded Galor. Discrete dynamical systems. Springer Science & Business Media, 2007.
Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of gans using variational inequalities. arXiv preprint arXiv: 01531, 2018.
Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Eigengame: Pea as a nash equilibrium. International Conference on Learning Representations, 2020.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advonces in neural information processing systems, 27, 2014.
11

---

Published in Transactions on Machine Learning Research (07/2023)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commanications of the ACM, 63(11): 139 144, 2020.
Ya-Ping Hsich, Panayotis Mertikopoulos, and Volkan Cevher. The limits of min-max optimization algorithms: convergence to spurious non-critical sets. arXiv preprint arXie:2006. 09065,, 2020.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-noncooeann minimax optimization? In International conference on machine learning, pp. 4880-48888 PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiu:14122.680 6980, 2014.
Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon, 12: 747-756, 1976.
Jason D Lee, Max Simchowitz, Michacl I Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Conference on learning theory, pp. 1246 1257. PMLR, 2016.
Jason D Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael Jordan, and Benjamin Recht. First-order methods almost always avoid strict saddle points. Mathematical programming. 176(1):311-33) 2019.
Nicolas Loizou, Hugo Berard, Alexia Jolicoeur-Martineaut Pascal Vincent, Simon Lacoste-Julien, and loannis Mitliagkas. Stochastic hamiltonian gradient methods for smooth games, In International Conference on Machine Learning, pp.
PMLR, 2020.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp. 1540 1552. PMLR, 2020.
James Martens al. Deep learning via hessian-free free optimization. In ICML, volume 27, pp. 735-742, 2010.
Eric Mazundar, Lillian J Ratliff, Michael I Jordan, and S Shankar Sastry. Policy-gradient algorithms have no guarantees of convergence in linear quadratic games. In AAMAS, 2020a.
Eric Mazumdar, Lillian J Ratliff, and Shankar Sastry. On gradient- based learning in continuous games. STAM Journal on Mathematics of Data Science, 2(1):103-13)) 2020b.
Eric V Mazundar, Michael Jordan, and S Shankar Sastry. On finding local nash equilibria (and only local nash equilibria) in zero-sum games. arXiv preprint (:Xxt:1901.00820 2019.
Panayotis Mcrtikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasckhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In International Conference on Learning Representations, 2018.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numeries of gans. Advances in neural information processing systems, 30, 2017.
Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. A unified analysis srata--gradiet and optimistic gradient methods for saddle point problems: Proximal point approach. In International Conference on Artificial Intelligence and Statistics, pp. 1497-1507. PMLR, 2020.
Peter Ochs, René Ranftl, Thomas Brox, and Thomas Pock. Bilevel optimization with nonsmooth lower level problems. In Scale Space and Variational Methods in Computer Vision: 5th International Conference, SSVM 2015, Lège-Cap Ferret, France, May 31-June 4, 2015, Proceedings 5, pp. 654-665. Springer, 2015.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International conference on machine learning, pp. 737-746. PMTRR, 2016.
12

---

Published in Transactions on Machine Learning Research (07/2023)
Leonid Denisovich Popov. A modification of the arrow-hurwicz method for search of saddle points. Mathe- matical notes of the Academy of Sciences of the USSR, 28(5):845 848, 1980.
Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In Conference on PMLR, 2013.
Learning Theory, pp.
Lillian J Ratliff, Samuel A Burden, and S Shankar Sastry. Characterization and computation of local nash equilibria in continuous games.. In 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 917-924. IEEE, 2013.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Represenlations, 2018.
Ernest K Ryu and Stephen Boyd. Primer monotone operator methods. Appl. Comput. Math, 15(1):3 43,, 2016.
Florian Schäler and Anima Anandkumar. Competitive gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
Tijmen
Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a rumning average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26 31, 2012.
John Von Neumann and Oskar Morgenstern. Theory of gomes and economic behavior. Princeton university press, 2007.
Heinrich Von Stackelberg, Market structure and equilibrium. Springer Science & Business Media, 2010.
Yuanhao Wang, Guodong Zhang, and Jimmy Ba. On solving minimax optimization locally: A follow-the-ridge approach. In International Conference on Learning Representations, 2019.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, pp. 5-32, 1992.
Guodong Zhang, Xuchan Bao, Laurent Lessard, and Roger Grosso. A unified analysis of first-order methods for smooth games via integral quadratic constraints. Journal of Machine Learwing Reseurch, 12:1-39, 2021a.
Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger Grosse. Near-optimal local convergence of alternating gradient descent-asoent for minimax optimization. arXiv preprint #xXx:2102.00446 2021b.
Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Policy optimization provably converges to nash equilibria in zero-sum linear quadratic games, Advances in Neural Information Processing Systems, 32, 2019a.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E. Hinton. Lookahead optimizer: k steps forward, step back. Advances in Neural Information Processing Systems, 32, 2019b.
13


Reviews:
[{"review_id": "8LFRUY37b8", "reviewer": "Reviewer_cbWv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's main strength is the contributions seem sound and are relevant to the TMLR community.  Though the algorithm itself is somewhat incremental, the consequence of it provably converging to DNE in two-player general-sum games is important.\n\nI appreciate that the experimental evaluation is broad and more thorough than similar papers in this area.  The experiments themselves are synthetic, but the LQR and GAN domains are practical and non-trivial.  The GAN experiment not suffering mode collapse is particularly impressive.\n\nThe main paper is reasonably easy to follow.  The introduction and background sections thoroughly describe and motivate the problem.  The paper also does a good job of describing prior work and its contributions fit in.\n\nAt a high level, the paper's main weakness is that in places it is not as precise as it could or should be.  I do not believe this is pedantry on my part.  A consequence of this impreciseness is that some contributions are overstated while others are hard to properly evaluate.  As a concrete example, the abstract explicitly states that Double FTR converges to DNE in two-player general-sum games, but the regularity assumptions seem to exclude certain games of this class.  I will fully enumerate my concerns on this front in the requested changes.  That said, I do believe this concern can be resolved.\n\nAdditionally, I have some concerns about the proof of Theorem 1 that hopefully can be explained or resolved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Double Follow-the-Ridge (Double FTR), an algorithm that converges to local differential Nash equilibria in two-player general-sum differentiable games.  In contrast, Gradient Descent/Ascent may diverge or converge to non-Nash critical points even in zero-sum games, and Hamiltonian Gradient Descent may converge to non-Nash critical points in general-sum games.\n\nThe paper proves that if Double FTR/preconditioned Double FTR converges then it converges to a local differentiable Nash equilibrium under some regularity assumptions and step-size restrictions.\n\nFinally, the paper experimentally demonstrates the performance of preconditioned Double FTR in a toy 2D zero-sum game, a general-sum linear quadratic game, and a synthetic GAN trained to learn a mixture of Gaussians.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The abstract states \"[Double FTR] locally converges to and only to DNE in general-sum two-player differentiable games\".  The conclusion additionally qualifies the convergence is proven under \"mild assumptions\".  First, is it true that double FTR always converges if a suitable DNE exists (i.e., one that satisfies the regularity assumptions)?  I believe it should be possible to diverge, or at the very least the theorem only proves convergence within a neighborhood of a suitable DNE.  To me, the statements imply the algorithm is guaranteed to converge.  Second, I prefer \"regularity assumptions\" to \"mild assumptions\" unless some text is added to convince the reader that the assumptions are indeed mild (as opposed to just standard).\n\nThe above statement is more with respect to Assumption 2 than Assumption 1.  It would be beneficial to give an example of a game where a DNE exists that satisfies Assumption 1, but not Assumption 2.\n\nThe text below Assumption 2 implies that adding L2 regularization will guarantee Assumption 2 holds.  Is this true?  Typically, adding regularization to an optimization problem perturbs the problem (and thus its solution) and additionally can affect its convergence rate.  To me, the text implies that L2 regularization simply fixes the issue.  A discussion of the drawbacks would be beneficial.\n\nThere is no convergence rate given for Double FTR.  Some discussion as to why this is the case would be beneficial, and/or potentially under what conditions one would expect Double FTR to converge slowly or quickly.\n\nThe experiments section states it shows that double FTR converges to DNE, but the appendix states that all experiments are done using preconditioned DNE.  Additionally, as far as I can tell there is some degree of approximation in the gradients and Hessians for the second two experiments.  I'd expect these additions to make things more difficult for double FTR.  Is there a reason results for non-preconditioned FTR are not included?\n\nIn particular for the GAN experiment, is there any reason to believe that Assumption 1 and 2 are satisfied?  Would a highly over-parameterized network fail to satisfy these assumptions?\n\nIn the proof, it is not obvious why Assumptions 1 and 2 imply $\\omega_{\\text{FTR}} = 0$ implies $\\omega_{\\text{GDA}} = 0$.\n\nIt seems that there perhaps needs to be additional conditions on the preconditioners for preconditioned double FTR.  In particular, do they not need to be continuous at least around DNE?\n\nIn the appendix describing the GAN experiments, the statement \"double FTR experiences training instability very close to full convergence\" is concerning.  Can this be confirmed to be a result of the gradient and Hessian approximations?\n\nMinor points:\n - In 6.2, GDA is referred to as \"gradient-play\".\n - In 6.3, \"we use conjugate gradient to approximate the Hessian inverse\" should be \"we use the conjugate gradient method to approximately compute vector products with the inverse Hessian\".\n - Appendix B should be referenced from the main text or removed.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hltwjIY28c", "reviewer": "Reviewer_Tmz4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of this paper lies primarily in the proposed algorithm and its convergence guarantees as proven in the appendix (I checked the proofs and everything looks correct). Convergence to sensible solution concepts in differentiable games is a difficult and important problem for domains like GANs and multiagent systems (as exemplified in experiments).\n\nFigures 1, 2, & 3 were helpful to my understanding. However, Figures 4, 5, & 6 were less enlightening. This is partly due to minor issues that can be clarified (e.g., adding colorbars), but also due to a mismatch between claims and plotted results (my main concern described below regarding Figure 6). Figures 7 & 8 were nicely illustrative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an algorithm for n-player, differentiable, general-sum games called Double Follow-the-Ridge (double-FTR). Double-FTR is guaranteed to locally converge to and only to differential Nash equilibria. To my knowledge this is the first algorithm with any such guarantee. The guarantees rely on a few mild assumptions, which the authors provide sensible \"fixes\" for when they are not met. The authors empirically test double-FTR against a few standard baselines on a variety of qualitatively different differentiable games.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical (Addressed)**: I have one main critique, specifically that the results of the parameterized bilinear game (sec 6.3, Figure 6) do not seem to support the claim that double-FTR \"converges to the unique equilibrium $(x,y)=(0,0)$\". My understanding of the game is that in order to converge to the unique equilibrium, any algorithm must find that $\\theta = \\phi = [\\mu, \\sigma] = [0, 0]^\\top$. This should be reflected as $\\mu \\rightarrow 0$ in Figure 6 (left) and $\\log(\\sigma) \\rightarrow -\\infty$ in Figure 6 (right). I am content with the left Figure, but the right Figure does not show double-FTR log-variance parameters tending toward $-\\infty$. The analysis in this paper focuses entirely on the deterministic (non-stochastic) setting, so I do not think showing strong performance in the stochastic setting is necessary for acceptance. However, Figure 6 does not appear to support the claim I quoted above. Please rectify this. Also, please define the function $r(x,y)$ so that others can replicate your work. I answered YES to Claims and Evidence, but with this one caveat. *The reviewers reviewed this experiment, which addresses my concern.*\n\n**Minor (Addressed)**:\n- Terminology: The authors use several notions of Nash equilibria in their writing. Sometimes, local Nash equilibria (LNE) and differential Nash equilibria (DNE) are conflated. For example, the abstract should more accurately claim convergence to DNE, not LNE. This mistake is repeated in the paragraph before Section 2. Moreover, Figure 2 caption drops any qualifier and just states NE rather than clarifying the specific type. Please be rigorous with your word choice throughout the paper as it is critical to your proofs and central claim of your paper. In addition, Section 4, paragraph 2, uses \"minimax\" [solution] rather than NE -- try not to introduce new terminology if it is unnecessary. Stick to using DNE, LNE, LASE if you can as these are the terms you have more rigorously defined and referenced.\n- LNE: After defining local Nash equilibrium, it might help the reader's understanding if you then re-stated LNE conditions in the case that  player losses are twice differentiable. For example, stating something like Def 2.1 but where positive definiteness is replaced with positive semi-definiteness. You make a remark along these lines in the paragraph after Def 2.1, but it's a little unclear and given this is a journal submission, you have the space to make things explicit.\n- Figure 1 examples: Can you please add examples or references to examples for each of the distinct regions in Figure 1? Can you point out LNEs as well? Are they effectively the same as DNE, but DNE does not include its boundary? You may find [1] helpful in referring to an example.\n- Explanation of reason for Assumption 2: Reading through the proof in the appendix, it appears the full rank assumption is more about ensuring that fixed points of Double-FTR are also fixed points of GDA. This makes more sense to me than the vague, hand-wavy explanation given in the sentence following Assumption 2.\n- Hamiltonian Gradient Descent: You state that \"in general, it is not guaranteed to converge only to local Nash equilibria\". This sounds to me like HGD converges to other types of equilibria or fixed points. It may be worth stating that HGD can converge to completely non-sensical fixed points (e.g., unstable fixed points).\n- Figure 4: Please enlarge the tick labels, axis labels, and add a colorbar indicating what the colors indicate in the plot (green = low?, blue = high?). Lastly, in the paragraph below Figures 4 & 5, the authors say \"It confirms that the solution double-FTR converges to is indeed a Nash equilibrium\". Do you mean DNE?\n- Figure 5: Is each of the four plots in (a) a slice? Can you label them slices 1-4? And what do you mean by slice? Is this a random projection? Or are you just showing a axis-aligned subspace? Also, by \"quadratic function defined by the current $J_{GDA}$, do you mean $z^\\top J_{GDA} z$? Please define this clearly in the caption text.\n\n**Typos / Suggestions (Addressed)**\n- Sec 1, par 1: \"gradient descent ENJOYS a convergence guarantee to local minima\"\n- Sec 1, par 6: \"issues using gradient-descent style algorithmS\"\n- Sec 2, par 1: \"denote such A game as\"\n- Sec 2.2, above Def 2.2: \"characterize the STABILITY AROUND fixed points\"\n- Sec 2.2, below Def 2.3: \"Note that IN THE LIMIT $\\gamma \\rightarrow 0$\"\n- Sec 3.2, Remark 1: Please include zeros in your matrix rather than leave cells empty.\n- Sec 3.2, last par: \"Remark 1 also sheds LIGHT\"\n- Sec 5, par 2: \"with zero real parts AND large imaginary parts\"\n- Appx a, Theorem 1 (1): \"be a strictly stable fixed POINT of equation 2\"\n- Appx b, “the To avoid notation clutter” -- remove \"the\"\n- Notation: 1e-3 vs $10^{-3}$. The latter seems more formal, but up to the editor\n\n[1] @inproceedings{letcherimpossibility,\n  title={On the Impossibility of Global Convergence in Multi-Loss Optimization},\n  author={Letcher, Alistair},\n  booktitle={International Conference on Learning Representations}\n}", "broader_impact_concerns": "I do not have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QwLTdV9UNT", "reviewer": "Reviewer_GLp1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and clearly presents the setting, context and result. It is rather simple to follow. \n- The problem of computing local NE in general-sum two player games is important and difficult, hence the interest for this work\n\n\nWeakness:\n\n- Assumption 1 seems pretty restrictive, and I would have liked some discussion on why it is needed and how restrictive it is\n- The proof of the main (and only) Theorem is pretty straightforward and seems to present no difficulty\n- The presented method requires the computation of the Hessian (its inverse). As mentioned in the experiments, it is not suitable to modern neural network architectures, since computing the Hessian is very cumbersome. Extending the current double FTR algorithm to an algorithm that does not need second order gradient (but e.g. only an approximation of it) would be a big improvement", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of finding (and only finding) Differential Nash Equilibria for general-sum two-player games. In particular, the authors propose a double Follow the Ridge strategy, adapted from Wang et al (2019), which presents local convergence guarantees towards Differential Nash Equilibria under non-denegerate assumptions on the Hessian.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest the following changes/additions to the paper:\n- the title is misleading to me, as the main result is not about *Local Nash Equilibria* but *Differential Nash Equilibria* and would thus prefer a modification of the title\n- I would like a thorough discussion about Assumption 1\n- About the second part of Theorem 1 (the eigenvalues are real), the authors shortly discuss it, but it does not bring enough insight about the significance of this property to me. I would also like more discussion of why it is so beneficial\n- In the related work, I would also mention some literature on bilevel optimization, which considers optimization problems similar to general-sum two-player games. In particular, most of the bilevel optimisation methods also seem to use some factor with the Hessian inverse in their iterations ([1] for example).\n\n[1] Arbel, Michael, and Julien Mairal. \"Non-Convex Bilevel Games with Critical Point Selection Maps.\" arXiv preprint arXiv:2207.04888 (2022).", "broader_impact_concerns": "Not concerned", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were supportive of the authors' changes and felt they added clarity as well as providing care to ensure the validity of the paper's claims.  I think the novelty of the work, combined with the careful empirical experimentation suggest this paper may be deserving of a "Featured Certification".
Decision: Accept (poster)