Title: Approximating Nash Equilibria in Normal-Form Games via Unbiased Stochastic Optimization
Venue: NeurIPS 2023
Accepted: no
Keywords: game theory

stochastic optimization

nash equilbrium

normal-form game

x-armed bandits
Abstract: We propose the first, to our knowledge,  loss function for approximate Nash equilibria of normal-form games that is amenable to unbiased Monte Carlo estimation. This construction allows us to deploy standard non-convex stochastic optimization techniques for approximating Nash equilibria, resulting in novel algorithms  with provable guarantees. We complement our theoretical analysis with experiments demonstrating that stochastic gradient descent can outperform previous state-of-the-art approaches.

1

Abstract

We propose the first, to our knowledge, loss function for approximate Nash equilibria of normal-form games that is amenable to unbiased Monte Carlo estimation. This construction allows us to deploy standard non-convex stochastic optimization techniques for approximating Nash equilibria, resulting in novel algorithms with provable guarantees. We complement our theoretical analysis with experiments demonstrating that stochastic gradient descent can outperform previous state-of-the-art approaches.

1 Introduction 8

Nash equilibrium famously encodes stable behavioral outcomes in multi-agent systems and is arguably 9 the most influential solution concept in game theory. Formally speaking, if n players independently 10 choose n , possibly mixed, strategies ( x i for i ∈ [ n ] ) and their joint strategy ( x = ∏ i x i ) constitutes a 11 Nash equilibrium , then no player has any incentive to unilaterally deviate from their strategy. This 12 concept has sparked extensive research in various fields, ranging from economics [30] to machine 13 learning [16], and has even inspired behavioral theory generalizations such as quantal response 14 equilibria which allow for more realistic models of boundedly rational agents [28]. 15

Unfortunately, when considering Nash equilibria beyond the special case of the 2 -player, zero-sum 16 scenario, two significant challenges arise. First, it becomes unclear how a group of n independent 17 players would collectively identify a Nash equilibrium when multiple equilibria are possible, giving 18 rise to the equilibrium selection problem [18]. Secondly, even approximating a single Nash equilib19 rium is known to be computationally intractable and specifically PPAD-complete [11]. Combining 20 both problems together, e.g., testing for the existence of equilibria with welfare greater than some 21 fixed threshold is NP-hard and it is in fact even hard to approximate (i.e., finding a Nash equilibrium 22 with welfare greater than ω for any ω > 0 , even when the best equilibrium has welfare 1 -ω ) [2]. 23

24

25

26

27

28

From a machine learning (ML) practitioner's perspective, however, such computational complexity results hardly give pause for thought as collectively we have become all too familiar with the unreasonable effectiveness of ML heuristics in circumventing such obstacles. Famously, non-convex optimization is NP-hard, even if the goal is to compute a local minimizer [31], however, stochastic gradient descent (and variants thereof) succeed in training models with billions of parameters [7].

Unfortunately, computational techniques for Nash equilibrium have so far not achieved anywhere 29 near the same level of success. In contrast, most modern Nash equilibrium solvers for n -player, 30 m -action, general-sum, normal-form games (NFGs) are practically restricted to a handful of players 31 and/or actions per player except in special cases (e.g., symmetric [38] or mean-field games [34]). This 32 is partially due to the fact that an NFG is represented by a tensor with an exponential nm n entries; 33 even reading this description into memory can be computationally prohibitive. More to the point, any 34

computational technique that presumes exact computation of the expectation of any function sampled 35 according to x similarly does not have any hope of scaling beyond small instances. 36

This inefficiency arguably lies at the core of the differential success between ML optimization and 37 equilibrium computation. For example, numerous techniques exist that reduce the problem of Nash 38 equilibrium computation to finding the minimum of the expectation of a random variable (see related 39 work section). Unfortunately, unlike the source of randomness in ML applications where batch 40 learning suffices to easily produce unbiased estimators, these techniques do not extend easily to game 41 theory which incorporates non-linear functions such as maximum, best-response amongst others. 42 This raises our motivating goal: 43

Can we solve for Nash equilibria via unbiased stochastic optimization?

Our results. Following in the successful steps of the interplay between ML and stochastic optimiza44 tion, we reformulate the approximation of Nash equilibria in an NFG as a stochastic non-convex 45 optimization problem admitting unbiased Monte-Carlo estimation. This enables the use of powerful 46 solvers and advances in parallel computing to efficiently enumerate Nash equilibria for n -player, 47 general-sum games. Furthermore, this re-casting allows practitioners to incorporate other desirable 48 objectives into the problem such as 'find an approximate Nash equilibrium with welfare above ω ' 49 or 'find an approximate Nash equilibrium nearest the current observed joint strategy' resolving the 50 equilibrium selection problem in effectively ad-hoc and application tailored manner. Concretely, we 51 make the following contributions by producing: 52

53

54

A loss function L ( x ) 1) whose global minima coincide with interior Nash equilibria in normal form games, 2) admits unbiased Monte-Carlo estimation, and 3) is Lipschitz and bounded.

55

56

57

58

59

60

61

62

63

64

65

66

A loss function L τ ( x ) 1) whose global minima coincide with logit equilibria (QREs) in normal form games, 2) admits unbiased Monte-Carlo estimation, and 3) is Lipschitz and bounded.

An efficient randomized algorithm for approximating Nash equilibria in a novel class of games. The algorithm emerges by employing a recent X -armed bandit approach to L τ ( x ) and connecting its stochastic optimization guarantees to approximate Nash guarantees. For large games, this enables approximating equilibria faster than the game can even be read into memory.

An empirical comparison of stochastic gradient descent against state-of-the-art baselines for approximating NEs in large games. In some games, vanilla SGD actually improves upon previous state-of-the-art; in others, SGD is slowed by saddle points, a familiar challenge in deep learning [12].

Overall, this perspective showcases a promising new route to approximating equilibria at scale in practice. We conclude the paper with discussion for future work.

2 Preliminaries

In an n -player, normal-form game, each player i ∈ { 1 , . . . , n } has a strategy set A i = 67 { a i 1 , . . . , a im i } consisting of m i pure strategies. These strategies can be naturally indexed, so 68 we redefine A i = { 1 , . . . , m i } as an abuse of notation. Each player i also has a utility function, 69 u i : A = ∏ i A i → [0 , 1] , (equiv. 'payoff tensor') that maps joint actions to payoffs in the unit70 interval . Note that equilibria are invariant to payoff shift and scale [27] so we are effectively assuming 71 we know bounds on possible payoffs. We denote the average cardinality of the players' action sets 72 by ¯ m = 1 n ∑ k m k and maximum by m ∗ = max k m k . Player i may play a mixed strategy by 73 sampling from a distribution over their pure strategies. Let player i 's mixed strategy be represented 74 by a vector x i ∈ ∆ m i -1 where ∆ m i -1 is the ( m i -1) -dimensional probability simplex embedded 75 in R m i . Each function u i is then extended to this domain so that u i ( x ) = ∑ a ∈ A u i ( a ) ∏ j x ja j 76 where x = ( x 1 , . . . , x n ) and a j ∈ A j denotes player j 's component of the joint action a ∈ A . For 77 convenience, let x -i denote all components of x belonging to players other than player i . 78

The joint strategy x ∈ ∏ i ∆ m i -1 is a Nash equilibrium if and only if, for all i ∈ { 1 , . . . , n } , 79 u i ( z i , x -i ) ≤ u i ( x ) for all z i ∈ ∆ m i -1 , i.e., no player has any incentive to unilaterally deviate from 80 x . Nash is typically relaxed with /epsilon1 -Nash, our focus: u i ( z i , x -i ) ≤ u i ( x ) + /epsilon1 for all z i ∈ ∆ m i -1 . 81

∇ i x i as player i 's utility gradient. And for convenience, denote by H i il = E x -il [ u i ( a i , a l , x -il )] the 84 bimatrix game approximation [20] between players i and l with all other players marginalized out; 85 x -il denotes all strategies belonging to players other than i and l and u i ( a i , a l , x -il ) separates out l 's 86 strategy x l from the rest of the players x -i . Similarly, denote by T i ilq = E x -ilq [ u i ( a i , a l , a q , x -ilq )] 87 the 3 -player tensor approximation to the game. Note player i 's utility can now be written succinctly 88 as u i ( x i , x -i ) = x /latticetop i ∇ i x i = x /latticetop i H i il x l = x i T i ilq x l x q for any l, q where we use Einstein notation for 89 tensor arithmetic. For convenience, define diag ( z ) as the function that places a vector z on the 90 diagonal of a square matrix, and diag3 : z ∈ R d → R d × d × d as a 3-tensor of shape ( d, d, d ) where 91 diag3 ( z ) iii = z i . Following convention from differential geometry, let T v M be the tangent space 92 of a manifold M at v . For the interior of the d -action simplex ∆ d -1 , the tangent space is the same at 93 every point, so we drop the v subscript, i.e., T ∆ d -1 . We denote the projection of a vector z ∈ R d 94 onto this tangent space as Π T ∆ d -1 ( z ) = z -1 d 1 /latticetop z . We drop d when the dimensionality is clear 95 from the context. Finally, let U ( S ) denote a discrete uniform distribution over elements from set S . 96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

3 Related Work

Representing the problem of computing a Nash equilibrium as an optimization problem is not new. A variety of loss functions and pseudo-distance functions have been proposed. Most of them measure some function of how much each player can exploit the joint strategy by unilaterally deviating:

As argued in the introduction, we believe it is important to be able to subsample payoff tensors of normal-form games in order to scale to large instances. As Nash equilibria can consist of mixed strategies, it is advantageous to be able to sample from an equilibrium to estimate its exploitability /epsilon1 . However none of these losses is amenable to unbiased estimation under sampled play. Each of the functions currently explored in the literature is biased under sampled play either because 1) a random variable appears as the argument of a complex, nonlinear (non-polynomial) function or because 2) how to sample play is unclear. Exploitability, Nikaido-Isoda (NI) [32] (also known by NashConv [21] and ADI [15]), as well as fully-differentiable options ([36], p. 106, Eqn 4.31) introduce bias when a max over payoffs is estimated using samples from x . Gradient-based NI [35] requires projecting the result of a gradient-ascent step onto the simplex; for the same reason as the max , this is prohibitive because it is a nonlinear operation which introduces bias. Lastly, unconstrained optimization approaches ([36], p. 106) that instead penalize deviation from the simplex lose the ability to sample from strategies when iterates are no longer proper distributions. Table 1 summarizes these complications.

4 Nash Equilibrium as Stochastic Optimization

We will now develop our proposed loss function which is amenable to unbiased estimation. Our key technical insight is to pay special attention to the geometry of the simplex. To our knowledge, prior works have failed to recognize the role of the tangent space T ∆ . Proofs are in the appendix.

4.1 Stationarity on the Simplex Interior

Lemma 1. Assuming player i 's utility, u i ( x i , x -i ) , is concave in its own strategy x i , a strategy in the interior of the simplex is a best response BR i if and only if it has zero projected-gradient 1 norm:

126

127

128

129

130

131

132

133

134

135

136

145

146

147

148

149

4.4 Unbiased Estimation

As discussed in Section 3, a primary obstacle to unbiased estimation of L ( x ) is the presence of complex, nonlinear functions of random variables, with the projection of a point onto the simplex being one such example (see Π ∆ in Table 1). However, Π T ∆ , the projection onto the tangent space of the simplex, is linear ! This is the key that allows us to design an unbiased estimator (Lemma 5).

/negationslash

Our proposed loss requires computing the squared norm of the expected value of the gradient 150 under the players' mixed strategies, i.e., the l -th entry of player k 's gradient equals ∇ k x kl = 151 E a -k ∼ x -k u k ( a kl , a -k ) . By analogy, consider a random variable Y . In general, E [ Y ] 2 = E [ Y 2 ] . 152 This means that we cannot just sample projected-gradients and then compute their average norm to 153 estimate our loss. However, consider taking two independent samples from two corresponding identi154 cally distributed, independent random variables Y (1) and Y (2) . Then E [ Y (1) ] 2 = E [ Y (1) ] E [ Y (2) ] = 155

In NFGs, each player's utility is linear in x i , thereby satisfying the concavity condition of Lemma 1. 121

122

4.2 Projected Gradient Norm as Loss

An equivalent description of a Nash equilibrium is a joint strategy x where every player's strategy is 123 a best response to the equilibrium (i.e., x i = BR i so that /epsilon1 i ( x ) = 0 ). Lemma 1 states that any interior 124 best response has zero projected-gradient norm, which inspires the following loss function 125

where η k > 0 represent scalar weights, or equivalently, step sizes to be explained next.

Proposition 1. The loss L is equivalent to NashConv , but where player k 's best response is approximated by a single step of projected-gradient ascent with step size η k : aBR k = x k + η k Π T ∆ ( ∇ k x k ) .

This connection was already pointed out in prior work for unconstrained problems [15, 35], but this result is the first for strategies constrained to the simplex.

4.3 Connection to True Exploitability

In general, we can bound exploitability in terms of the projected-gradient norm as long as each player's utility is concave (this result extends beyond gradients to subgradients of non-smooth functions).

Lemma 2. The amount a player can gain by exploiting a joint strategy x is upper bounded by a quantity proportional to the norm of the projected-gradient:

This bound is not tight on the boundary of the simplex, which can be seen clearly by considering x k 137 to be part of a pure strategy equilibrium. In that case, this analysis assumes x k can be improved upon 138 by a projected-gradient ascent step (via the equivalence pointed out in Proposition 1). However, that 139 is false because the probability of a pure strategy cannot be increased beyond 1 . We mention this to 140 provide further intuition for why L ( x ) is only valid for interior equilibria. 141

Note that || Π T ∆ ( ∇ k x k ) || ≤ || ∇ k x k || because Π T ∆ is a projection. Therefore, this improves the naive 142 bounds on exploitability and distance to best responses given using the 'raw' gradient ∇ k x k . 143

Lemma 3. The exploitability of a joint strategy x , is upper bounded by a function of L ( x ) : 144

/negationslash

E [ Y (1) Y (2) ] by properties of expected value over products of independent random variables. This is 156 a common technique to construct unbiased estimates of expectations over polynomial functions of 157 random variables. Proceeding in this way, define ∇ k (1) x k as a random variable distributed according to 158 the distribution induced by all other players' mixed strategies ( j = k ). Let ∇ k (2) x k be independent and 159 distributed identically to ∇ k (1) x k . Then 160

161

162

163

164

165

166

167

168

169

170

where ˆ ∇ k ( p ) x k is an unbiased estimator of player k 's gradient. This unbiased estimator can be constructed in several ways. The most expensive, an exact estimator, is constructed by marginalizing player k 's payoff tensor over all other players' strategies. However, a cheaper estimate can be obtained at the expense of higher variance by approximating this marginalization with a Monte Carlo estimate of the expectation. Specifically, if we sample a single action for each of the remaining players, we can construct an unbiased estimate of player k 's gradient by considering the payoff of each of its actions against the sampled background strategy. Lastly, we can consider constructing a Monte Carlo estimate of player k 's gradient by sampling only a single action from player k to represent their entire gradient. Each of these approaches is outlined in Table 2 along with the query complexity [3] of computing the estimator and bounds on the values it can take (derived via Lemma 19).

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

We can extend Lemma 3 to one that holds under T samples with probability 1 -δ by applying, for example, a Hoeffding bound: /epsilon1 ≤ f ( ˆ L ( x ) + O ( √ 1 T ln(1 / δ ) ) .

4.5 Interior Equilibria

We discussed earlier that L ( x ) captures interior equilibria. But some games may only have pure equilibria. We show how to circumvent this shortcoming by considering quantal response equilibria (QREs), specifically, logit equilibria. By adding an entropy bonus to each player's utility, we can

guarantee all equilibria are interior,

still obtain unbiased estimates of our loss,

maintain an upper bound on the exploitability /epsilon1 of any approximate equilibrium in the original game (i.e., the game without an entropy bonus).

Define u τ k ( x ) = u k ( x ) + τ S ( x k ) where the Shannon entropy S ( x k ) = -∑ l x kl ln( x kl ) is a 1 -strongly concave function with respect to the 1 -norm [6]. Also define L τ ( x ) as before except where ∇ k x k is replaced with ∇ k τ x k = ∇ x k u τ k ( x ) , i.e., the gradient of player k 's utility with the entropy bonus. It is well known that Nash equilibria of entropy-regularized games satisfy the conditions for logit equilibria [23], which are solutions to the fixed point equation x k = softmax ( ∇ k x k τ ) . The appearance of the softmax makes clear that all probabilities have positive mass at positive temperature.

Recall that in order to construct an unbiased estimate of our loss, we simply needed to construct 187 unbiased estimates of player gradients. The introduction of the entropy term to player k 's utility is 188 special in that it depends entirely on known quantities, i.e., the player's own mixed strategy. We 189 can directly and deterministically compute τ dS dx k = -τ (ln( x k ) + 1 ) and add this to our estimator of 190 ∇ k ( p ) x k : ˆ ∇ k τ ( p ) x k = ˆ ∇ k ( p ) x k + τ dS dx k . Consider our refined loss function with changes in blue: 191

10-1

1.0

≤f(c) with t = 0

1.0 =

≤ f(c) with  = 75×10-3

1.0

ε≤f(c) with  = 100×10-3

1.0

ε ≤ f(c) with  = 125×103

1.0

ε ≤ f(c) with  = 150×10-3

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

10

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.0

0.0

0.2

Player 2 Prob[Swerve]

0.4

0.6

0.8

1.0

0.0

0.0

0.2

Player 2 Prob[Swerve]

0.4

0.6

0.8

1.0

0.0

0.0

0.2

Player 2 Prob[Swerve]

0.4

0.6

0.8

1.0

0.0

0.0

0.2

Player 2 Prob[Swerve]

0.4

0.6

0.8

1.0

0.0

0.0

0.2

Player 2 Prob[Swerve]

0.4

0.6

0.8

1.0

PrisonersDilemma

Player1Prob[Cooperate]

1.0

≤f(c) with t = 0

1.0

≤f(c) with t = 75×10-3

1.0

ε ≤f(c) with  = 100×10-3

1.0

≤ f(c) with t = 125×10-3

1.0

≤ f(c) with t= 150×10-3

0.8 -

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.6

10°

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.0

0.0

Player 2 Prob[Cooperate]

0.2

0.4

0.6

0.8

1.0

0.0

0.0

Player 2 Prob[Cooperate]

0.2

0.4

0.6

0.8

1.0

0.0 +

0.0

Player 2 Prob[Cooperate]

0.2

0.4

0.6

0.8

1.0

0.0

0.0

Player 2 Prob[Cooperate]

0.2

0.4

0.6

0.8

1.0

0.0

0.0

Player 2 Prob[Cooperate]

0.2

0.4

0.6

0.8

1.0

As mentioned above, the utilities with entropy bonuses are still concave, therefore, a similar bound 192 to Lemma 2 applies. We use this to prove the QRE counterpart to Lemma 3 where /epsilon1 QRE is the 193 exploitability of an approximate equilibrium in a game with entropy bonuses. 194

195

196

197

198

199

200

201

202

203

204

205

206

Lemma 4. The entropy regularized exploitability, /epsilon1 QRE , of a joint strategy x , is upper bounded as:

Lastly, we establish a connection between quantal response equilibria and Nash equilibria that allows us to approximate Nash equilibria in the original game via minimizing our modified loss L τ ( x ) .

Lemma 14 ( L τ Scores Nash Equilibria) . Let L τ ( x ) be our proposed entropy regularized loss function with payoffs bounded in [0 , 1] and x be an approximate QRE. Then it holds that

where W is the Lambert function: W ( 1 / e ) = W (exp( -1)) ≈ 0 . 278 .

This upper bound is plotted as a heatmap for familiar games in Figure 1. Notice how pure equilibria are not visible as minima for zero temperature, but appear for slightly warmer temperatures.

5 Analysis

In the preceding section we established a loss function that upper bounds the exploitability of an approximate equilibrium. In addition, the zeros of this loss function have a one-to-one correspondence with quantal response equilibria (which approximate Nash equilibria at low temperature).

Here, we derive properties that suggest it is 'easy' to optimize. While this function is generally 207 non-convex and may suffer from a proliferation of saddle points and local maxima (Figure 2) , it is 208 Lipschitz continuous (over a subset of the interior) and bounded. These are two commonly made 209 assumptions in the literature on non-convex optimization, which we leverage in Section 6. In addition, 210 we can derive its gradient, its Hessian, and characterize its behavior around global minima. 211

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

ProliferationofSaddlePoints

.10)

10-1

0

2p Sheriff

上

10-4

Exploitability

m

3p Colored Trails

10-7

4pBridge

5p Goofspiel

10-10

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Indexofcriticalpoint(α)

Lemma 15. The gradient of L τ ( x ) with respect to player l 's strategy x l is 212

/negationslash where B ll = -τ [ I -1 m l 11 /latticetop ] diag ( 1 x l ) and B kl = [ I -1 m k 11 /latticetop ] H k kl for k = l .

Lemma 17. The Hessian of L τ ( x ) can be written

where ˜ B kl = √ η k B kl , Π T ∆ ( ˜ ∇ τ ) = [ η 1 Π T ∆ ( ∇ 1 τ x 1 ) , . . . , η n Π T ∆ ( ∇ n τ x n )] , and we augment T (the 3 -player approximation to the game, T k lqk ) so that T l lll = τ diag3 ( 1 x 2 l ) .

At an equilibrium, the latter term disappears because Π T ∆ ( ∇ k τ x k ) = 0 for all k (Lemma 1). If X was R n ¯ m , then we could simply check if ˜ B is full-rank to determine if Hess /follows 0 . However, X is a simplex product, and we only care about curvature in directions toward which we can update our equilibrium. Toward that end, define M to be the n ( ¯ m +1) × n ¯ m matrix that stacks ˜ B on top of a repeated identity matrix that encodes orthogonality to the simplex:

where Π T ∆ ( z ∈ R a × b ) = [ I a -1 a 1 a 1 /latticetop a ] z subtracts the mean from each column of z and 1 x i is shorthand for diag ( 1 x i ) . If M ( x ) z = 0 for a nonzero vector z ∈ R n ¯ m , this implies there exists a z that 1) is orthogonal to the ones vectors of each simplex (i.e., is a valid equilibrium update direction) and 2) achieves zero curvature in the direction z , i.e., z /latticetop ( ˜ B /latticetop ˜ B ) z = z /latticetop ( Hess ) z = 0 , and so Hess is not positive definite. Conversely, if M ( x ) is of rank n ¯ m for a quantal response equilibrium x , then the Hessian of L τ at x in the tangent space of the simplex product ( X = ∏ i X i ) is positive definite. In this case, we call x well -isolated because it implies it is not connected to any other equilibria.

By analyzing the rank of M , we can confirm that many classical matrix games including RockPaper-Scissors, Chicken, Matching Pennies, and Shapley's game all induce strongly convex L τ 's at zero temperature (i.e., they have unique mixed Nash equilibria). In contrast, a game like Prisoner's Dilemma has a unique pure strategy that will not be captured by our loss at zero temperature.

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

Shapleys (β = 0.5)

GAMUT (D7-CovariantGame)

Blotto(10c,4f,3p)

10-3

10-1

0.3

0.20

FTRL (s = 1)

RM (S = 1)

M

1011

1015

10-7

10-3

102

0.2

0.1

0.15

0.10

0.05

yQREauto (s = ∞o)

yQREauto (s = 1)

yQREauto (s = 102)

SGD (s = 102)

SGD (s = 2)

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

0.0

2.5

5.0

7.5

10.0

0.00

0.0

2.5

5.0

7.5

10.0

SGD (s = ∞)

Iterations (Thousands)

Iterations (Thousands)

Iterations (Thousands)

Iterations (Thousands)

6 Algorithms

We have formally transformed the approximation of Nash equilibria in NFGs into a stochastic optimization problem. To our knowledge, this is the first such formulation that allows one-shot unbiased Monte-Carlo estimation which is critical to introduce the use of powerful algorithms capable of solving high dimensional optimization problems. We explore two off-the-shelf approaches.

Stochastic gradient descent is the workhorse of high-dimensional stochastic optimization. It comes with guaranteed convergence to stationary points [10], however, it may converge to local, rather than global minima. It also enjoys implicit gradient regularization [4], seeking 'flat' minima and performs approximate Bayesian inference [26]. Despite the lack of global convergence guarantee, in the next section, we find it performs well empirically in games previously examined by the literature.

We explore one other algorithmic approach to non-convex optimization based on minimizing regret, which enjoys finite time convergence rates. X -armed bandits [8] systematically explore the space of solutions by refining a mesh over the joint strategy space, trading off exploration versus exploitation of promising regions. 2 Several approaches exist [5, 37] with open source implementations (e.g., [24]).

6.1 High Probability, Polynomial Convergence Rates

We use a recent X -armed bandit approach called BLiN [14] to establish a high probability ˜ O ( T -1 / 4 ) convergence rate to Nash equilibria in n -player, general-sum games under mild assumptions. The quality of this approximation improves as τ → 0 , at the same time increasing the constant on the convergence rate via the Lipschitz constant √ ˆ L defined below. For clarity, we assume users provide a temperature in the form τ = 1 ln(1 /p ) with p ∈ (0 , 1) which ensures all equilibria have probability mass greater than p m ∗ for all actions (Lemma 9). Lower p corresponds with lower temperature.

The following convergence rate depends on bounds on the exploitability in terms of the loss (Lemma 14), bounds on the magnitude of estimates of the loss (Lemma 8), Lipschitz bounds on the infinity norm of the gradient (Corollary 2), and the number of distinct strategies ( n ¯ m = ∑ k m k ).

Theorem 1 (BLiN PAC Rate) . Assume η k = η = 2 / ˆ L , τ = 1 ln(1 /p ) , and a previously pulled arm is returned uniformly at random (i.e., t ∼ U ([ T ]) ). Then for any w > 0

with probability (1 -w -1 )(1 -2 T -2 ) where W is the Lambert function ( W (1 /e ) ≈ 0 . 278 ), m ∗ = max k m k , c ≤ 1 4 n ¯ m ˆ L ( ln( m ∗ ) ln(1 /p ) +2 ) 2 ≤ 1 4 ( ln( m ∗ ) ln(1 /p ) +2 ) upper bounds the range of stochastic estimates of L τ (see Lemma 8), and ˆ L = ( ln( m ∗ ) ln(1 /p ) +2 )( m ∗ 2 p ln(1 /p ) + n ¯ m ) (see Corollary 2).

This result depends on the near-optimality [37] or zooming -dimension d z = n ¯ m ( α hi -α lo α lo α hi ) ∈ [0 , ∞ ) (Theorem 2) where α lo and α hi denote the degree of the polynomials that lower and upper bound the function L τ ◦ s locally around an equilibrium. For example, in the case where the Hessian is positive definite, α lo = α hi = 2 and d z = 0 . Here, s : [0 , 1] n ( ¯ m -1) → ∏ i ∆ m i -1 is any function that maps from the unit hypercube to a product of simplices; we analyze two such maps in the appendix.

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

7-Player, Symmetric, 2-Action Game

Joint Strategy

101

TrueExploitability

Exploitability of

10-1

(E)

ε≤f(C)

10-3.

SymmetricJ

ε≤f(Ct=0.01)

10-5

1.14

1.60

0.82

0.56

BanditSamples

0.69

10-7.

0.0

0.2

0.4

0.6

0.8

1.0

Probability of Selecting Action 1 of 2

Note that Theorem 1 implies that for games whose corresponding L τ has zooming dimension d z = 0 , NEs can be approximated with high probability in polynomial time. This general property is difficult to translate concisely into game theory parlance. For this reason, we present the following more interpretable corollary which applies to a more restricted class of games.

Corollary 1. Consider the class of NFGs with at least one QRE( τ ) whose local polymatrix approximation indicates it is isolated (i.e., M from equation (12) is rankn ¯ m implies Hess /follows 0 implies d z = n ¯ m ( 2 -2 4 ) = 0 ). Then by Theorem 1, BLiN is a fully polynomial-time randomized approximation scheme (FPRAS) for QREs and is a PRAS for NEs of games in this class.

To convey the impact of stochastic optimization guarantees more concretely, assume we are given that an interior well-isolated NE exists. Then for a 20 -player, 50 -action game, it is 1000 × cheaper to compute a 1 / 100 -NE with probability 95% than it is to just list the nm n payoffs that define the game.

6.2 Empirical Evaluation

Figure 3 shows SGD is competitive with scalable techniques to approximating NEs. Shapley's game induces a strongly convex L (see Section 5) leading to SGD's strong performance. Blotto shows signs of convergence to low, but nonzero /epsilon1 , demonstrating the challenges of local minima.

We demonstrate BLiN (applied to L τ ) on a 7 -player, symmetric, 2 -action game. Figure 4 shows the bandit algorithm discovers two equilibria, settling on one near x = [0 . 7 , 0 . 3] × 7 with a wider basin of attraction (and higher welfare). In theory, BLiN can enumerate all NEs as T →∞ .

7 Conclusion

In this work, we proposed a stochastic loss for approximate Nash equilibria in normal-form games. 286 An unbiased loss estimator of Nash equilibria is the 'key' to the stochastic optimization 'door' 287 which holds a wealth of research innovations uncovered over several decades. Thus, it allows the 288 development of new algorithmic techniques for computing equilibria. We consider bandit and vanilla 289 SGD methods in this work, but theses are only two of the many options now at our disposal (e.g, 290 adaptive methods [1], Gaussian processes [9], evolutionary algorithms [17], etc.). Such approaches as 291 well as generalizations of these techniques to imperfect-information games are promising directions 292 for future work. Similarly to how deep learning research first balked at and then marched on to train 293 neural networks via NP-hard non-convex optimization, we hope computational game theory can 294 march ahead to make useful equilibrium predictions of large multiplayer systems. 295

296

297

298

References

K. Antonakopoulos, P. Mertikopoulos, G. Piliouras, and X. Wang. Adagrad avoids saddle points. In International Conference on Machine Learning , pages 731-771. PMLR, 2022.

P. Austrin, M. Braverman, and E. Chlamtáˇ c. Inapproximability of NP-complete variants of Nash 299 equilibrium. In Approximation, Randomization, and Combinatorial Optimization. Algorithms 300 and Techniques: 14th International Workshop, APPROX 2011, and 15th International Workshop, 301 RANDOM 2011, Princeton, NJ, USA, August 17-19, 2011. Proceedings , pages 13-25. Springer, 302 2011. 303

Y. Babichenko. Query complexity of approximate Nash equilibria. Journal of the ACM (JACM) , 304 63(4):36:1-36:24, 2016. 305

306

307

D. Barrett and B. Dherin. Implicit gradient regularization. In International Conference on Learning Representations , 2020.

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

P. L. Bartlett, V . Gabillon, and M. Valko. A simple parameter-free and adaptive approach to optimization under a minimal local smoothness assumption. In Algorithmic Learning Theory , pages 184-206. PMLR, 2019.

A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

S. Bubeck, R. Munos, G. Stoltz, and C. Szepesvári. X -armed bandits. Journal of Machine Learning Research , 12(5), 2011.

D. Calandriello, L. Carratino, A. Lazaric, M. Valko, and L. Rosasco. Scaling gaussian process optimization by evaluating a few unique candidates multiple times. In International Conference on Machine Learning , pages 2523-2541. PMLR, 2022.

A. Cutkosky, H. Mehta, and F. Orabona. Optimal stochastic non-smooth non-convex optimization through online-to-non-convex conversion. arXiv preprint arXiv:2302.03775 , 2023.

C. Daskalakis, P. W. Goldberg, and C. H. Papadimitriou. The complexity of computing a Nash equilibrium. Communications of the ACM , 52(2):89-97, 2009.

Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems , 27, 2014.

A. Deligkas, J. Fearnley, A. Hollender, and T. Melissourgos. Pure-circuit: Strong inapproximability for PPAD. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 159-170. IEEE, 2022.

Y. Feng, T. Wang, et al. Lipschitz bandits with batched feedback. Advances in Neural Information Processing Systems , 35:19836-19848, 2022.

I. Gemp, R. Savani, M. Lanctot, Y. Bachrach, T. Anthony, R. Everett, A. Tacchetti, T. Eccles, and J. Kramár. Sample-based approximation of Nash in large many-player games via gradient descent. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 507-515, 2022.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems , 27, 2014.

N. Hansen, S. D. Müller, and P. Koumoutsakos. Reducing the time complexity of the de340 randomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary 341 computation , 11(1):1-18, 2003. 342

J. C. Harsanyi, R. Selten, et al. A general theory of equilibrium selection in games. MIT Press 343 Books , 1, 1988. 344

E. Hazan, K. Singh, and C. Zhang. Efficient regret minimization in non-convex games. In 345 International Conference on Machine Learning , pages 1433-1441. PMLR, 2017. 346

347

348

E. Janovskaja. Equilibrium points in polymatrix games. Lithuanian Mathematical Journal , 8 (2):381-384, 1968.

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, and T. Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems , pages 4190-4203, 2017.

M. Lanctot, E. Lockhart, J.-B. Lespiau, V. Zambaldi, S. Upadhyay, J. Pérolat, S. Srinivasan, F. Timbers, K. Tuyls, S. Omidshafiei, D. Hennes, D. Morrill, P. Muller, T. Ewalds, R. Faulkner, J. Kramár, B. D. Vylder, B. Saeta, J. Bradbury, D. Ding, S. Borgeaud, M. Lai, J. Schrittwieser, T. Anthony, E. Hughes, I. Danihelka, and J. Ryan-Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR , abs/1908.09453, 2019. URL http://arxiv.org/abs/1908.09453 .

S. Leonardos, G. Piliouras, and K. Spendlove. Exploration-exploitation in multi-agent competition: convergence with bounded rationality. Advances in Neural Information Processing Systems , 34:26318-26331, 2021.

W. Li, H. Li, J. Honorio, and Q. Song. Pyxab - a python library for X -armed bandit and online blackbox optimization algorithms, 2023. URL https://arxiv.org/abs/2303.04030 .

C. K. Ling, F. Fang, and J. Z. Kolter. What game are we playing? end-to-end learning in normal and extensive form games. arXiv preprint arXiv:1805.02777 , 2018.

S. Mandt, M. D. Hoffman, and D. M. Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research , 18:1-35, 2017.

L. Marris, I. Gemp, and G. Piliouras. Equilibrium-invariant embedding, metric space, and fundamental set of 2x2 normal-form games. arXiv preprint arXiv:2304.09978 , 2023.

R. D. McKelvey and T. R. Palfrey. Quantal response equilibria for normal form games. Games and Economic Behavior , 10(1):6-38, 1995.

D. Milec, J. ˇ Cern` y, V . Lis` y, and B. An. Complexity and algorithms for exploiting quantal opponents in large two-player games. Proceedings of the AAAI Conference on Artificial Intelligence , 35(6):5575-5583, 2021.

P. R. Milgrom and R. J. Weber. A theory of auctions and competitive bidding. Econometrica: Journal of the Econometric Society , pages 1089-1122, 1982.

K. G. Murty and S. N. Kabadi. Some NP-complete problems in quadratic and nonlinear programming. Technical report, 1985.

H. Nikaidô and K. Isoda. Note on non-cooperative convex games. Pacific Journal of Mathematics , 5(1):807815, 1955.

E. Nudelman, J. Wortman, Y. Shoham, and K. Leyton-Brown. Run the GAMUT: A comprehensive approach to evaluating game-theoretic algorithms. In AAMAS , volume 4, pages 880-887, 2004.

J. Pérolat, S. Perrin, R. Elie, M. Laurière, G. Piliouras, M. Geist, K. Tuyls, and O. Pietquin. Scaling mean field games by online mirror descent. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , 2022.

A. Raghunathan, A. Cherian, and D. Jha. Game theoretic optimization via gradient-based Nikaido-Isoda function. In International Conference on Machine Learning , pages 5291-5300. PMLR, 2019.

Y. Shoham and K. Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical 389 foundations . Cambridge University Press, 2008. 390

M. Valko, A. Carpentier, and R. Munos. Stochastic simultaneous optimistic optimization. In 391 International Conference on Machine Learning , pages 19-27. PMLR, 2013. 392

B. Wiedenbeck and E. Brinkman. Data structures for deviation payoffs. In Proceedings of the 393 22nd International Conference on Autonomous Agents and Multiagent Systems , 2023. 394

Y. Zhou, J. Li, and J. Zhu. Identify the Nash equilibrium in static games with random payoffs. 395 In International Conference on Machine Learning , pages 4160-4169. PMLR, 2017. 396

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulates a Lipschitz loss function that makes computing approximate interior Nash equilibria in normal-form games amenable to unbiased Monte Carlo estimation, opening the door to using a number of scalable stochastic optimization techniques. They also provide a loss function with similar properties but under the notion of quantal-response equilibrium (QRE). The authors also provide certain illustrative experiments to support their claims.\n\nSTRENGTHS:\nThis paper provides a novel approach to computing equilibria in multi-player games. In particular, the authors derive loss functions that make equilibrium computation amenable to scalable methods from stochastic optimization. Given the lack of scalable algorithms for computing solutions concepts such as the Nash equilibrium, this is a promising approach, and has the potential to bring many new insights to equilibrium computation. In particular, the idea proposed for deriving an unbiased estimator (Section 4.4) is interesting, and addresses many of the pitfalls of other commonly used loss functions in the more challenging constrained setting. Furthermore, the presentation and the writing are overall clear, and the authors accurately place their results into the existing literature.\n\nWEAKNESSES:\nThere are a number of issues that weaken the contribution of the paper. First, the underlying assumption that there is an interior Nash equilibrium is very strong. For one, if there is an interior NE it is known that it can be computed in polynomial time via linear programming, which significantly weakens the motivation regarding the hardness of NE. There appears to be some confusion regarding this point. For example, Corollary 3 in the Appendix claims a new FPTAS for computing interior NE in polymatrix games, which I believe is known (beyond polymatrix games); there is perhaps still some benefit in using the proposed methodology in practice, but no evidence of that is provided in the paper. (As an aside, it would be helpful to clarify in the prelimaries that by interior you mean relative interior.) Beyond the very restrictive assumption of having an interior Nash equilibrium, the authors provide similar results for QRE, but that is a significantly weaker equilibrium concept. I would also strongly recommend clarifying in the abstract that your results apply for interior NE; as it is written currently it is very misleading. \n\nBesides the issue mentioned above, there is an underlying premise in the proposed methodology which I find unconvincing: Why should we expect local optima in the formulated loss functions to give meaningful guarantees? The fact that this turned out to be the case in many ML applications is not enough to justify this proposition. It is a significant weakness that the proposed method has no theoretical finite-time guarantees of reaching a Nash equilibrium. \n\nAnd unfortunately the experiments do not offer enough evidence to support this approach. Indeed, there are many issues in the experiments that can be significantly improved. First, the games experimented on are overly small; for example, Shapley's game is completely toy, no meaningful conclusions can be drawn from it. Since the main message of this paper is about scalability, I expected to see experiments on much bigger games. It would be helpful if the proposed theory applied in extensive-form games for which there are many large benchmark games in the literature, but the current method is tailored to normal-form games. Besides this issue, I am very confused regarding the compared benchmarks. It is claimed in the last sentence of the abstract that the method often outperforms prior state of the art, but the main algorithms compared against are RM and FTRL. These algorithms will not even find an NE in finite-time, how can you claim that those are the state of the art? In particular, in Lines 966-969 it is claimed that those are the two most popular scalable stochastic algorithms for approximating NE; I strongly disagree with this claim. I would suggest trying some other benchmarks, such as the Lemke-Howson algorithm; a mixed-integer programming approach; or the algorithm presented in \"Exclusion Method for Finding Nash Equilibrium in Multiplayer Games.\"\n\nAnother issue is on the proof of Corollary 1. For a constant $\\epsilon$, it is claimed that you have a poly-time algorithm (since it is a PRAS), but you also say that the temperature parameter has to be exponentially small to achieve that. So it seems that even for a constant $\\epsilon$ you need an exponential number of iterations to converge. \n\nOverall, although the approach proposed is promising, there are a number of issues that have to be addressed before the paper is ready for publication.\n\nQUESTIONS:\nSome minor issues:\n\n1. There are many missing punctuation marks in the equations throughout the paper\n2. There are many overfull equations in the Appendix; I would recommend fixing those in the revised version\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach for determining the Nash equilibrium of normal form games, utilizing a solution to a non-convex stochastic optimization problem. It defines the Nash equilibria in normal form games as the global minima of a specifically cunstructed loss function. Moreover, a randomized algorithm is developed to resolve this newly proposed loss function. Finally, empirical results further verify the theoretical analysis.\n\nSTRENGTHS:\nThough the idea of loss function has been proposed before, this paper contributes to the discourse with several innovative insights that enhance the understanding and applicability of loss functions. For example, this paper restricts the parameter to the simplex, which is the key of making stochastic gradient unbiased.\n\nRegarding the quality and clarity, this paper is sufficiently complete. It also provides clear backgrounds, which make it easy to understand how this loss function comes from. It is not completely new but it has something new.\n\nWEAKNESSES:\nThe motivation of this work is not sufficiently clear. I could understand solving a Nash equilibrium may not be efficient but I don't think proposing a NE solver via unbiased stochastic optimization will make it better. \n\nIt is unclear how this method is better than some existing NE solver such as Lemke–Howson algorithm.\n\nQUESTIONS:\n1. I agree that there is a gap between the success of using SGD solving non-convex optimization problem and the failure of efficiently computing Nash equilibria. Why does this motivate the goal: \"Can we solve for Nash equilibria via unbiased stochastic optimization\"? To my understanding, solving a non-convex optimization problem is still very hard. \n\n2.  Solving a non-convex optimization problem may lead to a stationary point instead of the global minima. Why does this proposed method is better than using an existing NE solver such as Lemke–Howson algorithm to ensure obtaining a NE?\n\nLIMITATIONS:\nThis is a theoretical work so there is no negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the computation of Nash equilibria (NE) of normal-form games and proposes a new loss function: the (weighted) sum of the squared norms of the projections of each player gradient onto the tangent space of the unit simplex. The authors show that this loss function is a meaningful surrogate of exploitability when the game has an interior equilibria. Then, the authors provide methods to efficiently construct unbiased estimators of the loss function via unbiased estimation of each player gradients. To extend these results to handle games with only pure equilibria, the authors propose surrogate player utility functions via entropy regularization (with coefficient $\\tau$, the \"temperature\") and show how the modified loss function (based on the modified game with surrogate player utility functions) captures the exploitability of the original game. Next, the authors derive gradient and Hessian expressions for the modified loss function. Leveraging a recent bandit optimization method BLiN, and assuming a sufficiently large temperature (which degrades the convergence rate), the authors provide a high-probability convergence guarantee for computing NE using this approach (loss function + BLiN). Experiments on SGD and BLiN show the effectiveness of the proposed approaches.\n\nSTRENGTHS:\n- Novel observation of the connection between projection of player gradient to simplex tangent space and best response, which lead to the loss function proposed in this paper.\n- Extensive studies of the newly proposed loss function in terms of its gradient, Hessian, and other properties.\n\nWEAKNESSES:\nSome technical details seem to require further clarification. See **Questions**.\n\nQUESTIONS:\n- Since BLiN is technically a zeroth order method (pulling an arm <==> sampling a function value), can you elaborate/repeat, somewhere around Theorem 1, what is the oracle passed into BLiN? I believe it should be a Monte-Carlo approximation through (6) but with the player gradients being the ones with temperatures (entropy regularization). In other words, please point out what needs to be computed in each step of BLiN.\n- As stated in 229-231, if a NFG has a unique equilibrium which is also mixed, then $\\mathcal{L}$ is strongly convex. Based on earlier results in this paper, are there other conditions that ensure strong or non-strong convexity of $\\mathcal{L}$ (or $\\mathcal{L}^\\tau$)? It would be helpful to state them explicitly, as many stochastic optimization methods can exploit (strong) convexity.\n\nLIMITATIONS:\nThis is a methodological work that does not have immediate or potential negative societal impact. The limitations are on the technical contributions and are discussed above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This work studies solving Nash Equilibria (NE) by stochastic unbiased optimization. The main contribution is providing a new loss function based on the gradient norm of the utility function, and finding the NE by using standard stochastic optimization methods (like Lipschitz bandit algorithms and stochastic gradient descent (SGD)). The authors also carried our experiment results on several games to show the scalability of their proposed methods.

STRENGTHS:
The presentation of this work is very clear. The experiment results are comprehensive and back up the main claims of this work. The results are also significant as they point out a new way to solve the NE problem in general.

WEAKNESSES:
More remarks are supposed to be added to the main text. For example, 

- What does 's' mean in the legend of Figure 3? 

- In Table 1, why the obstacle of NI method is 'max of random variable'? I did not see any max operator in the definition of the loss function of NI. 

- In Table 1, for the unconstrained method, can the authors show one concrete example to show why it 'lose the ability to sample from strategies when iterates are no longer proper distributions', as stated in line 113?

QUESTIONS:
The same to the 'Weaknesses' section.

LIMITATIONS:
This work aims to solve an open problem about the algorithmic game theory, thus it does not need to address the potential negative societal impacts of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes a loss function (optimization problem) for normal-form games to estimate Nash equilibria which can be solved via unbiased stochastic optimization. They do this by relating their proposed loss function with exploitability. They also provide theoretical guarantees (under some technical conditions) of using bandit stochastic gradient algorithms to solve their proposed problem. They show the applicability of their method by conducting some numerical experiments.

STRENGTHS:
The paper tackles an important problem in the game theory of estimating Nash equilibria using optimization. It proposes a potentially scalable solution and provides theoretical guarantees for the same. The paper is well-written (albeit a bit notation-heavy) and the content is easy to follow.

WEAKNESSES:
The authors propose an optimization problem with possible unbiased estimators. However, the proposed problem is still non-convex and it is not clear to me whether it can be solved efficiently with SGD with a potentially large number of saddle points. I understand the analogy to deep learning problems but recent work ([1] and related papers) have shown that those problems carry some interesting structure. Similar properties are unknown (and are perhaps more difficult to establish) for the proposed function.

Du, S., Lee, J., Li, H., Wang, L., & Zhai, X. (2019, May). Gradient descent finds global minima of deep neural networks. In International conference on machine learning (pp. 1675-1685).

QUESTIONS:
Could authors comment on the applicability of their methods on real problems in the context of my comments in the "Weaknesses" section?

LIMITATIONS:
The limitations are addressed adequately by the authors.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08