Title: Towards Characterizing the First-order Query Complexity of Learning (Approximate) Nash Equilibria in Zero-sum Matrix Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: game theory

minimax optimization

lower bounds
Abstract: In the first-order query model for zero-sum $K\times K$ matrix games, players observe the expected pay-offs for all their possible actions under the randomized action played by their opponent. This classical model has received renewed interest after the discovery by Rakhlin and Sridharan that $\epsilon$-approximate Nash equilibria can be computed efficiently from $O(\frac{\ln K}{\epsilon})$ instead of $O(\frac{\ln K}{\epsilon^2})$ queries. Surprisingly, the optimal number of such queries, as a function of both $\epsilon$ and $K$, is not known. We make progress on this question on two fronts. First, we fully characterise the query complexity of learning exact equilibria ($\epsilon=0$), by showing that they require a number of queries that is linear in $K$, which means that it is essentially as hard as querying the whole matrix, which can also be done with $K$ queries. Second, for $\epsilon > 0$, the current query complexity upper bound stands at $O(\min(\frac{\ln(K)}{\epsilon} , K))$. We argue that, unfortunately, obtaining a matching lower bound is not possible with existing techniques: we prove that no lower bound can be derived by constructing hard matrices whose entries take values in a known countable set, because such matrices can be fully identified by a single query. This rules out, for instance, reducing to an optimization problem over the hypercube by encoding it as a binary payoff matrix. We then introduce a new technique for lower bounds, which allows us to obtain lower bounds of order $\tilde\Omega(\log(\frac{1}{K\epsilon})$ for any $\epsilon \leq 1 / (cK^4)$, where $c$ is a constant independent of $K$. We further discuss possible future directions to improve on our techniques in order to close the gap with the upper bounds.

Towards Characterizing the First-order Query Complexity of Learning (Approximate) Nash Equilibria in Zero-sum Matrix Games

Abstract

In the first-order query model for zero-sum K × K matrix games, players observe the expected pay-offs for all their possible actions under the randomized action played by their opponent. This classical model has received renewed interest after the discovery by Rakhlin and Sridharan that ε -approximate Nash equilibria can be computed efficiently from O (ln K/ε ) instead of O (ln K/ε 2 ) queries. Surprisingly, the optimal number of such queries, as a function of both ε and K , is not known. We make progress on this question on two fronts. First, we fully characterise the query complexity of learning exact equilibria ( ε = 0 ), by showing that they require a number of queries that is linear in K , which means that it is essentially as hard as querying the whole matrix, which can also be done with K queries. Second, for ε > 0 , the current query complexity upper bound stands at O (min(ln( K ) /ε, K )) . We argue that, unfortunately, obtaining a matching lower bound is not possible with existing techniques: we prove that no lower bound can be derived by constructing hard matrices whose entries take values in a known countable set, because such matrices can be fully identified by a single query. This rules out, for instance, reducing to an optimization problem over the hypercube by encoding it as a binary payoff matrix. We then introduce a new technique for lower bounds, which allows us to obtain lower bounds of order ˜ Ω(log( 1 Kε )) for any ε ⩽ 1 / ( cK 4 ) , where c is a constant independent of K . We further discuss possible future directions to improve on our techniques in order to close the gap with the upper bounds.

1 Introduction

Computing the saddle point

for convex-concave functions f : X ×Y → R is of general interest throughout optimization, economics and machine learning. We study the computation of an approximate saddle point ( x ⋆ , y ⋆ ) , satisfying max y ∈Y f ( x ⋆ , y ) -min x ∈X f ( x, y ⋆ ) ⩽ 2 ε for some given ε ⩾ 0 . Starting with an unknown f from a known class F , we consider sequential learning in the first-order feedback model, where the learner gets to observe gradients of the objective. Formally, each query ( x, y ) results in feedback ( ∇ x f ( x, y ) , ∇ y f ( x, y )) . We are interested in the following question:

How many first-order queries are necessary and sufficient for a sequential learner to output an approximate saddle point for any f ∈ F ?

Characterizing the query complexity of learning saddle points is of theoretical interest for understanding the hardness of computing equilibria, and for certifying the optimality of upper bounds.

In this work, we restrict our attention to the special case of zero-sum matrix games, where X and Y are finite-dimensional probability simplices and f is bilinear. For this canonical setting, the optimal query complexity is, surprisingly, unresolved. Indeed, computation algorithms are known since [3], up to [37], while lower bounds remain elusive, leaving the optimal query complexity still unknown.

Obtaining lower bounds is not only of fundamental interest in itself, but may also lead to interesting new techniques, since none of the existing proof strategies are applicable. New ideas provided here could prove useful to tackle other problems.

1.1 Contributions

We make progress towards characterizing the first-order query complexity of approximate Nash equilibria in finite-action zero-sum games, as a function of the number of actions K and approximation level ε . Our contributions are the following:

Lower bounds We provide the first lower bounds on the first-order query complexity for zero-sum matrix games with bounded entries. We show that K/ 2 -1 queries are necessary to compute an exact equilibrium (Theorem 11 in Section 4), and at least Ω(log(1 /εK ) / log( K )) queries are required for an ε -equilibrium if ε ⩽ 1 /K 4 (Theorem 17 in Section 4). More than the concrete rates, we believe that the structure of our proof is of particular interest, and might be of use beyond this specific setting.

Upper Bounds We show that if the game matrix has entries in a known countable set, then the learner can recover the full matrix in one single first-order query (Theorem 4 in Section 3). This result can be interpreted both as an upper bound on the query complexity for matrices on countable entry sets, and as an impossibility result, precluding the use of simple predefined sets of matrices as candidate objectives for proving lower bounds. This, together with the lack of rotational invariance of the action sets, sheds light on why this setting turns out to be surprisingly resistant to proving lower bounds using well-established techniques.

1.2 Related Work on Lower Bounds: Lower Bounds

Below we review the literature on lower bound techniques for related problems and settings. We find that none of the existing techniques apply to our setting, for three particular reasons. They either:

Build matrices from a set of matrices with entries lying in a finite alphabet. In Theorem 4, we show that, for game matrices with entries in a known countable set, a single first-order query suffices to infer the exact Nash equilibrium. This shows in particular that any such matrix construction cannot lead to a lower bound in the first-order model.

Rely crucially on rotational invariance of the action set, which limits the approach to the ℓ 2 -constrained or unconstrained cases, ruling out the probability simplex.

Assume that the players select actions in the span of the observed gradients. This span assumption is not suited when the action set is not an ℓ 2 ball, in which case the span of the gradients has no natural embedding into the action set; most algorithms leave the span, e.g. exponential weights.

First-order Query Complexity of Minimax Optimization Existing lower bounds [24, 33] for minmax optimization are known for finite-dimensional Euclidean spaces where X , Y are either Euclidean balls or the whole space. Both these cases benefit from the property that the action sets are invariant under rotation, allowing for a step-by-step reduction to lower dimensional instances, as shown in the seminal work of [29, 30]. The notable exception of [14] also does not cover the case we study: they consider the computational hardness of non-convex non-concave optimisation.

In the unconstrained case with curvature, that is, if F is the set of strongly convex, strongly concave functions with marginal condition numbers κ x and κ y , the query complexity is settled to be Θ( √ κ x κ y log(1 /ε )) ; see [24, 43] for lower bounds and [27] and references therein for upper bounds.

For the constrained case, [33] provide lower bounds for bilinear saddle-point problems. They establish a query complexity of order Ω( L f D X D Y /ε ) , where D X and D Y are the respective diameters of the

constraint sets X and Y , and L f measures the Lipschitz regularity of f . However, their techniques rely crucially on rotational invariance and sequentially adapted constraint sets, and they ask the open question of whether similar lower bounds can also be shown for fixed constraint sets.

Other Query Models The query complexity of minimax optimization has also been studied under different feedback models. Recall that we focus on the bilinear case, where f : p, q ↦→ p ⊺ Mq for some game matrix M . [16] study the query complexity of (approximate) Nash equilibria of the game, i.e., (approximate) saddle points of f , under a query model where the learner chooses single entries of the matrix to observe. They show, among other results, that in order to compute an ε -equilibrium in K × K zero-sum games, the number of entries queried needs to be at least Ω( K log K ) when ε = O (1 / log K ) . [22] consider a setting in which the learner observes a best reponse to their query ( p, q ) , i.e., i ⋆ ∈ argmin( Mq ) i and j ⋆ ∈ argmin -( M ⊺ p ) j . In that setting, they show that to compute a 1 / 4 -equilibrium, at least Ω( √ K/ (log K ) 2 ) best-response queries are necessary. Since a first-order query brings strictly more information than either a best-response query or an entry query, these lower bounds do not have direct consequences for our query model. Both of these references prove their lower bounds by building hard matrices with entries in a known finite set (e.g., { 0 , 1 } ).

Other Lower Bounds in Saddle Point and Equilibria Computation For multiplayer games, strong lower bounds on the computational complexity of exact Nash equilibria have been uncovered: PPAD-hardness for computing the Nash-equilibrium in a general game, see [7-9], or in a non-convexconcave zero-sum game [14]. Regarding query complexity, [2, 21] study the single-entry query complexity for approximate correlated equilibria and Nash equilibria of games with many players. As in other references mentioned, all these bounds are built on matrices with entries in a finite set. Due to their intrinsically combinatorial nature, these techniques are less common for numerical algorithms. One of the classical methods for sample complexity lower bounds was introduced by [31] for first-order optimization. It was successfully extended to saddle-point problems [43] and recently also to unconstrained zero-sum games by [24]. However, these techniques rely on the assumption that the next iterate is chosen in the span of the previous oracle information (cf. Definition 1 in [24] or Assumption 2.1.4 in [31]). Moreover, the game constructed by [24] for a Nesterov-style lower bound results in matrices with entries from a finite alphabet. For the same reason, no proof technique inspired by lower bounds via Rademacher random variables, as in [32], can ever work.

1.2.1 Other Related Work

Upper Bounds for Finite Action Zero-Sum Games Early on, [3, 38] show that under bestresponse dynamics, the average plays converge to an equilibrium. The connection to regret bounds was established by [17], who deduce a O (log( K ) /ε 2 ) query complexity, by means of a construction akin to online-to-batch conversion. More recently, [11, 12] obtained the first fast rates of order O ( c ( K ) /ε ) , through an ingenious learning mechanism. [37] rediscovered the Optimistic Online Mirror Descent algorithm [36], and pioneered optimistic online learning regret bounds; they yield to this the day the fastest rates of O (log( K ) /ε ) and the simplest algorithm. The instance-dependent linear-rate upper bounds of order λ ( M ) ln(1 /ε ) by [19, 41] are essentially incomparable to the worst-case O (log( K ) /ε ) ; they are superior when λ ( M ) ≪ K , and vacuous in the typical case λ ( M ) ≈ K (see the discussion below Theorem 1).

Our focus is the quality of the inferred saddle point (this optimization perspective is sometimes called pure exploration ). Part of the literature regards the queries as actual moves made, and consequently prioritizes other objectives. In particular, many focus on studying uncoupled dynamics , that is, sequences of actions that separate the observations of the individual p and q players without allowing communication between the players. Another recent theme of interest is the last-iterate convergence of such dynamics, see e.g., [23].

Upper Bounds for Minmax Optimization The vast majority of the known upper bounds in minimax optimization (and more generally in variational inequality problems), outside of finite-action zero-sum games mentioned above, concern unconstrained settings [1, 20, 28]. The literature on constrained settings is more sparse: [4] recently proved the rate of convergence of the projected extragradient method [26]. [42] adapt interior methods to handle constraints. [18] propose a fast algorithm when either the action sets are strongly convex, or the objective is strongly-convex strongly-concave. Recently, non-convex-concave settings have also attracted attention, see [27] and references therein.

Upper Bounds for General-Sum Games and Correlated Equilibria For multiplayer games, uncoupled dynamics do not converge (in any sense) to Nash equilibria. However, regret-based procedures can find correlated and coarse correlated equilibria [5, 39, 35]. In particular, internal regret guarantees for individual players in general-sum multiplayer games were recently improved from the generic O (1 /ε 2 ) for any sequences of losses, to O (1 /ε 4 / 3 ) in [40], to O (1 /ε 6 / 5 ) in [6] and to O (log(1 /ε ) /ε ) in [13, 15] (we omit the dependence on the number of actions and players). Polynomial-time methods for efficient computation of exact correlated equilibria are designed by [25, 34], using their formulation as solutions to a linear program.

2 Setting and Notation

General Notation Given a set of numbers A ⊂ R , we denote by M K ( A ) the set of K × K matrices with entries in A ; we mainly consider the bounded-entries class M K ([ -1 , 1]) . A K × K zero-sum game between a minimizing p -player (or row player) and a maximizing q -player (or column player) is represented by a matrix M ∈ M K ( R ) ; we restrict our attention to square K × K games. For any pair of plays ( p, q ) , the expected loss vector of the p -player (resp. q -player) is Mq ∈ R K (resp. -M ⊺ p ∈ R K ). The suboptimality gap at ( p, q ) is

The gap g ( M,p,q ) is non-negative for any M,p and q . The pair of plays ( p, q ) is said to be an ε -Nash equilibrium if g ( M,p,q ) ⩽ 2 ε ; Nash equilibria are 0 -Nash equilibria. Most games we will consider will possess a unique Nash equilibrium p, q , and this equilibrium will be fully supported, i.e., all components of p and q are positive. Fully supported equilibria in finite games are also equalizing strategies, meaning that the loss vectors are then isotropic; in finite zero-sum games, this entails that M ⊺ p = Mq = v 1 , where v is the game-value. We follow the notation convention to abbreviate min( x, v ) = x ∧ v and to hide polylogarithmic terms by ˜ Ω or ˜ O .

First-Order Query Model and Objectives The first-order query model is an interaction protocol between a learner and a game matrix, defined as follows. Fix a time horizon T ∈ N , and a set of candidate matrices M ⊆ M K ( R ) . Over the course of T rounds, the learner sequentially picks (queries) pairs of plays ( p t , q t ) and observes the expected losses ( Mq t , -M ⊺ p t ) , where the matrix M ∈ M is fixed and unknown to the learner.

We examine the number of interactions necessary to compute an approximate equilibrium. At the end of the T rounds, the learner recommends a pair ( p, q ) . Given a fixed gap level ε ⩾ 0 and a set of candidate matrices M , we say a strategy achieves query complexity T ( ε ; M ) if for any matrix M ∈ M , the strategy outputs a pair p, q such that g ( M,p,q ) ⩽ 2 ε when T ⩾ T ( ε ; M ) . (The dependence on M is omitted when clear from context.) We also occasionally refer to the query complexity of recovering the full matrix, which is a different task but in the same query model. In this case, the learner recommends a full matrix and achieves query complexity T if the matrix guess is correct for any true matrix M ∈ M .

The topic of this paper is the study of the optimal query complexity of finding ε -Nash equilibria in the first-order model, for the set of matrices with bounded entries M K ([ -1 , 1]) .

3 Upper Bounds

In this section we collect upper bounds for the first-order query complexity of approximate Nash equilibria. These bounds are either well-established, trivial or exploit in a miraculously striking fashion the difference between learning saddle points for matrix games taking entries in a countable (e.g., Q ) or uncountable (e.g., [ -1 , 1] ) sets of values.

3.1 Query Complexity over M K ([ -1 , 1]) : Regret and Elementary Strategies

We list some well-known results from the literature and state them in terms of first-order query complexity. The results described here are compiled in Figure 1. We use the notation u n : m to denote the family ( u k ) n ⩽ k ⩽ m .

0

c/K

4

c

log

K

K

1

2

0

1

2

c

log

K

K

2

-

1

K

˜ Ω

(

log

1

)

by

Theorem 17

O

(

log

K

ε

)

Rakhlin and

Sridharan [37]

T

(

ε

)

= 1

Daskalakis et al.

[10]

ε

Queries

T

(

ε

)

1

-

1

K

0

1

2

c

log

K

K

2

-

1

K

T

(

ε

) = 0

Prop. 3

O ( ε -1 log K ) Queries from Optimistic Online Learning The current best upper bounds on the query complexity of ε -equilibria for zero-sum games are derived from online learning methods. As is well-known [5, Chapter 7], if we denote by ̂ p T (resp. ̂ q T ) the average of p 1: T (resp. q 1: T ) then

This last term is exactly the sum of the regrets suffered by each player on their respective losses: the gap of the average plays is smaller than the sum of the average regrets over T rounds. This relationship between regret and gap provides a fruitful way to upper bound the query complexity of computing equilibiria. Specifically, [37] observed that if players follow the Optimistic Mirror Descent strategy then the sum of the regrets stays smaller than O (log K ) ; see the paragraph following Proposition 6 in the mentioned reference.

Theorem 1 (Consequence of Rakhlin and Sridharan [37]) . There exists an absolute constant c > 0 such that the first-order query complexity over M K ([ -1 , 1]) is

A wide stream of literature leverages the connection between regret and equilibria, and study the dynamics of pairs of learning algorithms, including the two principal flavors of optimistic online algorithms, Optimistic Mirror Descent and Optimistic Follow-the-Regularized-Leader, cf. Section 1.2.

Instance-Dependent Rates of Convergence Many existing bounds show rates of convergence exponential in T , with instance-dependent constants 1 , see, e.g., [19, 41]. However, we show in Example 18 that for a zero-sum two-player bilinear game, these constants can be large, even on simple game matrices. If the constants are too large, the consequences on query complexity are vacuous, as they might require more than K queries to obtain non-trivial ( ε < 1 ) equilibria. Although these results give a very interesting analysis beyond the worst-case, they are beyond the scope of this work.

Upper Bounds for Large ε We conclude this section with two elementary strategies that provide equilibria with large approximation values of ε ⩾ 1 / 2 .

Theorem 2 (Theorem. 3.1 in [10]) . For any ε ⩾ 1 / 2 , the query complexity over M K ([ -1 , 1]) of finding an ε -equilibrium is T ( ε ) ⩽ 2 .

The following proposition is a well-known observation (see, e.g., [16]) that having the learner select a pair of uniform plays always gives an approximate equilibrium.

Proposition 3. For any ε ⩾ 1 -1 /K , the query complexity over M K ([ -1 , 1]) of finding an ε -equilibrium is T ( ε ) = 0 .

3.2 Recovering the Matrix: Upper and Lower Bounds

In this section we show that the first-order query complexity for recovering the full game matrix (a task harder than computing an equilibrium) is between Θ(1) , independent of the number of actions, and Θ( K ) , depending on the set of candidate matrices. An important consequence of these results is that several known standard lower bound techniques cannot be applied to provide lower bounds on the easier task of learning equilibria.

Matrix Sets with Countable Alphabet Many existing methods rely on building hard matrices with entries in a finite alphabet: we prove this cannot lead to a lower bound.

Theorem 4. Let A ⊂ R be a countable set of at least two numbers. Then the first-order query complexity over M K ( A ) of recovering the full matrix is 1 .

This result exploits infinite precision in the feedback, and as such does not provide a reasonable algorithm that the players would use should they know that the matrix belongs to M K ( A ) . The intent of this statement is to show that any attempt at a lower bound that builds matrices with entries in a fixed countable set (and in particular with integer entries) is doomed to fail after one query.

Remark 5. The proof of Theorem 4 does not provide an explicit algorithm. However, if the learner knows that the entries of the matrix are in a finite set A ⊂ Q , an explicit strategy can be easily described. In this case, A is of the form { a 1 /r, . . . , a n /r } with a i ∈ Z and r ∈ N . Consider a base b = max( a 1: n ) -min( a 1: n ) and consider a query with p ∝ ( b -1 , b -2 , . . . , b -K ) (the choice for q is irrelevant here). The learner can deduce the full matrix M from the single observation M ⊺ p . Of course, this strategy has no practical interest as soon as either A or K is moderately large, since it requires arbitrary precision in the outputs.

Difficulty of Recovering the Matrix Exactly In the first-order query model, the learner receives 2 K numbers every round, so in order to fully determine an arbitrary K × K matrix one needs at least K 2 / (2 K ) = K/ 2 rounds. It turns out that this bound is not tight, and we may need exactly K rounds in the worst case, because of redundancies between the 2 K numbers we observe per round:

Theorem 6. The first-order query complexity over M K ([ -1 , 1]) of recovering the full matrix is K .

Intuitively, it is overkill to recover the matrix exactly to output an ε -Nash equilibrium. In the next section, when ε ≪ 1 /K K (in particular when ε = 0 ), we show that first-order queries do not provide useful information to find an ε -equilibrium faster than it takes to reveal the whole matrix.

4 A New Lower Bound on the Query Complexity of Approximate Equilibria

We switch perspective, attempting to make the life of any learner hard. For this, we need to think about responding to queries to keep the learner unwitting.

4.1 Overview and Notation

After t time steps, queries p 1: t and q 1: t have been made and the outputs provided to the learner are ℓ ( q ) 1: t and ℓ ( p ) 1: t . Let us denote the set of matrices with entries in [ -1 , 1] that are consistent with the observations after t time steps by

we sometimes refer to this as the set of candidate matrices after t rounds of observations. We omit the dependence on the sequence of queries ( p s , q s ) and outputs ( ℓ ( p ) s , ℓ ( q ) s ) to reduce clutter, as it will always be clear from the context.

We say a sequence of matrices M 1: T is adapted to the queries p 1: T , q 1: T , if it gives consistent outputs to the queries, i.e., if for all s ⩽ t ⩽ T , M ⊺ t p s = M ⊺ s p s and M t q s = M s q s ; in other words, M 1: T is adapted if M t +1 ∈ E t for all t .

Instead of defining directly the answers to the queries, we equivalently build a sequence of adapted matrices. Formally, at round t +1 , given E t and ( p t +1 , q t +1 ) , we select a matrix M t +1 ∈ E t and output ( ℓ ( p ) t +1 , ℓ ( q ) t +1 ) = ( M t +1 q t +1 , -M ⊺ t +1 p t +1 ) .

Let us fix some common technical notation that we use in the proofs, to measure distances in matrix space. For a matrix M ∈ M K ( R ) , and for r, s ∈ [1 , ∞ ] , we denote the operator norm of M by ∥ M ∥ r,s = sup x ∈ R K : ∥ x ∥ r =1 ∥ Mx ∥ s . Recall that for y, z ∈ R K , we have ∥ yz ⊺ ∥ r,s = ∥ y ∥ s ∥ z ∥ r ′ , where r ′ ∈ [1 , ∞ ] is such that 1 /r + 1 /r ′ = 1 . In particular ∥ M ∥ 1 , ∞ = max i,j ∈ [ K ] | M i,j | , and ∥ yz ⊺ ∥ 1 , ∞ = ∥ y ∥ ∞ ∥ z ∥ ∞ . If F is a closed convex set subset of R K , we denote by Proj F ( x ) the orthogonal projection of x on F .

Common Structure of the Proofs Both proofs of Theorems 11 and 17 follow the same template:

Assume an ε -equilibrium is found after T first-order queries.

-Find a necessary condition on this equilibrium formulated as constraints on the observed losses and the sequence of queries. Precisely, we observe that under some initial assumptions on the game matrix, the all-ones vector 1 needs to lie near the span of the outputs observed by the player, regardless of the queries and outputs.

-Ensure that this necessary condition cannot be met by any pair of mixed actions ( p, q ) by building an appropriate adapted sequence of matrices. In our case, we ensure that the span of the losses of the q -player stays far from the all-ones vector 1 .

This proof structure is a promising way to derive the exact query complexity of (approximate) Nash equilibria. We use it to provide the first non-trivial lower bound for this setting.

The next two sections implement this template to prove query complexity lower bounds for exact ε = 0 (Theorem 11) and approximate ε > 0 (Theorem 17) equilibria. Detailed proofs are in Appendix D. While we could have deduced the final lower bound for exact equilibria directly from the approximate equilibria case, we describe the result separately to ease exposition. Indeed, the bound for the exact case contains the main ideas but is technically simpler.

4.2 Proof: Exact Equilibrium Case

Step I: Common Equilibria are in the Span of Past Queries We start by defining an initial set of candidate matrices with some special properties. These properties are used to ensure that the common equilibrium is fully mixed, and therefore an equalizing strategy for both players.

Definition 7. A set of matrices B 0 ⊂ M K ([ -1 , 1]) satisfies the assumptions A (0) if

For all M ∈ B 0 , all equilibria of M are fully mixed,

There exists a matrix M ∈ B 0 with non-zero value.

By Lemma 20 (App. D), any ball centered at (1 / 2) I K with small enough radius satisfies A (0) , e.g.,

The following lemma states that after t first-order queries, if at least one matrix in B 0 is still a candidate matrix, then any common equilibrium to all candidate matrices must lie in the span of the queries. Since equilibria of matrices in B 0 are fully mixed, and are thus equalizing strategies, this further implies that the all-ones vector 1 must lie in the span of the outputs to the queries.

̸

Lemma 8. Let B 0 be a set of matrices satisfying A (0) . Assume in addition that E t ∩ B 0 = ∅ and that E t ∩ M K (( -1 , 1)) = ∅ . If there exists a common Nash equilibrium ( p, q ) to all matrices in E t , then p ∈ Span( p 1: t ) and q ∈ Span( q 1: t ) .

̸

̸

Corollary 9. Under the assumptions of Lemma 8, there exists a value v = 0 such that

Step II: Sequential construction Given Corollary 9, to prove a query complexity lower bound, it suffices to build the answers to the queries p 1: t and q 1: t in a way that ensures that

-the vector 1 never belongs to the span of the observations of (say) the q -player,

̸

-there is at least one remaining candidate matrix in B 0 , i.e., E t ∩ B 0 = ∅ .

For technical reasons (cf. the assumptions of Lemma 8), we also need to make sure that there is enough space left in E t . We do so by keeping a candidate matrix M t ∈ E t ∩ M K (( -1 , 1)) , away from the border of the initial set of candidate matrices (i.e., with entries stricly between -1 and 1 ).

̸

Lemma 10. Fix a time horizon T ⩽ K/ 2 -1 . For any sequence of queries p 1: t , q 1: t , there exists a sequence of matrices M 1: T in M K (( -1 , 1)) adapted to p 1: T , q 1: T that defines losses ℓ ( q ) 1: T for which, for any v = 0 ,

Conclusion Combining Lemmas 8 and 10, against any learning strategy, we have built a sequence of outputs for which there is no common equilibrium to all remaining candidate matrices after T rounds, as long as T ⩽ ( K -3) / 2 ; this proves the following theorem.

Theorem 11. The first-order query complexity over M K ([ -1 , 1]) of finding a Nash equilibrium is

The next section tackles the lower bound for approximate equilibria. Its proof follows the same template as for the exact case, although the technical complexity increases.

4.3 Proof: Approximate Equilibria Case

Step I: Common Equilibria Are Close to the Span of Queries In the following, we say a probability distribution over [ K ] is δ -supported if p ( i ) ⩾ δ for all i . We say a pair of distributions ( p, q ) is δ -supported if both p and q are δ -supported. We start by defining a quantitative version of Definition 7 for approximate equilibria. In order to retain the property that losses at ε -equilibria stay close to an isotropic vector, we add a requirement on the support of the equilibria.

Definition 12. A set of matrices B ε,δ ⊂ M K ([ -1 , 1]) satisfies the assumption A ( ε, δ ) if

For all M ∈ B ε,δ , all ε -equilibria of M are δ -supported,

There exists a matrix M ∈ B ε,δ with non-zero value.

For example, by Corollary 21, the ∥ · ∥ 1 , ∞ -ball centered at (1 / 2) I K and of radius (1 / 16) K 2 satisfies this condition for any ε ⩽ 1 / (16 K 2 ) and δ ⩽ 1 / (2 K ) .

The following proposition is a quantitative version of Lemma 8 for approximate equilibria, in which we show that any common approximate equilibrium to all matrices in E t needs to be close to the span of the queries.

̸

Lemma 13. For ε, δ > 0 , let B ε,δ be a set of matrices satisfying A ( ε, δ ) , cf. Definition 12. Assume that E t ∩ B ε,δ = ∅ , and that the relative interior of E t contains a ball of radius r t measured in ∥ · ∥ 1 , ∞ -norm. If there exits a common ε -Nash equilibrium ( p, q ) sto all matrices in E t , then

Corollary 14. Under the assumptions of Lemma 13, there exists a game matrix M ∈ B ε,δ such that the value v ∈ R of M satisfies ∥ v 1 -Proj Span( ℓ ( q ) 1: t ) ( v 1 ) ∥ ⩽ 4 √ Kε/ ( δr t ) .

Step II: Construction of Matrices We design an adapted sequence of matrices that keeps v 1 away from the span of the observed losses.

Lemma 15. Let B be a closed ball in ∥ · ∥ 1 , ∞ -norm of radius r ⩽ 1 / 2 contained in M K ([ -1 , 1]) . Fix a time horizon T ⩽ K/ 2 -1 . For any sequence of queries ( p t , q t ) t ⩽ T , there exists a sequence of matrices M 1: T in B adapted to ( p t , q t ) that defines losses ℓ ( q ) 1: T for which, for any v ⩾ 0 ,

and such that E t contains a ball of radius r/ 2 in its relative interior.

The proof relies on a decomposition of the squared distance of v 1 to the span of observed losses at time t +1 through a Pythagorean identity, relating it to the span at time t . By carefully choosing the new matrix M t +1 as a function of the query p t +1 , q t +1 , we manage to ensure that the squared distance decreases only by a constant factor.

Remark 16. The radius of the ball inside E t stays constant at r/ 2 , even though E t gets smaller as t increases. This is made possible by choosing M t very close to M t -1 (roughly at a distance of r/ ( KT ) measured in ∥ · ∥ 1 , ∞ ), effectively ensuring that M t stays far from the boundary of M K ([ -1 , 1]) .

Step III: Conclusion We now combine Corollary 14 and Lemma 15 to obtain a lower bound on the best achievable gap after T rounds, which directly translates to a query complexity lower bound. Theorem 17. In the first-order query model on M K ([ -1 , 1]) , for any algorithm the worst-case gap after T ⩽ ( K -3) / 2 time steps is at least

Therefore, the query complexity of finding an ε -equilibrium for any ε ⩽ 1 / ( e 2 11 K 4 ) is at least

4.4 Potential Improvement and Discussion

There is still a wide disparity between upper and lower bounds on first-order query complexity. The lower bound Theorem 17 is most probably not tight, so let us discuss potential ways to improve it.

We believe the most promising approach for improvement is to find a different proxy for the gap. Our proof uses the distance to the span of the losses ∥ v 1 -Proj Span( ℓ ( q ) 1: t ) ∥ 2 for v ⩾ 1 / (2 K ) , which is convenient because it is a distance, but we need to introduce a strong restriction on the set of candidate matrices B ε,δ to be able to relate it to the gap, namely the restriction to a ball around 1 2 I K of radius O (1 /K 2 ) . In order to make significant progress it is therefore essential to enlarge the class of candidate matrices significantly compared to B ε,δ , and therefore to modify the proxy for the gap.

5 Discussion: Future Work and Conclusion

We study the first-order query complexity of computing approximate Nash equilibria for two-player zero-sum matrix games. We review upper bounds coming from online learning, and discuss existing lower bounds for related problems including alternative query models. Taking stock, we arrive at the surprising state of affairs that for this fundamental problem no lower bounds are known. We then offer some explanation for this current state of affairs: the first-order query model is powerful enough to identify any matrix from a fixed countable set in a single query; this rules out many techniques. We then turn to lower bounds. We design an adaptive adversary that answers incoming learner queries in such a way that the remaining consistent matrices do not share a common Nash equilibrium for as long as possible. Our approach is based on a quantity serving as a 'potential function': namely the distance of the all-ones vector to the span of the observations. We discuss in detail the result, future scope and limits of our proposed technique.

As can be seen in Figure 1, we are still far from matching lower and upper bounds. The most intriguing possibility for resolution would be if it were to turn out that the upper bounds (i.e. algorithms) are improvable. Current upper bounds come from online learning regret bounds, with algorithms falling in the category of uncoupled dynamics. We would love to know if these algorithm templates are in fact optimal for query complexity.

To cycle back to our motivation of computing saddle points in general, future work could attempt to extend our lower bound technique to different constraint sets, to functions possibly exhibiting curvature, and to instance-dependent rates. Another direction would be to generalize to multi-player games and investigate the query complexity of weaker solution concepts.

Limitations and Broader Impact The main limit of this work is that it only partially resolves the first-order query complexity of approximate NE in finite games, calling for tighter analysis. Regarding broader impact, results are mainly theoretical and do not entail direct societal consequences.

Acknowledgments and Disclosure of Funding

Sachs and Van Erven were supported by the Netherlands Organization for Scientific Research (NWO) under grant number VI.Vidi.192.095. Part of the research was performed while Hadiji was at the University of Amsterdam. During that time he was supported by the same grant number VI.Vidi.192.095.

References

Waïss Azizian, Ioannis Mitliagkas, Simon Lacoste-Julien, and Gauthier Gidel. A tight and unified analysis of gradient-based methods for a whole spectrum of differentiable games. In International conference on artificial intelligence and statistics , pages 2863-2873. PMLR, 2020.

Yakov Babichenko. Query complexity of approximate Nash equilibria. Journal of the ACM (JACM) , 63(4):1-24, 2016.

George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation , 13 (1):374, 1951.

Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Tight last-iterate convergence of the extragradient method for constrained monotone variational inequalities. arXiv preprint arXiv:2204.09228 , 2022.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Xi Chen and Binghui Peng. Hedging in games: Faster convergence of external and swap regrets. Advances in Neural Information Processing Systems , 33:18990-18999, 2020.

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Computing Nash equilibria: Approximation and smoothed complexity. Electron. Colloquium Comput. Complex. , 2006.

Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing two-player Nash equilibria. J. ACM , 56(3), may 2009.

Constantinos Daskalakis, Paul Goldberg, and Christos Papadimitriou. The complexity of computing a Nash equilibrium. SIAM J. Comput. , 39:195-259, 02 2009.

Constantinos Daskalakis, Aranyak Mehta, and Christos Papadimitriou. A note on approximate Nash equilibria. Theoretical Computer Science , 410(17):1581-1588, 2009.

Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms , pages 235-254. SIAM, 2011.

Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. Games and Economic Behavior , 92:327-348, 2015.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Advances in Neural Information Processing Systems , volume 34, pages 27604-27616, 2021.

Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1466-1478, 2021.

Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. Near-optimal no-regret learning dynamics for general convex games. Advances in Neural Information Processing Systems , 35:39076-39089, 2022.

John Fearnley, Martin Gairing, Paul W Goldberg, and Rahul Savani. Learning equilibria of games via payoff queries. Journal of Machine Learning Research , 16:1305-1344, 2015.

Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior , 29(1-2):79-103, 1999.

Gauthier Gidel, Tony Jebara, and Simon Lacoste-Julien. Frank-wolfe algorithms for saddle point problems. In Artificial Intelligence and Statistics , pages 362-371. PMLR, 2017.

Andrew Gilpin, Javier Peña, and Thomas Sandholm. First-order algorithm with o (ln(1 /ε )) convergence for ε -equilibrium in two-person zero-sum games. In Proceedings of the 23rd National Conference on Artificial Intelligence , AAAI'08, pages 75-82, 2008.

Noah Golowich, Sarath Pattathil, Constantinos Daskalakis, and Asuman Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems, 2020.

Sergiu Hart and Noam Nisan. The query complexity of correlated equilibria. Games and Economic Behavior , 108:401-410, 2018.

Elad Hazan and Tomer Koren. The computational power of optimization in online learning. In Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing , STOC '16, pages 128-141, 2016.

Yu-Guan Hsieh, Kimon Antonakopoulos, and P. Mertikopoulos. Adaptive learning in continuous games: Optimal regret bounds and convergence to Nash equilibrium. In Annual Conference Computational Learning Theory , 2021.

Adam Ibrahim, Waïss Azizian, Gauthier Gidel, and Ioannis Mitliagkas. Linear lower bounds and conditioning of differentiable games. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020 , volume 119, pages 4583-4593. PMLR, 2020.

Albert Xin Jiang and Kevin Leyton-Brown. Polynomial-time computation of exact correlated equilibrium in compact games. In Proceedings of the 12th ACM conference on Electronic commerce , pages 119-126, 2011.

Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

Tianyi Lin, Chi Jin, and Michael I. Jordan. Near-optimal algorithms for minimax optimization. In Annual Conference Computational Learning Theory , 2020.

Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of o(1/k) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization , 30(4):3230-3251, 2020.

Arkadi S Nemirovsky. On optimality of Krylov's information when solving linear operator equations. Journal of Complexity , 7(2):121-130, 1991.

Arkadi S Nemirovsky. Information-based complexity of linear operator equations. Journal of Complexity , 8(2):153-175, 1992.

Yurii Nesterov. Introductory lectures on convex optimization - a basic course. In Applied Optimization , 2014.

Francesco Orabona and Dávid Pál. Scale-free online learning. Theoretical Computer Science , 716:50-69, 2018. Special Issue on ALT 2015.

Yuyuan Ouyang and Yangyang Xu. Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems. Mathematical Programming , 185(1):1-35, 2021.

Christos H Papadimitriou and Tim Roughgarden. Computing correlated equilibria in multiplayer games. Journal of the ACM (JACM) , 55(3):1-29, 2008.

Georgios Piliouras, Ryann Sim, and Stratis Skoulakis. Beyond time-average convergence: Nearoptimal uncoupled online learning via clairvoyant multiplicative weights update. Advances in Neural Information Processing Systems , 35:22258-22269, 2022.

Leonid Denisovich Popov. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR , 28:845-848, 1980.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems , 26, 2013.

Julia Robinson. An iterative method of solving a game. Annals of mathematics , pages 296-301, 1951.

Gilles Stoltz and Gábor Lugosi. Learning correlated equilibria in games with compact sets of strategies. Games and Economic Behavior , 59(1):187-208, 2007.

Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems , 28, 2015.

Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations , 2020.

Tong Yang, Michael I Jordan, and Tatjana Chavdarova. Solving constrained variational inequalities via an interior point method. arXiv preprint arXiv:2206.10575 , 2022.

Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming , 194(1-2):901-935, 2022.

A Proofs of Lemmas 8 and 10

Proof of Lemma 8 . Since p, q is an equilibrium to at least one matrix in B 0 ∩E t , it is fully supported; in particular p is an equalizing strategy for any M ∈ E t and M ⊺ p = v M 1 for some number v M ∈ R . Furthermore, the value v M is actually independent of M ∈ E t since, v M = ⟨ M ⊺ p, q 1 ⟩ = ⟨ p, ℓ ( p ) 1 ⟩ . Therefore, for any M,M ′ ∈ E t , we have ( M -M ′ ) ⊺ p = 0 . Let us now define ¯ p = p -Proj Span( p 1: t ) ( p ) , and consider the direction ¯ pu ⊺ q ∈ M K ( R ) for some arbitrary non-zero vector u q orthogonal to q 1 , . . . , q t . Fix some matrix M ∈ E t ∩M K (( -1 , 1)) ; a non-empty set by assumption. Then for α ∈ R small enough, the matrix M ′ = M + α ¯ pe ⊺ q is still in E t , since M is not on its border (all entries are away from {-1 , 1 } ). Therefore,

implying that ∥ ¯ p ∥ = 0 , i.e. that p ∈ Span( p 1: t ) ; similar reasoning shows that q ∈ Span( q 1: t ) .

Proof of Lemma 10 . We build the sequence M 1: T incrementally by moving at step t +1 in directions chosen as a function of the new queries p t +1 and q t +1 , ensuring by induction that 1 / ∈ Span( ℓ ( q ) 1: T ) . Initialize the sequence at M 0 = (1 / 2) I K . Now for t ⩾ 0 , let us assume that we have correctly built M 1: t , inducing losses such that 1 / ∈ Span( ℓ ( q ) 1: t ) , and let us define M t +1 . (Note that the initialization of the induction is valid with the convention that 1 / ∈ { 0 } = Span( ∅ ) .)

If p t +1 is in the span of p 1: t , then set M t +1 = M t and the induction holds, since the span of the losses is left unchanged. Otherwise set M t +1 = M t + ¯ p t +1 ∥ ¯ p t +1 ∥ 2 u ⊺ t , where ¯ p t +1 = p t +1 -

Proj Span( p 1: t ) ( p t +1 ) , and u t is a non-zero vector orthogonal to the vectors q 1: t , to ℓ ( q ) 1: t , to 1 and to M ⊺ t p t +1 ; the existence of such a u t is guaranteed since 2 t +2 < K , ensuring that there is at least one dimension orthogonal to those 2 t +2 vector. By choosing the norm of u t to be small enough, we can make sure that M t +1 ∈ E t ∩M K (( -1 , 1)) . Then, ℓ ( q ) t +1 = M ⊺ t +1 p t +1 = M ⊺ t p t +1 + u t , and 1 is not in the span of ℓ ( q ) 1:( t +1) . Indeed, assume by contradiction that there exists real numbers α 1:( t +1) such that

then α t +1 > 0 since by induction 1 / ∈ Span( ℓ ( q ) 1: t ) . Then, taking dot products with u t on both sides of the equation above, we obtain 0 = ⟨ u t , 1 ⟩ = ⟨ u t , α t +1 ℓ ( q ) t +1 ⟩ = α t +1 ∥ u t ∥ 2 , leading to a contradiction. Therefore 1 / ∈ Span( ℓ ( q ) 1: t ) at all times t ⩽ T .

B Details for Upper bounds

B.1 Instance-Dependent Query Complexity

The following example shows that the bounds in [19] can be vacuous for our setting. Since, as noted by the authors, Theorem 5 by [41] is equivalent to the results in [19], the following example is only with respect to the latter.

Example 18. For this example, we use the notation from [19]. Consider the game matrix M = I K . Observe that M has a unique Nash-equilibrium, hence, by considering p = 1 K 1 and q = δ i it can be seen that δ ( M ) is at most √ K K -1 1 K ≈ 1 K . The bound is defined with respect to the condition number κ ( M ) = √ λ max M ⊺ M/δ ( M ) , where λ max ( M ) denotes the largest eigenvalue of M . Hence, for our example, κ ( M ) is at least of the order of K , which makes a bound of κ ( M ) log 1 ε for this specific example vacuous. Note that for other game matrices, the results give valuable insights into guarantees beyond the worst-case.

B.2 Constant Queries for Games from a Finite Alphabet

Proof . Let F be the smallest subfield of R that contains A , then F is countable and, R can be seen as an infinite-dimensional vector space over F . Recall that a family of real numbers ( x 1 , . . . , x n )

(each seen as a vector over the field F ) is linearly independent if for any λ 1 , . . . , λ n ∈ F we have λ 1 x 1 + · · · + λ n x n = 0 if and only if λ 1 = · · · = λ n = 0 .

We claim that if a player, say the q -player, queries an action such that the components q 1 , . . . , q K form a linearly independent family, then they can compute the whole matrix M with just one observation. Indeed, if M,M ′ ∈ M K ( A ) , yield the same output after one query, then for any i ∈ [ K ] :

Therefore, by the independence of ( q i ) i ∈ [ K ] , this implies that M i,j = M ′ i,j for any ( i, j ) . In other words, no two different matrices can give the same output after one query under q .

We are now left to show that there exists a play ( q 1 , . . . , q K ) with coordinates forming a linearly independent family. We prove this using the probabilistic method. Consider a sequence of random variables ( U 1 , . . . , U K ) i.i.d. and uniformly distributed over [0 , 1] . Then with probability 1 , the U i are independent over F . Indeed, we can upper bound the probability that they are dependent by a union bound and use of the tower rule as

̸

̸

̸

̸

̸

The last equality holds because, for any i ∈ [ K ] , conditionally on { U j | j = i } , the span of { U j | j = i } is a countable set, therefore the probability that U i belongs to that set is null. This concludes the proof.

̸

C First-Order Query Complexity of Recovering the Game Matrix

Proof of Theorem 6 . Clearly, we can fully reconstruct M from the queries p t = q t = e t for t = 1 , . . . , K , where e t denotes the standard basis vector in direction t . It turns out that this is optimal.

To show this, note that each query ( p, q ) provides us with constraints

for some loss vectors a and b . These may equivalently be expressed as linear constraints in the Hilbert space of matrices A ∈ R K × K , with inner product ⟨ A,B ⟩ = Tr( A ⊺ B ) :

Among these 2 K constraints on M , there is (at least) one redundant constraint, because there always exist numbers λ 1 , . . . , λ K and γ 1 , . . . , γ K , at least one of which is nonzero, such that

Specifically, this holds for λ i = q i and γ j = -p j . It follows that a query ( p t , q t ) in round t will provide at most 2( K -t ) + 1 new constraints on top of the queries from rounds 1 , . . . , t -1 . To see this, note that p t will have at least one constraint in common with q 1 , . . . , q t -1 and q t will have at least one constraint in common with p 1 , . . . , p t , so the total number of new constraints is at most n t := 2 K -( t -1) -t = 2( K -t ) + 1 .

We now provide the following scenario in which M cannot be fully determined by strictly less than K queries. In rounds t = 1 , . . . , K -1 , the answer to queries ( p t , q t ) is always the two loss vectors ℓ ( p ) t = ℓ ( q ) t = 1 2 1 , which are compatible with the possibility that M equals 1 2 11 ⊺ , i.e. the matrix with all entries equal to 1 / 2 . We will show by induction that the dimension of the null-space (i.e. the

number of unconstrained dimensions of M ) of all constraints up to round t is at least ( K -t ) 2 . This is true for t = 0 , because the domain of M has K 2 dimensions, and, whenever it is true for t , then for t +1 it is at least

̸

Thus, after K -1 rounds, there remains at least one direction V ∈ R K × K in this null space with V = 0 . This means that the learner cannot distinguish the case M = 1 2 11 ⊺ from the case M = 1 2 11 ⊺ + 1 2 V/ max i,j | V i,j | , thus K -1 queries are not sufficient to fully determine M .

D Proofs and Technical Results for Section 4

D.1 Approximate Equilibria

D.1.1 Proofs of Main Results

Proof of Corollary 14. If ( p, q ) is a common ε -NE to all matrices in E t , then we have for any game matrix M ∈ E t ,

and therefore for any v > 0 ,

Now as δ > 0 , any M ∈ B ε,δ has a unique Nash equilibrium p ⋆ , q ⋆ , which is fully supported. Therefore, the value of M is v = p ⊺ Mq ⋆ , as q ⋆ is a an equalizing strategy. Now, using (1) in the proof of Lemma 13, which is valid for any matrix that admits p, q as an ε -NE,

Proof of Lemma 13 . First note that p and q are δ -supported, as ( p, q ) is an ε -equilibrium of at least one matrix in E t ∩ B ( ε, δ ) . For M ∈ E t , pick any j ⋆ ∈ arg min j ∈ [ K ] ( M ⊺ p ) j . As ( p, q ) is an ε -equilibrium for M ,

and therefore, dividing by q ( j ⋆ ) ⩾ δ ,

Hence, as min j ∈ [ K ] ( M ⊺ p ) j ⩽ p ⊺ Mq 1 ⩽ max j ∈ [ K ] ( M ⊺ p ) j ,

Since p ⊺ Mq 1 = p ⊺ ℓ ( p ) 1 has the same value for any M ∈ E t , we apply this inequality for any pair of matrices M,M ′ ∈ E t to deduce that

Let us now instantiate this identity with some well-chosen M and M ′ . Let u ∈ R K be a vector orthogonal to q 1 , . . . , q t , such that ∥ u ∥ ∞ = 1 (such a u exists as long as q 1 , . . . , q t do not span the whole of R K ). Next consider M and M ′ in E t such that

Such M and M ′ are guaranteed to exist in E t as ∥ M -M ′ ∥ = r t . Then, applying (2), we obtain

from which the claim follows. The same reasoning applies to obtain the bound on q .

Proof of Lemma 15 . Denote by Π t the projection on the span of the observed losses ℓ ( q ) 1: t for the q -player at time step t . Webuild the sequence ( M t ) incrementally by moving at step t +1 in directions chosen as a function of the new queries p t +1 and q t +1 .

We initialize the sequence at M 0 the center of B . Now for t ⩾ 0 , if p t +1 is in the span of p 1: t , then set M t +1 = M t , otherwise set

where ¯ p t +1 = p t +1 -Proj Span( p 1: t ) ( p t +1 ) , and u t is a non-zero vector orthogonal to the vectors q 1: t , to ℓ ( q ) 1: t , to 1 , and to M ⊺ t p t +1 ; the existence of such a u t is guaranteed since the only condition is that it is orthogonal to 2 t +2 < K vectors. (Note that u t does not depend on the value of v .) We set the norm of u t at a later stage of the proof. Then,

Then for any v ∈ R , the squared distance from the vector v 1 to the space Span( ℓ ( q ) 1: t +1 ) can be decomposed thanks to the Pythagorean equality, as the squared distance to the previous span minus the squared norm of the projection on the new orthogonal direction ℓ ( q ) t +1 -Π t ( ℓ ( q ) t +1 ) :

Observe that for any vectors a, b , we have ⟨ a -Π t ( a ) , b ⟩ = ⟨ a -Π t ( a ) , b -Π t ( b ) ⟩ = ⟨ a, b -Π t ( b ) ⟩ , as Π t is an orthogonal projection on a linear subspace. Using this identity, as well as the orthogonality conditions used to define u t (precisely, that Π t ( u t ) = 0 , and that u t is orthogonal to M ⊺ t p t +1 and Π t ( M ⊺ t p t +1 ) ), we obtain after applying Cauchy-Schwarz,

We also used the fact that projections reduce the norm, and the function x ↦→ x/ ( x +1) is increasing on (0 , + ∞ ) to obtain the final inequality.

Using this bound on D t to lower bound the distance to the span, we obtain

We are now left to choose the norm of u t ; the objective is to make it as big as possible under the constraint that the sequence ( M t ) stays in B . We set the norm of u t to be a constant multiple of ∥ M ⊺ t p t +1 ∥ , i.e., √

For α small enough, we can ensure that the whole sequence ( M t ) stays in the ball B , as

where we used ∥ M t x ∥ ⩽ ∥ M t ∥ 2 , 2 ∥ x ∥ ⩽ √ K ∥ x ∥ , and M ⊺ t ¯ p t +1 = M ⊺ t p t +1 since M t ∈ E t . Therefore by taking, α = ( r/ 2) 2 / ( KT 2 ) , we ensure that M t ∈ B as B contains a ball of radius r centered at M 0 . Furthermore, this choice ensures that E t contains the ball of radius r/ 2 centered at M t in its relative interior as for any matrix U with norm less than r/ 2 , we have ∥ M t + U -M 0 ∥ ⩽ ∥ M t -M 0 ∥ + ∥ U ∥ ⩽ r/ 2 , so M t + U ∈ B ⊂ M K ([0 , 1]) .

Plugging back the value of α into (3), we get

After T time steps, this implies that for any v ⩾ 0 ,

This is the claimed result.

Proof of Theorem 17 . Define B ε,δ = B ∥·∥ 1 , ∞ ( 1 2 I K , 1 16 K 2 ) , which is a ball of radius r = 1 / (16 K 2 ) . Given an algorithm, consider the sequence of matrices M 1: T generated by Lemma 15, applied with B = B ε,δ ; we know in particular that M t ∈ B ε,δ , so B ε,δ ∩ E t = ∅ .

̸

Assume now that the algorithm outputs a common ε -equilibrium to all matrices in E T . The assumptions of Corollary 14 hold, instantiated with B ε,δ and r t = 1 / (32 K 2 ) . Therefore there exists M ∈ B ε,δ , with value v such that

On the other hand, by Lemma 15 also instantiated with B = B ε,δ , we know that

This implies that

Finally, since v is the value of a matrix M ∈ B ε,δ ,

We conclude by replacing the constants by their values, δ = 1 / (2 K ) , and r t = r/ 2 = 1 / (32 K 2 ) . By Lemma 19 in Appendix D, we deduce that for (2 ε ) ⩽ 1 / ( e 2 10 K 4 ) ,

D.1.2 Technical Lemmas

Lemma 19. For any a, b, x > 0 , for any ε ⩽ a/e ,

Proof of Lemma 19. Assume then

thus, applying logarithms to both sides,

Reformulate as, since a ⩾ eε

Similarly, i.e., after rearranging,

Similarly

Therefore

Completing the proof.

Corollary 21. For any ε ⩽ 1 / (16 K 2 ) , for any game matrix M such that

all ε -NE ( p, q ) of M satisfy for all i, j ∈ [ K ]

Corollary 22. For any game matrix M such that

all exact Nash equilibria ( p, q ) are fully supported.

We now apply Lambert's W 0 function, which is increasing on its main branch [ -1 /e, + ∞ ) , and use a standard lower bound on W 0 to obtain

thus

which is the claimed bound.

Lemma 20. Let α ⩾ 0 and s > 0 be real numbers, let M ∈ M K ( R ) be a matrix such that

Then for any ε -Nash equilibrium ( p, q ) of M we have for any i, j ∈ [ K ]

Proof . Let ( p, q ) denote an ε -NE of M . Then

Now for any j ∈ [ K ] , since M is close to sI K ,

thus, using the fact that p is a probability vector,

Combining the equations (4), (5) and (6) above, we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the first-order query complexity of two-player zero-sum games. More precisely, given a bi-matrix zero-sum game $(A,-A^\\top)$ (with $A \\in \\mathbb{R}^{k\\times k}$) a first-order query is a set of strategies $(x,y)$. The answer to such a query is the set of vectors $(A y, -A^\\top x)$ respectively denoting the payoff vectors of the row (x) and the column (y) agent. The authors try to understand what is the necessary number of first-order queries so as to compute a min-max equilibrium of the respective zero-sum game once the matrix $A$ is unknown. The authors establish that in order to compute an exact min-max equilibrium at least $\\Omega(K)$ first-order queries are needed. The latter result coincides with the first-order complexity of the naive algorithm that first $learns$ the whole matrix $A$ and then computes an exact min-max equilibrium. Then the authors show that in order to compute an $\\epsilon$-approximate min-max equilibrium $\\Omega(\\log(1/\\epsilon K))$ first-order queries are needed. The latter result approximately matches the best-known upper bound ($O(\\frac{\\log K}{\\epsilon})$) obtained by Optimistic MWU.\n\nSTRENGTHS:\nI think that this is a solid work. The first-order complexity model is very well-motivated since it captures exactly the information exchange model of no-regret dynamics with full information feedback. Thus any lower-bound on the first-order complexity directly implies lower bounds on the convergence rates of such no-regret dynamics. The current state of the art convergence results of such no-regret dynamics is $O( \\frac{\\log K}{\\epsilon})$ that is approximately matched by the provided lower bound.\n\nI also think that the paper introduces some interesting techniques that can be of use so as to completely settle the first-order query complexity for computing min-max equilibrium.\n\nWEAKNESSES:\nThe only weakness of the paper is the fact that it does not match the current upper bound. That being said I believe that this work is an important first-step.\n\nQUESTIONS:\nMinor: You may want to include also this references for regret-based algorithms for the computation of CCE in general-sum games.\n\nBeyond Time-Average Convergence: Near-Optimal Uncoupled Online Learning via Clairvoyant Multiplicative Weights Update, Piliouras et al. 2022\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the query complexity lower bound of first-order methods for finding exact and approximate Nash equilibria in two-player zero-sum games. It is first established that if the matrix entries come from a fixed countable set, one query suffices to reconstruct the full matrix and hence to compute the exact Nash equilibrium. This rules out most existing lower bound constructions as they use rational or otherwise countable matrix entries. A new approach is then used to establish an $\\Omega(K)$ lower bound for finding the exact Nash and $\\tilde{\\Omega}(\\log 1/K\\epsilon)$ lower bound for finding an $\\epsilon$-approximate Nash (for sufficiently small $\\epsilon$), where $K$ is the number of actions.\n\nSTRENGTHS:\n- This paper studies a fundamental yet perhaps overlooked problem in game theory; as mentioned by the authors, it is indeed surprising that none of the established lower bounds apply to this setting. To the best of my knowledge, the results obtained in this paper are indeed the first non-trivial lower bounds for noiseless first-order methods in two-player zero-sum games.\n- The paper is well-written and offers a good explanation (Theorem 4) why it is non-straightforward to adapt existing constructions to two-player zero-sum games.\n\nWEAKNESSES:\n- The current lower bound rate is still quite far from the existing upper bound in terms of the dependence on $\\epsilon$. \n\nTypos:\nLine 16: Although not exactly a typo, the submodular optimization problem is never mentioned in the main text.\nLine 59: \"build build\"\nLine 519: the first term should be $\\lambda_1x_1$ instead.\n\nQUESTIONS:\nLines 287 and 322: a common Nash equilibrium for all matrices in $\\varepsilon_t$ may not exist, right? Does the statement here mean that \"suppose that a common Nash equilibrium exist\"?\n\nLIMITATIONS:\nThis work is mainly theoretical and does not entail direct societal consequences", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
Define the gap of a two player zero sum game with payoff matrix M and mixed strategies p and q as: 
gap(M,p,q):=max_j (M^Tp)_j - min_i(Mq)_i
An equilibrium is called \eps approximate if the gap is at most 2\eps. An important question is to compute the approximate Nash equilibrium by observing the gradients of the objectives in a sequential manner. Of course the number of observations / queries needs to be optimized. Rakhlin and Sridharan showed that O(\ln K/\eps^2) queries suffice for K X K matrix games. The authors here improve their bound to O(\ln K/\eps) - which is optimal - for small \eps (\eps < 1/K^4). They also show a lower bound that rules out certain algorithms for improving the upper bound to match the lower bound for all \eps.

STRENGTHS:
- Learning the \eps approximate Nash equilibrium is an interesting and important open problem in its own right, not to mention several applications.
-  The techniques in the paper are interesting by themselves.

WEAKNESSES:
- The actual result is fairly technical and may not be of broad interest since the improvement is marginal (only for \eps < 1/K^4).

QUESTIONS:
- How robust are your techniques/bounds when the observations are noisy? Often we don't have access to precise gradients.

LIMITATIONS:
Yes, they have been presented clearly.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the complexities associated with identifying a mixed Nash equilibrium within the context of a zero-sum game. Specifically, it scrutinizes situations where the algorithm does not directly observe the payoff matrix. Instead, it learns about it by querying its product with either a column vector or row vector. The challenge of computing Nash equilibria in this model has been extensively studied, however, prior to this study, little was known about lower boundaries.\n\nThe study's central findings include a K-2 query lower bound for calculating an exact equilibrium and a lower bound that exceeds the constant when 1/epsilon is super-polynomial in K for calculating an epsilon-approximate equilibrium. The methodology for deriving these lower bounds involves the identification of an adversary capable of answering an arbitrary series of first-order queries. This adversary ensures that the all-ones vector remains considerably distanced from the linear span of query responses for the maximum possible steps, while concurrently guaranteeing that the set of matrices congruent with query responses remains non-empty.\n\nEarly on, the paper also introduces a minor result highlighting the complexities inherent in proving lower boundaries within this model. If provided with a countable set of numbers known to encompass all the entries of the payoff matrix, a single first-order query is sufficient to reconstruct these entries and consequently, identify a mixed equilibrium. Although this result is simple, its proof capitalizes on the assumption that responses to first-order queries comprise vectors of infinite-precision real numbers. While this algorithm lacks practicality (an aspect openly acknowledged in the paper), its existence accentuates the challenge associated with establishing lower bounds in a model that facilitates such powerful queries.\n\nSTRENGTHS:\nn/a\n\nWEAKNESSES:\nThe paper has some discernible weaknesses which impact its overall effectiveness and value. They are listed as follows:\n\n1. Lack of Clarity: The exposition in the paper lacks clarity and coherence which makes it difficult to follow the arguments and understand the results. Better structuring and clear, concise language would greatly improve the readability and accessibility of the paper. \n\n2. Absence of Experiments: There are no empirical evaluations or experiments provided to support the theoretical results. Experimental results are key in illustrating the practical applications and feasibility of the proposed methods. Therefore, including such results is crucial for providing a comprehensive understanding of the paper's contributions.\n\n3. Uncertain Significance of the Results: While the results of the paper are novel, their actual importance and impact on the field are uncertain. The paper would greatly benefit from a discussion explaining the broader implications of the results, their potential applications, and how they advance the current state of knowledge in the field. Without such context, it is hard to assess the true significance of the findings.\n\nQUESTIONS:\nThis paper leaves me in a state of indecision, and being the more positive reviewer, this doesn't appear to be a promising sign for the paper. The rather tepid scores possibly reflect my collective sentiment towards the main result, which is the lower bound. It appears to be quantitatively modest, and we're uncertain about the significance of obtaining upper/lower bounds in this exact query model that permits the use of \"funny bit tricks\". It's unclear how vital these aspects are in terms of contributing to the field, and this ambiguity might be affecting our overall reception of the paper.\n\nLet me give some recommendations about improving the clarity of your theoretical paper:\n\n1. **Abstract and Introduction**: Ensure that these sections provide a clear and concise overview of the key ideas and contributions of the paper. Avoid using overly technical terms and jargon in these sections, as many readers will try to get an understanding of your work from these parts before diving into the main content. \n\n2. **Theoretical Concepts**: Each theoretical concept, model or algorithm that you introduce should be clearly defined and explained. When introducing a new concept, briefly review its background and the relevant literature. Also, explicitly state the assumptions you are making.\n\n3. **Equations and Theorems**: Always explain the intuition and significance behind every equation or theorem before diving into rigorous mathematical proofs. Make sure to annotate your equations adequately and define all terms and variables used. Whenever possible, accompany your mathematical results with visualizations or intuitive examples.\n\nThis crucially misses in the main part of your results\n\n4. **Results Section**: In the results section, don't just state your results, but also explain their implications. Make sure to clearly relate your results back to the problems and questions you outlined in the introduction. \n\n5. **Language and Flow**: Aim for a clear, concise, and precise language. Avoid long and convoluted sentences. Make sure each section and subsection follows logically from the previous ones, helping to guide the reader through your narrative.\n\n6. **Discussion and Conclusion**: Use these sections to clearly summarize your contributions and their implications. Discuss how your results fit into the bigger picture of your research field. \n\n7. **Appendix**: Put additional details, proofs, and technicalities that are not essential for understanding the main ideas of your paper in the appendix. This way, you can keep the main text more readable without losing any necessary detail.\n\nAdd explanatory paragraphs after theorems\n\n\n\nI strongly believe that the paper is not  submitted in the correct conference venue.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the query complexity of learning (approximate) Nash equilibria in zero-sum matrix games. The authors provide lower bounds on the query complexity for exact and approximate equilibria. They also show that if the game matrix has entries in a known countable set, the learner can recover the full matrix in one single first-order query. The paper discusses related work on lower and upper bounds for similar problems. The authors present upper bounds for the query complexity of finding approximate equilibria and introduce two strategies for obtaining equilibria with large approximation values. They also propose a potential improvement to the lower bounds by finding a different proxy for the gap. The paper includes proofs, examples, and explanations to support the theorems and corollaries related to the query complexity of learning game matrices. Overall, the paper contributes to the understanding of the query complexity of learning Nash equilibria in matrix games and provides insights into the lower and upper bounds for this problem.\n\nSTRENGTHS:\nThe paper presents original and valuable contributions to the understanding of the query complexity of learning Nash equilibria in matrix games. It introduces new results and insights, providing lower bounds on the query complexity for both exact and approximate equilibria, and proposing a potential improvement to the lower bounds. The quality of the paper is high, with rigorous proofs, clear examples, and a comprehensive discussion of related work. The paper is also well-structured and clearly written, making it accessible to readers. In terms of significance, the paper’s contributions can inform the development of more efficient algorithms for learning Nash equilibria in matrix games, with practical applications in various domains. Overall, the paper is a valuable addition to the existing literature in this field.\n\nWEAKNESSES:\nThe paper provides valuable insights into the query complexity of learning Nash equilibria in matrix games. However, the novelty of the results is somewhat limited as they largely build upon existing literature, such as the connection between regret and equilibria.\n\nThe paper partially resolves the first-order query complexity of approximate Nash Equilibria in finite games, but further analysis is needed for tighter results. The paper’s focus is primarily on theoretical analysis and proofs, without experimental evaluation, and the results are mainly theoretical with no direct societal consequences.\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nThe author has clearly acknowledged the limitations of their work, specifically that it only partially resolves the first-order query complexity of approximate Nash Equilibria in finite games and further analysis is needed for tighter results. In terms of broader impact, the results are mainly theoretical and do not have direct societal consequences.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}