SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hedi Hadiji
Laboratoire des signaux et systèmes Univ. Paris-Saclay, CNRS, CentraleSupélec
Sarah Sachs, Tim van Even
Korteweg-de Vries Institute for Mathematics  
University of Amsterdam
Wouter Koolen
Centrum Wiskunde & Informatica and University of Twente
In the first-order query model for zero-sum  $K \times K$  matrix games, players observe the expected pay-offs for all their possible actions under the randomized action played by their opponent. This classical model has received renewed interest after the discovery by Rakhlin and Sridharan that  $\varepsilon$ -approximate Nash equilibria can be computed efficiently from  $O(\ln K / \varepsilon)$  instead of  $O(\ln K / \varepsilon^2)$  queries. Surprisingly, the optimal number of such queries, as a function of both  $\varepsilon$  and  $K$ , is not known. We make progress on this question on two fronts. First, we fully characterise the query complexity of learning exact equilibria  $(\varepsilon = 0)$ , by showing that they require a number of queries that is linear in  $K$ , which means that it is essentially as hard as querying the whole matrix, which can also be done with  $K$  queries. Second, for  $\varepsilon > 0$ , the current query complexity upper bound stands at  $O(\min(\ln(K) / \varepsilon, K))$ . We argue that, unfortunately, obtaining a matching lower bound is not possible with existing techniques: we prove that no lower bound can be derived by constructing hard matrices whose entries take values in a known countable set, because such matrices can be fully identified by a single query. This rules out, for instance, reducing to an optimization problem over the hypercube by encoding it as a binary payoff matrix. We then introduce a new technique for lower bounds, which allows us to obtain lower bounds of order  $\tilde{\Omega}(\log\left(\frac{1}{K\varepsilon}\right))$  for any  $\varepsilon \leqslant 1/(cK^4)$ , where  $c$  is a constant independent of  $K$ . We further discuss possible future directions to improve on our techniques in order to close the gap with the upper bounds.
Computing the saddle point
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} f (x, y)
$$
for convex-concave functions  $f: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  is of general interest throughout optimization, economics and machine learning. We study the computation of an approximate saddle point  $(x_{\star}, y_{\star})$ , satisfying  $\max_{y \in \mathcal{Y}} f(x_{\star}, y) - \min_{x \in \mathcal{X}} f(x, y_{\star}) \leqslant 2\varepsilon$  for some given  $\varepsilon \geqslant 0$ . Starting with an unknown  $f$  from a known class  $\mathcal{F}$ , we consider sequential learning in the first-order feedback model, where the learner gets to observe gradients of the objective. Formally, each query  $(x, y)$  results in feedback  $(\nabla_x f(x, y), \nabla_y f(x, y))$ . We are interested in the following question:
How many first-order queries are necessary and sufficient for a sequential learner to output an approximate saddle point for any  $f \in \mathcal{F}$ ?
Characterizing the query complexity of learning saddle points is of theoretical interest for understanding the hardness of computing equilibria, and for certifying the optimality of upper bounds.
In this work, we restrict our attention to the special case of zero-sum matrix games, where  $\mathcal{X}$  and  $\mathcal{Y}$  are finite-dimensional probability simplices and  $f$  is bilinear. For this canonical setting, the optimal query complexity is, surprisingly, unresolved. Indeed, computation algorithms are known since [3], up to [37], while lower bounds remain elusive, leaving the optimal query complexity still unknown.
Obtaining lower bounds is not only of fundamental interest in itself, but may also lead to interesting new techniques, since none of the existing proof strategies are applicable. New ideas provided here could prove useful to tackle other problems.
We make progress towards characterizing the first-order query complexity of approximate Nash equilibria in finite-action zero-sum games, as a function of the number of actions  $K$  and approximation level  $\varepsilon$ . Our contributions are the following:
Lower bounds We provide the first lower bounds on the first-order query complexity for zero-sum matrix games with bounded entries. We show that  $K / 2 - 1$  queries are necessary to compute an exact equilibrium (Theorem 11 in Section 4), and at least  $\Omega (\log (1 / \varepsilon K) / \log (K))$  queries are required for an  $\varepsilon$ -equilibrium if  $\varepsilon \leqslant 1 / K^4$  (Theorem 17 in Section 4). More than the concrete rates, we believe that the structure of our proof is of particular interest, and might be of use beyond this specific setting.
Upper Bounds We show that if the game matrix has entries in a known countable set, then the learner can recover the full matrix in one single first-order query (Theorem 4 in Section 3). This result can be interpreted both as an upper bound on the query complexity for matrices on countable entry sets, and as an impossibility result, precluding the use of simple predefined sets of matrices as candidate objectives for proving lower bounds. This, together with the lack of rotational invariance of the action sets, sheds light on why this setting turns out to be surprisingly resistant to proving lower bounds using well-established techniques.
1.2 Related Work on Lower Bounds: Lower Bounds
Below we review the literature on lower bound techniques for related problems and settings. We find that none of the existing techniques apply to our setting, for three particular reasons. They either:
- Build matrices from a set of matrices with entries lying in a finite alphabet. In Theorem 4, we show that, for game matrices with entries in a known countable set, a single first-order query suffices to infer the exact Nash equilibrium. This shows in particular that any such matrix construction cannot lead to a lower bound in the first-order model.
- Rely crucially on rotational invariance of the action set, which limits the approach to the  $\ell_2$ -constrained or unconstrained cases, ruling out the probability simplex.
- Assume that the players select actions in the span of the observed gradients. This span assumption is not suited when the action set is not an  $\ell_2$  ball, in which case the span of the gradients has no natural embedding into the action set; most algorithms leave the span, e.g. exponential weights.
First-order Query Complexity of Minimax Optimization Existing lower bounds [24, 33] for minmax optimization are known for finite-dimensional Euclidean spaces where  $\mathcal{X},\mathcal{Y}$  are either Euclidean balls or the whole space. Both these cases benefit from the property that the action sets are invariant under rotation, allowing for a step-by-step reduction to lower dimensional instances, as shown in the seminal work of [29, 30]. The notable exception of [14] also does not cover the case we study: they consider the computational hardness of non-convex non-concave optimisation.
In the unconstrained case with curvature, that is, if  $\mathcal{F}$  is the set of strongly convex, strongly concave functions with marginal condition numbers  $\kappa_{x}$  and  $\kappa_{y}$ , the query complexity is settled to be  $\Theta (\sqrt{\kappa_x\kappa_y}\log (1 / \varepsilon))$ ; see [24, 43] for lower bounds and [27] and references therein for upper bounds.
For the constrained case, [33] provide lower bounds for bilinear saddle-point problems. They establish a query complexity of order  $\Omega (L_fD_{\mathcal{X}}D_{\mathcal{Y}} / \varepsilon)$ , where  $D_{\mathcal{X}}$  and  $D_{\mathcal{Y}}$  are the respective diameters of the
constraint sets  $\mathcal{X}$  and  $\mathcal{Y}$ , and  $L_{f}$  measures the Lipschitz regularity of  $f$ . However, their techniques rely crucially on rotational invariance and sequentially adapted constraint sets, and they ask the open question of whether similar lower bounds can also be shown for fixed constraint sets.
Other Query Models The query complexity of minimax optimization has also been studied under different feedback models. Recall that we focus on the bilinear case, where  $f: p, q \mapsto p^{\mathsf{T}}Mq$  for some game matrix  $M$ . [16] study the query complexity of (approximate) Nash equilibria of the game, i.e., (approximate) saddle points of  $f$ , under a query model where the learner chooses single entries of the matrix to observe. They show, among other results, that in order to compute an  $\varepsilon$ -equilibrium in  $K \times K$  zero-sum games, the number of entries queried needs to be at least  $\Omega(K\log K)$  when  $\varepsilon = \mathcal{O}(1/\log K)$ . [22] consider a setting in which the learner observes a best response to their query  $(p,q)$ , i.e.,  $i^{\star} \in \mathrm{argmin}(Mq)_i$  and  $j^{\star} \in \mathrm{argmin}(-(M^{\top}p)_j)$ . In that setting, they show that to compute a 1/4-equilibrium, at least  $\Omega(\sqrt{K}/(\log K)^2)$  best-response queries are necessary. Since a first-order query brings strictly more information than either a best-response query or an entry query, these lower bounds do not have direct consequences for our query model. Both of these references prove their lower bounds by building hard matrices with entries in a known finite set (e.g.,  $\{0,1\}$ ).
Other Lower Bounds in Saddle Point and Equilibria Computation For multiplayer games, strong lower bounds on the computational complexity of exact Nash equilibria have been uncovered: PPAD-hardness for computing the Nash-equilibrium in a general game, see [7-9], or in a non-convex-concave zero-sum game [14]. Regarding query complexity, [2, 21] study the single-entry query complexity for approximate correlated equilibria and Nash equilibria of games with many players. As in other references mentioned, all these bounds are built on matrices with entries in a finite set. Due to their intrinsically combinatorial nature, these techniques are less common for numerical algorithms. One of the classical methods for sample complexity lower bounds was introduced by [31] for first-order optimization. It was successfully extended to saddle-point problems [43] and recently also to unconstrained zero-sum games by [24]. However, these techniques rely on the assumption that the next iterate is chosen in the span of the previous oracle information (cf. Definition 1 in [24] or Assumption 2.1.4 in [31]). Moreover, the game constructed by [24] for a Nesterov-style lower bound results in matrices with entries from a finite alphabet. For the same reason, no proof technique inspired by lower bounds via Rademacher random variables, as in [32], can ever work.
1.2.1 Other Related Work
Upper Bounds for Finite Action Zero-Sum Games Early on, [3, 38] show that under best-response dynamics, the average plays converge to an equilibrium. The connection to regret bounds was established by [17], who deduce a  $\mathcal{O}(\log (K) / \varepsilon^2)$  query complexity, by means of a construction akin to online-to-batch conversion. More recently, [11, 12] obtained the first fast rates of order  $\mathcal{O}(c(K) / \varepsilon)$ , through an ingenious learning mechanism. [37] rediscovered the Optimistic Online Mirror Descent algorithm [36], and pioneered optimistic online learning regret bounds; they yield to this the day the fastest rates of  $\mathcal{O}(\log (K) / \varepsilon)$  and the simplest algorithm. The instance-dependent linear-rate upper bounds of order  $\lambda (M)\ln (1 / \varepsilon)$  by [19, 41] are essentially incomparable to the worst-case  $\mathcal{O}(\log (K) / \varepsilon)$ ; they are superior when  $\lambda (M)\ll K$ , and vacuous in the typical case  $\lambda (M)\approx K$  (see the discussion below Theorem 1).
Our focus is the quality of the inferred saddle point (this optimization perspective is sometimes called pure exploration). Part of the literature regards the queries as actual moves made, and consequently prioritizes other objectives. In particular, many focus on studying uncoupled dynamics, that is, sequences of actions that separate the observations of the individual  $p$  and  $q$  players without allowing communication between the players. Another recent theme of interest is the last-iterate convergence of such dynamics, see e.g., [23].
Upper Bounds for Minmax Optimization The vast majority of the known upper bounds in minimax optimization (and more generally in variational inequality problems), outside of finite-action zero-sum games mentioned above, concern unconstrained settings [1, 20, 28]. The literature on constrained settings is more sparse: [4] recently proved the rate of convergence of the projected extra-gradient method [26]. [42] adapt interior methods to handle constraints. [18] propose a fast algorithm when either the action sets are strongly convex, or the objective is strongly-convex strongly-concave. Recently, non-convex-concave settings have also attracted attention, see [27] and references therein.
Upper Bounds for General-Sum Games and Correlated Equilibria For multiplayer games, uncoupled dynamics do not converge (in any sense) to Nash equilibria. However, regret-based procedures can find correlated and coarse correlated equilibria [5, 39, 35]. In particular, internal regret guarantees for individual players in general-sum multiplayer games were recently improved from the generic  $\mathcal{O}(1 / \varepsilon^2)$  for any sequences of losses, to  $O(1 / \varepsilon^{4 / 3})$  in [40], to  $O(1 / \varepsilon^{6 / 5})$  in [6] and to  $O(\log (1 / \varepsilon) / \varepsilon)$  in [13, 15] (we omit the dependence on the number of actions and players). Polynomial-time methods for efficient computation of exact correlated equilibria are designed by [25, 34], using their formulation as solutions to a linear program.
2 Setting and Notation
General Notation Given a set of numbers  $A \subset \mathbb{R}$ , we denote by  $\mathcal{M}_K(A)$  the set of  $K \times K$  matrices with entries in  $A$ ; we mainly consider the bounded-entries class  $\mathcal{M}_K([-1,1])$ . A  $K \times K$  zero-sum game between a minimizing  $p$ -player (or row player) and a maximizing  $q$ -player (or column player) is represented by a matrix  $M \in \mathcal{M}_K(\mathbb{R})$ ; we restrict our attention to square  $K \times K$  games. For any pair of plays  $(p,q)$ , the expected loss vector of the  $p$ -player (resp.  $q$ -player) is  $Mq \in \mathbb{R}^K$  (resp.  $-M^{\intercal}p \in \mathbb{R}^{K}$ ). The suboptimality gap at  $(p,q)$  is
$$
g (M, p, q) = \max  _ {j \in [ K ]} (M ^ {\intercal} p) _ {j} - \min  _ {i \in [ K ]} (M q) _ {i}.
$$
The gap  $g(M, p, q)$  is non-negative for any  $M, p$  and  $q$ . The pair of plays  $(p, q)$  is said to be an  $\varepsilon$ -Nash equilibrium if  $g(M, p, q) \leqslant 2\varepsilon$ ; Nash equilibria are 0-Nash equilibria. Most games we will consider will possess a unique Nash equilibrium  $p, q$ , and this equilibrium will be fully supported, i.e., all components of  $p$  and  $q$  are positive. Fully supported equilibria in finite games are also equalizing strategies, meaning that the loss vectors are then isotropic; in finite zero-sum games, this entails that  $M^{\mathsf{T}}p = Mq = v\mathbf{1}$ , where  $v$  is the game-value. We follow the notation convention to abbreviate  $\min(x, v) = x \wedge v$  and to hide polylogarithmic terms by  $\tilde{\Omega}$  or  $\tilde{O}$ .
First-Order Query Model and Objectives The first-order query model is an interaction protocol between a learner and a game matrix, defined as follows. Fix a time horizon  $T \in \mathbb{N}$ , and a set of candidate matrices  $\mathcal{M} \subseteq \mathcal{M}_K(\mathbb{R})$ . Over the course of  $T$  rounds, the learner sequentially picks (queries) pairs of plays  $(p_t, q_t)$  and observes the expected losses  $(Mq_t, -M^\top p_t)$ , where the matrix  $M \in \mathcal{M}$  is fixed and unknown to the learner.
We examine the number of interactions necessary to compute an approximate equilibrium. At the end of the  $T$  rounds, the learner recommends a pair  $(p,q)$ . Given a fixed gap level  $\varepsilon \geqslant 0$  and a set of candidate matrices  $\mathcal{M}$ , we say a strategy achieves query complexity  $T(\varepsilon; \mathcal{M})$  if for any matrix  $M \in \mathcal{M}$ , the strategy outputs a pair  $p,q$  such that  $g(M,p,q) \leqslant 2\varepsilon$  when  $T \geqslant T(\varepsilon; \mathcal{M})$ . (The dependence on  $\mathcal{M}$  is omitted when clear from context.) We also occasionally refer to the query complexity of recovering the full matrix, which is a different task but in the same query model. In this case, the learner recommends a full matrix and achieves query complexity  $T$  if the matrix guess is correct for any true matrix  $M \in \mathcal{M}$ .
The topic of this paper is the study of the optimal query complexity of finding  $\varepsilon$ -Nash equilibria in the first-order model, for the set of matrices with bounded entries  $\mathcal{M}_K([-1,1])$ .
3 Upper Bounds
In this section we collect upper bounds for the first-order query complexity of approximate Nash equilibria. These bounds are either well-established, trivial or exploit in a miraculously striking fashion the difference between learning saddle points for matrix games taking entries in a countable (e.g.,  $\mathbb{Q}$ ) or uncountable (e.g.,  $[-1, 1]$ ) sets of values.
3.1 Query Complexity over  $\mathcal{M}_K([-1,1])$ : Regret and Elementary Strategies
We list some well-known results from the literature and state them in terms of first-order query complexity. The results described here are compiled in Figure 1. We use the notation  $u_{n:m}$  to denote the family  $(u_k)_{n \leqslant k \leqslant m}$ .
Figure 1: Upper (blue) and lower (red) bounds on the first-order query complexity of computing an  $\varepsilon$ -equilibrium for  $K\times K$  matrix games with entries in  $[-1,1]$ . See Sections 3 and 4 for details.
$O(\varepsilon^{-1}\log K)$  Queries from Optimistic Online Learning The current best upper bounds on the query complexity of  $\varepsilon$ -equilibria for zero-sum games are derived from online learning methods. As is well-known [5, Chapter 7], if we denote by  $\widehat{p}_T$  (resp.  $\widehat{q}_T$ ) the average of  $p_{1:T}$  (resp.  $q_{1:T}$ ) then
$$
\begin{array}{l} T g (M, \widehat {p} _ {T}, \widehat {q} _ {T}) = T \max  _ {j \in [ K ]} (M ^ {\intercal} \widehat {p} _ {T}) _ {j} - T \min  _ {i \in [ K ]} (M \widehat {q} _ {T}) _ {i} \\ = \sum_ {t = 1} ^ {T} \left\langle p _ {t}, M q _ {t} \right\rangle - T \min  _ {i \in [ K ]} (M \widehat {q} _ {T}) _ {i} + \sum_ {t = 1} ^ {T} \left\langle q _ {t}, - M ^ {\top} p _ {t} \right\rangle - T \min  _ {i \in [ K ]} (- M ^ {\top} \widehat {p} _ {T}) _ {j} \\ \leqslant \sum_ {t = 1} ^ {T} \left\langle p _ {t}, M q _ {t} \right\rangle - \sum_ {t = 1} ^ {T} \min  _ {i \in [ K ]} (M q _ {t}) _ {i} + \sum_ {t = 1} ^ {T} \left\langle q _ {t}, - M ^ {\intercal} p _ {t} \right\rangle - \sum_ {t = 1} ^ {T} \min  _ {i \in [ K ]} (- M ^ {\intercal} p _ {t}) _ {j}. \\ \end{array}
$$
This last term is exactly the sum of the regrets suffered by each player on their respective losses: the gap of the average plays is smaller than the sum of the average regrets over  $T$  rounds. This relationship between regret and gap provides a fruitful way to upper bound the query complexity of computing equilibria. Specifically, [37] observed that if players follow the Optimistic Mirror Descent strategy then the sum of the regrets stays smaller than  $\mathcal{O}(\log K)$ ; see the paragraph following Proposition 6 in the mentioned reference.
Theorem 1 (Consequence of Rakhlin and Sridharan [37]). There exists an absolute constant  $c > 0$  such that the first-order query complexity over  $\mathcal{M}_K([-1,1])$  is
$$
T (\varepsilon) \leqslant \left(c \frac {\log K}{\varepsilon}\right) \wedge K.
$$
A wide stream of literature leverages the connection between regret and equilibria, and study the dynamics of pairs of learning algorithms, including the two principal flavors of optimistic online algorithms, Optimistic Mirror Descent and Optimistic Follow-the-Regularized-Leader, cf. Section 1.2.
Instance-Dependent Rates of Convergence Many existing bounds show rates of convergence exponential in  $T$ , with instance-dependent constants<sup>1</sup>, see, e.g., [19, 41]. However, we show in Example 18 that for a zero-sum two-player bilinear game, these constants can be large, even on simple game matrices. If the constants are too large, the consequences on query complexity are vacuous, as they might require more than  $K$  queries to obtain non-trivial ( $\varepsilon < 1$ ) equilibria. Although these results give a very interesting analysis beyond the worst-case, they are beyond the scope of this work.
1More precisely: a parameter similar to a condition number of the game matrix
Upper Bounds for Large  $\varepsilon$  We conclude this section with two elementary strategies that provide equilibria with large approximation values of  $\varepsilon \geqslant 1 / 2$ .
Theorem 2 (Theorem. 3.1 in [10]). For any  $\varepsilon \geqslant 1/2$ , the query complexity over  $\mathcal{M}_K([-1,1])$  of finding an  $\varepsilon$ -equilibrium is  $T(\varepsilon) \leqslant 2$ .
The following proposition is a well-known observation (see, e.g., [16]) that having the learner select a pair of uniform plays always gives an approximate equilibrium.
Proposition 3. For any  $\varepsilon \geqslant 1 - 1 / K$ , the query complexity over  $\mathcal{M}_K([-1,1])$  of finding an  $\varepsilon$ -equilibrium is  $T(\varepsilon) = 0$ .
3.2 Recovering the Matrix: Upper and Lower Bounds
In this section we show that the first-order query complexity for recovering the full game matrix (a task harder than computing an equilibrium) is between  $\Theta(1)$ , independent of the number of actions, and  $\Theta(K)$ , depending on the set of candidate matrices. An important consequence of these results is that several known standard lower bound techniques cannot be applied to provide lower bounds on the easier task of learning equilibria.
Matrix Sets with Countable Alphabet Many existing methods rely on building hard matrices with entries in a finite alphabet: we prove this cannot lead to a lower bound.
Theorem 4. Let  $\mathcal{A} \subset \mathbb{R}$  be a countable set of at least two numbers. Then the first-order query complexity over  $\mathcal{M}_K(\mathcal{A})$  of recovering the full matrix is 1.
This result exploits infinite precision in the feedback, and as such does not provide a reasonable algorithm that the players would use should they know that the matrix belongs to  $\mathcal{M}_K(\mathcal{A})$ . The intent of this statement is to show that any attempt at a lower bound that builds matrices with entries in a fixed countable set (and in particular with integer entries) is doomed to fail after one query.
Remark 5. The proof of Theorem 4 does not provide an explicit algorithm. However, if the learner knows that the entries of the matrix are in a finite set  $\mathcal{A} \subset \mathbb{Q}$ , an explicit strategy can be easily described. In this case,  $\mathcal{A}$  is of the form  $\{a_1 / r, \ldots, a_n / r\}$  with  $a_i \in \mathbb{Z}$  and  $r \in \mathbb{N}$ . Consider a base  $b = \max(a_{1:n}) - \min(a_{1:n})$  and consider a query with  $p \propto (b^{-1}, b^{-2}, \ldots, b^{-K})$  (the choice for  $q$  is irrelevant here). The learner can deduce the full matrix  $M$  from the single observation  $M^{\top}p$ . Of course, this strategy has no practical interest as soon as either  $\mathcal{A}$  or  $K$  is moderately large, since it requires arbitrary precision in the outputs.
Difficulty of Recovering the Matrix Exactly In the first-order query model, the learner receives  $2K$  numbers every round, so in order to fully determine an arbitrary  $K \times K$  matrix one needs at least  $K^2 / (2K) = K / 2$  rounds. It turns out that this bound is not tight, and we may need exactly  $K$  rounds in the worst case, because of redundancies between the  $2K$  numbers we observe per round:
Theorem 6. The first-order query complexity over  $\mathcal{M}_K([-1,1])$  of recovering the full matrix is  $K$ .