SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuitively, it is overkill to recover the matrix exactly to output an  $\varepsilon$ -Nash equilibrium. In the next section, when  $\varepsilon \ll 1 / K^K$  (in particular when  $\varepsilon = 0$ ), we show that first-order queries do not provide useful information to find an  $\varepsilon$ -equilibrium faster than it takes to reveal the whole matrix.
4 A New Lower Bound on the Query Complexity of Approximate Equilibria
We switch perspective, attempting to make the life of any learner hard. For this, we need to think about responding to queries to keep the learner unwitting.
4.1 Overview and Notation
After  $t$  time steps, queries  $p_{1:t}$  and  $q_{1:t}$  have been made and the outputs provided to the learner are  $\ell_{1:t}^{(q)}$  and  $\ell_{1:t}^{(p)}$ . Let us denote the set of matrices with entries in  $[-1,1]$  that are consistent with the observations after  $t$  time steps by
$$
\mathcal {E} _ {t} = \left\{M \in \mathcal {M} _ {K} ([ - 1, 1 ]) \mid \forall s \in [ t ], M ^ {\intercal} p _ {s} = - \ell_ {s} ^ {(q)} \quad \text {a n d} \quad M q _ {s} = \ell_ {s} ^ {(p)} \right\};
$$
we sometimes refer to this as the set of candidate matrices after  $t$  rounds of observations. We omit the dependence on the sequence of queries  $(p_s, q_s)$  and outputs  $(\ell_s^{(p)}, \ell_s^{(q)})$  to reduce clutter, as it will always be clear from the context.
We say a sequence of matrices  $M_{1:T}$  is adapted to the queries  $p_{1:T}, q_{1:T}$ , if it gives consistent outputs to the queries, i.e., if for all  $s \leqslant t \leqslant T$ ,  $M_t^\top p_s = M_s^\top p_s$  and  $M_t q_s = M_s q_s$ ; in other words,  $M_{1:T}$  is adapted if  $M_{t+1} \in \mathcal{E}_t$  for all  $t$ .
Instead of defining directly the answers to the queries, we equivalently build a sequence of adapted matrices. Formally, at round  $t + 1$ , given  $\mathcal{E}_t$  and  $(p_{t+1}, q_{t+1})$ , we select a matrix  $M_{t+1} \in \mathcal{E}_t$  and output  $(\ell_{t+1}^{(p)}, \ell_{t+1}^{(q)}) = (M_{t+1}q_{t+1}, -M_{t+1}^{\mathsf{T}}p_{t+1})$ .
Let us fix some common technical notation that we use in the proofs, to measure distances in matrix space. For a matrix  $M \in \mathcal{M}_K(\mathbb{R})$ , and for  $r, s \in [1, \infty]$ , we denote the operator norm of  $M$  by  $\| M \|_{r,s} = \sup_{x \in \mathbb{R}^K : \| x \|_r = 1} \| M x \|_s$ . Recall that for  $y, z \in \mathbb{R}^K$ , we have  $\| y z^\top \|_{r,s} = \| y \|_s \| z \|_{r'}$  where  $r' \in [1, \infty]$  is such that  $1/r + 1/r' = 1$ . In particular  $\| M \|_{1,\infty} = \max_{i,j \in [K]} |M_{i,j}|$ , and  $\| y z^\top \|_{1,\infty} = \| y \|_\infty \| z \|_\infty$ . If  $\mathcal{F}$  is a closed convex set subset of  $\mathbb{R}^K$ , we denote by  $\operatorname{Proj}_{\mathcal{F}}(x)$  the orthogonal projection of  $x$  on  $\mathcal{F}$ .
Common Structure of the Proofs Both proofs of Theorems 11 and 17 follow the same template:
- Assume an  $\varepsilon$ -equilibrium is found after  $T$  first-order queries.
- Find a necessary condition on this equilibrium formulated as constraints on the observed losses and the sequence of queries. Precisely, we observe that under some initial assumptions on the game matrix, the all-ones vector  $\mathbf{1}$  needs to lie near the span of the outputs observed by the player, regardless of the queries and outputs.
- Ensure that this necessary condition cannot be met by any pair of mixed actions  $(p, q)$  by building an appropriate adapted sequence of matrices. In our case, we ensure that the span of the losses of the  $q$ -player stays far from the all-ones vector  $\mathbf{1}$ .
This proof structure is a promising way to derive the exact query complexity of (approximate) Nash equilibria. We use it to provide the first non-trivial lower bound for this setting.
The next two sections implement this template to prove query complexity lower bounds for exact  $\varepsilon = 0$  (Theorem 11) and approximate  $\varepsilon > 0$  (Theorem 17) equilibria. Detailed proofs are in Appendix D. While we could have deduced the final lower bound for exact equilibria directly from the approximate equilibria case, we describe the result separately to ease exposition. Indeed, the bound for the exact case contains the main ideas but is technically simpler.
4.2 Proof: Exact Equilibrium Case
Step I: Common Equilibria are in the Span of Past Queries We start by defining an initial set of candidate matrices with some special properties. These properties are used to ensure that the common equilibrium is fully mixed, and therefore an equalizing strategy for both players.
Definition 7. A set of matrices  $B_0 \subset \mathcal{M}_K([-1,1])$  satisfies the assumptions  $A(0)$  if
- For all  $M \in B_0$ , all equilibria of  $M$  are fully mixed,
- There exists a matrix  $M \in B_0$  with non-zero value.
By Lemma 20 (App. D), any ball centered at  $(1/2)I_K$  with small enough radius satisfies  $A(0)$ , e.g.,
$$
B _ {0} = \mathcal {B} _ {\| \cdot \| _ {1, \infty}} \left(\frac {1}{2} I _ {K}, \frac {1}{1 6 K ^ {2}}\right) = \left\{M \in \mathcal {M} _ {K} (\mathbb {R}): \left| M _ {i, j} - \frac {1}{2} \delta_ {i = j} \right| \leqslant \frac {1}{1 6 K ^ {2}} \quad \forall i, j \in [ K ] \right\}.
$$
The following lemma states that after  $t$  first-order queries, if at least one matrix in  $B_0$  is still a candidate matrix, then any common equilibrium to all candidate matrices must lie in the span of the queries. Since equilibria of matrices in  $B_0$  are fully mixed, and are thus equalizing strategies, this further implies that the all-ones vector  $\mathbf{1}$  must lie in the span of the outputs to the queries.
Lemma 8. Let  $B_0$  be a set of matrices satisfying  $A(0)$ . Assume in addition that  $\mathcal{E}_t \cap B_0 \neq \emptyset$  and that  $\mathcal{E}_t \cap \mathcal{M}_K((-1,1)) \neq \emptyset$ . If there exists a common Nash equilibrium  $(p,q)$  to all matrices in  $\mathcal{E}_t$ , then  $p \in \operatorname{Span}(p_{1:t})$  and  $q \in \operatorname{Span}(q_{1:t})$ .
Corollary 9. Under the assumptions of Lemma 8, there exists a value  $v \neq 0$  such that
$$
v \mathbf {1} \in \operatorname {S p a n} \left(\ell_ {1: t} ^ {(q)}\right) \cap \operatorname {S p a n} \left(\ell_ {1: t} ^ {(p)}\right).
$$
Step II: Sequential construction Given Corollary 9, to prove a query complexity lower bound, it suffices to build the answers to the queries  $p_{1:t}$  and  $q_{1:t}$  in a way that ensures that
- the vector  $\mathbf{1}$  never belongs to the span of the observations of (say) the  $q$ -player,
- there is at least one remaining candidate matrix in  $B_0$ , i.e.,  $\mathcal{E}_t \cap B_0 \neq \emptyset$ .
For technical reasons (cf. the assumptions of Lemma 8), we also need to make sure that there is enough space left in  $\mathcal{E}_t$ . We do so by keeping a candidate matrix  $M_t \in \mathcal{E}_t \cap \mathcal{M}_K((-1,1))$ , away from the border of the initial set of candidate matrices (i.e., with entries strictly between  $-1$  and  $1$ ).
Lemma 10. Fix a time horizon  $T \leqslant K / 2 - 1$ . For any sequence of queries  $p_{1:t}, q_{1:t}$ , there exists a sequence of matrices  $M_{1:T}$  in  $\mathcal{M}_K((-1,1))$  adapted to  $p_{1:T}, q_{1:T}$  that defines losses  $\ell_{1:T}^{(q)}$  for which, for any  $v \neq 0$ ,
$$
v \mathbf {1} \notin \operatorname {S p a n} (\ell_ {1: T} ^ {(q)}).
$$
Conclusion Combining Lemmas 8 and 10, against any learning strategy, we have built a sequence of outputs for which there is no common equilibrium to all remaining candidate matrices after  $T$  rounds, as long as  $T \leqslant (K - 3) / 2$ ; this proves the following theorem.
Theorem 11. The first-order query complexity over  $\mathcal{M}_K([-1,1])$  of finding a Nash equilibrium is
$$
T (0) \geqslant K / 2 - 1.
$$
The next section tackles the lower bound for approximate equilibria. Its proof follows the same template as for the exact case, although the technical complexity increases.
4.3 Proof: Approximate Equilibria Case
Step I: Common Equilibria Are Close to the Span of Queries In the following, we say a probability distribution over  $[K]$  is  $\delta$ -supported if  $p(i) \geqslant \delta$  for all  $i$ . We say a pair of distributions  $(p, q)$  is  $\delta$ -supported if both  $p$  and  $q$  are  $\delta$ -supported. We start by defining a quantitative version of Definition 7 for approximate equilibria. In order to retain the property that losses at  $\varepsilon$ -equilibria stay close to an isotropic vector, we add a requirement on the support of the equilibria.
Definition 12. A set of matrices  $B_{\varepsilon, \delta} \subset \mathcal{M}_K([-1,1])$  satisfies the assumption  $A(\varepsilon, \delta)$  if
- For all  $M \in B_{\varepsilon, \delta}$ , all  $\varepsilon$ -equilibria of  $M$  are  $\delta$ -supported,
- There exists a matrix  $M \in B_{\varepsilon, \delta}$  with non-zero value.
For example, by Corollary 21, the  $\| \cdot \|_{1,\infty}$ -ball centered at  $(1 / 2)I_K$  and of radius  $(1 / 16)K^2$  satisfies this condition for any  $\varepsilon \leqslant 1 / (16K^2)$  and  $\delta \leqslant 1 / (2K)$ .
The following proposition is a quantitative version of Lemma 8 for approximate equilibria, in which we show that any common approximate equilibrium to all matrices in  $\mathcal{E}_t$  needs to be close to the span of the queries.
Lemma 13. For  $\varepsilon, \delta > 0$ , let  $B_{\varepsilon, \delta}$  be a set of matrices satisfying  $A(\varepsilon, \delta)$ , cf. Definition 12. Assume that  $\mathcal{E}_t \cap B_{\varepsilon, \delta} \neq \emptyset$ , and that the relative interior of  $\mathcal{E}_t$  contains a ball of radius  $r_t$  measured in  $\| \cdot \|_{1,\infty}$ -norm. If there exists a common  $\varepsilon$ -Nash equilibrium  $(p, q)$  sto all matrices in  $\mathcal{E}_t$ , then
$$
\| p - \operatorname {P r o j} _ {\operatorname {S p a n} (p _ {1: t})} (p) \| \leqslant \frac {2 \varepsilon}{\delta r _ {t}} \quad a n d \quad \| q - \operatorname {P r o j} _ {\operatorname {S p a n} (q _ {1: t})} (q) \| \leqslant \frac {2 \varepsilon}{\delta r _ {t}}.
$$
Corollary 14. Under the assumptions of Lemma 13, there exists a game matrix  $M \in B_{\varepsilon, \delta}$  such that the value  $v \in \mathbb{R}$  of  $M$  satisfies  $\| v \mathbf{1} - \mathrm{Proj}_{\mathrm{Span}(\ell_{1:t}^{(q)})}(v \mathbf{1}) \| \leqslant 4 \sqrt{K} \varepsilon / (\delta r_t)$ .
Step II: Construction of Matrices We design an adapted sequence of matrices that keeps  $v1$  away from the span of the observed losses.
Lemma 15. Let  $B$  be a closed ball in  $\| \cdot \|_{1,\infty}$ -norm of radius  $r \leqslant 1/2$  contained in  $\mathcal{M}_K([-1,1])$ . Fix a time horizon  $T \leqslant K/2 - 1$ . For any sequence of queries  $(p_t, q_t)_{t \leqslant T}$ , there exists a sequence of matrices  $M_{1:T}$  in  $B$  adapted to  $(p_t, q_t)$  that defines losses  $\ell_{1:T}^{(q)}$  for which, for any  $v \geqslant 0$ ,
$$
\| v \mathbf {1} - \operatorname {P r o j} _ {\operatorname {S p a n} \left(\ell_ {1: T} ^ {(q)}\right)} (v \mathbf {1}) \| ^ {2} \geqslant v ^ {2} K \left(\frac {r ^ {2}}{8 K T ^ {2}}\right) ^ {T + 1},
$$
and such that  $\mathcal{E}_t$  contains a ball of radius  $r / 2$  in its relative interior.
The proof relies on a decomposition of the squared distance of  $v1$  to the span of observed losses at time  $t + 1$  through a Pythagorean identity, relating it to the span at time  $t$ . By carefully choosing the new matrix  $M_{t + 1}$  as a function of the query  $p_{t + 1}, q_{t + 1}$ , we manage to ensure that the squared distance decreases only by a constant factor.
Remark 16. The radius of the ball inside  $\mathcal{E}_t$  stays constant at  $r / 2$ , even though  $\mathcal{E}_t$  gets smaller as  $t$  increases. This is made possible by choosing  $M_t$  very close to  $M_{t - 1}$  (roughly at a distance of  $r / (KT)$  measured in  $\| \cdot \|_{1,\infty}$ ), effectively ensuring that  $M_t$  stays far from the boundary of  $\mathcal{M}_K([-1,1])$ .
Step III: Conclusion We now combine Corollary 14 and Lemma 15 to obtain a lower bound on the best achievable gap after  $T$  rounds, which directly translates to a query complexity lower bound.
Theorem 17. In the first-order query model on  $\mathcal{M}_K([-1,1])$ , for any algorithm the worst-case gap after  $T \leqslant (K - 3)/2$  time steps is at least
$$
\varepsilon \geqslant \frac {1}{2 ^ {1 0} K ^ {4}} \left(\frac {1}{2 ^ {1 1 / 2} K ^ {5 / 2} T}\right) ^ {T + 1}.
$$
Therefore, the query complexity of finding an  $\varepsilon$ -equilibrium for any  $\varepsilon \leqslant 1 / (e2^{11}K^4)$  is at least
$$
T (\varepsilon) \geqslant \left(\frac {- \log \left(2 ^ {1 1} K ^ {4} \varepsilon\right)}{\log \left(2 ^ {1 1 / 2} K ^ {5 / 2}\right) + \log \left(- \log \left(2 ^ {1 1} K ^ {4} \varepsilon\right)\right)} - 1\right) \wedge (K / 2 - 1).
$$
4.4 Potential Improvement and Discussion
There is still a wide disparity between upper and lower bounds on first-order query complexity. The lower bound Theorem 17 is most probably not tight, so let us discuss potential ways to improve it.
We believe the most promising approach for improvement is to find a different proxy for the gap. Our proof uses the distance to the span of the losses  $\| v\mathbf{1} - \mathrm{Proj}_{\mathrm{Span}(\ell_{1:t}^{(q)})}\|^2$  for  $v \geqslant 1/(2K)$ , which is convenient because it is a distance, but we need to introduce a strong restriction on the set of candidate matrices  $B_{\varepsilon,\delta}$  to be able to relate it to the gap, namely the restriction to a ball around  $\frac{1}{2} I_K$  of radius  $O(1/K^2)$ . In order to make significant progress it is therefore essential to enlarge the class of candidate matrices significantly compared to  $B_{\varepsilon,\delta}$ , and therefore to modify the proxy for the gap.
5 Discussion: Future Work and Conclusion
We study the first-order query complexity of computing approximate Nash equilibria for two-player zero-sum matrix games. We review upper bounds coming from online learning, and discuss existing lower bounds for related problems including alternative query models. Taking stock, we arrive at the surprising state of affairs that for this fundamental problem no lower bounds are known. We then offer some explanation for this current state of affairs: the first-order query model is powerful enough to identify any matrix from a fixed countable set in a single query; this rules out many techniques. We then turn to lower bounds. We design an adaptive adversary that answers incoming learner queries in such a way that the remaining consistent matrices do not share a common Nash equilibrium for as long as possible. Our approach is based on a quantity serving as a "potential function": namely the distance of the all-ones vector to the span of the observations. We discuss in detail the result, future scope and limits of our proposed technique.
As can be seen in Figure 1, we are still far from matching lower and upper bounds. The most intriguing possibility for resolution would be if it were to turn out that the upper bounds (i.e. algorithms) are improvable. Current upper bounds come from online learning regret bounds, with algorithms falling in the category of uncoupled dynamics. We would love to know if these algorithm templates are in fact optimal for query complexity.
To cycle back to our motivation of computing saddle points in general, future work could attempt to extend our lower bound technique to different constraint sets, to functions possibly exhibiting curvature, and to instance-dependent rates. Another direction would be to generalize to multi-player games and investigate the query complexity of weaker solution concepts.
Limitations and Broader Impact The main limit of this work is that it only partially resolves the first-order query complexity of approximate NE in finite games, calling for tighter analysis. Regarding broader impact, results are mainly theoretical and do not entail direct societal consequences.
Sachs and Van Erven were supported by the Netherlands Organization for Scientific Research (NWO) under grant number VI.Vidi.192.095. Part of the research was performed while Hadiji was at the University of Amsterdam. During that time he was supported by the same grant number VI.Vidi.192.095.
[1] Waiss Azizian, Ioannis Mitlagkas, Simon Lacoste-Julien, and Gauthier Gidel. A tight and unified analysis of gradient-based methods for a whole spectrum of differentiable games. In International conference on artificial intelligence and statistics, pages 2863-2873. PMLR, 2020.
[2] Yakov Babichenko. Query complexity of approximate Nash equilibria. Journal of the ACM (JACM), 63(4):1-24, 2016.
[3] George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13 (1):374, 1951.
[4] Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Tight last-iterate convergence of the extragradient method for constrained monotone variational inequalities. arXiv preprint arXiv:2204.09228, 2022.
[5] Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[6] Xi Chen and Binghui Peng. Hedging in games: Faster convergence of external and swap regrets. Advances in Neural Information Processing Systems, 33:18990-18999, 2020.
[7] Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Computing Nash equilibria: Approximation and smoothed complexity. *Electron. Colloquium Comput. Complex.*, 2006.
[8] Xi Chen, Xiaotie Deng, and Shang-Hua Teng. Settling the complexity of computing two-player Nash equilibria. J. ACM, 56(3), may 2009.
[9] Constantinos Daskalakis, Paul Goldberg, and Christos Papadimitriou. The complexity of computing a Nash equilibrium. SIAM J. Comput., 39:195-259, 02 2009.
[10] Constantinos Daskalakis, Aranyak Mehta, and Christos Papadimitriou. A note on approximate Nash equilibria. Theoretical Computer Science, 410(17):1581-1588, 2009.
[11] Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms, pages 235-254. SIAM, 2011.
[12] Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. Games and Economic Behavior, 92:327-348, 2015.
[13] Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Advances in Neural Information Processing Systems, volume 34, pages 27604-27616, 2021.
[14] Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1466-1478, 2021.
[15] Gabriele Farina, Ioannis Anagnostides, Haipeng Luo, Chung-Wei Lee, Christian Kroer, and Tuomas Sandholm. Near-optimal no-regret learning dynamics for general convex games. Advances in Neural Information Processing Systems, 35:39076–39089, 2022.
[16] John Fearnley, Martin Gairing, Paul W Goldberg, and Rahul Savani. Learning equilibria of games via payoff queries. Journal of Machine Learning Research, 16:1305-1344, 2015.
[36] Leonid Denisovich Popov. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR, 28:845-848, 1980.
[37] Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems, 26, 2013.
[38] Julia Robinson. An iterative method of solving a game. Annals of mathematics, pages 296-301, 1951.
[39] Gilles Stoltz and Gábor Lugosi. Learning correlated equilibria in games with compact sets of strategies. Games and Economic Behavior, 59(1):187-208, 2007.
[40] Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E Schapire. Fast convergence of regularized learning in games. Advances in Neural Information Processing Systems, 28, 2015.
[41] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations, 2020.
[42] Tong Yang, Michael I Jordan, and Tatjana Chavdarova. Solving constrained variational inequalities via an interior point method. arXiv preprint arXiv:2206.10575, 2022.
[43] Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming, 194(1-2):901-935, 2022.
A Proofs of Lemmas 8 and 10
Proof of Lemma 8. Since  $p, q$  is an equilibrium to at least one matrix in  $B_0 \cap \mathcal{E}_t$ , it is fully supported; in particular  $p$  is an equalizing strategy for any  $M \in \mathcal{E}_t$  and  $M^\intercal p = v_M \mathbf{1}$  for some number  $v_M \in \mathbb{R}$ . Furthermore, the value  $v_M$  is actually independent of  $M \in \mathcal{E}_t$  since,  $v_M = \langle M^\intercal p, q_1 \rangle = \langle p, \ell_1^{(p)} \rangle$ . Therefore, for any  $M, M' \in \mathcal{E}_t$ , we have  $(M - M')^\intercal p = \mathbf{0}$ . Let us now define  $\bar{p} = p - \mathrm{Proj}_{\mathrm{Span}(p_{1:t})}(p)$ , and consider the direction  $\bar{p} u_q^\intercal \in \mathcal{M}_K(\mathbb{R})$  for some arbitrary non-zero vector  $u_q$  orthogonal to  $q_1, \ldots, q_t$ . Fix some matrix  $M \in \mathcal{E}_t \cap \mathcal{M}_K((-1,1))$ ; a non-empty set by assumption. Then for  $\alpha \in \mathbb{R}$  small enough, the matrix  $M' = M + \alpha \bar{p} e_q^\intercal$  is still in  $\mathcal{E}_t$ , since  $M$  is not on its border (all entries are away from  $(-1,1)$ ). Therefore,
$$
\mathbf {0} = \left(M - M ^ {\prime}\right) ^ {\intercal} p = \alpha \langle p, \bar {p} \rangle e _ {q} = \alpha \| \bar {p} \| ^ {2} u _ {q};
$$
implying that  $\| \bar{p} \| = 0$ , i.e. that  $p \in \operatorname{Span}(p_{1:t})$ ; similar reasoning shows that  $q \in \operatorname{Span}(q_{1:t})$ .
Proof of Lemma 10. We build the sequence  $M_{1:T}$  incrementally by moving at step  $t + 1$  in directions chosen as a function of the new queries  $p_{t + 1}$  and  $q_{t + 1}$ , ensuring by induction that  $\mathbf{1} \notin \operatorname{Span}(\ell_{1:T}^{(q)})$ .
Initialize the sequence at  $M_0 = (1/2) I_K$ . Now for  $t \geqslant 0$ , let us assume that we have correctly built  $M_{1:t}$ , inducing losses such that  $\mathbf{1} \notin \operatorname{Span}(\ell_{1:t}^{(q)})$ , and let us define  $M_{t+1}$ . (Note that the initialization of the induction is valid with the convention that  $\mathbf{1} \notin \{\mathbf{0}\} = \operatorname{Span}(\varnothing)$ .)
If  $p_{t+1}$  is in the span of  $p_{1:t}$ , then set  $M_{t+1} = M_t$  and the induction holds, since the span of the losses is left unchanged. Otherwise set  $M_{t+1} = M_t + \frac{\bar{p}_{t+1}}{\|\bar{p}_{t+1}\|^2} u_t^{\mathsf{T}}$ , where  $\bar{p}_{t+1} = p_{t+1} - \mathrm{Proj}_{\mathrm{Span}(p_{1:t})}(p_{t+1})$ , and  $u_t$  is a non-zero vector orthogonal to the vectors  $q_{1:t}$ , to  $\ell_{1:t}^{(q)}$ , to  $\mathbf{1}$  and to  $M_t^{\mathsf{T}} p_{t+1}$ ; the existence of such a  $u_t$  is guaranteed since  $2t + 2 < K$ , ensuring that there is at least one dimension orthogonal to those  $2t + 2$  vector. By choosing the norm of  $u_t$  to be small enough, we can make sure that  $M_{t+1} \in \mathcal{E}_t \cap \mathcal{M}_K((-1,1))$ . Then,  $\ell_{t+1}^{(q)} = M_{t+1}^{\mathsf{T}} p_{t+1} = M_t^{\mathsf{T}} p_{t+1} + u_t$ , and  $\mathbf{1}$  is not in the span of  $\ell_{1:(t+1)}^{(q)}$ . Indeed, assume by contradiction that there exists real numbers  $\alpha_{1:(t+1)}$  such that
$$
\mathbf {1} = \sum_ {s = 1} ^ {t} \alpha_ {s} \ell_ {s} ^ {(q)} + \alpha_ {t + 1} \ell_ {t + 1} ^ {(q)},
$$
then  $\alpha_{t + 1} > 0$  since by induction  $\mathbf{1}\notin \mathrm{Span}(\ell_{1:t}^{(q)})$ . Then, taking dot products with  $u_{t}$  on both sides of the equation above, we obtain  $0 = \langle u_t,\mathbf{1}\rangle = \langle u_t,\alpha_{t + 1}\ell_{t + 1}^{(q)}\rangle = \alpha_{t + 1}\| u_t\| ^2$ , leading to a contradiction. Therefore  $\mathbf{1}\notin \mathrm{Span}(\ell_{1:t}^{(q)})$  at all times  $t\leqslant T$ .
B Details for Upper bounds
B.1 Instance-Dependent Query Complexity