SlowGuess/NeurIPS_2023_OCR huggingface dataset:
since  $\| \hat{z}^{(h + 1,1)} - z^{(h + 1,\star)}\| _2^2\leq 2\| \hat{z}^{(h + 1,1)} - z^{(h,\star)}\| _2^2 +2\| z^{(h,\star)} - z^{(h + 1,\star)}\| _2^2$  by the triangle inequality and Young's inequality. Moreover, for  $t\geq 2$
$$
\begin{array}{l} \left\| \boldsymbol {u} _ {z} ^ {(t)} - \boldsymbol {u} _ {z} ^ {(t - 1)} \right\| _ {2} ^ {2} = \left\| F ^ {(t)} (\boldsymbol {z} ^ {(t)}) - F ^ {(t - 1)} (\boldsymbol {z} ^ {(t - 1)}) \right\| _ {2} ^ {2} \leq 2 L ^ {2} \left\| \boldsymbol {z} ^ {(t)} - \boldsymbol {z} ^ {(t - 1)} \right\| _ {2} ^ {2} \\ + 2 \| F ^ {(t)} \left(\boldsymbol {z} ^ {(t - 1)}\right) - F ^ {(t - 1)} \left(\boldsymbol {z} ^ {(t - 1)}\right) \| _ {2} ^ {2}, \\ \end{array}
$$
by  $L$ -smoothness. As a result,
$$
\sum_ {t = 1} ^ {T - 1} \| \boldsymbol {u} _ {z} ^ {(t + 1)} - \boldsymbol {u} _ {z} ^ {(t)} \| _ {2} ^ {2} \leq 2 L ^ {2} \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {z} ^ {(t)} - \boldsymbol {z} ^ {(t - 1)} \| _ {2} ^ {2} + 2 \mathcal {V} _ {\nabla f} ^ {(H)},
$$
and the claimed bound on the second-order path length follows. Finally, the second claim of the theorem follows from Claim A.10 using convexity-concavity, analogously to Theorem 3.3.  $\square$
We conclude this subsection by pointing out an improved variation-dependent regret bound, which follows directly from Theorem A.21 (cf. Corollary A.12).
Corollary A.22. In the setup of Theorem A.21, the maximum of the two players' regrets,  $\max \{\mathrm{Reg}_{\mathcal{L},x}^{(T)},\mathrm{Reg}_{\mathcal{L},y}^{(T)}\}$ , can be upper bounded by
$$
\frac {D _ {\mathcal {Z}} ^ {2}}{\eta} + 1 6 \eta L ^ {2} D _ {\mathcal {Z}} ^ {2} + (3 2 \eta^ {3} L ^ {2} + \eta) \| F (\boldsymbol {z} ^ {(1)}) \| _ {2} ^ {2} + 3 2 \eta L ^ {2} \mathcal {S} _ {N E} ^ {(H)} + (6 4 \eta^ {3} L ^ {2} + 2 \eta) \mathcal {V} _ {\nabla f} ^ {(H)}.
$$
Thus, setting the learning rate optimally implies that  $\mathrm{Reg}_{\mathcal{L},x}^{(T)},\mathrm{Reg}_{\mathcal{L},y}^{(T)} = O\left(\sqrt{1 + \mathcal{S}_{\mathrm{NE}}^{(H)} + \mathcal{V}_{\nabla f}^{(H)}}\right)$ .
A.3 Proofs from Section 3.3
In this subsection, we provide the proofs from Section 3.3.
A.3.1 Potential games
We first characterize the behavior of (online) gradient descent (GD) in time-varying potential games; we recall that GD is equivalent to OG under the prediction  $\boldsymbol{m}_x^{(t)} = \mathbf{0}$  for all  $t$ . Below we give the formal definition of a potential game.
Definition A.23 (Potential game). An  $n$ -player game admits a potential if there exists a function  $\Phi : \times_{i=1}^{n} \mathcal{X}_i \to \mathbb{R}$  such that for any player  $i \in [[n]]$ , any joint strategy profile  $\pmb{x}_{-i} \in \times_{i' \neq i} \mathcal{X}_{i'}$ , and any pair of strategies  $\pmb{x}_i, \pmb{x}_i' \in \mathcal{X}_i$ ,
$$
\Phi (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {- i}) - \Phi (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {- i}) = u _ {i} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {- i}) - u _ {i} (\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {- i}).
$$
The key ingredient in the proof of Theorem 3.7 is the following bound on the second-order path length of the dynamics.
Proposition A.24. Suppose that each player employs  $GD$  with a sufficiently small learning rate  $\eta > 0$  and initialization  $(\pmb{x}_1^{(1)}, \ldots, \pmb{x}_n^{(1)}) \in \mathsf{X}_{i=1}^n \mathcal{X}_i$  in a sequence of time-varying potential games. Then,
$$
\frac {1}{2 \eta} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} ^ {(t + 1)} - \boldsymbol {x} _ {i} ^ {(t)} \| _ {2} ^ {2} \leq \sum_ {t = 1} ^ {T} \left(\Phi^ {(t)} \left(\boldsymbol {x} _ {1} ^ {(t + 1)}, \dots , \boldsymbol {x} _ {n} ^ {(t + 1)}\right) - \Phi^ {(t)} \left(\boldsymbol {x} _ {1} ^ {(t)}, \dots , \boldsymbol {x} _ {n} ^ {(t)}\right)\right). \tag {47}
$$
This bound can be derived from [2, Theorem 4.3]. Now, we note that if  $\Phi^{(1)} = \Phi^{(2)} = \dots = \Phi^{(T)}$ , the right-hand side of (47) telescopes, thereby implying that the second-order path-length is bounded. More generally, the right-hand side of (47) can be upper bounded by
$$
2 \Phi_ {\max } + \sum_ {t = 1} ^ {T - 1} \left(\Phi^ {(t)} \left(\boldsymbol {x} _ {1} ^ {(t + 1)}, \dots , \boldsymbol {x} _ {n} ^ {(t + 1)}\right) - \Phi^ {(t + 1)} \left(\boldsymbol {x} _ {1} ^ {(t + 1)}, \dots , \boldsymbol {x} _ {n} ^ {(t + 1)}\right)\right) \leq 2 \Phi_ {\max } + \mathcal {V} _ {\Phi} ^ {(T)}, \tag {48}
$$
where  $\Phi_{\mathrm{max}}$  is an upper bound on  $|\Phi^{(t)}(\cdot)|$  for any  $t \in [[T]]$ , and  $\mathcal{V}_{\Phi}^{(T)}$  is the variation measure of the potential functions we introduced in Section 3.3. Namely,  $\mathcal{V}_{\Phi}^{(T)} := \sum_{t=1}^{T-1} d(\Phi^{(t)}, \Phi^{(t+1)})$ , where  $d: (\Phi, \Phi') \mapsto \max_{z \in \times_{i=1}^n \mathcal{X}_i} (\Phi(z) - \Phi'(z))$ . Furthermore, similarly to Claim A.10, we know that the Nash equilibrium gap in the  $t$ -th potential game can be bounded in terms of  $\sum_{i=1}^n \|x_i^{(t+1)} - x_i^{(t)}\|_2$ . As a result, combining this property with Proposition A.24 and (48) establishes Theorem 3.7, the statement of which is recalled below.
Theorem 3.7. Suppose that each player employs (online)  $GD$  with a sufficiently small learning rate in a sequence of time-varying potential games. Then,  $\sum_{t=1}^{T} \left( \mathrm{EQGAP}^{(t)}(\boldsymbol{z}^{(t)}) \right)^2 = O(\Phi_{max} + \mathcal{V}_{\Phi}^{(T)})$ , where  $\Phi_{max}$  is such that  $|\Phi^{(t)}(\cdot)| \leq \Phi_{max}$ .
A.3.2 General-sum games
We next turn out attention to general-sum multi-player games in normal form using the well-known bilinear formulation of correlated equilibria (Section 3.3). More precisely, this zero-sum game is played between a mediator, who selects strategies from the set of correlated distributions  $\Xi := \Delta(\mathsf{X}_{i=1}^{n}\mathcal{A}_{i})$ , and the  $n$  players. Per the usual description of correlated equilibria, a mediator recommends to each player  $i \in [[n]]$  an action in  $\mathcal{A}_{i}$ , and then that player selects an action that could be different from the one recommended by the mediator. As a result, each pure strategy of Player  $i$  corresponds to a different mapping from  $\mathcal{A}_{i}$  to  $\mathcal{A}_{i}$ . Player  $i$  wants to maximize the deviation benefit, while the mediator wants to minimize it. We let  $\bar{\mathcal{X}}_{i}$  be the set of (mixed) strategies of Player  $i$ . The identity mapping  $d_{i}: \mathcal{A}_{i} \ni a_{i} \mapsto a_{i}$  will be referred to as the direct (or obedient) strategy, as it prescribes following the recommendation of the mediator. As a result, this game can be naturally cast as the bilinear saddle-point problem in the form of (4). By the existence of correlated equilibria [50], it follows that there exists a mediator strategy  $\pmb{\mu}^{\star} \in \Xi$  such that  $\bar{\pmb{x}}_{i}^{\top}\mathbf{A}_{i}^{\top}\pmb{\mu}^{\star} \leq 0$ , for any player  $i \in [[n]]$  and  $\bar{\pmb{x}}_{i} \in \bar{\mathcal{X}}_{i}$ ; in words, there is no benefit for any player to deviate from the recommendation of the mediator.
Now, to establish Property 3.8, let us first define the regret of any player  $i \in [[n]]$  as
$$
\mathrm {R e g} _ {i} ^ {(T)} (\bar {\pmb {x}} _ {i} ^ {\star}) := \sum_ {t = 1} ^ {T} \langle \bar {\pmb {x}} _ {i} ^ {\star} - \bar {\pmb {x}} _ {i} ^ {(t)}, (\mathbf {A} _ {i} ^ {(t)}) ^ {\top} \pmb {\mu} ^ {(t)} \rangle ,
$$
where  $\bar{\pmb{x}}_i^{\star}\in \tilde{\mathcal{X}}_i$ , so that  $\sum_{i = 1}^{n}\mathrm{Reg}_i^{(T)}$  is easily seen to be equal to the regret of Player max in (4). Further, the dynamic regret of the mediator—Player min in (4)—can be expressed as
$$
\mathrm {D R e g} _ {\mu} ^ {(T)} (\boldsymbol {\mu} ^ {(1, \star)}, \dots , \boldsymbol {\mu} ^ {(T, \star)}) := \sum_ {t = 1} ^ {T} \langle \boldsymbol {\mu} ^ {(t)} - \boldsymbol {\mu} ^ {(t, \star)}, \sum_ {i = 1} ^ {n} \mathbf {A} _ {i} ^ {(t)} \bar {\boldsymbol {x}} _ {i} ^ {(t)} \rangle .
$$
The key idea in the proof of Property 3.8 is that the time-invariant direct strategy  $(d_1, \ldots, d_n)$  suffices to guarantee Property 3.1 as long as the mediator is selecting a sequence of CE.
Property 3.8. Suppose that  $\Xi \ni \pmb{\mu}^{(t,\star)}$  is a correlated equilibrium of the game at any time  $t\in [[T]]$ . Then,  $\mathrm{DReg}_{\mu}^{(T)}(\pmb{\mu}^{(1,\star)},\dots,\pmb{\mu}^{(T,\star)}) + \sum_{i = 1}^{n}\mathrm{Reg}_i^{(T)}\geq 0$ .
$$
\mathrm {D R e g} _ {\mu} ^ {(T)} + \sum_ {i = 1} ^ {n} \mathrm {R e g} _ {i} ^ {(T)} (\bar {\pmb {x}} _ {i} ^ {\star}) = \sum_ {i = 1} ^ {n} \sum_ {t = 1} ^ {T} \langle \bar {\pmb {x}} _ {i} ^ {\star}, (\mathbf {A} _ {i} ^ {(t)}) ^ {\top} \pmb {\mu} ^ {(t)} \rangle - \sum_ {t = 1} ^ {T} \langle \pmb {\mu} ^ {(t, \star)}, \sum_ {i = 1} ^ {n} \mathbf {A} _ {i} ^ {(t)} \bar {\pmb {x}} _ {i} ^ {(t)} \rangle .
$$
Now for any correlated equilibrium  $\pmb{\mu}^{(t,\star)}$  of the  $t$ -th game, we have that  $\langle \pmb{\mu}^{(t,\star)}, \mathbf{A}_i^{(t)} \bar{\pmb{x}}_i^{(t)} \rangle \leq 0$  for any player  $i \in [n]$ ,  $\bar{\pmb{x}}_i \in \bar{\mathcal{X}}_i$ , and time  $t \in [T]$ , which in turn implies that  $-\sum_{t=1}^{T} \langle \pmb{\mu}^{(t,\star)}, \sum_{i=1}^{n} \mathbf{A}_i^{(t)} \bar{\pmb{x}}_i^{(t)} \rangle \geq 0$ . Moreover,  $\sum_{i=1}^{n} \max_{\bar{\pmb{x}}_i^* \in \bar{\mathcal{X}}_i} \sum_{t=1}^{T} \langle \bar{\pmb{x}}_i^*, (\mathbf{A}_i^{(t)})^\top \pmb{\mu}^{(t)} \rangle \geq \sum_{i=1}^{n} \sum_{t=1}^{T} \langle \pmb{d}_i, (\mathbf{A}_i^{(t)})^\top \pmb{\mu}^{(t)} \rangle = 0$ , by definition of the direct strategy  $d_i \in \bar{\mathcal{X}}_i$ . This concludes the proof.
Property 3.8 in conjunction with the argument of Theorem 3.3 readily imply Theorem 3.9. We should note here that the variation measure  $\mathcal{V}_{\mathbf{A}}^{(T)} := \sum_{i=1}^{n} \sum_{t=1}^{T-1} \| \mathbf{A}_i^{(t+1)} - \mathbf{A}_i^{(t)} \|_2^2$  is also immediately bounded in terms of the second-order variation of the sum of the players' utility tensors. It is also worth noting that the description of the bilinear formulation (4) grows exponentially with the number of players; this is unavoidable in explicitly represented (normal-form) games, but it is worth investigating whether more compact representations exist in succinct classes of games, such as multi-player polymatrix.
Next, we provide the main implication of Theorem 3.9 in the meta-learning setting, which is similar to the meta-learning guarantee of Proposition A.14 we established earlier in two-player zero-sum games. Below, we denote by  $\Xi^{(h,\star)}$  the set of correlated equilibria of the  $h$ -th game in the meta-learning sequence.
Corollary A.25 (Meta-learning in general games). Suppose that each player employs DGD in (4) with a suitable learning rate  $\eta > 0$  and the prediction of (37) in a meta-learning general-sum problem with  $H \in \mathbb{N}$  games, each repeated for  $m \in \mathbb{N}$  consecutive iterations. Then, for an average game,
$$
O \left(\frac {1}{\epsilon^ {2} H} + \frac {\mathcal {V} _ {C E} ^ {(H)}}{\epsilon^ {2} H}\right) \tag {49}
$$
iterations suffice so that the mediator reaches an  $\epsilon$ -approximate correlated equilibrium, where
$$
\mathcal {V} _ {C E} ^ {(H)} := \inf  _ {\boldsymbol {\mu} ^ {(h, \star)} \in \Xi^ {(h, \star)}} \| \boldsymbol {\mu} ^ {(h + 1, \star)} - \boldsymbol {\mu} ^ {(h, \star)} \| _ {2}.
$$
In particular, in the meta-learning regime,  $H \gg 1$ , the iteration-complexity bound (49) is dominated by the (algorithm-independent) similarity metric of the correlated equilibria  $\frac{\mathcal{V}_{\mathrm{CE}}^{(H)}}{H}$ . Corollary A.25 establishes significant gains when  $\frac{\mathcal{V}_{\mathrm{CE}}^{(H)}}{H} \ll 1$ .
Finally, we conclude this subsection by providing a variation-dependent regret bound in general-sum multi-player games. To do so, we combine Corollary A.12 with Theorem 3.9, leading to the following guarantee.
Corollary A.26 (Regret in general-sum games). In the setup of Theorem 3.9,
$$
\operatorname {R e g} _ {\mu} ^ {(T)}, \operatorname {R e g} _ {i} ^ {(T)} = O \left(\frac {1}{\eta} + \eta \left(1 + \mathcal {V} _ {C E} ^ {(T)} + \mathcal {V} _ {\mathbf {A}} ^ {(T)}\right)\right),
$$
for any player  $i\in [[n]]$
In particular, if one selects optimally the learning rate, Corollary A.26 implies that the individual regret of each player is bounded by  $O\left(\sqrt{1 + \mathcal{V}_{\mathrm{CE}}^{(T)} + \mathcal{V}_{\mathbf{A}}^{(T)}}\right)$ . We note again that there are techniques that would allow (nearly) recovering such regret guarantees without having to know the variation measures in advance [93].
A.4 Proofs from Section 3.4
Finally, in this subsection we present the proofs omitted from Section 3.4. We begin with Proposition 3.10, the statement of which is recalled below. We first recall that a regularizer  $\phi_x$ , 1-strongly convex with respect to a norm  $\| \cdot \|$ , is said to be  $G$ -smooth if  $\| \nabla \phi_x(\pmb{x}) - \nabla \phi_x(\pmb{x}') \|_* \leq G \| \pmb{x} - \pmb{x}' \|$  for all  $\pmb{x}, \pmb{x}'$ .
Proposition 3.10. Suppose that both players in a (static) two-player zero-sum game employ OMD with a smooth regularizer. Then,  $\mathrm{DReg}_x^{(T)},\mathrm{DReg}_y^{(T)} = O(\sqrt{T})$
Proof. First, using Claim A.10, it follows that the dynamic regret  $\mathrm{DReg}_x^{(T)}$  of Player  $x$  up to time  $T$  can be bounded as
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\max  _ {\boldsymbol {x} ^ {(t, \star)} \in \mathcal {X}} \left\{\langle \boldsymbol {x} ^ {(t, \star)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle \right\} - \langle \boldsymbol {x} ^ {(t)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\left(\frac {G D _ {\mathcal {X}}}{\eta} + \| \boldsymbol {u} _ {x} ^ {(t)} \| _ {*}\right) \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)} \| + \frac {G D _ {\mathcal {X}}}{\eta} \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t)} \|\right), \tag {50} \\ \end{array}
$$
where  $G > 0$  is the smoothness parameter of the regularizer, and  $\eta > 0$  is the learning rate. We further know that  $\sum_{t=1}^{T}\left(\|\pmb{x}^{(t)} - \hat{\pmb{x}}^{(t)}\|^2 + \|\pmb{x}^{(t)} - \hat{\pmb{x}}^{(t+1)}\|^2\right) = O(1)$  for any instance of 0MD in a two-player zero-sum game [2], which in turn implies that  $\sum_{t=1}^{T}\left(\|\pmb{x}^{(t)} - \hat{\pmb{x}}^{(t)}\|^2 + \|\pmb{x}^{(t)} - \hat{\pmb{x}}^{(t+1)}\|\right) = O(\sqrt{T})$  by Cauchy-Schwarz. Thus, combining with (50) we have shown that  $\mathrm{DReg}_x^{(T)} = O(\sqrt{T})$ . Similar reasoning yields that  $\mathrm{DReg}_y^{(T)} = O(\sqrt{T})$ , concluding the proof.
Let us next make a certain refinement of Observation 3.11. Staying on (static) bilinear saddle-point problems, we consider the unconstrained setting where  $\mathcal{X} = \mathbb{R}^{d_x}$  and  $\mathcal{Y} = \mathbb{R}^{d_y}$ , and we focus on the following update rule for  $\tau \in \mathbb{N}$ .
$$
\boldsymbol {x} ^ {(\tau + 1)} = \boldsymbol {x} ^ {(\tau)} - 2 \eta \mathbf {A} \boldsymbol {y} ^ {(\tau)} + \eta \mathbf {A} \boldsymbol {y} ^ {(\tau - 1)},
$$
$$
\boldsymbol {y} ^ {(\tau + 1)} = \boldsymbol {y} ^ {(\tau)} + 2 \eta \mathbf {A} ^ {\top} \boldsymbol {x} ^ {(\tau)} - \eta \mathbf {A} ^ {\top} \boldsymbol {x} ^ {(\tau - 1)}, \tag {51}
$$
where  $\pmb{x}^{(0)},\pmb{x}^{(-1)}\in \mathcal{X}$  and  $\pmb{y}^{(0)},\pmb{y}^{(-1)}\in \mathcal{Y}$ . This is an instance of optimistic follow the regularized leader (OFTRL) [78], although our discussion here also covers the corresponding OMD variant. Then, summing (51) for  $\tau = 1,2,\dots ,t$  we have
$$
\begin{array}{l} \sum_ {\tau = 1} ^ {t} \boldsymbol {x} ^ {(\tau + 1)} = \sum_ {\tau = 1} ^ {t} \boldsymbol {x} ^ {(\tau)} - 2 \eta \mathbf {A} \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {y} ^ {(\tau)}\right) + \eta \mathbf {A} \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {y} ^ {(\tau - 1)}\right), \\ \sum_ {\tau = 1} ^ {t} \boldsymbol {y} ^ {(\tau + 1)} = \sum_ {\tau = 1} ^ {t} \boldsymbol {y} ^ {(\tau)} + 2 \eta \mathbf {A} ^ {\top} \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {x} ^ {(\tau)}\right) - \eta \mathbf {A} ^ {\top} \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {x} ^ {(\tau - 1)}\right), \\ \end{array}
$$
in turn implying that
$$
\begin{array}{l} \bar {\boldsymbol {x}} ^ {(t + 1)} = \frac {1}{t + 1} \bar {\boldsymbol {x}} ^ {(1)} + \eta \mathbf {A} \frac {1}{t + 1} \boldsymbol {y} ^ {(0)} + \frac {t}{t + 1} \bar {\boldsymbol {x}} ^ {(t)} - 2 \eta \frac {t}{t + 1} \mathbf {A} \bar {\boldsymbol {y}} ^ {(t)} + \eta \frac {t - 1}{t + 1} \mathbf {A} \bar {\boldsymbol {y}} ^ {(t - 1)}, \\ \bar {\boldsymbol {y}} ^ {(t + 1)} = \frac {1}{t + 1} \bar {\boldsymbol {y}} ^ {(1)} - \eta \mathbf {A} ^ {\top} \frac {1}{t + 1} \boldsymbol {x} ^ {(0)} + \frac {t}{t + 1} \bar {\boldsymbol {y}} ^ {(t)} + 2 \eta \frac {t}{t + 1} \mathbf {A} ^ {\top} \bar {\boldsymbol {x}} ^ {(t)} + \eta \frac {t - 1}{t + 1} \mathbf {A} ^ {\top} \bar {\boldsymbol {x}} ^ {(t - 1)}. \tag {52} \\ \end{array}
$$
Above, we define  $\bar{\pmb{x}}^{(t)}\coloneqq \frac{1}{t}\sum_{\tau = 1}^{t}\pmb{x}^{(\tau)}$  and  $\bar{\pmb{y}}^{(t)}\coloneqq \frac{1}{t}\sum_{\tau = 1}^{t}\pmb{y}^{(\tau)}$ . That is, the update rule (52) describes the evolution of the time average of (51). As a result, this implies that (52) converges in a last-iterate sense with a  $T^{-1}$  rate, simply because OFTRL incurs  $O(1)$  regret [78]. This is conceptually interesting as it suggests a simple way to analyze the convergence of the last iterate solely through a regret-based analysis. In fact, (52) is a variant of the so-called Halpern's iteration [32]--a classical technique in optimization--that incorporates optimism; a similar update rule was recently shown to exhibit a new form of acceleration by Yoon and Ryu [90], and has engendered considerable interest ever since. Returning to Observation 3.11, one can use (52) to guarantee  $O(\log T)$  dynamic regret in the standard feedback model, although it is not clear how to extend this argument in the constrained setting (cf. [14]).
General-sum games We next extend our scope beyond bilinear saddle-point problems. We first observe that obtaining sublinear dynamic regret is precluded in general-sum games. In particular, we note that the computational-hardness result below (Proposition 3.12) holds beyond the online learning setting. It should be stressed that, at least in normal-form games, without imposing computational or memory restrictions there are trivial online algorithms that guarantee even  $O(1)$  dynamic regret by first exploring the payoff tensors and then computing a Nash equilibrium [26]. We suspect that under the memory limitations imposed by Daskalakis et al. [26] there could be unconditional information-theoretic lower bounds, but that is left for future work.
Proof. We will make use of the fact that computing a Nash equilibrium in two-player (normal-form) games to a sufficiently small accuracy  $\epsilon = O(1)$  requires superpolynomial time, unless the exponential-time hypothesis for PPAD fails [73]. Indeed, suppose that there exist polynomial-time algorithms that always guarantee that  $\sum_{i=1}^{n} \mathrm{DReg}_i^{(T)} \leq \epsilon T$  for some polynomial  $T \in \mathbb{N}$ , where  $n := 2$ . Then, this implies that there exists a time  $t \in [[T]]$  such that
$$
\max _ {\pmb {x} _ {1} ^ {(t, \star)} \in \mathcal {X} _ {1}} \langle \pmb {x} _ {1} ^ {(t, \star)}, \pmb {u} _ {1} ^ {(t)} \rangle - \langle \pmb {x} _ {1} ^ {(t)}, \pmb {u} _ {1} ^ {(t)} \rangle + \max _ {\pmb {x} _ {2} ^ {(t, \star)} \in \mathcal {X} _ {2}} \langle \pmb {x} _ {2} ^ {(t, \star)}, \pmb {u} _ {2} ^ {(t)} \rangle - \langle \pmb {x} _ {2} ^ {(t)}, \pmb {u} _ {2} ^ {(t)} \rangle \leq \epsilon ,
$$
which in turn implies that  $(\pmb{x}_1^{(t)},\pmb{x}_2^{(t)})$  is an  $\epsilon$ -approximate Nash equilibrium. Further, such a time  $t\in [[T]]$  can be identified in polynomial time since  $T\leq \mathrm{poly}(|\mathcal{A}_1|,|\mathcal{A}_2|)$ . This concludes the proof.
We also note the following computational hardness result, which rests on the more standard complexity assumption that PPAD is not contained in P. As in Proposition 3.12, below we tacitly assume that the underlying algorithm is deterministic.
Proposition A.27. Unless PPAD  $\subseteq$  P, no polynomial-time algorithm guarantees  $\sum_{i=1}^{n} \mathrm{DReg}_i^{(T)} \leq \text{poly}(|\mathcal{A}_1|, |\mathcal{A}_2|) T^{1-\omega}$  for all  $T \in \mathbb{N}$ , where  $\omega \in (0,1]$  is an absolute constant, even if  $n = 2$ .
The proof follows directly from the PPAD-hardness result of Chen et al. [19]. Of course, if we do not operate in the online learning model, it is computationally trivial to come up with algorithms that guarantee that one of the two players will have 0 dynamic regret by simply best responding to the strategy of the other player.
Finally, we provide the proof of Theorem 3.13, the detailed version of which is provided below.
Theorem A.28 (Detailed version of Theorem 3.13). Consider an  $n$ -player game such that  $\| \nabla_{\boldsymbol{x}_i} u_i(\boldsymbol{z}) - \nabla_{\boldsymbol{x}_i} u_i(\boldsymbol{z}') \|_2 \leq L \| \boldsymbol{z} - \boldsymbol{z}' \|_2$ , where  $\boldsymbol{z}, \boldsymbol{z}' \in \mathbb{X}_{i=1}^n \mathcal{X}_i$  for any player  $i \in [[n]]$ . Then, if all players employ OGDP with learning rate  $\eta > 0$  it holds that
1.  $\sum_{i=1}^{n} K\text{-}\mathrm{DReg}_i^{(T)} = O(K\sqrt{n}LD_{\mathcal{Z}}^2)$  if  $\eta = \Theta\left(\frac{1}{L\sqrt{n}}\right)$ ;
2.  $K\text{-}\mathrm{DReg}_i^{(T)} = O(K^{3 / 4}T^{1 / 4}n^{1 / 4}L^{1 / 2}D_{\mathcal{X}_i}^{3 / 2}),$  for any  $i\in \llbracket n\rrbracket$  , if  $\eta = \Theta \left(\frac{K^{1 / 4}D_{\mathcal{X}_i}^{1 / 2}}{n^{1 / 4}L^{1 / 2}T^{1 / 4}}\right)$
Proof. First, applying Lemma A.1 subject to the constraint that  $\sum_{t=1}^{T-1} \mathbb{1}\{\pmb{x}_i^{(t+1,\star)} \neq \pmb{x}_i^{(t,\star)}\} \leq K-1$  gives that for any Player  $i \in [[n]]$ ,
$$
\begin{array}{l} K - D R e g _ {i} ^ {(T)} \leq \frac {D _ {\mathcal {X} _ {i}} ^ {2}}{2 \eta} (2 K - 1) + \eta \| \boldsymbol {u} _ {i} ^ {(1)} \| _ {2} ^ {2} + \eta \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {u} _ {i} ^ {(t + 1)} - \boldsymbol {u} _ {i} ^ {(t)} \| _ {2} ^ {2} \\ - \frac {1}{4 \eta} \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {x} _ {i} ^ {(t + 1)} - \boldsymbol {x} _ {i} ^ {(t)} \| _ {2} ^ {2}. \tag {53} \\ \end{array}
$$
Further, by  $L$ -smoothness we have that
$$
\| \boldsymbol {u} _ {i} ^ {(t + 1)} - \boldsymbol {u} _ {i} ^ {(t)} \| _ {2} ^ {2} = \| \nabla_ {\boldsymbol {x} _ {i}} u _ {i} (\boldsymbol {z} ^ {(t + 1)}) - \nabla_ {\boldsymbol {x} _ {i}} u _ {i} (\boldsymbol {z} ^ {(t)}) \| _ {2} ^ {2} \leq L ^ {2} \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} ^ {(t + 1)} - \boldsymbol {x} _ {i} ^ {(t)} \| _ {2} ^ {2},
$$
for any  $t \in [[T - 1]]$ , where  $(\pmb{x}_1^{(t)}, \dots, \pmb{x}_n^{(t)}) = \pmb{z}^{(t)} \in \times_{i=1}^n \mathcal{X}_i$  is the joint strategy profile at time  $t$ . Thus, summing (53) over all  $i \in [[n]]$  and taking  $\eta \leq \frac{1}{2L\sqrt{n}}$  implies that  $\sum_{i=1}^n K\text{-DReg}_i^{(T)} \leq \frac{2K-1}{2\eta} \sum_{i=1}^n D_{\mathcal{X}_i}^2 + \eta \sum_{i=1}^n \| \pmb{u}_i^{(1)} \|_2^2$ , yielding the first part of the statement since  $D_{\mathcal{Z}}^2 = \sum_{i=1}^n D_{\mathcal{X}_i}^2$ , where we recall the notation  $\mathcal{Z} := \times_{i=1}^n \mathcal{X}_i$ . The second part follows directly from (53) using the stability property of OGD:  $\| \pmb{x}_i^{(t+1)} - \pmb{x}_i^{(t)} \|_2 = O(\eta)$ , for any time  $t \in [[T - 1]]$ .
Remark A.29. It is not hard to show that Item 2 above can be improved to  $O_{K,T}(K)$  using clairvoyant mirror descent (CMD) [68], but under a stronger feedback model. In particular, using the (squared) Euclidean regularizer it is direct to show (see [39, 65]) that the dynamic regret of CMD is bounded solely by the first-order variation of the sequence of comparators, which suffices for this claim. It is open whether similar results can be obtained in the standard feedback model.
B Experimental examples