SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we developed a framework for characterizing iterate-convergence of no-regret learning algorithms—primarily optimistic gradient descent (OGD)—in time-varying games. There are many promising avenues for future research. Besides closing the obvious gaps we highlighted in Section 3.4, it is important to characterize the behavior of no-regret learning algorithms in other fundamental time-varying multiagent settings, such as Stackelberg (security) games [5]. Moreover, our results operate in the full-feedback model where each player receives feedback on all possible actions. Extending the scope of our framework to capture partial-feedback models as well is another interesting direction for future work.
5The same asymptotic bound was independently established by Cai and Zheng [14] but in the standard feedback model.
We are grateful to anonymous referees for providing valuable and constructive feedback. We also thank Vince Conitzer and Caspar Oesterheld for many helpful comments. This material is based on work supported by the National Science Foundation under grants IIS-1901403 and CCF-1733556 and by the ARO under award W911NF2210266.
[1] I. Anagnostides, G. Farina, C. Kroer, C. Lee, H. Luo, and T. Sandholm. Uncoupled learning dynamics with  $O(\log T)$  swap regret in multiplayer games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[2] I. Anagnostides, I. Panageas, G. Farina, and T. Sandholm. On last-iterate convergence beyond zero-sum games. In International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 536–581. PMLR, 2022.
[3] R. Aumann. Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics, 1:67-96, 1974.
[4] W. Azizian, I. Mitliagkas, S. Lacoste-Julien, and G. Gidel. A tight and unified analysis of gradient-based methods for a whole spectrum of differentiable games. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS, volume 108 of Proceedings of Machine Learning Research, pages 2863–2873. PMLR, 2020.
[5] M. Balcan, A. Blum, N. Haghtalab, and A. D. Procaccia. Commitment without regrets: Online learning in stackelberg security games. In Proceedings of the ACM Conference on Economics and Computation (EC), pages 61-78. ACM, 2015.
[6] L. Baudin, M. Scarsini, and X. Venel. Strategic behavior and no-regret learning in queueing systems. CoRR, abs/2302.03614, 2023.
[7] H. H. Bauschke, W. M. Moursi, and X. Wang. Generalized monotone operators and their averaged resolvents. Math. Program., 189(1):55-74, 2021.
[8] M. Bowling, N. Burch, M. Johanson, and O. Tammelin. Heads-up limit hold'em poker is solved. Science, 347(6218), January 2015.
[9] N. Brown and T. Sandholm. Strategy-based warm starting for regret minimization in games. In AAAI Conference on Artificial Intelligence (AAAI), 2016.
[10] N. Brown and T. Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-424, 2018.
[11] N. Brown and T. Sandholm. Solving imperfect-information games via discounted regret minimization. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
[12] N. Brown and T. Sandholm. Superhuman AI for multiplayer poker. Science, 365(6456):885-890, 2019.
[13] Y. Cai and W. Zheng. Accelerated single-call methods for constrained min-max optimization. CoRR, abs/2210.03096, 2022.
[14] Y. Cai and W. Zheng. Doubly optimal no-regret learning in monotone games. In International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 3507-3524. PMLR, 2023.
[15] Y. Cai, O. Candogan, C. Daskalakis, and C. H. Papadimitriou. Zero-sum polymatrix games: A generalization of minmax. Math. Oper. Res., 41(2):648-655, 2016.
[16] Y. Cai, A. Oikonomou, and W. Zheng. Finite-time last-iterate convergence for learning in multiplayer games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[90] T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with  $O(1 / k^2)$  rate on squared gradient norm. In International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 12098-12109. PMLR, 2021.
[91] B. H. Zhang, G. Farina, I. Anagnostides, F. Cacciamani, S. M. McAleer, A. A. Haupt, A. Celli, N. Gatti, V. Conitzer, and T. Sandholm. Steering no-regret learners to optimal equilibria. CoRR, abs/2306.05221, 2023.
[92] L. Zhang, T. Yang, J. Yi, R. Jin, and Z. Zhou. Improved dynamic regret for non-degenerate functions. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), pages 732-741, 2017.
[93] M. Zhang, P. Zhao, H. Luo, and Z. Zhou. No-regret learning in time-varying zero-sum games. In International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 26772-26808. PMLR, 2022.
[94] P. Zhao, Y. Zhang, L. Zhang, and Z. Zhou. Dynamic regret of convex and smooth functions. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[95] M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning (ICML), pages 928-936, 2003.
A Omitted proofs
In this section, we provide the proofs from the main body (Section 3).
A.1 Proofs from Section 3.1
First, we start with the proofs of the dynamic RVU bounds (Lemmas A.1 and A.2 below). Before we proceed, it will be useful to express the update rule (OGD) in the following equivalent form:
$$
\boldsymbol {x} ^ {(t)} := \underset {\boldsymbol {x} \in \mathcal {X}} {\arg \max } \left\{\Psi_ {x} ^ {(t)} (\boldsymbol {x}) := \langle \boldsymbol {x}, \boldsymbol {m} _ {x} ^ {(t)} \rangle - \frac {1}{\eta} \mathcal {B} _ {\phi_ {x}} (\boldsymbol {x} \| \hat {\boldsymbol {x}} ^ {(t)}) \right\}, \tag {5}
$$
$$
\hat {\boldsymbol {x}} ^ {(t + 1)} := \underset {\hat {\boldsymbol {x}} \in \mathcal {X}} {\arg \max } \left\{\hat {\Psi} _ {x} ^ {(t)} (\hat {\boldsymbol {x}}) := \langle \hat {\boldsymbol {x}}, \boldsymbol {u} _ {x} ^ {(t)} \rangle - \frac {1}{\eta} \mathcal {B} _ {\phi_ {x}} (\hat {\boldsymbol {x}} \| \hat {\boldsymbol {x}} ^ {(t)}) \right\}.
$$
Here,  $\mathcal{B}_{\phi_x}(\cdot \| \cdot)$  denotes the Bregman divergence induced by the (squared) Euclidean regularizer  $\phi_x:\pmb {x}\mapsto \frac{1}{2}\| \pmb {x}\| _2^2$ ; namely,  $\mathcal{B}_{\phi_x}(\pmb {x}\parallel \pmb{x}')\coloneqq \phi (\pmb {x}) - \phi (\pmb{x}') - \langle \nabla \phi (\pmb{x}'),\pmb {x} - \pmb{x}'\rangle = \frac{1}{2}\| \pmb {x} - \pmb{x}'\| _2^2$  for  $\pmb {x},\pmb{x}'\in \mathcal{X}$ . The update rule (5) for general Bregman divergences will be referred to as optimistic mirror descent (OMD).
A.1.1 Dynamic RVU bounds
The first key ingredient that we need for the proof of Theorem 3.3 is the property of regret bounded by variation in utilities (RVU), in the sense of Syrgkanis et al. [78], but with respect to dynamic regret; such a bound is established below, and is analogous to that obtained by Zhang et al. [93].
Lemma A.1 (RVU bound for dynamic regret). Consider any sequence of utilities  $(\pmb{u}_x^{(1)},\dots,\pmb{u}_x^{(T)})$  up to time  $T\in \mathbb{N}$ . The dynamic regret (1) of OGD with respect to any sequence of comparators  $(\pmb{x}^{(1,\star)},\dots,\pmb{x}^{(T,\star)})\in \mathcal{X}^T$  can be bounded by
$$
\begin{array}{l} \frac {D _ {\mathcal {X}} ^ {2}}{2 \eta} + \frac {D _ {\mathcal {X}}}{\eta} \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} + \eta \sum_ {t = 1} ^ {T} \| \boldsymbol {u} _ {x} ^ {(t)} - \boldsymbol {m} _ {x} ^ {(t)} \| _ {2} ^ {2} \\ - \frac {1}{2 \eta} \sum_ {t = 1} ^ {T} \left(\| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t)} \| _ {2} ^ {2} + \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)} \| _ {2} ^ {2}\right). \tag {6} \\ \end{array}
$$
In the special case of external regret—  $\pmb{x}^{(1,\star)} = \pmb{x}^{(2,\star)} = \dots = \pmb{x}^{(T,\star)}$  (6) recovers the bound for OGD of Syrgkanis et al. [78]. The key takeaway from Lemma A.1 is that the overhead of dynamic regret in (6) grows with the first-order variation of the sequence of comparators.
Proof of Lemma A.1. First, by  $(1 / \eta)$ -strong convexity of the function  $\Psi_x^{(t)}$  (defined in (5)) for any time  $t\in [[T]]$ , we have that
$$
\langle \boldsymbol {x} ^ {(t)}, \boldsymbol {m} _ {x} ^ {(t)} \rangle - \frac {1}{2 \eta} \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t)} \| _ {2} ^ {2} - \langle \hat {\boldsymbol {x}} ^ {(t + 1)}, \boldsymbol {m} _ {x} ^ {(t)} \rangle + \frac {1}{2 \eta} \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \hat {\boldsymbol {x}} ^ {(t)} \| _ {2} ^ {2} \geq \frac {1}{2 \eta} \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)} \| _ {2} ^ {2}, \tag {7}
$$
where we used [75, Lemma 2.8; p. 135]. Similarly, by  $(1 / \eta)$ -strong convexity of the function  $\hat{\Psi}_x^{(t)}$  (defined in (5)) for any time  $t\in [[T]]$ , we have that for any comparator  $\pmb{x}^{(t,\star)}\in \mathcal{X}$
$$
\begin{array}{l} \langle \hat {\boldsymbol {x}} ^ {(t + 1)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle - \frac {1}{2 \eta} \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \hat {\boldsymbol {x}} ^ {(t)} \| _ {2} ^ {2} - \langle \boldsymbol {x} ^ {(t, \star)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle + \frac {1}{2 \eta} \| \boldsymbol {x} ^ {(t, \star)} - \hat {\boldsymbol {x}} ^ {(t)} \| _ {2} ^ {2} \\ \geq \frac {1}{2 \eta} \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2}. \tag {8} \\ \end{array}
$$
Thus, adding (7) and (8),
$$
\begin{array}{l} \langle \boldsymbol {x} ^ {(t, \star)} - \hat {\boldsymbol {x}} ^ {(t + 1)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle + \langle \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t)}, \boldsymbol {m} _ {x} ^ {(t)} \rangle \leq \frac {1}{2 \eta} \left(\| \hat {\boldsymbol {x}} ^ {(t)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2} - \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2}\right) \\ - \frac {1}{2 \eta} \left(\| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t)} \| _ {2} ^ {2} + \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)} \| _ {2} ^ {2}\right). \tag {9} \\ \end{array}
$$
We further see that
$$
\begin{array}{l} \langle \boldsymbol {x} ^ {(t, \star)} - \boldsymbol {x} ^ {(t)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle = \langle \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)}, \boldsymbol {m} _ {x} ^ {(t)} - \boldsymbol {u} _ {x} ^ {(t)} \rangle + \langle \boldsymbol {x} ^ {(t, \star)} - \hat {\boldsymbol {x}} ^ {(t + 1)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle \\ + \langle \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t)}, \boldsymbol {m} _ {x} ^ {(t)} \rangle . \tag {10} \\ \end{array}
$$
Now the first term on the right-hand side can be upper bounded using the fact that, by (7) and (8),
$$
\langle \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)}, \boldsymbol {m} _ {x} ^ {(t)} - \boldsymbol {u} _ {x} ^ {(t)} \rangle \geq \frac {1}{\eta} \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t)} \| _ {2} ^ {2} \Rightarrow \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t)} \| _ {2} \leq \eta \| \boldsymbol {m} _ {x} ^ {(t)} - \boldsymbol {u} _ {x} ^ {(t)} \| _ {2},
$$
by Cauchy-Schwarz, in turn implying that  $\langle \pmb{x}^{(t)} - \hat{\pmb{x}}^{(t + 1)},\pmb{m}_x^{(t)} - \pmb{u}_x^{(t)}\rangle \leq \eta \| \pmb{m}_x^{(t)} - \pmb{u}_x^{(t)}\| _2^2$ . Thus, the proof follows by combining this bound with (9) and (10), along with the fact that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\| \hat {\boldsymbol {x}} ^ {(t)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2} - \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2}\right) \leq \| \hat {\boldsymbol {x}} ^ {(1)} - \boldsymbol {x} ^ {(1, \star)} \| _ {2} ^ {2} \\ + \sum_ {t = 1} ^ {T - 1} \left(\| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, \star)} \| _ {2} ^ {2} - \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2}\right) \\ \leq D _ {\mathcal {X}} ^ {2} + 2 D _ {\mathcal {X}} \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2}, \\ \end{array}
$$
where the last bound follows since
$$
\begin{array}{l} \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, \star)} \| _ {2} ^ {2} - \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} ^ {2} \leq 2 D _ {\mathcal {X}} \left| \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, \star)} \| _ {2} - \| \hat {\boldsymbol {x}} ^ {(t + 1)} - \boldsymbol {x} ^ {(t, \star)} \| _ {2} \right| \\ \leq 2 D _ {\mathcal {X}} \left\| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \right\| _ {2}, \\ \end{array}
$$
where we recall that  $D_{\mathcal{X}}$  denotes the  $\ell_2$ -diameter of  $\mathcal{X}$ .
As an aside, we remark that assuming that  $\pmb{m}_x^{(t)} \coloneqq \mathbf{0}$  and  $\| \pmb{u}_x^{(t)} \|_2 \leq 1$  for any  $t \in [T]$ , Lemma A.1 implies that dynamic regret can be upper bounded by  $O\left(\sqrt{(1 + \sum_{t=1}^{T-1} \| \pmb{x}^{(t+1,\star)} - \pmb{x}^{(t,\star)} \|_2 T}\right)$ , for any (bounded)—potentially adversarially selected—sequence of utilities  $(\pmb{u}_x^{(1)}, \dots, \pmb{u}_x^{(T)})$ , for  $\eta \coloneqq \sqrt{\frac{D_x^2}{2T} + \frac{D_x \sum_{t=1}^{T-1} \| \pmb{x}^{(t+1,\star)} - \pmb{x}^{(t,\star)} \|_2}{T}}$ , which is a well-known result in online optimization [95]; while that requires setting the learning rate based on the first-order variation of the (optimal) comparators, there are standard techniques that would allow bypassing that assumption.
Next, we provide an extension of Lemma A.1 to the more general OMD algorithm under a broad class of regularizers.
Lemma A.2 (Extension of Lemma A.1 beyond Euclidean regularization). Consider a 1-strongly convex continuously differentiable regularizer  $\phi$  with respect to a norm  $\|\cdot\|$  such that (i)  $\|\nabla \phi(\pmb{x})\|_* \leq G$  for any  $\pmb{x}$ , and (ii)  $\mathcal{B}_{\phi_x}(\pmb{x} \parallel \pmb{x}') \leq L\|\pmb{x} - \pmb{x}'\|$  for any  $\pmb{x}, \pmb{x}'$ . Then, for any sequence of utilities  $(\pmb{u}_x^{(1)}, \ldots, \pmb{u}_x^{(T)})$  up to time  $T \in \mathbb{N}$  the dynamic regret (1) of  $\mathcal{OMD}$  with respect to any sequence of comparators  $(\pmb{x}^{(1,\star)}, \ldots, \pmb{x}^{(T,\star)}) \in \mathcal{X}^T$  can be bounded as
$$
\begin{array}{l} \frac {\mathcal {B} _ {\phi_ {x}} \left(\boldsymbol {x} ^ {(1 , \star)} \| \hat {\boldsymbol {x}} ^ {(1)}\right)}{\eta} + \frac {L + 2 G}{\eta} \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \| + \eta \sum_ {t = 1} ^ {T} \| \boldsymbol {u} _ {x} ^ {(t)} - \boldsymbol {m} _ {x} ^ {(t)} \| _ {*} ^ {2} \\ - \frac {1}{2 \eta} \sum_ {t = 1} ^ {T} \left(\| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t)} \| ^ {2} + \| \boldsymbol {x} ^ {(t)} - \hat {\boldsymbol {x}} ^ {(t + 1)} \| ^ {2}\right). \\ \end{array}
$$
The proof is analogous to that of Lemma A.1, and relies on the well-known three-point identity for the Bregman divergence:
$$
\mathcal {B} _ {\phi_ {x}} \left(\boldsymbol {x} \| \boldsymbol {x} ^ {\prime}\right) = \mathcal {B} _ {\phi_ {x}} \left(\boldsymbol {x} \| \boldsymbol {x} ^ {\prime \prime}\right) + \mathcal {B} _ {\phi_ {x}} \left(\boldsymbol {x} ^ {\prime \prime} \| \boldsymbol {x} ^ {\prime}\right) - \langle \boldsymbol {x} - \boldsymbol {x} ^ {\prime \prime}, \nabla \phi \left(\boldsymbol {x} ^ {\prime}\right) - \nabla \phi \left(\boldsymbol {x} ^ {\prime \prime}\right) \rangle . \tag {11}
$$
In particular, along with the assumptions of Lemma A.2 imposed on the regularizer  $\phi_x$ , (11) implies that the term  $\sum_{t=1}^{T-1}\left(\mathcal{B}_{\phi_x}(\boldsymbol{x}^{(t+1,\star)} \parallel \hat{\boldsymbol{x}}^{(t+1)}) - \mathcal{B}_{\phi_x}(\boldsymbol{x}^{(t,\star)} \parallel \hat{\boldsymbol{x}}^{(t+1)})\right)$  is equal to
$$
\begin{array}{l} \sum_ {t = 1} ^ {T - 1} \left(\mathcal {B} _ {\phi_ {x}} (\boldsymbol {x} ^ {(t + 1, \star)} \| \boldsymbol {x} ^ {(t, \star)}) - \langle \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)}, \nabla \phi (\hat {\boldsymbol {x}} ^ {(t + 1)}) - \nabla \phi (\boldsymbol {x} ^ {(t, \star)}) \rangle\right) \\ \leq (L + 2 G) \sum_ {t = 1} ^ {T - 1} \| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \|, \\ \end{array}
$$
since  $\mathcal{B}_{\phi_x}(\pmb{x}^{(t + 1,\star)}\parallel \pmb{x}^{(t,\star)})\leq L\| \pmb{x}^{(t + 1,\star)} - \pmb{x}^{(t,\star)}\|$  (by assumption) and
$$
\begin{array}{l} \langle \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)}, \nabla \phi (\hat {\boldsymbol {x}} ^ {(t + 1)}) - \nabla \phi (\boldsymbol {x} ^ {(t, \star)}) \rangle \\ \leq \left\| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \right\| \left\| \nabla \phi \left(\hat {\boldsymbol {x}} ^ {(t + 1)}\right) - \nabla \phi \left(\boldsymbol {x} ^ {(t, \star)}\right) \right\| _ {*} (12) \\ \leq \left(\| \nabla \phi (\hat {\boldsymbol {x}} ^ {(t + 1)}) \| _ {*} + \| \nabla \phi (\boldsymbol {x} ^ {(t, \star)}) \| _ {*}\right) \| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \| (13) \\ \leq 2 G \| \boldsymbol {x} ^ {(t + 1, \star)} - \boldsymbol {x} ^ {(t, \star)} \|, (14) \\ \end{array}
$$
where (12) follows from the Cauchy-Schwarz inequality; (13) uses the triangle inequality for the dual norm  $\| \cdot \|_{*}$ ; and (14) follows from the assumption of Lemma A.2 that  $\| \nabla \phi (\cdot)\|_{*}\leq G$ . The rest of the proof of Lemma A.2 is analogous to Lemma A.1, and it is therefore omitted. One important question here is whether Lemma A.2 can be extended under a broader class of regularizers; as we explain in Section 3.4, this is related to improving Theorem 3.13.
A.1.2 Nonnegativity of dynamic regret
We next proceed with the proof of Property 3.1. For completeness, we first prove the following special case [93]; the proof of Property 3.1 is then analogous.
Property A.3 (Special case of Property 3.1). Suppose that  $\mathcal{Z} \ni z^{(t,\star)} = (\pmb{x}^{(t,\star)},\pmb{y}^{(t,\star)})$  is a Nash equilibrium of the  $t$ -th game, for any time  $t \in [[T]]$ . Then, for  $s_x^{(T)} = (\pmb{x}^{(t,\star)})_{1 \leq t \leq T}$  and  $s_y^{(T)} = (\pmb{y}^{(t,\star)})_{1 \leq t \leq T}$ ,
$$
\mathrm {D R e g} _ {x} ^ {(T)} (s _ {x} ^ {(T)}) + \mathrm {D R e g} _ {y} ^ {(T)} (s _ {y} ^ {(T)}) \geq 0.
$$
Proof. Let  $v^{(t)} \coloneqq \langle \pmb{x}^{(t,\star)}, \mathbf{A}^{(t)}\pmb{y}^{(t,\star)} \rangle$  be the value of the  $t$ -th game, for some  $t \in [T]$ . Then, we have that  $v^{(t)} = \langle \pmb{x}^{(t,\star)}, \mathbf{A}^{(t)}\pmb{y}^{(t,\star)} \rangle \leq \langle \pmb{x}, \mathbf{A}^{(t)}\pmb{y}^{(t,\star)} \rangle$  for any  $\pmb{x} \in \mathcal{X}$ , since  $\pmb{x}^{(t,\star)}$  is a best response to  $\pmb{y}^{(t,\star)}$ ; similarly,  $v^{(t)} = \langle \pmb{x}^{(t,\star)}, \mathbf{A}^{(t)}\pmb{y}^{(t,\star)} \rangle \geq \langle \pmb{x}^{(t,\star)}, \mathbf{A}^{(t)}\pmb{y} \rangle$  for any  $\pmb{y} \in \mathcal{Y}$ . Hence,  $\langle \pmb{x}^{(t)}, \mathbf{A}^{(t)}\pmb{y}^{(t,\star)} \rangle - \langle \pmb{x}^{(t,\star)}, \mathbf{A}^{(t)}\pmb{y}^{(t)} \rangle \geq 0$ , or equivalently,  $\langle \pmb{x}^{(t,\star)}, \pmb{u}_x^{(t)} \rangle + \langle \pmb{y}^{(t,\star)}, \pmb{u}_y^{(t)} \rangle \geq 0$ . But given that the game is zero-sum, it holds that  $\langle \pmb{x}^{(t)}, \pmb{u}_x^{(t)} \rangle + \langle \pmb{y}^{(t)}, \pmb{u}_y^{(t)} \rangle = 0$ , so the last inequality can be in turn cast as
$$
\langle \boldsymbol {x} ^ {(t, \star)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle - \langle \boldsymbol {x} ^ {(t)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle + \langle \boldsymbol {y} ^ {(t, \star)}, \boldsymbol {u} _ {y} ^ {(t)} \rangle - \langle \boldsymbol {y} ^ {(t)}, \boldsymbol {u} _ {y} ^ {(t)} \rangle \geq 0,
$$
for any  $t \in [[T]]$ . As a result, summing over all  $t \in [[T]]$  we have shown that
$$
\begin{array}{l} \mathrm {D R e g} _ {x} ^ {(T)} \left(\boldsymbol {x} ^ {(1, \star)}, \dots , \boldsymbol {x} ^ {(T, \star)}\right) + \mathrm {D R e g} _ {y} ^ {(T)} \left(\boldsymbol {y} ^ {(1, \star)}, \dots , \boldsymbol {y} ^ {(T, \star)}\right) \\ = \sum_ {t = 1} ^ {T} \langle \boldsymbol {x} ^ {(t, \star)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle - \langle \boldsymbol {x} ^ {(t)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle + \langle \boldsymbol {y} ^ {(t, \star)}, \boldsymbol {u} _ {y} ^ {(t)} \rangle - \langle \boldsymbol {y} ^ {(t)}, \boldsymbol {u} _ {y} ^ {(t)} \rangle \geq 0. \\ \end{array}
$$
$$
\mathrm {D R e g} _ {x} ^ {(T)} \left(s _ {x} ^ {(T)}\right) + \mathrm {D R e g} _ {y} ^ {(T)} \left(s _ {y} ^ {(T)}\right) \geq - 2 \sum_ {t = 1} ^ {T} \epsilon^ {(t)}.
$$
Proof. Given that  $(\pmb{x}^{(t,\star)},\pmb{y}^{(t,\star)})\in \mathcal{Z}$  is an  $\epsilon^{(t)}$ -approximate Nash equilibrium of the  $t$ -th game, it follows that  $\langle \pmb{x}^{(t,\star)},\mathbf{A}^{(t)}\pmb{y}^{(t,\star)}\rangle \leq \langle \pmb{x}^{(t)},\mathbf{A}^{(t)}\pmb{y}^{(t,\star)}\rangle +\epsilon_x^{(t)}$  and  $\langle \pmb{x}^{(t,\star)},\mathbf{A}^{(t)}\pmb{y}^{(t,\star)}\rangle \geq \langle \pmb{x}^{(t,\star)},\mathbf{A}^{(t)}\pmb{y}^{(t)}\rangle -\epsilon_y^{(t)}$ , for some  $\epsilon_x^{(t)},\epsilon_y^{(t)}\leq \epsilon_t^{(t)}$ . Thus, we have that  $\langle \pmb{x}^{(t)},\mathbf{A}^{(t)}\pmb{y}^{(t,\star)}\rangle \geq \langle \pmb{x}^{(t,\star)},\mathbf{A}^{(t)}\pmb{y}^{(t)}\rangle -\epsilon_x^{(t)} - \epsilon_y^{(t)}$ , or equivalently,  $\langle \pmb{x}^{(t,\star)},\pmb{u}_x^{(t)}\rangle +\langle \pmb{y}^{(t,\star)},\pmb{u}_y^{(t)}\rangle \geq -\epsilon_x^{(t)} - \epsilon_y^{(t)}\geq -2\epsilon_t^{(t)}$ . As a result,
$$
\langle \boldsymbol {x} ^ {(t, \star)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle - \langle \boldsymbol {x} ^ {(t)}, \boldsymbol {u} _ {x} ^ {(t)} \rangle + \langle \boldsymbol {y} ^ {(t, \star)}, \boldsymbol {u} _ {y} ^ {(t)} \rangle - \langle \boldsymbol {y} ^ {(t)}, \boldsymbol {u} _ {y} ^ {(t)} \rangle \geq - 2 \epsilon^ {(t)}, \tag {15}
$$
for any  $t \in [[T]]$ , and the statement follows by summing (15) over all  $t \in [[T]]$ .
In fact, as we show below (in Property A.4), Property A.3 is a more general consequence of the minimax theorem. In particular, for a nonlinear online learning problem, we define dynamic regret with respect to a sequence of comparators  $(\pmb{x}^{(1,\star)},\dots ,\pmb{x}^{(T,\star)})\in \mathcal{X}^T$  as follows:
$$
\operatorname {D R e g} _ {x} ^ {(T)} \left(\boldsymbol {x} ^ {(1, \star)}, \dots , \boldsymbol {x} ^ {(T, \star)}\right) := \sum_ {t = 1} ^ {T} \left(u _ {x} ^ {(t)} \left(\boldsymbol {x} ^ {(t, \star)}\right) - u _ {x} ^ {(t)} \left(\boldsymbol {x} ^ {(t)}\right)\right), \tag {16}
$$
where  $u_x^{(1)}, \ldots, u_x^{(T)} : x \mapsto \mathbb{R}$  are the continuous utility functions observed by the learner, which could be in general nonconcave, and  $(\pmb{x}^{(t)})_{1 \leq t \leq T}$  is the sequence of strategies produced by the learner; (16) generalizes the notion of dynamic regret (1) in online linear optimization, that is, when  $u_x^{(t)} : x \mapsto \langle x, \pmb{u}_x^{(t)} \rangle$ , where  $\pmb{u}_x^{(t)} \in \mathbb{R}^{d_x}$  for any time  $t \in [T]$ .
Property A.4. Suppose that  $f^{(t)}: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  is a continuous function such that for any  $t \in [[T]]$ ,
$$
\min  _ {\boldsymbol {x} \in \mathcal {X}} \max  _ {\boldsymbol {y} \in \mathcal {Y}} f ^ {(t)} (\boldsymbol {x}, \boldsymbol {y}) = \max  _ {\boldsymbol {y} \in \mathcal {Y}} \min  _ {\boldsymbol {x} \in \mathcal {X}} f ^ {(t)} (\boldsymbol {x}, \boldsymbol {y}).
$$