SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ioannis Panageas
ipanagea@ics.uci.edu
Gabriele Farina
Most of the literature on learning in games has focused on the restrictive setting where the underlying repeated game does not change over time. Much less is known about the convergence of no-regret learning algorithms in dynamic multiagent settings. In this paper, we characterize the convergence of optimistic gradient descent (OGD) in time-varying games. Our framework yields sharp convergence bounds for the equilibrium gap of OGD in zero-sum games parameterized on natural variation measures of the sequence of games, subsuming known results for static games. Furthermore, we establish improved second-order variation bounds under strong convexity-concavity, as long as each game is repeated multiple times. Our results also extend to time-varying general-sum multi-player games via a bilinear formulation of correlated equilibria, which has novel implications for meta-learning and for obtaining refined variation-dependent regret bounds, addressing questions left open in prior papers. Finally, we leverage our framework to also provide new insights on dynamic regret guarantees in static games.
Most of the classical results in the literature on learning in games—exemplified by, among others, the work of Hart and Mas-Colell [49], Foster and Vohra [42], Freund and Schapire [43]—rest on the assumption that the underlying repeated game remains invariant. Yet, in many paradigmatic learning environments that is unrealistic [35, 93, 17, 59]. One such class is settings where the underlying game is actually changing, such as routing problems on the internet [58], online advertising auctions [55], and dynamic mechanism design [67, 31]. Another such class consists of settings in which many similar games need to be solved [48]. For example, one may want to solve variations of a game for the purpose of sensitivity analysis with respect to the modeling assumptions used to construct the game model.
Despite the considerable interest in such dynamic multiagent environments, much less is known about the convergence of no-regret learning algorithms in time-varying games. No-regret dynamics are natural learning algorithms that have desirable convergence properties in static settings. Also, the state-of-the-art algorithms for finding minimax equilibria in two-player zero-sum games are based on advanced forms of no-regret dynamics [37, 11]. Indeed, all the superhuman milestones in poker have used them in the equilibrium-finding module of their architectures [8, 10, 12].
In this paper, we seek to fill this knowledge gap by understanding properties of no-regret dynamics in time-varying games. In particular, we primarily investigate the convergence of optimistic gradient descent (OGD) [20, 72] in time-varying games. Unlike traditional no-regret learning algorithms, such as (online) gradient descent, OGD has been recently shown to exhibit last-iterate convergence in static (two-player) zero-sum games [27, 46, 16, 47]. For the more challenging scenario where the underlying game can vary in every round, a fundamental question arises: Under what conditions on the sequence of games does OGD approximate (with high probability) the sequence of Nash equilibria?
In this paper, we develop a framework that enables us to characterize the convergence of OGD, and generalizations thereof, in a number of fundamental time-varying multiagent settings.
Bilinear saddle-point problems First, building on the work of Zhang et al. [93], we identify natural variation measures on the sequence of games whose sublinear growth guarantees that almost all iterates of OGD under a constant learning rate are approximate Nash equilibria in time-varying (two-player) zero-sum games (Corollary 3.4). More precisely, in Theorem 3.3 we derive a sharp non-asymptotic characterization of the equilibrium gap of OGD as a function of the minimal first-order variation of approximate Nash equilibria and the second-order variation of the payoff matrices. We stress that our variation measure that depends on the deviation of approximate Nash equilibria of the games can be arbitrarily smaller than the one based on (even the least varying) sequence of exact Nash equilibria (Proposition 3.2), thereby significantly sharpening the measure considered by Zhang et al. [93]. It is also a compelling property, in light of the multiplicity of Nash equilibria, that the variation of the Nash equilibria is measured in terms of the most favorable—i.e., one that minimizes the variation—such sequence.
From a technical standpoint, our analysis revolves around a connection between the convergence of OGD in time-varying games and dynamic regret. In particular, the first key observation is that dynamic regret cannot be too negative under any sequence of approximate Nash equilibria (Property 3.1); this was first observed by Zhang et al. [93] under a sequence of exact Nash equilibria. We also generalize this property by connecting it to the admission of a minimax theorem (Property A.4), as well as the so-called MVI property in the more general context of time-varying variational inequalities (VIs); this will enable us to extend our scope to certain multi-player games such as polymatrix zero-sum [15] (Theorem A.16). By combining Property 3.1 with a dynamic RVU bound [93], we obtain a variation-dependent bound for the second-order path length of OGD under a constant learning rate in time-varying zero-sum games. In turn, this leads to our first main result, Theorem 3.3, discussed above. In the special case of static games, our theorem reduces to a tight  $T^{-1/2}$  rate. We also instantiate our results to the special setting of meta-learning, where each game is repeated multiple times.
Strongly convex-concave games Moreover, for strongly convex-concave time-varying games, we obtain a refined second-order variation bound on the sequence of Nash equilibria, as long as each game is repeated multiple times (Theorem 3.5); this is inspired by an improved second-order bound for dynamic regret under analogous conditions due to Zhang et al. [92]. As a byproduct of our techniques, we point out that any no-regret learners are approaching a Nash equilibrium under strong convexity-concavity (Proposition 3.6; cf. [85]). Those results apply even in non-strongly convex-concave settings by suitably trading off the magnitude of a regularizer that makes the game strongly convex-concave. This offers significant gains in the meta-learning setting as well.
General-sum multi-player games Next, we extend our results to time-varying general-sum multi-player games by expressing correlated equilibria via a bilinear saddle-point problem (BSPP); while this formulation goes back to the early work of Hart and Schmeidler [50], it is rarely employed in the literature on learning in games. By leveraging a structural property of the underlying BSPP, we manage to obtain convergence bounds parameterized on the variation of the correlated equilibria (Theorem 3.9). To illustrate the power of our framework, we immediately recover natural and algorithm-independent similarity measures for the meta-learning setting (Proposition A.14) even in general games (Corollary A.25), thereby addressing an open question of Harris et al. [48]. Our techniques also imply new per-player regret bounds in zero-sum and general-sum games (Corollaries A.12 and A.26), the latter addressing a question left open by Zhang et al. [93], albeit under a different learning paradigm (Section 3.3 contains a discussion on this point). We further
parameterize the convergence of (vanilla) gradient descent in time-varying potential games in terms of the deviation of the potential functions (Theorem 3.7).
Finally, building on our techniques in time-varying games, we investigate the best dynamic-regret guarantees possible in static games (see also the work of Cai and Zheng [14]). We first note that instances of optimistic mirror descent guarantee  $O(\sqrt{T})$  dynamic per-player regret (Proposition 3.10), matching the known rate of (online) gradient descent but for the significantly weaker notion of external regret. We further point out that  $O(\log T)$  dynamic regret is attainable, but in a stronger two-point feedback model. In stark contrast, even obtaining sublinear dynamic regret for each player is precluded in general-sum games (Proposition 3.12). This motivates studying a relaxation of dynamic regret that constrains the number of switches in the comparator, for which we derive accelerates rates in general games (Theorem 3.13) by leveraging the techniques of Syrgkanis et al. [78] in conjunction with the dynamic RVU bound. Interesting, this relaxation of dynamic regret gives rise to a natural refinement of coarse correlated equilibria, recently investigated by Crippa et al. [22].
Even in static (two-player) zero-sum games, the pointwise convergence of no-regret learning algorithms is a tenuous affair. Indeed, traditional learning dynamics within the no-regret framework, such as (online) mirror descent, may even diverge away from the equilibrium; e.g., see [74, 60, 83, 45]. Notwithstanding the lack of pointwise convergence, the empirical frequency of no-regret learners is well-known to approach the set of Nash equilibria in zero-sum games [43], and the set of coarse correlated equilibria in general-sum games [49, 42]—a standard relaxation of the Nash equilibrium [63, 3]. Unfortunately, those classical results are of little use beyond static games, thereby offering a crucial impetus for investigating iterate-convergence in games with a time-varying component—a ubiquitous theme in many practical scenarios of interest [31, 67, 82, 44, 81, 71, 69, 89, 70, 6]. One concrete and quite broad example worth mentioning here revolves around the steering problem [91], where a mediator dynamically modifies the utilities of the game via nonnegative and vanishing payments so as to guide no-regret learners to desirable equilibria. In particular, when the payment function varies over time, the steering problem can be naturally phrased as a time-varying zero-sum game [91].
In this context, there has been a considerable effort endeavoring to extend the scope of traditional game-theoretic results to the time-varying setting, approached from a variety of different standpoints in prior work [55, 93, 17, 58, 59, 35, 41, 79, 40, 86]. In particular, our techniques in Section 3.1 are strongly related to the ones developed by Zhang et al. [93], but our primary focus is different: Zhang et al. [93] were mainly interested in obtaining variation-dependent regret bounds, while our results revolve around iterate-convergence of OGD (under a constant learning rate) to Nash equilibria. Minimizing regret and approaching Nash equilibria are two inherently distinct problems, although connections have emerged [2, 93], and are further cultivated in this paper. Moreover, in an independent work Feng et al. [40] established a surprising separation between the behavior of optimistic GDA and the extra-gradient method in time-varying unconstrained bilinear games; it is open whether such a discrepancy persists in the constrained setting as well. We also point out the concurrent work of Yan et al. [86] that focuses on improved guarantees for strongly monotone games; the results we obtain in Section 3.2 are complementary to theirs.
Another closely related direction is on meta-learning in games [48, 77, 34], where each game can be repeated for multiple iterations. Such considerations are motivated in part by a number of use-cases in which many "similar" games—or multiple game variations—ought to be solved [9], such as Poker with different stack-sizes. While the meta-learning problem is a special case of our general setting, our results are strong enough to have new implications for meta-learning in games, even though the algorithms considered herein are not tailored to operate in that setting.
Notation We let  $\mathbb{N} := \{1, 2, \ldots, \}$  be the set of natural numbers. For a number  $p \in \mathbb{N}$ , we let  $\llbracket p \rrbracket := \{1, \ldots, p\}$ . For a vector  $\boldsymbol{w} \in \mathbb{R}^d$ , we use  $\| \boldsymbol{w} \|_2$  to represent its Euclidean norm; we also overload that notation so that  $\| \cdot \|_2$  denotes the spectral norm when the argument is a matrix.
For a two-player zero-sum game, we denote by  $\mathcal{X} \subseteq \mathbb{R}^{d_x}$  and  $\mathcal{Y} \subseteq \mathbb{R}^{d_y}$  the strategy sets of the two players—namely, Player  $x$  and Player  $y$ , respectively—where  $d_x, d_y \in \mathbb{N}$  represent the corresponding
dimensions. It is assumed that  $\mathcal{X}$  and  $\mathcal{V}$  are nonempty convex and compact sets. For example, in the special case where  $\mathcal{X} \coloneqq \Delta^{d_x}$  and  $\mathcal{Y} \coloneqq \Delta^{d_y}$  each set corresponds to a probability simplex—the game is said to be in normal form. Further, we denote by  $D_{\mathcal{X}}$  the  $\ell_2$ -diameter of  $\mathcal{X}$ , and by  $\| \mathcal{X} \|_2$  the maximum  $\ell_2$ -norm attained by a point in  $\mathcal{X}$ . We will always assume that the strategy sets remain invariant, while the payoff matrix can change in each round. For notational convenience, we will denote by  $z \coloneqq (x, y)$  the concatenation of  $x$  and  $y$ , and by  $\mathcal{Z} \coloneqq \mathcal{X} \times \mathcal{Y}$  the Cartesian product of  $\mathcal{X}$  and  $\mathcal{Y}$ . In general  $n$ -player games, we instead use subscripts indexed by  $i \in [n]$  to specify quantities related to a player. Superscripts are typically reserved to identify the time index. Finally, to simplify the exposition, we often use the  $O(\cdot)$  notation to suppress time-independent parameters of the problem.
Dynamic regret We operate in the usual online learning setting under full feedback. Namely, at every time  $t \in \mathbb{N}$  the learner decides on a strategy  $\pmb{x}^{(t)} \in \mathcal{X}$ , and then subsequently observes a utility function  $\pmb{x} \mapsto \langle \pmb{x}, \pmb{u}_x^{(t)} \rangle$ , for  $\pmb{u}_x^{(t)} \in \mathbb{R}^{d_x}$ . A strong performance benchmark in this online setting is dynamic regret, defined for a time horizon  $T \in \mathbb{N}$  as follows:
$$
\operatorname {D R e g} _ {x} ^ {(T)} \left(s _ {x} ^ {(T)}\right) := \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {x} ^ {(t, \star)} - \boldsymbol {x} ^ {(t)}, \boldsymbol {u} _ {x} ^ {(t)} \right\rangle , \tag {1}
$$
where  $s_x^{(T)} \coloneqq (\pmb{x}^{(1,\star)},\dots,\pmb{x}^{(T,\star)}) \in \mathcal{X}^T$  above is the sequence of comparators; by setting  $\pmb{x}^{(1,\star)} = \pmb{x}^{(2,\star)} = \dots = \pmb{x}^{(T,\star)}$  in (1) we recover the standard notion of (external) regret (denoted simply by  $\mathrm{Reg}_x^{(T)}$ ), which is commonly used to establish convergence of the time-average strategies in static two-player zero-sum games [43]. On the other hand, the more general notion of dynamic regret, introduced in (1), has been extensively used in more dynamic environments; e.g., [94, 92, 53, 18, 51, 54, 62]. We also define  $\mathrm{DReg}_x^{(T)} \coloneqq \max_{s_x^{(T)} \in \mathcal{X}^T} \mathrm{DReg}_x^{(T)}(s_x^{(T)})$ . While ensuring  $o(T)$  dynamic regret is clearly hopeless in a truly adversarial environment, Section 3.4 reveals that non-trivial guarantees are possible when learning in zero-sum games (see also [14]).
Optimistic gradient descent Optimistic gradient descent (OGD) [20, 72] is a no-regret algorithm defined with the following update rule:
$$
\boldsymbol {x} ^ {(t)} := \Pi_ {\mathcal {X}} \left(\hat {\boldsymbol {x}} ^ {(t)} + \eta \boldsymbol {m} _ {x} ^ {(t)}\right),
$$
$$
\hat {\boldsymbol {x}} ^ {(t + 1)} := \Pi_ {\mathcal {X}} \left(\hat {\boldsymbol {x}} ^ {(t)} + \eta \boldsymbol {u} _ {x} ^ {(t)}\right). \tag {OGD}
$$
Here,  $\eta > 0$  is the learning rate;  $\hat{\pmb{x}}^{(1)} \coloneqq \arg \min_{\hat{\pmb{x}} \in \mathcal{X}} \| \hat{\pmb{x}} \|_2^2$  represents the initialization of OGD;  $m_x^{(t)} \in \mathbb{R}^{d_x}$  is the prediction vector at time  $t$ , and it is set as  $m_x^{(t)} \coloneqq u_x^{(t-1)}$  when  $t \geq 2$ , and  $m_x^{(1)} \coloneqq 0_{d_x}$ ; and finally,  $\Pi_{\mathcal{X}}(\cdot)$  represents the Euclidean projection to the set  $\mathcal{X}$ , which is well-defined, and can be further computed efficiently for structured sets, such as the probability simplex. For our purposes, we will posit access to a projection oracle for the set  $\mathcal{X}$ , in which case the update rule (OGD) is indeed efficiently implementable.
In a multi-player  $n$ -player game, each player  $i \in [[n]]$  is associated with a utility function  $u_{i} : \bigvee_{i=1}^{n} \mathcal{X}_{i} \to \mathbb{R}$ . We recall the following central definition [64].
Definition 2.1 (Approximate Nash equilibrium). A joint strategy profile  $(\pmb{x}_1^\star, \dots, \pmb{x}_n^\star) \in \mathbb{X}_{i=1}^n \mathcal{X}_i$  is an  $\epsilon$ -approximate Nash equilibrium (NE), for an  $\epsilon \geq 0$ , if for any player  $i \in [[n]]$  and any possible deviation  $\pmb{x}_i' \in \mathcal{X}_i$  it holds that  $u_i(\pmb{x}_1^\star, \dots, \pmb{x}_i^\star, \dots, \pmb{x}_n^\star) \geq u_i(\pmb{x}_1^\star, \dots, \pmb{x}_i', \dots, \pmb{x}_n^\star) - \epsilon$ .
3 Convergence in time-varying games
In this section, we formalize our results regarding convergence in time-varying games. We organize this section as follows: First, in Section 3.1, we study the convergence of OGD in time-varying bilinear saddle-point problems (BSPPs) and beyond, culminating in the non-asymptotic characterization of Theorem 3.3; Section 3.2 formalizes our improvements under strong convexity-concavity; we then extend our scope to time-varying multi-player general-sum and potential games in Section 3.3; and finally, Section 3.4 is concerned with dynamic regret guarantees in static games.
3.1 Bilinear saddle-point problems
We first study an online learning setting wherein two players interact in a sequence of time-varying BSPPs [93]. We assume that in every repetition  $t \in [[T]]$  the players select a pair of strategies  $(\pmb{x}^{(t)},\pmb{y}^{(t)}) \in \mathcal{X} \times \mathcal{Y}$ . Then, Player  $x$  receives the utility  $\pmb{u}_x^{(t)} \coloneqq -\mathbf{A}^{(t)}\pmb{y}^{(t)} \in \mathbb{R}^{d_x}$ , where  $\mathbf{A}^{(t)} \in \mathbb{R}^{d_x \times d_y}$  represents the payoff matrix at the  $t$ -th repetition; similarly, Player  $y$  receives the utility  $\pmb{u}_y^{(t)} \coloneqq (\mathbf{A}^{(t)})^\top \pmb{x}^{(t)} \in \mathbb{R}^{d_y}$ . We also extend our scope to a more general class of time-varying problems, as we formalize in Section 3.1.1. The proofs of this subsection are in Appendix A.1.
We commence by pointing out a crucial property: by selecting a sequence of approximate Nash equilibria as the comparators, the sum of the players' dynamic regrets cannot be too negative:
Property 3.1. Suppose that  $\mathcal{Z} \ni \pmb{z}^{(t,\star)} = (\pmb{x}^{(t,\star)},\pmb{y}^{(t,\star)})$  is an  $\epsilon^{(t)}$ -approximate Nash equilibrium of the  $t$ -th game. Then, for  $s_x^{(T)} = (\pmb{x}^{(t,\star)})_{1\leq t\leq T}$  and  $s_y^{(T)} = (\pmb{y}^{(t,\star)})_{1\leq t\leq T}$ ,
$$
\mathrm {D R e g} _ {x} ^ {(T)} (s _ {x} ^ {(T)}) + \mathrm {D R e g} _ {y} ^ {(T)} (s _ {y} ^ {(T)}) \geq - 2 \sum_ {t = 1} ^ {T} \epsilon^ {(t)}.
$$
A special case of this property was previously noted by Zhang et al. [93] by considering a sequence of exact Nash equilibria; the approximate version stated above leads to a significant improvement in the convergence bounds, as we shall see in the sequel. In fact, as we note in Property A.4, Property 3.1 applies even in certain (time-varying) nonconvex-nonconcave min-max optimization problems, and it is a consequence of the minimax theorem. Property 3.1 also holds for time-varying variational inequalities (VIs) that satisfy the so-called MVI property, as we elaborate further in Section 3.1.1.
Now, building on the work of Zhang et al. [93], let us introduce some natural measures of the games' variation. The first-order variation of the Nash equilibria was defined by Zhang et al. [93] for  $T \geq 2$  as  $\mathcal{V}_{\mathrm{NE}}^{(T)} \coloneqq \inf_{\boldsymbol{z}^{(t,\star)} \in \mathcal{Z}^{(t,\star)}, \forall t \in [T]} \sum_{t=1}^{T-1} \| \boldsymbol{z}^{(t+1,\star)} - \boldsymbol{z}^{(t,\star)} \|_2$ , where  $\mathcal{Z}^{(t,\star)}$  is the (nonempty) set of Nash equilibria of the  $t$ -th game. We recall that there can be a multiplicity of Nash equilibria [80]; as such, a compelling feature of the above variation measure is that it depends on the most favorable sequence of Nash equilibria—one that minimizes the first-order variation.
It is important, however, to come to terms with the well-known fact that Nash equilibria can change abruptly even under a "small" perturbation in the payoff matrix (see Example A.5), which is an important caveat of the variation  $\mathcal{V}_{\mathrm{NE}}^{(T)}$ . To address this, and in accordance with our more general Property 3.1, we consider a more favorable variation measure, defined as
$$
\mathcal {V} _ {\epsilon - \mathrm {N E}} ^ {(T)} := \inf  \left\{\sum_ {t = 1} ^ {T - 1} \| z ^ {(t + 1, \star)} - z ^ {(t, \star)} \| _ {2} + C \sum_ {t = 1} ^ {T} \epsilon^ {(t)} \right\}, \tag {2}
$$
for a sufficiently large parameter  $C > 0$  (see (28)); the infimum above is subject to  $\epsilon^{(t)}\in \mathbb{R}_{\geq 0}$  and  $\pmb {z}^{(t,\star)}\in \mathcal{Z}_{\epsilon^{(t)}}^{(t,\star)}$  for all  $t\in [[T]]$  , where  $\mathcal{Z}_{\epsilon^{(t)}}^{(t,\star)}$  is the set of  $\epsilon^{(t)}$  -approximate NE. It is evident that  $\mathcal{V}_{\epsilon -\mathrm{NE}}^{(T)}\leq \mathcal{V}_{\mathrm{NE}}^{(T)}$  since one can take  $\epsilon^{(1)} = \dots = \epsilon^{(T)} = 0$  ; in fact,  $\mathcal{V}_{\epsilon -\mathrm{NE}}^{(T)}$  can be arbitrarily smaller:
Proposition 3.2. For any  $T \geq 4$ , there is a sequence of  $T$  games such that  $\mathcal{V}_{NE}^{(T)} \geq \frac{T}{2}$  while  $\mathcal{V}_{\epsilon - NE}^{(T)} \leq \delta$ , for any  $\delta > 0$ .
This shows that  $\mathcal{V}_{\epsilon -\mathrm{NE}}^{(T)}$  is a more compelling variation measure compared to  $\nu_{\mathrm{NE}}^{(T)}$ . It is worth noting here that Zhang et al. [93] also considered the variation measure  $\mathcal{W}_{\mathbf{A}}^{(T)}\coloneqq \sum_{t = 1}^{T}\| \mathbf{A}^{(t)} - \bar{\mathbf{A}}\| _2$ , where  $\bar{\mathbf{A}}$  is the average payoff matrix and  $\| \cdot \| _2$  denotes the spectral norm. It is in fact not hard to construct instances such that  $\mathcal{V}_{\epsilon -\mathrm{NE}}^{(T)}\ll \min \{\mathcal{V}_{\mathrm{NE}}^{(T)},\mathcal{W}_{\mathbf{A}}^{(T)}\}$  (Proposition A.6).
Moreover, we also recall another quantity that captures the variation of the payoff matrices:  $\mathcal{V}_{\mathbf{A}}^{(T)} := \sum_{t=1}^{T-1} \| \mathbf{A}^{(t+1)} - \mathbf{A}^{(t)} \|_2^2$ . Unlike  $\mathcal{V}_{\mathrm{NE}}^{(T)}$ , the variation measure  $\mathcal{V}_{\mathbf{A}}^{(T)}$  depends on the second-order variation (of the payoff matrices), which could translate to a lower-order impact compared to  $\mathcal{V}_{\mathrm{NE}}^{(T)}$ .
<sup>1</sup>Any equivalent norm in the definition of  $\mathcal{W}_{\mathbf{A}}^{(T)}$  suffices for the purpose of Proposition A.6.
(see, e.g., Corollary A.9). We stress that while our convergence bounds will be parameterized based on  $\mathcal{V}_{\epsilon -\mathrm{NE}}^{(T)}$  and  $\mathcal{V}_{\mathbf{A}}^{(T)}$ , the underlying algorithm—namely OGD—will remain oblivious to those variation measures. We only make the mild assumption that players know in advance the value of  $L\coloneqq \max_{1\leq t\leq T}\| \mathbf{A}^{(t)}\| _2$ , so that they can tune the learning rate  $\eta$  appropriately.