Title: On the Convergence of No-Regret Learning Dynamics in Time-Varying Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: no-regret learning

optimistic gradient descent

time-varying games

dynamic regret
Abstract: Most of the literature on learning in games has focused on the restrictive setting where the underlying repeated game does not change over time. Much less is known about the convergence of no-regret learning algorithms in dynamic multiagent settings. In this paper, we characterize the convergence of optimistic gradient descent (OGD) in time-varying games. Our framework yields sharp convergence bounds for the equilibrium gap of OGD in zero-sum games parameterized on natural variation measures of the sequence of games, subsuming known results for static games. Furthermore, we establish improved second-order variation bounds under strong convexity-concavity, as long as each game is repeated multiple times. Our results also apply to time-varying general-sum multi-player games via a bilinear formulation of correlated equilibria, which has novel implications for meta-learning and for obtaining refined variation-dependent regret bounds, addressing questions left open in prior papers. Finally, we leverage our framework to also provide new insights on dynamic regret guarantees in static games.

On the Convergence of No-Regret Learning Dynamics in Time-Varying Games

Abstract

Most of the literature on learning in games has focused on the restrictive setting where the underlying repeated game does not change over time. Much less is known about the convergence of no-regret learning algorithms in dynamic multiagent settings. In this paper, we characterize the convergence of optimistic gradient descent ( OGD ) in time-varying games. Our framework yields sharp convergence bounds for the equilibrium gap of OGD in zero-sum games parameterized on natural variation measures of the sequence of games, subsuming known results for static games. Furthermore, we establish improved second-order variation bounds under strong convexity-concavity, as long as each game is repeated multiple times. Our results also extend to time-varying general-sum multi-player games via a bilinear formulation of correlated equilibria, which has novel implications for meta-learning and for obtaining refined variation-dependent regret bounds, addressing questions left open in prior papers. Finally, we leverage our framework to also provide new insights on dynamic regret guarantees in static games.

1 Introduction

Most of the classical results in the literature on learning in games-exemplified by, among others, the work of Hart and Mas-Colell [49], Foster and Vohra [42], Freund and Schapire [43]-rest on the assumption that the underlying repeated game remains invariant. Yet, in many paradigmatic learning environments that is unrealistic [35, 93, 17, 59]. One such class is settings where the underlying game is actually changing, such as routing problems on the internet [58], online advertising auctions [55], and dynamic mechanism design [67, 31]. Another such class consists of settings in which many similar games need to be solved [48]. For example, one may want to solve variations of a game for the purpose of sensitivity analysis with respect to the modeling assumptions used to construct the game model.

Despite the considerable interest in such dynamic multiagent environments, much less is known about the convergence of no-regret learning algorithms in time-varying games. No-regret dynamics are natural learning algorithms that have desirable convergence properties in static settings. Also, the state-of-the-art algorithms for finding minimax equilibria in two-player zero-sum games are based on advanced forms of no-regret dynamics [37, 11]. Indeed, all the superhuman milestones in poker have used them in the equilibrium-finding module of their architectures [8, 10, 12].

In this paper, we seek to fill this knowledge gap by understanding properties of no-regret dynamics in time-varying games. In particular, we primarily investigate the convergence of optimistic gradient descent ( OGD ) [20, 72] in time-varying games. Unlike traditional no-regret learning algorithms, such as (online) gradient descent, OGD has been recently shown to exhibit last-iterate convergence in static (two-player) zero-sum games [27, 46, 16, 47]. For the more challenging scenario where the underlying game can vary in every round, a fundamental question arises: Under what conditions on the sequence of games does OGD approximate (with high probability) the sequence of Nash equilibria?

1.1 Our results

In this paper, we develop a framework that enables us to characterize the convergence of OGD , and generalizations thereof, in a number of fundamental time-varying multiagent settings.

Bilinear saddle-point problems First, building on the work of Zhang et al. [93], we identify natural variation measures on the sequence of games whose sublinear growth guarantees that almost all iterates of OGD under a constant learning rate are approximate Nash equilibria in time-varying (two-player) zero-sum games (Corollary 3.4). More precisely, in Theorem 3.3 we derive a sharp non-asymptotic characterization of the equilibrium gap of OGD as a function of the minimal first-order variation of approximate Nash equilibria and the second-order variation of the payoff matrices. We stress that our variation measure that depends on the deviation of approximate Nash equilibria of the games can be arbitrarily smaller than the one based on (even the least varying) sequence of exact Nash equilibria (Proposition 3.2), thereby significantly sharpening the measure considered by Zhang et al. [93]. It is also a compelling property, in light of the multiplicity of Nash equilibria, that the variation of the Nash equilibria is measured in terms of the most favorablei.e. , one that minimizes the variation-such sequence.

From a technical standpoint, our analysis revolves around a connection between the convergence of OGD in time-varying games and dynamic regret . In particular, the first key observation is that dynamic regret cannot be too negative under any sequence of approximate Nash equilibria (Property 3.1); this was first observed by Zhang et al. [93] under a sequence of exact Nash equilibria. We also generalize this property by connecting it to the admission of a minimax theorem (Property A.4), as well as the so-called MVI property in the more general context of time-varying variational inequalities (VIs); this will enable us to extend our scope to certain multi-player games such as polymatrix zerosum [15] (Theorem A.16). By combining Property 3.1 with a dynamic RVU bound [93], we obtain a variation-dependent bound for the second-order path length of OGD under a constant learning rate in time-varying zero-sum games. In turn, this leads to our first main result, Theorem 3.3, discussed above. In the special case of static games, our theorem reduces to a tight T -1 / 2 rate. We also instantiate our results to the special setting of meta-learning , where each game is repeated multiple times.

Strongly convex-concave games Moreover, for strongly convex-concave time-varying games, we obtain a refined second-order variation bound on the sequence of Nash equilibria, as long as each game is repeated multiple times (Theorem 3.5); this is inspired by an improved second-order bound for dynamic regret under analogous conditions due to Zhang et al. [92]. As a byproduct of our techniques, we point out that any no-regret learners are approaching a Nash equilibrium under strong convexity-concavity (Proposition 3.6; cf . [85]). Those results apply even in non-strongly convex-concave settings by suitably trading off the magnitude of a regularizer that makes the game strongly convex-concave. This offers significant gains in the meta-learning setting as well.

General-sum multi-player games Next, we extend our results to time-varying general-sum multi-player games by expressing correlated equilibria via a bilinear saddle-point problem (BSPP); while this formulation goes back to the early work of Hart and Schmeidler [50], it is rarely employed in the literature on learning in games. By leveraging a structural property of the underlying BSPP, we manage to obtain convergence bounds parameterized on the variation of the correlated equilibria (Theorem 3.9). To illustrate the power of our framework, we immediately recover natural and algorithm-independent similarity measures for the meta-learning setting (Proposition A.14) even in general games (Corollary A.25), thereby addressing an open question of Harris et al. [48]. Our techniques also imply new per-player regret bounds in zero-sum and general-sum games (Corollaries A.12 and A.26), the latter addressing a question left open by Zhang et al. [93], albeit under a different learning paradigm (Section 3.3 contains a discussion on this point). We further

parameterize the convergence of (vanilla) gradient descent in time-varying potential games in terms of the deviation of the potential functions (Theorem 3.7).

Finally, building on our techniques in time-varying games, we investigate the best dynamic-regret guarantees possible in static games (see also the work of Cai and Zheng [14]). We first note that instances of optimistic mirror descent guarantee O ( √ T ) dynamic per-player regret (Proposition 3.10), matching the known rate of (online) gradient descent but for the significantly weaker notion of external regret. We further point out that O (log T ) dynamic regret is attainable, but in a stronger two-point feedback model. In stark contrast, even obtaining sublinear dynamic regret for each player is precluded in general-sum games (Proposition 3.12). This motivates studying a relaxation of dynamic regret that constrains the number of switches in the comparator, for which we derive accelerates rates in general games (Theorem 3.13) by leveraging the techniques of Syrgkanis et al. [78] in conjunction with the dynamic RVU bound. Interesting, this relaxation of dynamic regret gives rise to a natural refinement of coarse correlated equilibria, recently investigated by Crippa et al. [22].

1.2 Further related work

Even in static (two-player) zero-sum games, the pointwise convergence of no-regret learning algorithms is a tenuous affair. Indeed, traditional learning dynamics within the no-regret framework, such as (online) mirror descent, may even diverge away from the equilibrium; e.g. , see [74, 60, 83, 45]. Notwithstanding the lack of pointwise convergence, the empirical frequency of no-regret learners is well-known to approach the set of Nash equilibria in zero-sum games [43], and the set of coarse correlated equilibria in general-sum games [49, 42]-a standard relaxation of the Nash equilibrium [63, 3]. Unfortunately, those classical results are of little use beyond static games, thereby offering a crucial impetus for investigating iterate-convergence in games with a time-varying component-a ubiquitous theme in many practical scenarios of interest [31, 67, 82, 44, 81, 71, 69, 89, 70, 6]. One concrete and quite broad example worth mentioning here revolves around the steering problem [91], where a mediator dynamically modifies the utilities of the game via nonnegative and vanishing payments so as to guide no-regret learners to desirable equilibria. In particular, when the payment function varies over time, the steering problem can be naturally phrased as a time-varying zero-sum game [91].

In this context, there has been a considerable effort endeavoring to extend the scope of traditional game-theoretic results to the time-varying setting, approached from a variety of different standpoints in prior work [55, 93, 17, 58, 59, 35, 41, 79, 40, 86]. In particular, our techniques in Section 3.1 are strongly related to the ones developed by Zhang et al. [93], but our primary focus is different: Zhang et al. [93] were mainly interested in obtaining variation-dependent regret bounds, while our results revolve around iterate-convergence of OGD (under a constant learning rate) to Nash equilibria. Minimizing regret and approaching Nash equilibria are two inherently distinct problems, although connections have emerged [2, 93], and are further cultivated in this paper. Moreover, in an independent work Feng et al. [40] established a surprising separation between the behavior of optimistic GDA and the extra-gradient method in time-varying unconstrained bilinear games; it is open whether such a discrepancy persists in the constrained setting as well. We also point out the concurrent work of Yan et al. [86] that focuses on improved guarantees for strongly monotone games; the results we obtain in Section 3.2 are complementary to theirs.

Another closely related direction is on meta-learning in games [48, 77, 34], where each game can be repeated for multiple iterations. Such considerations are motivated in part by a number of use-cases in which many 'similar' games-or multiple game variations-ought to be solved [9], such as Poker with different stack-sizes. While the meta-learning problem is a special case of our general setting, our results are strong enough to have new implications for meta-learning in games, even though the algorithms considered herein are not tailored to operate in that setting.

2 Preliminaries

Notation We let N := { 1 , 2 , . . . , } be the set of natural numbers. For a number p ∈ N , we let [ [ p ] ] := { 1 , . . . , p } . For a vector w ∈ R d , we use ∥ w ∥ 2 to represent its Euclidean norm; we also overload that notation so that ∥ · ∥ 2 denotes the spectral norm when the argument is a matrix.

For a two-player zero-sum game, we denote by X ⊆ R d x and Y ⊆ R d y the strategy sets of the two players-namely, Player x and Player y , respectively-where d x , d y ∈ N represent the corresponding

dimensions. It is assumed that X and Y are nonempty convex and compact sets. For example, in the special case where X := ∆ d x and Y := ∆ d y -each set corresponds to a probability simplex-the game is said to be in normal form . Further, we denote by D X the ℓ 2 -diameter of X , and by ∥X∥ 2 the maximum ℓ 2 -norm attained by a point in X . We will always assume that the strategy sets remain invariant, while the payoff matrix can change in each round. For notational convenience, we will denote by z := ( x , y ) the concatenation of x and y , and by Z := X × Y the Cartesian product of X and Y . In general n -player games, we instead use subscripts indexed by i ∈ [ [ n ] ] to specify quantities related to a player. Superscripts are typically reserved to identify the time index. Finally, to simplify the exposition, we often use the O ( · ) notation to suppress time-independent parameters of the problem.

Dynamic regret We operate in the usual online learning setting under full feedback. Namely, at every time t ∈ N the learner decides on a strategy x ( t ) ∈ X , and then subsequently observes a utility function x ↦→⟨ x , u ( t ) x ⟩ , for u ( t ) x ∈ R d x . A strong performance benchmark in this online setting is dynamic regret , defined for a time horizon T ∈ N as follows:

where s ( T ) x := ( x (1 ,⋆ ) , . . . , x ( T,⋆ ) ) ∈ X T above is the sequence of comparators ; by setting x (1 ,⋆ ) = x (2 ,⋆ ) = · · · = x ( T,⋆ ) in (1) we recover the standard notion of (external) regret (denoted simply by Reg ( T ) x ), which is commonly used to establish convergence of the time-average strategies in static two-player zero-sum games [43]. On the other hand, the more general notion of dynamic regret, introduced in (1), has been extensively used in more dynamic environments; e.g. , [94, 92, 53, 18, 51, 54, 62]. We also define DReg ( T ) x := max s ( T ) x ∈X T DReg ( T ) x ( s ( T ) x ) . While ensuring o ( T ) dynamic regret is clearly hopeless in a truly adversarial environment, Section 3.4 reveals that non-trivial guarantees are possible when learning in zero-sum games (see also [14]).

Optimistic gradient descent Optimistic gradient descent ( OGD ) [20, 72] is a no-regret algorithm defined with the following update rule:

Here, η > 0 is the learning rate ; ˆ x (1) := arg min ˆ x ∈X ∥ ˆ x ∥ 2 2 represents the initialization of OGD ; m ( t ) x ∈ R d x is the prediction vector at time t , and it is set as m ( t ) x := u ( t -1) x when t ≥ 2 , and m (1) x := 0 d x ; and finally, Π X ( · ) represents the Euclidean projection to the set X , which is well-defined, and can be further computed efficiently for structured sets, such as the probability simplex. For our purposes, we will posit access to a projection oracle for the set X , in which case the update rule ( OGD ) is indeed efficiently implementable.

In a multi-player n -player game, each player i ∈ [ [ n ] ] is associated with a utility function u i : × n i =1 X i → R . We recall the following central definition [64].

Definition 2.1 (Approximate Nash equilibrium) . A joint strategy profile ( x ⋆ 1 , . . . , x ⋆ n ) ∈ × n i =1 X i is an ϵ -approximate Nash equilibrium (NE), for an ϵ ≥ 0 , if for any player i ∈ [ [ n ] ] and any possible deviation x ′ i ∈ X i it holds that u i ( x ⋆ 1 , . . . , x ⋆ i , . . . , x ⋆ n ) ≥ u i ( x ⋆ 1 , . . . , x ′ i , . . . , x ⋆ n ) -ϵ .

3 Convergence in time-varying games

In this section, we formalize our results regarding convergence in time-varying games. We organize this section as follows: First, in Section 3.1, we study the convergence of OGD in time-varying bilinear saddle-point problems (BSPPs) and beyond, culminating in the non-asymptotic characterization of Theorem 3.3; Section 3.2 formalizes our improvements under strong convexity-concavity; we then extend our scope to time-varying multi-player general-sum and potential games in Section 3.3; and finally, Section 3.4 is concerned with dynamic regret guarantees in static games.

3.1 Bilinear saddle-point problems

We first study an online learning setting wherein two players interact in a sequence of time-varying BSPPs [93]. We assume that in every repetition t ∈ [ [ T ] ] the players select a pair of strategies ( x ( t ) , y ( t ) ) ∈ X × Y . Then, Player x receives the utility u ( t ) x := -A ( t ) y ( t ) ∈ R d x , where A ( t ) ∈ R d x × d y represents the payoff matrix at the t -th repetition; similarly, Player y receives the utility u ( t ) y := ( A ( t ) ) ⊤ x ( t ) ∈ R d y . We also extend our scope to a more general class of time-varying problems, as we formalize in Section 3.1.1. The proofs of this subsection are in Appendix A.1.

We commence by pointing out a crucial property: by selecting a sequence of approximate Nash equilibria as the comparators, the sum of the players' dynamic regrets cannot be too negative :

Property 3.1. Suppose that Z ∋ z ( t,⋆ ) = ( x ( t,⋆ ) , y ( t,⋆ ) ) is an ϵ ( t ) -approximate Nash equilibrium of the t -th game. Then, for s ( T ) x = ( x ( t,⋆ ) ) 1 ≤ t ≤ T and s ( T ) y = ( y ( t,⋆ ) ) 1 ≤ t ≤ T ,

Aspecial case of this property was previously noted by Zhang et al. [93] by considering a sequence of exact Nash equilibria; the approximate version stated above leads to a significant improvement in the convergence bounds, as we shall see in the sequel. In fact, as we note in Property A.4, Property 3.1 applies even in certain (time-varying) nonconvex-nonconcave min-max optimization problems, and it is a consequence of the minimax theorem . Property 3.1 also holds for time-varying variational inequalities (VIs) that satisfy the so-called MVI property , as we elaborate further in Section 3.1.1.

Now, building on the work of Zhang et al. [93], let us introduce some natural measures of the games' variation. The first-order variation of the Nash equilibria was defined by Zhang et al. [93] for T ≥ 2 as V ( T ) NE := inf z ( t,⋆ ) ∈Z ( t,⋆ ) , ∀ t ∈ [ [ T ] ] ∑ T -1 t =1 ∥ z ( t +1 ,⋆ ) -z ( t,⋆ ) ∥ 2 , where Z ( t,⋆ ) is the (nonempty) set of Nash equilibria of the t -th game. We recall that there can be a multiplicity of Nash equilibria [80]; as such, a compelling feature of the above variation measure is that it depends on the most favorable sequence of Nash equilibria-one that minimizes the first-order variation.

It is important, however, to come to terms with the well-known fact that Nash equilibria can change abruptly even under a 'small' perturbation in the payoff matrix (see Example A.5), which is an important caveat of the variation V ( T ) NE . To address this, and in accordance with our more general Property 3.1, we consider a more favorable variation measure, defined as

for a sufficiently large parameter C > 0 (see (28)); the infimum above is subject to ϵ ( t ) ∈ R ≥ 0 and z ( t,⋆ ) ∈ Z ( t,⋆ ) ϵ ( t ) for all t ∈ [ [ T ] ] , where Z ( t,⋆ ) ϵ ( t ) is the set of ϵ ( t ) -approximate NE. It is evident that V ( T ) ϵ -NE ≤ V ( T ) NE since one can take ϵ (1) = · · · = ϵ ( T ) = 0 ; in fact, V ( T ) ϵ -NE can be arbitrarily smaller:

Proposition 3.2. For any T ≥ 4 , there is a sequence of T games such that V ( T ) NE ≥ T 2 while V ( T ) ϵ -NE ≤ δ , for any δ > 0 .

This shows that V ( T ) ϵ -NE is a more compelling variation measure compared to V ( T ) NE . It is worth noting here that Zhang et al. [93] also considered the variation measure W ( T ) A := ∑ T t =1 ∥ A ( t ) -¯ A ∥ 2 , where ¯ A is the average payoff matrix and ∥ · ∥ 2 denotes the spectral norm. 1 It is in fact not hard to construct instances such that V ( T ) ϵ -NE ≪ min {V ( T ) NE , W ( T ) A } (Proposition A.6).

Moreover, we also recall another quantity that captures the variation of the payoff matrices: V ( T ) A := ∑ T -1 t =1 ∥ A ( t +1) -A ( t ) ∥ 2 2 . Unlike V ( T ) NE , the variation measure V ( T ) A depends on the second-order variation (of the payoff matrices), which could translate to a lower-order impact compared to V ( T ) NE

(see, e.g. , Corollary A.9). We stress that while our convergence bounds will be parameterized based on V ( T ) ϵ -NE and V ( T ) A , the underlying algorithm-namely OGD -will remain oblivious to those variation measures. We only make the mild assumption that players know in advance the value of L := max 1 ≤ t ≤ T ∥ A ( t ) ∥ 2 , so that they can tune the learning rate η appropriately.

We are now ready to state the main result of this subsection. Below, we use the notation EQGAP ( t ) ( z ( t ) ) to represent the Nash equilibrium gap of ( x ( t ) , y ( t ) ) = z ( t ) ∈ Z at the t -th game. More precisely, EQGAP ( t ) ( z ( t ) ) := max { BR ( t ) x ( x ( t ) ) , BR ( t ) y ( y ( t ) ) } , where BR ( t ) x ( x ( t ) ) := max x ∈X {⟨ x , u ( t ) x ⟩}-⟨ x ( t ) , u ( t ) x ⟩ is the best response gap of Player x (and analogously for Player y ). Theorem 3.3 (Detailed version in Theorem A.11) . Suppose that both players employ OGD with learning rate η = 1 4 L in a sequence of time-varying BSPPs, where L := max 1 ≤ t ≤ T ∥ A ( t ) ∥ 2 . Then, ∑ T t =1 ( EQGAP ( t ) ( z ( t ) ) ) 2 = O ( 1 + V ( T ) ϵ -NE + V ( T ) A ) , where ( z ( t ) ) 1 ≤ t ≤ T is the sequence of joint strategy profiles produced by OGD .

The proof of this theorem is based on upper bounding the second-order path length of OGD , ∑ T t =1 ( ∥ z ( t ) -ˆ z ( t ) ∥ 2 2 + ∥ z ( t ) -ˆ z ( t +1) ∥ 2 2 ) , as a function of the variation measures V ( T ) ϵ -NE and V ( T ) A . This is shown via a dynamic RVU bound [93] (Lemmas A.1 and A.2) in conjunction with Property 3.1.

It is worth noting that when the deviation of the payoff matrices is controlled by the deviation of the players' strategies, in the sense that ∑ T -1 t =1 ∥ A ( t +1) -A ( t ) ∥ 2 2 ≤ W 2 ∑ T -1 t =1 ∥ z ( t +1) -z ( t ) ∥ 2 2 for some parameter W ∈ R > 0 , the variation measure V ( T ) A in Theorem 3.3 can be entirely eliminated; see Corollary A.9. The same in fact applies under an improved prediction mechanism (see Remark A.13); while that prediction is not implementable in the online learning setting, it can be used, for example, when the sequence of games is known in advance (as is the case in certain applications).

Wenext state some immediate consequences of Theorem 3.3. (Item 2 below follows from Theorem 3.3 by Jensen's inequality.)

Corollary 3.4. In the setting of Theorem 3.3,

In particular, in terms of asymptotic implications, if lim T → + ∞ V ( T ) ϵ -NE T , lim T → + ∞ V ( T ) A T = 0 , then (i) for any ϵ > 0 the fraction of iterates of OGD with at least an ϵ Nash equilibrium gap converges to 0 ; and (ii) the average Nash equilibrium gap of the iterates of OGD converges to 0 .

In the special case where V ( T ) ϵ -NE , V ( T ) A = O (1) , Theorem 3.3 recovers the T -1 / 2 iterate-convergence rate of OGD in static bilinear saddle-point problems.

Meta-learning Our results also have immediate applications in the meta-learning setting. More precisely, meta-learning in games is a special case of time-varying games which consists of a sequence of H ∈ N separate games, each of which is repeated for m ∈ N consecutive rounds, so that T := m × H . The central goal in meta-learning is to obtain convergence bounds parameterized by the similarity of the games; identifying suitable similarity metrics is a central question in that line of work.

In this context, we highlight that Theorem 3.3 shown above readily provides a meta-learning guarantee parameterized by the following notion of similarity between the Nash equilibria: inf z ( h,⋆ ) ∈Z ( h,⋆ ) , ∀ h ∈ [ [ H ] ] ∑ H -1 h =1 ∥ z ( h +1 ,⋆ ) -z ( h,⋆ ) ∥ 2 , where Z ( h,⋆ ) is the set of Nash equilibria of the h -th game in the meta-learning sequence, 2 as well as the similarity of the payoff matricescorresponding to the term V ( T ) A . In fact, under a suitable prediction-the one used by Harris et al. [48]-the dependence on V ( T ) A can be entirely removed; see Proposition A.14 for our formal result. A compelling aspect of our meta-learning guarantee is that the considered algorithm is oblivious to the boundaries of the meta-learning. It is also worth noting that our similarity metric can be arbitrarily

smaller than the one considered by Harris et al. [48] (Proposition A.15). We further provide some novel results on meta-learning in general-sum games in Section 3.3.

3.1.1 Beyond bilinear saddle-point problems

As we alluded to earlier, our approach can be generalized beyond time-varying bilinear saddlepoint problems to a more general class of time-varying variational inequality (VIs) as follows. Let F ( t ) : Z → Z be the (single-valued) operator of the VI problem at time t ∈ N . F ( t ) is said to satisfy the MVI property [61, 36] if there exists a point z ( t,⋆ ) ∈ Z such that ⟨ z -z ( t,⋆ ) , F ( t ) ( z ) ⟩ ≥ 0 for any z ∈ Z . For example, in the special case of a bilinear saddle-point problem we have that F : z := ( x , y ) ↦→ ( A y , -A ⊤ x ) , and the MVI property is satisfied by virtue of Von Neumann's minimax theorem. It is direct to see that Property 3.1 applies to any time-varying sequence of VIs with respect to ( z ( t,⋆ ) ) 1 ≤ t ≤ T as long as every operator in the sequence ( F (1) , . . . , F ( T ) ) satisfies the MVI property. (Even more broadly, it would suffice if almost all operators in the sequence-in that their fraction converges to 1 as T → + ∞ -satisfy the MVI property.) This observation enables extending Theorem 3.3 to a more general class of problems. As a concrete example, we provide a generalization of Theorem 3.3 (Theorem A.16) to polymatrix zero-sum games [15] in Appendix A.1.7. By contrast, in problems where the MVI property fails it appears that a much different approach is called for.

We finally point out that our framework could have certain implications for solving (static) general VIs, as we discuss in Appendix A.1.8.

3.2 Strongly convex-concave games

In this subsection, we show that under additional structure we can significantly improve the variation measures established in Theorem 3.3. More precisely, we first assume that each objective function f ( x , y ) is µ -strongly convex with respect to x and µ -strongly concave with respect to y . Our second assumption is that each game is played for multiple rounds m ∈ N , instead of only a single round; this is akin to the meta-learning setting. The key insight is that as long as m is large enough, m = Ω(1 /µ ) , those two assumptions suffice to obtain a second-order variation bound in terms of the sequence of Nash equilibria, S ( H ) NE := ∑ H -1 h =1 ∥ z ( h +1 ,⋆ ) -z ( h,⋆ ) ∥ 2 2 , where z ( h,⋆ ) is a Nash equilibrium of the h -th game. This significantly refines the result of Theorem 3.3, and is inspired by the improved dynamic regret bounds obtained by Zhang et al. [92]. Below we sketch the key ideas of the improvement; full proofs are deferred to Appendix A.2.

In this setting, it is assumed that Player x obtains the utility u ( t ) x := -∇ x f ( t ) ( x ( t ) , y ( t ) ) at every time t ∈ [ [ T ] ] , while its regret will be denoted by Reg ( T ) L ,y ; similar notation applies for Player y . The first observation is that, focusing on a single (static) game, under strong convexity-concavity the sum of the players' regrets are strongly nonnegative (Lemma A.18):

for any Nash equilibrium z ⋆ ∈ Z of the game. In turn, this can be cast in terms of dynamic regret over the sequence of the h games (Lemma A.19). Next, combining those dynamic-regret lower bounds with a suitable RVU-type property leads to a refined second-order path length bound as long as m = Ω(1 /µ ) , which in turn leads to our main result below. Before we present its statement, let us introduce the following measure of variation of the gradients: V ( H ) ∇ f := ∑ H -1 h =1 max z ∈Z ∥ F ( h +1) ( z ) -F ( h ) ( z ) ∥ 2 2 , where let F : z := ( x , y ) ↦→ ( ∇ x f ( x , y ) , -∇ y f ( x , y )) . This variation measure is analogous to V ( T ) A we introduced earlier for time-varying BSPPs.

Theorem 3.5 (Detailed version in Theorem A.21) . Let f ( h ) : X ×Y be a µ -strongly convex-concave and L -smooth function, for all h ∈ [ [ H ] ] . Suppose further that both players employ OGD with learning rate η = min { 1 8 L , 1 2 µ } for T ∈ N repetitions, where T = m × H and m ≥ 2 ηµ . Then, ∑ T t =1 ( EQGAP ( t ) ( z ( t ) ) ) 2 = O (1 + S ( H ) NE + V ( H ) ∇ f ) .

Our techniques also imply the improved regret bounds Reg ( T ) L ,x , Reg ( T ) L ,y = O (√ 1 + S ( H ) NE + V ( H ) ∇ f ) , under suitable tuning of the learning rate (see Corollary A.22).

There is another immediate but important implication of (3): any no-regret algorithm in a (static) strongly convex-concave setting ought to be approaching the Nash equilibrium; 3 in contrast, this property is spectacularly false in (general) monotone settings [60].

Proposition 3.6. Let f : X × Y → R be a µ -strongly convex-concave function. If players incur regrets such that Reg ( T ) L ,x +Reg ( T ) L ,y ≤ CT 1 -ω , for some parameters C > 0 and ω ∈ (0 , 1] , then for any ϵ > 0 and T > 2 C µϵ 2 there is a pair of strategies z ( t ) ∈ Z such that ∥ z ( t ) -z ⋆ ∥ 2 ≤ ϵ , where z ⋆ ∈ Z is a Nash equilibrium.

( ) 1 /ω

The insights of this subsection are also of interest in general monotone settings by incorporating a strongly convex regularizer; tuning its magnitude allows us to trade off between a better approximation and the benefits of strong convexity-concavity revealed in this subsection.

3.3 General-sum multi-player games

Next, we turn our attention to general-sum multi-player games. For simplicity, in this subsection we posit that the game is represented in normal form, so that each player i ∈ [ [ n ] ] has a finite set of available actions A i , and X i := ∆( A i ) . The proofs of this subsection are included in Appendix A.3.

Potential games First, we study the convergence of (online) gradient descent ( GD ) in time-varying potential games (see Definition A.23 for the formal description); we recall that unlike two-player zero-sum games, gradient descent is known to approach Nash equilibria in potential games. In our time-varying setup, it is assumed that each round t ∈ [ [ T ] ] corresponds to a different potential game described with a potential function Φ ( t ) . Wefurther let d : (Φ , Φ ′ ) ↦→ max z ∈ × n i =1 X i (Φ( z ) -Φ ′ ( z )) , so that V ( T ) Φ := ∑ T -1 t =1 d (Φ ( t ) , Φ ( t +1) ) ; we call attention to the fact that d ( · , · ) is not symmetric. Analogously to Theorem 3.3, we use EQGAP ( t ) ( z ( t ) ) ∈ R ≥ 0 to represent the NE gap of the joint strategy profile z ( t ) := ( x ( t ) 1 , . . . , x ( t ) n ) at the t -th game.

Theorem 3.7. Suppose that each player employs (online) GD with a sufficiently small learning rate in a sequence of time-varying potential games. Then, ∑ T t =1 ( EQGAP ( t ) ( z ( t ) ) ) 2 = O (Φ max + V ( T ) Φ ) , where Φ max is such that | Φ ( t ) ( · ) | ≤ Φ max.

We refer to Appendix B for some illustrative experiments related to Theorem 3.7.

General games Unfortunately, unlike the settings considered thus far, computing Nash equilibria in general games is computationally hard even under a crude approximation ϵ = Θ(1) [25, 19, 30]. Instead, learning algorithms are known to converge-in a time-average sense-to relaxations of the Nash equilibrium, known as (coarse) correlated equilibria . For our purposes, we will employ a bilinear formulation of correlated equilibria, which dates back to the seminal work of Hart and Schmeidler [50] (see also [84, Chapter 12] for an excellent exposition). This will allow us to translate the results of Section 3.1 to general multi-player games.

Specifically, correlated equilibria 4 can be phrased via a game between the n players and a mediator , an additional agent. At a high level, the mediator is endeavoring to identify a correlated strategy µ ∈ Ξ := ∆ ( × n i =1 A i ) for which no player has an incentive to deviate from the recommendation. In contrast, the players are trying to optimally deviate so as to maximize their own utility. More precisely, there exist matrices A 1 , . . . , A n , with each matrix A i depending solely on the utility of Player i , for which the bilinear saddle-point problem can be expressed as

where ¯ X i above is a suitable set of strategies; we elaborate more on this formulation in Appendix A.3. This zero-sum game has the property that there exists a strategy µ ⋆ ∈ Ξ such that max ¯ x i ∈ ¯ X i ( µ ⋆ ) ⊤ A i ¯ x i ≤ 0 , for any player i ∈ [ [ n ] ] , which is precisely a correlated equilibrium.

Before we proceed, it is important to note that the learning paradigm considered here deviates from the traditional one in that orchestrating the protocol requires an additional learning agent, resulting in a less decentralized protocol. Yet, the dynamics induced by solving (4) via online algorithms remain uncoupled [49], in the sense that each player obtains feedback-corresponding to the deviation benefit-that depends solely on its own utility.

Now in the time-varying setting, the matrices A 1 , . . . , A n that capture the players' utilities can change in each repetition. Crucially, we show that the structure of the induced bilinear problem (4) is such that there is a sequence of correlated equilibria that guarantee nonnegative dynamic regret; this refines Property 3.1 in that only one player's strategies suffice to guarantee nonnegativity, even if the strategy of the other player remains invariant. Below, we denote by DReg ( T ) µ the dynamic regret of Player min in (4), and by Reg ( T ) i the regret of each player i ∈ [ [ n ] ] up to time T ∈ N , so that the regret of Player max in (4) can be expressed as ∑ n i =1 Reg ( T ) i .

As a result, this enables us to apply Theorem 3.3 parameterized on (i) the variation of the CE V ( T ) CE := inf µ ( t,⋆ ) ∈ Ξ ( t,⋆ ) , ∀ t ∈ [ [ T ] ] ∑ T -1 t =1 ∥ µ ( t +1 ,⋆ ) -µ ( t,⋆ ) ∥ 2 , where Ξ ( t,⋆ ) denotes the set of CE of the t -th game, and (ii) the variation in the players' utilities V ( T ) A := ∑ n i =1 ∑ T -1 t =1 ∥ A ( t +1) i -A ( t ) i ∥ 2 2 ; below, we denote by CEGAP ( t ) ( µ ( t ) ) the CE gap of µ ( t ) ∈ Ξ at the t -th game.

Property 3.8. Suppose that Ξ ∋ µ ( t,⋆ ) is a correlated equilibrium of the game at any time t ∈ [ [ T ] ] . Then, DReg ( T ) µ ( µ (1 ,⋆ ) , . . . , µ ( T,⋆ ) ) + ∑ n i =1 Reg ( T ) i ≥ 0 .

Theorem 3.9. Suppose that each player employs OGD in a sequence of time-varying BSPPs (4) with a sufficiently small learning rate. Then, ∑ T t =1 ( CEGAP ( t ) ( µ ( t ) ) ) 2 = O (1 + V ( T ) CE + V ( T ) A ) .

There are further interesting implications of our framework that are worth highlighting. First, we obtain meta-learning guarantees for general games that depend on the (algorithm-independent) similarity of the correlated equilibria (Corollary A.25); that was left as an open question by Harris et al. [48], where instead algorithm-dependent similarity metrics were derived. Further, by applying Corollary A.12, we derive natural variation-dependent per-player regret bounds in general games (Corollary A.26); this addresses a question left by Zhang et al. [93], albeit under a different learning paradigm. We suspect that obtaining such results-parameterized on the variation of the CE-are not possible without the presence of the additional agent, as in (4).

3.4 Dynamic regret bounds in static games

Finally, in this subsection we switch gears by investigating dynamic regret guarantees when learning in static games. The proofs of this subsection are included in Appendix A.4.

First, we point out that while traditional no-regret learning algorithms guarantee O ( √ T ) external regret, instances of OMD -a generalization of OGD ; see (5) in Appendix A-in fact guarantee O ( √ T ) dynamic regret in two-player zero-sum games, which is a much stronger performance measure:

Proposition 3.10. Suppose that both players in a (static) two-player zero-sum game employ OMD with a smooth regularizer. Then, DReg ( T ) x , DReg ( T ) y = O ( √ T ) .

In proof, the dynamic regret of each player under OMD with a smooth regularizer can be bounded by the first-order path length of that player's strategies, which in turn can be bounded by O ( √ T ) given that the second-order path length is O (1) (Theorem A.7). In fact, Theorem A.7 readily extends Proposition 3.10 to time-varying zero-sum games as well, implying that

A question that arises from Proposition 3.10 is whether the O ( √ T ) guarantee for dynamic regret of OMD can be improved in the online learning setting. Below, we point out a significant improvement to

O (log T ) , but under a stronger two-point feedback model; 5 namely, we posit that in every round each player can select an additional auxiliary strategy, and each player then gets to additionally observe the utility corresponding to the auxiliary strategies. Notably, this is akin to how the extra-gradient method works [52] (also cf . [72, Section 4.2] for multi-point feedback models in the bandit setting).

Observation 3.11. Under two-point feedback, there exist learning algorithms that guarantee DReg ( T ) x , DReg ( T ) y = O (log T ) in two-player zero-sum games.

In particular, it suffices for each player to employ OMD , but with the twist that the first strategy in each round is the time-average of OMD ; the auxiliary strategy is the standard output of OMD . Then, the dynamic regret of each player will grow as O ( ∑ T t =1 1 t ) = O (log T ) since the duality gap of the average strategies is decreasing with a rate of T -1 [72]. It is an interesting question whether the bound of Observation 3.11 can be improved to O (1) [14]; we conjecture that there is a lower bound of Ω(log T ) .

General-sum games In stark contrast, no (computationally efficient) sublinear dynamic regret guarantees are possible in general-sum games:

Proposition 3.12. Any polynomial-time algorithm incurs ∑ n i =1 DReg ( T ) i ≥ CT for any polynomial T ∈ N , even if n = 2 and C > 0 is an absolute constant, unless ETH for PPAD is false [73].

̸

Indeed, this follows immediately since computing a Nash equilibrium to O (1) precision in two-player games requires superpoylnomial time [73]. As such, Proposition 3.12 applies beyond the online learning setting. This motivates considering a relaxation of dynamic regret, wherein the sequence of comparators is subject to the constraint ∑ T -1 t =1 ✶ { x ( t +1 ,⋆ ) i = x ( t,⋆ ) i } ≤ K -1 , for some parameter K ∈ N [22]; this will be referred to as K -DReg ( T ) i . Naturally, external regret coincides with K -DReg ( T ) i under K = 1 . In this context, we employ Lemma A.1 to bound K -DReg ( T ) under OGD :

Theorem 3.13 (Precise version in Theorem A.28) . Suppose that all n players employ OGD with a suitable learning rate in an L -smooth game. Then, for any K ∈ N ,

One question that arises here is whether the per-player bound of O ( K 3 / 4 T 1 / 4 ) (Item 2) can be improved to ˜ O ( K ) , where ˜ O ( · ) hides logarithmic factors. The main challenge is that, even for K = 1 , all known methods that obtain ˜ O (1) [29, 68, 1, 38] rely on non-smooth regularizers that violate the preconditions of Lemma A.2-our dynamic RVU bound beyond (squared) Euclidean regularization; yet, we point out that it is possible under a slightly stronger feedback model (Remark A.29). We finally highlight that the game-theoretic significance of the solution concept that arises as the limit point of no-regret learners when K -DReg = o ( T ) was recently investigated by Crippa et al. [22].

4 Conclusions and future work

In this paper, we developed a framework for characterizing iterate-convergence of no-regret learning algorithms-primarily optimistic gradient descent ( OGD )-in time-varying games. There are many promising avenues for future research. Besides closing the obvious gaps we highlighted in Section 3.4, it is important to characterize the behavior of no-regret learning algorithms in other fundamental time-varying multiagent settings, such as Stackelberg (security) games [5]. Moreover, our results operate in the full-feedback model where each player receives feedback on all possible actions. Extending the scope of our framework to capture partial-feedback models as well is another interesting direction for future work.

Acknowledgments and Disclosure of Funding

We are grateful to anonymous referees for providing valuable and constructive feedback. We also thank Vince Conitzer and Caspar Oesterheld for many helpful comments. This material is based on work supported by the National Science Foundation under grants IIS-1901403 and CCF-1733556 and by the ARO under award W911NF2210266.

References

I. Anagnostides, G. Farina, C. Kroer, C. Lee, H. Luo, and T. Sandholm. Uncoupled learning dynamics with O (log T ) swap regret in multiplayer games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

I. Anagnostides, I. Panageas, G. Farina, and T. Sandholm. On last-iterate convergence beyond zero-sum games. In International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research , pages 536-581. PMLR, 2022.

R. Aumann. Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics , 1:67-96, 1974.

W. Azizian, I. Mitliagkas, S. Lacoste-Julien, and G. Gidel. A tight and unified analysis of gradient-based methods for a whole spectrum of differentiable games. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS , volume 108 of Proceedings of Machine Learning Research , pages 2863-2873. PMLR, 2020.

M. Balcan, A. Blum, N. Haghtalab, and A. D. Procaccia. Commitment without regrets: Online learning in stackelberg security games. In Proceedings of the ACM Conference on Economics and Computation (EC) , pages 61-78. ACM, 2015.

L. Baudin, M. Scarsini, and X. Venel. Strategic behavior and no-regret learning in queueing systems. CoRR , abs/2302.03614, 2023.

H. H. Bauschke, W. M. Moursi, and X. Wang. Generalized monotone operators and their averaged resolvents. Math. Program. , 189(1):55-74, 2021.

M. Bowling, N. Burch, M. Johanson, and O. Tammelin. Heads-up limit hold'em poker is solved. Science , 347(6218), January 2015.

N. Brown and T. Sandholm. Strategy-based warm starting for regret minimization in games. In AAAI Conference on Artificial Intelligence (AAAI) , 2016.

N. Brown and T. Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

N. Brown and T. Sandholm. Solving imperfect-information games via discounted regret minimization. In AAAI Conference on Artificial Intelligence (AAAI) , 2019.

N. Brown and T. Sandholm. Superhuman AI for multiplayer poker. Science , 365(6456):885-890, 2019.

Y. Cai and W. Zheng. Accelerated single-call methods for constrained min-max optimization. CoRR , abs/2210.03096, 2022.

Y. Cai and W. Zheng. Doubly optimal no-regret learning in monotone games. In International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 3507-3524. PMLR, 2023.

Y. Cai, O. Candogan, C. Daskalakis, and C. H. Papadimitriou. Zero-sum polymatrix games: A generalization of minmax. Math. Oper. Res. , 41(2):648-655, 2016.

Y. Cai, A. Oikonomou, and W. Zheng. Finite-time last-iterate convergence for learning in multiplayer games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

A. R. Cardoso, J. D. Abernethy, H. Wang, and H. Xu. Competing against nash equilibria in adversarially changing zero-sum games. In International Conference on Machine Learning (ICML) , volume 97 of Proceedings of Machine Learning Research , pages 921-930. PMLR, 2019.

N. Cesa-Bianchi, P. Gaillard, G. Lugosi, and G. Stoltz. Mirror descent meets fixed share (and feels no regret). In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 989-997, 2012.

X. Chen, X. Deng, and S.-H. Teng. Settling the complexity of computing two-player Nash equilibria. Journal of the ACM , 2009.

C.-K. Chiang, T. Yang, C.-J. Lee, M. Mahdavi, C.-J. Lu, R. Jin, and S. Zhu. Online optimization with gradual variations. In Conference on Learning Theory , pages 6-1, 2012.

P. L. Combettes and T. Pennanen. Proximal methods for cohypomonotone operators. SIAM J. Control. Optim. , pages 731-742, 2004.

L. Crippa, Y. Gur, and B. Light. Equilibria in repeated games under no-regret with dynamic benchmarks. CoRR , abs/2212.03152, 2023.

C. D. Dang and G. Lan. On the convergence properties of non-euclidean extragradient methods for variational inequalities with generalized monotone operators. Comput. Optim. Appl. , pages 277-310, 2015.

C. Daskalakis. Non-concave games: A challenge for game theory's next 100 years, 2022.

C. Daskalakis, P. Goldberg, and C. Papadimitriou. The complexity of computing a Nash equilibrium. SIAM Journal on Computing , 2008.

C. Daskalakis, A. Deckelbaum, and A. Kim. Near-optimal no-regret algorithms for zero-sum games. In Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2011.

C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng. Training gans with optimism. In 6th International Conference on Learning Representations, ICLR . OpenReview.net, 2018.

C. Daskalakis, D. J. Foster, and N. Golowich. Independent policy gradient methods for competitive reinforcement learning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

C. Daskalakis, M. Fishelson, and N. Golowich. Near-optimal no-regret learning in general games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 27604-27616, 2021.

A. Deligkas, J. Fearnley, A. Hollender, and T. Melissourgos. Pure-circuit: Strong inapproximability for PPAD. In Proceedings of the Annual Symposium on Foundations of Computer Science (FOCS) , pages 159-170. IEEE, 2022.

Y. Deng, V. Mirrokni, and S. Zuo. Non-clairvoyant dynamic mechanism design with budget constraints and beyond. In Proceedings of the 22nd ACM Conference on Economics and Computation , Proceedings of the ACM Conference on Economics and Computation (EC). Association for Computing Machinery, 2021.

J. Diakonikolas. Halpern iteration for near-optimal and parameter-free monotone inclusion and strong solutions to variational inequalities. In J. D. Abernethy and S. Agarwal, editors, Conference on Learning Theory (COLT) , volume 125 of Proceedings of Machine Learning Research , pages 1428-1451. PMLR, 2020.

J. Diakonikolas, C. Daskalakis, and M. I. Jordan. Efficient methods for structured nonconvexnonconcave min-max optimization. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS , volume 130 of Proceedings of Machine Learning Research , pages 2746-2754. PMLR, 2021.

Z. Duan, W. Huang, D. Zhang, Y. Du, J. Wang, Y. Yang, and X. Deng. Is nash equilibrium approximator learnable? In Autonomous Agents and Multi-Agent Systems , pages 233-241. ACM, 2023.

B. Duvocelle, P. Mertikopoulos, M. Staudigl, and D. Vermeulen. Multiagent online learning in time-varying games. Mathematics of Operations Research , 2022.

F. Facchinei and J.-S. Pang. Finite-dimensional variational inequalities and complementarity problems . Springer, 2003.

G. Farina, C. Kroer, and T. Sandholm. Faster game solving via predictive Blackwell approachability: Connecting regret matching and mirror descent. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

G. Farina, I. Anagnostides, H. Luo, C. Lee, C. Kroer, and T. Sandholm. Near-optimal noregret learning for general convex games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

G. Farina, C. Kroer, C. Lee, and H. Luo. Clairvoyant regret minimization: Equivalence with nemirovski's conceptual prox method and extension to general convex games. CoRR , abs/2208.14891, 2022.

Y. Feng, H. Fu, Q. Hu, P. Li, I. Panageas, B. Peng, and X. Wang. On the last-iterate convergence in time-varying zero-sum games: Extra gradient succeeds where optimism fails. CoRR , abs/2310.02604, 2023.

T. Fiez, R. Sim, S. Skoulakis, G. Piliouras, and L. J. Ratliff. Online learning in periodic zero-sum games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 10313-10325, 2021.

D. Foster and R. Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior , 21:40-55, 1997.

Y. Freund and R. Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior , 29:79-103, 1999.

D. F. Garrett. Dynamic mechanism design: Dynamic arrivals and changing values. Games and Economic Behavior , 104:595-612, 2017.

A. Giannou, E. Vlatakis-Gkaragkounis, and P. Mertikopoulos. Survival of the strictest: Stable and unstable equilibria under regularized learning with partial information. In Conference on Learning Theory (COLT) , volume 134 of Proceedings of Machine Learning Research , pages 2147-2148. PMLR, 2021.

N. Golowich, S. Pattathil, and C. Daskalakis. Tight last-iterate convergence rates for no-regret learning in multi-player games. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

E. Gorbunov, A. B. Taylor, and G. Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

K. Harris, I. Anagnostides, G. Farina, M. Khodak, S. Wu, and T. Sandholm. Meta-learning in games. In The Eleventh International Conference on Learning Representations, ICLR . OpenReview.net, 2023.

S. Hart and A. Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68:1127-1150, 2000.

S. Hart and D. Schmeidler. Existence of correlated equilibria. Mathematics of Operations Research , 14(1):18-25, 1989.

E. Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In International Conference on Machine Learning (ICML) , page 393-400. Association for Computing Machinery, 2009.

Y. Hsieh, F. Iutzeler, J. Malick, and P. Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 6936-6946, 2019.

A. Jadbabaie, A. Rakhlin, S. Shahrampour, and K. Sridharan. Online Optimization : Competing with Dynamic Comparators. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics , pages 398-406, 2015.

D. S. Kalhan, A. S. Bedi, A. Koppel, K. Rajawat, H. Hassani, A. K. Gupta, and A. Banerjee. Dynamic online learning via frank-wolfe algorithm. IEEE Trans. Signal Process. , 69:932-947, 2021.

T. Lykouris, V. Syrgkanis, and ´ E. Tardos. Learning and efficiency in games with dynamic population. In Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 120-129. SIAM, 2016.

O. Mangoubi and N. K. Vishnoi. Greedy adversarial equilibrium: an efficient alternative to nonconvex-nonconcave min-max optimization. In Proceedings of the Annual Symposium on Theory of Computing (STOC) , pages 896-909. ACM, 2021.

E. Mazumdar, L. J. Ratliff, and S. S. Sastry. On gradient-based learning in continuous games. SIAM J. Math. Data Sci. , pages 103-131, 2020.

I. Menache and A. E. Ozdaglar. Network Games: Theory, Models, and Dynamics . Synthesis Lectures on Communication Networks. Morgan & Claypool Publishers, 2011.

P. Mertikopoulos and M. Staudigl. Equilibrium tracking and convergence in dynamic games. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 930-935. IEEE, 2021.

P. Mertikopoulos, C. H. Papadimitriou, and G. Piliouras. Cycles in adversarial regularized learning. In Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 2703-2717. SIAM, 2018.

P. Mertikopoulos, B. Lecouat, H. Zenati, C. Foo, V. Chandrasekhar, and G. Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In 7th International Conference on Learning Representations, ICLR . OpenReview.net, 2019.

A. Mokhtari, S. Shahrampour, A. Jadbabaie, and A. Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In 55th IEEE Conference on Decision and Control, CDC 2016 , pages 7195-7201. IEEE, 2016.

H. Moulin and J.-P. Vial. Strategically zero-sum games: The class of games whose completely mixed equilibria cannot be improved upon. International Journal of Game Theory , 7(3-4): 201-221, 1978.

J. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences , 36:48-49, 1950.

A. Nemirovski. Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1), 2004.

M. Nouiehed, M. Sanjabi, T. Huang, J. D. Lee, and M. Razaviyayn. Solving a class of nonconvex min-max games using iterative first order methods. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 14905-14916, 2019.

C. H. Papadimitriou, G. Pierrakos, A. Psomas, and A. Rubinstein. On the complexity of dynamic mechanism design. Games and Economic Behavior , 134:399-427, 2022.

G. Piliouras, R. Sim, and S. Skoulakis. Beyond time-average convergence: Near-optimal uncoupled online learning via clairvoyant multiplicative weights update. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

J. I. Poveda, M. Krstic, and T. Basar. Fixed-time seeking and tracking of time-varying nash equilibria in noncooperative games. In American Control Conference, ACC 2022, Atlanta, GA, USA, June 8-10, 2022 , pages 794-799. IEEE, 2022.

J. Rahme, S. Jelassi, and S. M. Weinberg. Auction learning as a two-player game. In 9th International Conference on Learning Representations, ICLR , 2021.

A. A. Raja and S. Grammatico. Payoff distribution in robust coalitional games on time-varying networks. IEEE Trans. Control. Netw. Syst. , 9(1):511-520, 2022.

A. Rakhlin and K. Sridharan. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems , pages 3066-3074, 2013.

A. Rubinstein. Settling the complexity of computing approximate two-player nash equilibria. In Proceedings of the Annual Symposium on Foundations of Computer Science (FOCS) , pages 258-265. IEEE Computer Society, 2016.

Y. Sato, E. Akiyama, and J. D. Farmer. Chaos in learning a simple two-person game. Proceedings of the National Academy of Sciences , 99(7):4748-4751, 2002.

S. Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4, 2012.

C. Song, Z. Zhou, Y. Zhou, Y. Jiang, and Y. Ma. Optimistic dual extrapolation for coherent non-monotone variational inequalities. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

D. Sychrovsky, M. Sustr, E. Davoodi, M. Lanctot, and M. Schmid. Learning not to regret. CoRR , abs/2303.01074, 2023.

V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems , pages 2989-2997, 2015.

F. A. Taha, K. Rokade, and F. Parise. Gradient dynamics in linear quadratic network games with time-varying connectivity and population fluctuation. CoRR , abs/2309.07871, 2023.

E. van Damme. Stability and perfection of Nash equilibria , volume 339. Springer, 1991.

N. Van Long. A survey of dynamic games in economics , volume 1. World Scientific, 2010.

X. Venel. Regularity of dynamic opinion games. Games and Economic Behavior , 126:305-334, 2021.

E. Vlatakis-Gkaragkounis, L. Flokas, T. Lianeas, P. Mertikopoulos, and G. Piliouras. No-regret learning and mixed nash equilibria: They do not mix. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

B. Von Stengel. Game theory basics . Cambridge University Press, 2021.

Z. Wang, Y. Shen, M. Zavlanos, and K. H. Johansson. No-regret learning in strongly monotone games converges to a nash equilibrium, 2022.

Y. Yan, P. Zhao, and Z. Zhou. Fast rates in time-varying strongly monotone games. In International Conference on Machine Learning (ICML) , volume 202 of Proceedings of Machine Learning Research , pages 39138-39164. PMLR, 2023.

J. Yang, N. Kiyavash, and N. He. Global convergence and variance-reduced optimization for a class of nonconvex-nonconcave minimax problems. CoRR , abs/2002.09621, 2020.

Y. Yang and C. Ma. O ( T -1 ) convergence of optimistic-follow-the-regularized-leader in twoplayer zero-sum markov games. In The Eleventh International Conference on Learning Representations, ICLR . OpenReview.net, 2023.

M. Ye and G. Hu. Distributed seeking of time-varying nash equilibrium for non-cooperative games. IEEE Trans. Autom. Control. , 60(11):3000-3005, 2015.

T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with O (1 /k 2 ) rate on squared gradient norm. In International Conference on Machine Learning (ICML) , volume 139 of Proceedings of Machine Learning Research , pages 12098-12109. PMLR, 2021.

B. H. Zhang, G. Farina, I. Anagnostides, F. Cacciamani, S. M. McAleer, A. A. Haupt, A. Celli, N. Gatti, V. Conitzer, and T. Sandholm. Steering no-regret learners to optimal equilibria. CoRR , abs/2306.05221, 2023.

L. Zhang, T. Yang, J. Yi, R. Jin, and Z. Zhou. Improved dynamic regret for non-degenerate functions. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 732-741, 2017.

M. Zhang, P. Zhao, H. Luo, and Z. Zhou. No-regret learning in time-varying zero-sum games. In International Conference on Machine Learning (ICML) , volume 162 of Proceedings of Machine Learning Research , pages 26772-26808. PMLR, 2022.

P. Zhao, Y. Zhang, L. Zhang, and Z. Zhou. Dynamic regret of convex and smooth functions. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2020.

M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning (ICML) , pages 928-936, 2003.

A Omitted proofs

In this section, we provide the proofs from the main body (Section 3).

A.1 Proofs from Section 3.1

First, we start with the proofs of the dynamic RVU bounds (Lemmas A.1 and A.2 below). Before we proceed, it will be useful to express the update rule ( OGD ) in the following equivalent form:

Here, B ϕ x ( · ∥ · ) denotes the Bregman divergence induced by the (squared) Euclidean regularizer ϕ x : x ↦→ 1 2 ∥ x ∥ 2 2 ; namely, B ϕ x ( x ∥ x ′ ) := ϕ ( x ) -ϕ ( x ′ ) -⟨∇ ϕ ( x ′ ) , x -x ′ ⟩ = 1 2 ∥ x -x ′ ∥ 2 2 , for x , x ′ ∈ X . The update rule (5) for general Bregman divergences will be referred to as optimistic mirror descent ( OMD ) .

A.1.1 Dynamic RVU bounds

The first key ingredient that we need for the proof of Theorem 3.3 is the property of regret bounded by variation in utilities (RVU) , in the sense of Syrgkanis et al. [78], but with respect to dynamic regret; such a bound is established below, and is analogous to that obtained by Zhang et al. [93].

Lemma A.1 (RVU bound for dynamic regret) . Consider any sequence of utilities ( u (1) x , . . . , u ( T ) x ) up to time T ∈ N . The dynamic regret (1) of OGD with respect to any sequence of comparators ( x (1 ,⋆ ) , . . . , x ( T,⋆ ) ) ∈ X T can be bounded by

In the special case of external regretx (1 ,⋆ ) = x (2 ,⋆ ) = · · · = x ( T,⋆ ) -(6) recovers the bound for OGD of Syrgkanis et al. [78]. The key takeaway from Lemma A.1 is that the overhead of dynamic regret in (6) grows with the first-order variation of the sequence of comparators.

Proof of Lemma A.1. First, by (1 /η ) -strong convexity of the function Ψ ( t ) x (defined in (5)) for any time t ∈ [ [ T ] ] , we have that

where we used [75, Lemma 2.8; p. 135]. Similarly, by (1 /η ) -strong convexity of the function ˆ Ψ ( t ) x (defined in (5)) for any time t ∈ [ [ T ] ] , we have that for any comparator x ( t,⋆ ) ∈ X ,

Thus, adding (7) and (8),

We further see that

Now the first term on the right-hand side can be upper bounded using the fact that, by (7) and (8),

by Cauchy-Schwarz, in turn implying that ⟨ x ( t ) -ˆ x ( t +1) , m ( t ) x -u ( t ) x ⟩ ≤ η ∥ m ( t ) x -u ( t ) x ∥ 2 2 . Thus, the proof follows by combining this bound with (9) and (10), along with the fact that

where the last bound follows since

where we recall that D X denotes the ℓ 2 -diameter of X

As an aside, we remark that assuming that m ( t ) x := 0 and ∥ u ( t ) x ∥ 2 ≤ 1 for any t ∈ [ [ T ] ] , Lemma A.1 implies that dynamic regret can be upper bounded by O (√ (1 + ∑ T -1 t =1 ∥ x ( t +1 ,⋆ ) -x ( t,⋆ ) ∥ 2 ) T ) , while that requires setting the learning rate based on the first-order variation of the (optimal) com- for any (bounded)-potentially adversarially selected-sequence of utilities ( u (1) x , . . . , u ( T ) x ) , for η := √ D 2 X 2 T + D X ∑ T -1 t =1 ∥ x ( t +1 ,⋆ ) -x ( t,⋆ ) ∥ 2 T , which is a well-known result in online optimization [95]; parators, there are standard techniques that would allow bypassing that assumption.

Next, we provide an extension of Lemma A.1 to the more general OMD algorithm under a broad class of regularizers.

Lemma A.2 (Extension of Lemma A.1 beyond Euclidean regularization) . Consider a 1 -strongly convex continuously differentiable regularizer ϕ with respect to a norm ∥·∥ such that (i) ∥∇ ϕ ( x ) ∥ ∗ ≤ G for any x , and (ii) B ϕ x ( x ∥ x ′ ) ≤ L ∥ x -x ′ ∥ for any x , x ′ . Then, for any sequence of utilities ( u (1) x , . . . , u ( T ) x ) up to time T ∈ N the dynamic regret (1) of OMD with respect to any sequence of comparators ( x (1 ,⋆ ) , . . . , x ( T,⋆ ) ) ∈ X T can be bounded as

The proof is analogous to that of Lemma A.1, and relies on the well-known three-point identity for the Bregman divergence:

In particular, along with the assumptions of Lemma A.2 imposed on the regularizer ϕ x , (11) implies that the term ∑ T -1 t =1 ( B ϕ x ( x ( t +1 ,⋆ ) ∥ ˆ x ( t +1) ) -B ϕ x ( x ( t,⋆ ) ∥ ˆ x ( t +1) ) ) is equal to

since B ϕ x ( x ( t +1 ,⋆ ) ∥ x ( t,⋆ ) ) ≤ L ∥ x ( t +1 ,⋆ ) -x ( t,⋆ ) ∥ (by assumption) and

where (12) follows from the Cauchy-Schwarz inequality; (13) uses the triangle inequality for the dual norm ∥ · ∥ ∗ ; and (14) follows from the assumption of Lemma A.2 that ∥∇ ϕ ( · ) ∥ ∗ ≤ G . The rest of the proof of Lemma A.2 is analogous to Lemma A.1, and it is therefore omitted. One important question here is whether Lemma A.2 can be extended under a broader class of regularizers; as we explain in Section 3.4, this is related to improving Theorem 3.13.

A.1.2 Nonnegativity of dynamic regret

We next proceed with the proof of Property 3.1. For completeness, we first prove the following special case [93]; the proof of Property 3.1 is then analogous.

Property A.3 (Special case of Property 3.1) . Suppose that Z ∋ z ( t,⋆ ) = ( x ( t,⋆ ) , y ( t,⋆ )) ) is a Nash equilibrium of the t -th game, for any time t ∈ [ [ T ] ] . Then, for s ( T ) x = ( x ( t,⋆ ) ) 1 ≤ t ≤ T and s ( T ) y = ( y ( t,⋆ ) ) 1 ≤ t ≤ T ,

Proof. Let v ( t ) := ⟨ x ( t,⋆ ) , A ( t ) y ( t,⋆ ) ⟩ be the value of the t -th game, for some t ∈ [ [ T ] ] . Then, we have that v ( t ) = ⟨ x ( t,⋆ ) , A ( t ) y ( t,⋆ ) ⟩ ≤ ⟨ x , A ( t ) y ( t,⋆ ) ⟩ for any x ∈ X , since x ( t,⋆ ) is a best response to y ( t,⋆ ) ; similarly, v ( t ) = ⟨ x ( t,⋆ ) , A ( t ) y ( t,⋆ ) ⟩ ≥ ⟨ x ( t,⋆ ) , A ( t ) y ⟩ for any y ∈ Y . Hence, ⟨ x ( t ) , A ( t ) y ( t,⋆ ) ⟩ - ⟨ x ( t,⋆ ) , A ( t ) y ( t ) ⟩ ≥ 0 , or equivalently, ⟨ x ( t,⋆ ) , u ( t ) x ⟩ + ⟨ y ( t,⋆ ) , u ( t ) y ⟩ ≥ 0 . But given that the game is zero-sum, it holds that ⟨ x ( t ) , u ( t ) x ⟩ + ⟨ y ( t ) , u ( t ) y ⟩ = 0 , so the last inequality can be in turn cast as

for any t ∈ [ [ T ] ] . As a result, summing over all t ∈ [ [ T ] ] we have shown that

Property 3.1. Suppose that Z ∋ z ( t,⋆ ) = ( x ( t,⋆ ) , y ( t,⋆ ) ) is an ϵ ( t ) -approximate Nash equilibrium of the t -th game. Then, for s ( T ) x = ( x ( t,⋆ ) ) 1 ≤ t ≤ T and s ( T ) y = ( y ( t,⋆ ) ) 1 ≤ t ≤ T ,

Proof. Given that ( x ( t,⋆ ) , y ( t,⋆ ) ) ∈ Z is an ϵ ( t ) -approximate Nash equilibrium of the t -th game, it follows that ⟨ x ( t,⋆ ) , A ( t ) y ( t,⋆ ) ⟩ ≤ ⟨ x ( t ) , A ( t ) y ( t,⋆ ) ⟩ + ϵ ( t ) x and ⟨ x ( t,⋆ ) , A ( t ) y ( t,⋆ ) ⟩ ≥ ⟨ x ( t,⋆ ) , A ( t ) y ( t ) ⟩ -ϵ ( t ) y , for some ϵ ( t ) x , ϵ ( t ) y ≤ ϵ ( t ) . Thus, we have that ⟨ x ( t ) , A ( t ) y ( t,⋆ ) ⟩ ≥ ⟨ x ( t,⋆ ) , A ( t ) y ( t ) ⟩-ϵ ( t ) x -ϵ ( t ) y , or equivalently, ⟨ x ( t,⋆ ) , u ( t ) x ⟩ + ⟨ y ( t,⋆ ) , u ( t ) y ⟩ ≥ -ϵ ( t ) x -ϵ ( t ) y ≥ -2 ϵ ( t ) . As a result,

for any t ∈ [ [ T ] ] , and the statement follows by summing (15) over all t ∈ [ [ T ] ] .

In fact, as we show below (in Property A.4), Property A.3 is a more general consequence of the minimax theorem. In particular, for a nonlinear online learning problem, we define dynamic regret with respect to a sequence of comparators ( x (1 ,⋆ ) , . . . , x ( T,⋆ ) ) ∈ X T as follows:

where u (1) x , . . . , u ( T ) x : x ↦→ R are the continuous utility functions observed by the learner, which could be in general nonconcave, and ( x ( t ) ) 1 ≤ t ≤ T is the sequence of strategies produced by the learner; (16) generalizes the notion of dynamic regret (1) in online linear optimization, that is, when u ( t ) x : x ↦→⟨ x , u ( t ) x ⟩ , where u ( t ) x ∈ R d x for any time t ∈ [ [ T ] ] .

Property A.4. Suppose that f ( t ) : X × Y → R is a continuous function such that for any t ∈ [ [ T ] ] ,

Let also x ( t,⋆ ) ∈ arg min x ∈X max y ∈Y f ( t ) ( x , y ) and y ( t,⋆ ) ∈ arg max y ∈Y min x ∈X f ( t ) ( x , y ) , for any t ∈ [ [ T ] ] . Then, for s ( T ) x = ( x ( t,⋆ ) ) 1 ≤ t ≤ T and s ( T ) y = ( y ( t,⋆ ) ) 1 ≤ t ≤ T ,

Proof. By definition of dynamic regret (16), it suffices to show that f ( t ) ( x ( t ) , y ( t,⋆ ) ) ≥ f ( t ) ( x ( t,⋆ ) , y ( t ) ) , for any time t ∈ [ [ T ] ] . Indeed,

where (17) and (21) are obvious; (18) and (20) follow from the definition of y ( t,⋆ ) ∈ Y and x ( t,⋆ ) ∈ X , respectively; and (19) holds by assumption. This concludes the proof.

A.1.3 Variation of the Nash equilibria

In our next example, we point out the standard observation that an arbitrarily small change in the entries of the payoff matrix can lead to a substantial deviation in the Nash equilibrium. Example A.5 . Consider a 2 × 2 (two-player) zero-sum game, where X := ∆ 2 , Y := ∆ 2 , described by the payoff matrix for some δ > 0 . Then, it is easy to see that the unique Nash equilibrium of this game is such that x ⋆ , y ⋆ := ( 1 3 , 2 3 ) ∈ ∆ 2 . Suppose now that the original payoff matrix (22) is perturbed to a new matrix

The new (unique) Nash equilibrium now reads x ⋆ , y ⋆ := ( 2 3 , 1 3 ) ∈ ∆ 2 . We conclude that an arbitrarily small deviation in the entries of the payoff matrix can lead to a non-trivial change in the Nash equilibrium.

Next, we leverage the simple observation of the example above to establish Proposition 3.2, the statement of which is recalled below.

Proposition 3.2. For any T ≥ 4 , there is a sequence of T games such that V ( T ) NE ≥ T 2 while V ( T ) ϵ -NE ≤ δ , for any δ > 0 .

Proof. We consider a sequence of T ≥ 4 games such that X , Y := ∆ 2 and where A , A ′ are the payoff matrices defined in (22) and (23), and are parameterized by δ > 0 (Example A.5). Then, the exact Nash equilibria read

As a result, it follows that V ( T ) NE := ∑ T -1 t =1 ∥ z ( t +1 ,⋆ ) -z ( t,⋆ ) ∥ 2 = 2 3 ( T -1) ≥ T 2 , for T ≥ 4 . In contrast, it is clear that V ( T ) ϵ -NE ≤ CδT , which follows by simply considering the sequence of strategies wherein both players are always selecting actions uniformly at random; we recall that C > 0 here is the value that parameterizes V ( T ) ϵ -NE . Thus, taking δ := δ ′ CT , for some arbitrarily small δ ′ > 0 , concludes the proof.

In the above sequence of games, the variation measure W ( T ) A is in fact also arbitrarily small as δ → 0 ; this reflects the fact that, although the exact Nash equilibria change abruptly, the sequence of payoff matrices exhibits small variation. Nevertheless, we next point out a modified sequence of games where V ϵ -NE remains small even though both W ( T ) A and V ( T ) NE can be unbounded.

Proposition A.6. For any T ≥ 4 , there is a sequence of games such that V ( T ) NE , W ( T ) A = Ω( T ) while V ( T ) ϵ -NE ≤ δ , for any δ > 0 .

Proof. We consider a sequence of T ≥ 4 games such that X , Y := ∆ 2 and where A , A ′ are the payoff matrices defined in (22) and (23), and 1 2 × 2 denotes the all-ones 2 × 2 matrix. Given that x ⊤ ( A + 1 2 × 2 ) y = x ⊤ A y + 1 , for any x , y ∈ ∆ 2 , it follows that the set of ϵ -approximate Nash equilibria of each game coincides with the set of ϵ -approximate Nash equilibria of the corresponding game in Proposition 3.2. Thus, by the same argument as in Proposition 3.2 earlier, we have that V ( T ) NE ≥ T 2 and V ( T ) ϵ -NE ≤ CδT . It is also clear that W ( T ) A = Ω( T ) , concluding the proof.

In words, the phenomenon identified above occurs because W ( T ) A is affected by perturbations in the payoff matrices that do not strategically alter the game-namely, the addition of the matrix 1 2 × 2 .

A.1.4 Proof of Theorem 3.3

Next, we proceed with the proof of one of our main results, namely Theorem 3.3. The key ingredient is Theorem A.7 below, which bounds the second-order path length of OGD in terms of the considered variation measures.

Theorem A.7. Suppose that both players employ OGD with learning rate η ≤ 1 4 L in a time-varying bilinear saddle-point problem, where L := max 1 ≤ t ≤ T ∥ A ( t ) ∥ 2 . Then, for any time horizon T ∈ N ,

Proof of Theorem A.7. First, for any t ≥ 2 we have that ∥ u ( t ) x -m ( t ) x ∥ 2 2 is equal to

where (24) uses the triangle inequality for the norm ∥ · ∥ 2 along with the inequality 2 ab ≤ a 2 + b 2 for any a, b ∈ R ; (25) follows from the definition of the operator norm; and (26) uses the assumption that ∥ A ( t ) ∥ 2 ≤ L and ∥ y ∥ 2 ≤ ∥Y∥ 2 for any y ∈ Y . A similar derivaiton shows that for t ≥ 2 ,

Further, for t = 1 we have that ∥ u (1) x -m (1) x ∥ 2 = ∥ u (1) x ∥ 2 = ∥ -A (1) y (1) ∥ 2 ≤ L ∥Y∥ 2 , and ∥ u (1) y -m (1) y ∥ 2 = ∥ u (1) y ∥ 2 = ∥ ( A (1) ) ⊤ x (1) ∥ 2 ≤ L ∥X∥ 2 . Next, we will use the following simple corollary, which follows similarly to Lemma A.1.

Corollary A.8. For any sequence s ( T ) z := ( z ( t,⋆ ) ) 1 ≤ t ≤ T , the dynamic regret DReg ( T ) z ( s ( T ) z ) := DReg ( T ) x ( s ( T ) x ) + DReg ( T ) y ( s ( T ) y ) can be bounded by

where m ( t ) z := ( m ( t ) x , m ( t ) y ) and u ( t ) z := ( u ( t ) x , u ( t ) y ) for any t ∈ [ [ T ] ] .

As a result, combining (27) and (26) with Corollary A.8 applied for the dynamic regret of both players with respect to the sequence of comparators (( x ( t,⋆ ) , y ( t,⋆ ) )) 1 ≤ t ≤ T yields that DReg ( T ) x ( x (1 ,⋆ ) , . . . , x ( T,⋆ ) ) + DReg ( T ) y ( y (1 ,⋆ ) , . . . , y ( T,⋆ ) ) is upper bounded by

where we used the fact that

for any leaning rate η ≤ 1 4 L . Finally, using the fact that DReg ( T ) x ( x (1 ,⋆ ) , . . . , x ( T,⋆ ) ) + DReg ( T ) y ( y (1 ,⋆ ) , . . . , y ( T,⋆ ) ) ≥ -2 ∑ T t =1 ϵ ( t ) for a suitable sequence of ϵ ( t ) -approximate Nash equilibria (Property 3.1)-one that attains the variation measure V ( T ) ϵ -NE -yields that

where in the above derivation it suffices if the parameter C of V ( T ) ϵ -NE is such that

Thus, rearranging the last displayed inequality concludes the proof.

Next, we refine this theorem in time-varying games in which the deviation of the payoff matrices is bounded by the deviation of the players' strategies, in the following formal sense.

Corollary A.9. Suppose that both players employ OGD with learning rate η ≤ min { 1 4 L , 1 8 W ∥Z∥ } in a time-varying bilinear saddle-point problem, where L := max 1 ≤ t ≤ T ∥ A ( t ) ∥ 2 and V ( T ) A ≤ W 2 ∑ T -1 t =1 ∥ z ( t +1) -z ( t ) ∥ 2 2 , for some parameter W ∈ R > 0 . Then, for any time horizon T ∈ N ,

Proof. Following the proof of Theorem A.7, we have that for any η ≤ 1 4 L ,

Further, for η ≤ 1 8 W ∥Z∥ 2 ,

Thus, we have shown that

and rearranging concludes the proof.

Thus, in such time-varying games it is the first-order variation term, V ( T ) NE , that will drive our convergence bounds.

Now before proving Theorem 3.3, we state the connection between the equilibrium gap and the deviation of the players' strategies ( ∥ z ( t ) -ˆ z ( t ) ∥ 2 + ∥ z ( t ) -ˆ z ( t +1) ∥ 2 ) . In particular, the following claim can be extracted by [2, Claim A.14]. (We caution that we use a slightly different indexing for the secondary sequence (ˆ x ( t ) i ) in the definition of OMD (5) compared to [2].)

Claim A.10. Suppose that the sequences ( x ( t ) ) 1 ≤ t ≤ T and (ˆ x ( t ) ) 1 ≤ t ≤ T +1 are produced by OMD under a G -smooth regularizer 1 -strongly convex with respect to a norm ∥ · ∥ . Then, for any time t ∈ [ [ T ] ] and any x ∈ X ,

We are now ready to prove Theorem 3.3, the precise version of which is stated below.

Theorem A.11 (Detailed version of Theorem 3.3) . Suppose that both players employ OGD with learning rate η = 1 4 L in a time-varying bilinear saddle-point problem, where L := max 1 ≤ t ≤ T ∥ A ( t ) ∥ 2 . Then,

where ( z ( t ) ) 1 ≤ t ≤ T is the sequence of joint strategy profiles produced by OGD .

Proof. Let us fix t ∈ [ [ T ] ] . For convenience, we denote by BR ( t ) x ( x ( t ) ) := max x ∈X {⟨ x , u ( t ) x ⟩} -⟨ x ( t ) , u ( t ) x ⟩ , the best response gap of Player's x strategy x ( t ) ∈ X , and similarly for BR ( t ) y ( y ( t ) ) .

By definition, it holds that EQGAP ( t ) ( z ( t ) ) := max { BR ( t ) x ( x ( t ) ) , BR ( t ) y ( y ( t ) ) } . By Claim A.10, we have that

where (29) follows from Claim A.10 for G = 1 (since the squared Euclidean regularizer ϕ x : x ↦→ 1 2 ∥ x ∥ 2 2 ) is trivially 1 -smooth; (30) uses the fact that η := 1 4 L and ∥ u ( t ) x ∥ 2 = ∥-A ( t ) y ( t ) ∥ 2 ≤ L ∥Y∥ ; and (31) follows from the triangle inequality. A similar derivation shows that

Thus,

where the last bound uses (31) and (32). Combining (33) with Theorem A.7 concludes the proof.

A.1.5 Variation-dependent regret bounds

Here we state for completeness an implication of Theorem 3.3 for deriving variation-dependent regret bounds in time-varying bilinear saddle-point problems; cf . [93].

Corollary A.12. In the setup of Theorem A.7, it holds that

and

Proof. First, applying Lemma A.1 under x (1 ,⋆ ) = · · · = x ( T,⋆ ) , we have

and similarly,

Now, by Theorem A.7 we have

Further,

Combining this bound with (36) and (35) gives the claimed regret bound on Reg ( T ) y , and a similar derivation also gives the claimed bound on Reg ( T ) x .

Hence, selecting optimally the learning rate gives an O (√ 1 + V ( T ) NE + V ( T ) A ) bound on the individual regret of each player; while that optimal value depends on the variation measures, which are not known to the learners, there are techniques that would allow bypassing this [93]. Corollary A.12 can also be readily parameterized in terms of the improved variation measure V ( T ) ϵ -NE .

A.1.6 Meta-learning

We next provide the implication of Theorem 3.3 in the meta-learning setting. We first make a remark regarding the effect of the prediction of OGD to Theorem 3.3, and how that relates to an assumption present in earlier work on this problem [48].

Remark A.13 (Improved predictions) . Throughout Section 3.1, we have considered the standard prediction m ( t ) x := u ( t -1) x = -A ( t -1) y ( t -1) for t ≥ 2 , and similarly for Player y . It is easy to see that using the predictions

for t ≥ 1 (where z (0) := ˆ z (1) ) entirely removes the dependency on V ( T ) A on all our convergence bounds. While such a prediction cannot be implemented in the standard online learning model, there are settings in which we might, for example, know the sequence of matrices in advance; the meta-learning setting offers such examples, and indeed, Harris et al. [48] use the improved prediction of (37).

Proposition A.14 (Meta-learning) . Suppose that both players employ OGD with learning rate η = 1 4 L , where L := max 1 ≤ h ≤ H ∥ A ( h ) ∥ 2 , and the prediction of (37) in a meta-learning bilinear saddle-point problem with H ∈ N games, each repeated for m ∈ N consecutive iterations. Then, for an average game,

iterations suffice to reach an ϵ -approximate Nash equilibrium, where P := 4 L 2 (4 D Z + ∥Z∥ 2 ) 2 D 2 Z , P ′ := 8 L 2 (4 D Z + ∥Z∥ 2 ) 2 D Z , and

The proof is a direct application of Theorem A.11, where we remark that the term depending on V ( T ) A and the term 4 η 2 L 2 ∥Z∥ 2 2 from Theorem A.11 are eliminated because of the improved prediction of Remark A.13. More precisely, if we let m ( h ) ∈ N be the number of iterations required so that the equilibrium gap is at most ϵ at the h -th game, then Theorem A.11 implies that

Solving with respect to 1 H ∑ H h =1 m ( h ) thus yields Proposition A.14.

The first term in the iteration complexity bound (38) vanishes in the meta-learning regime-as the number of games increases H ≫ 1 -while the second term is proportional to V ( H ) NE H , a natural similarity measure; (38) always recovers the m -1 / 2 rate, but offers significant gains if the games as similar, in the sense that V ( H ) NE H ≪ 1 . It is worth noting that, unlike the similarity measure derived by Harris et al. [48], V ( H ) NE H depends on the order of the games. As a result, a natural problem that arises in settings where the sequence of games is known in advance is to optimize the order of the games so as to minimize V ( H ) NE ; the key challenge of course is to minimize V ( H ) NE without actually solving each game since that without defeat the purpose. We further remark that Proposition A.14 can be readily extended even if each game in the meta-learning sequence is not repeated for the same number of iterations.

Another natural question in this context is to compare the similarity metric of Proposition A.14 compared to that obtained by Harris et al. [48]-namely min z ∈Z ∑ H h =1 ∥ z ( h,⋆ ) -z ∥ 2 2 , where we can assume here that z ( h,⋆ ) is the unique Nash equilibrium of the h -th game. Below, we observe that V ( H ) NE can be arbitrarily smaller.

Proposition A.15. There exists a sequence of H games such that V ( H ) NE = O (1) while min z ∈Z ∑ H h =1 ∥ z ( h,⋆ ) -z ∥ 2 2 = Ω( H ) .

Proof. We let X , Y = ∆ 3 . We consider a sequence of H games so that x ( h,⋆ ) = y ( h,⋆ ) for all h ∈ [ [ H ] ] , and the sequence of points ( x ( h,⋆ ) ) 1 ≤ h ≤ H forms a regular H -gon within the relative interior of ∆ 3 with a constant radius. We note that a unique fully-mixed equilibrium (( α, β, γ ) , ( α, β, γ )) ∈ ∆ 3 × ∆ 3 can be obtained by considering the diagonal payoff matrix

It is easy to see that the above sequence of games establishes the claim.

A.1.7 Polymatrix zero-sum games

As we claimed earlier in Section 3.1.1, Theorem 3.3 can be extended to more general time-varying VIs as long as the MVI property holds. For concreteness, here we focus on a multi-player generalization of two-player zero-sum games, namely polymatrix zero-sum games [15]. More precisely, here it is assumed that there is an underlying n -node graph so that each player is uniquely associated with a node in the graph. The utility of each player i ∈ [ [ n ] ] can be expressed as ∑ i ′ ∈N i x ⊤ i A i,i ′ x i ′ , where N i denotes the set of neighbors of Player i and A i,i ′ ∈ R d i × d i ′ . It is also assumed that A ⊤ i,i ′ = -A i ′ ,i , for any edge { i, i ′ } . If z := ( x 1 , . . . , x n ) ∈ Z , the corresponding operator is defined as

It is easy to see that F is monotone , in that ⟨ z -z ′ , F ( z ) -F ( z ′ ) ⟩ ≥ 0 , for any z , z ′ ∈ Z . Now, for any ϵ -approximate Nash equilibrium z ⋆ it holds, by definition, that ⟨ z -z ⋆ , F ( z ⋆ ) ⟩ ≥ -nϵ , for any z ∈ Z . Thus, by monotonicity, ⟨ z -z ⋆ , F ( z ) ⟩ ≥ ⟨ z -z ⋆ , F ( z ⋆ ) ⟩ ≥ -nϵ , for any z ∈ Z . In other words, approximate Nash equilibria approximately satisfy the MVI property. This suffices to directly extend Property 3.1 to time-varying polymatrix zero-sum games. The rest of the argument of Theorem 3.3 is analogous, leading to the following result. Below, we define V ( T ) ϵ -NE as in (2), for a suitable parameter C > 0 , and V ( T ) A := ∑ T -1 t =1 ∑ n i =1 ∑ i ′ ∈N i ′ ∥ A ( t +1) i,i ′ -A ( t ) i,i ′ ∥ 2 2 .

Theorem A.16. Suppose that each player employs OGD with a sufficiently small learning rate in a sequence of time-varying polymatrix zero-sum games. Then, ∑ T t =1 ( EQGAP ( t ) ( z ( t ) ) ) 2 = O ( 1 + V ( T ) ϵ -NE + V ( T ) A ) , where ( z ( t ) ) 1 ≤ t ≤ T is the sequence of joint strategies produced by OGD .

A.1.8 General variational inequalities

Although our main focus in this paper is on the convergence of learning algorithms in time-varying games, our techniques could also be of interest for solving (static) general variational inequality (VI) problems.

In particular, let F : Z → Z be a single-valued operator. Solving general VIs is well-known to be computationally intractable, and so instead focus has been on identifying broad subclasses that elude those intractability barriers (see the references below). Our framework in Section 3.1 motivates introducing the following measure of complexity for a VI problem:

subject to

In words, (39) expresses the infimum first-order variation that a sequence of comparators must have in order to guarantee nonnegative dynamic regret (40); it is evident that (40) always admits a feasible sequence, namely s ( T ) z := ( z ( t ) ) 1 ≤ t ≤ T . We note that, in accordance to our results in Section 3.1, one can also consider an approximate version of the complexity measure (39), which could behave much more favorably (recall Proposition 3.2).

Now in a (static) bilinear saddle-point problem, it holds that C ( F ) = 0 given that there exists a static comparator that guarantees nonnegativity of the dynamic regret. More broadly, our techniques imply O ( poly (1 /ϵ )) iteration-complexity bounds for any VI problem such that C ( F ) ≤ CT 1 -ω , for a time-independent parameter C > 0 and ω ∈ (0 , 1] :

Proposition A.17. Consider a variational inequality problem described with the operator F : Z → Z such that F is L -Lipschitz continuous, in the sense that ∥ F ( z ) -F ( z ′ ) ∥ 2 ≤ L ∥ z -z ′ ∥ 2 , and C ( F ) ≤ CT 1 -ω for C > 0 and ω ∈ (0 , 1] . Then, OGD with learning rate η = 1 4 L reaches an ϵ -strong solution z ⋆ ∈ Z in O ( ϵ -2 /ω ) iterations; that is, ⟨ z -z ⋆ , F ( z ⋆ ) ⟩ ≥ -ϵ for any z ∈ Z .

This result should be viewed as part of an ongoing effort to characterize the class of variational inequalities (VIs) that are amenable to efficient algorithms; see [33, 13, 4, 7, 21, 23, 56, 57, 66, 76, 87, 24], and the many references therein.

It is worth comparing (39) with another natural complexity measure, namely inf z ⋆ ∈Z ∑ T t =1 ⟨ z ( t ) -z ⋆ , F ( z ( t ) ) ⟩ ; the latter measures how negative (external) regret can be, and has already proven useful in certain settings that go bilinear saddle-point problems [88], although unlike (39) it does not appear to be of much use in characterizing time-varying bilinear saddle-point problems. In this context, O ( poly (1 /ϵ )) iteration-complexity bounds can also be established whenever

inf z ⋆ ∈Z ∑ T t =1 ⟨ z ( t ) -z ⋆ , F ( z ( t ) ) ⟩ ≥ -CT 1 -ω for a time-invariant C > 0 , or · inf z ⋆ ∈Z ∑ T t =1 ⟨ z ( t ) -z ⋆ , F ( z ( t ) ) ⟩ ≥ -C ∑ T -1 t =1 ∥ z ( t +1) -z ( t ) ∥ 2 2 , for a sufficiently small C > 0 .

Following [88], identifying VIs that satisfy those relaxed conditions but not the MVI property is an interesting direction. In particular, it is important to understand if those relaxations can shed led more light into the convergence properties of OGD in Shapley's two-player zero-sum stochastic games .

A.2 Proofs from Section 3.2

In this subsection, we provide the proofs from Section 3.2, leading to our main result in Theorem 3.5.

Let us first introduce some additional notation. We let f ( t ) : X × Y → R be a continuously differentiable function for any t ∈ [ [ T ] ] . 6 We recall that in Section 3.2 it is assumed that the objective

function changes only after m ∈ N (consecutive) repetitions, which is akin to the meta-learning setting. Analogously to our setup for bilinear saddle-point problems (Section 3.1), it is assumed that Player x is endeavoring to minimizing the objective function, while Player y is trying to maximize it. We will denote by Reg ( T ) L ,x ( x ⋆ ) := ∑ T t =1 ⟨ x ( t ) -x ⋆ , -u ( t ) x ⟩ and Reg ( T ) L ,y ( y ⋆ ) := ∑ T t =1 ⟨ y ⋆ -y ( t ) , u ( t ) y ⟩ , where u ( t ) x := -∇ x f ( x ( t ) , y ( t ) ) and u ( t ) y := ∇ y f ( x ( t ) , y ( t ) ) for any t ∈ [ [ T ] ] ; similar notation is used for DReg ( T ) L ,x , DReg ( T ) L ,y .

Furthermore, we let s ( T ) z = (( x ( t,⋆ ) , y ( t,⋆ ) )) 1 ≤ t ≤ T , so that x ( t,⋆ ) = x ( h,⋆ ) and y ( t,⋆ ) = y ( h,⋆ ) for any t ∈ [ [ T ] ] such that ⌊ ( t -1) /m ⌋ = h ∈ [ [ H ] ] . The first important step in our analysis is that, following the proof of Lemma A.1,

where ˆ x ( h,k ) := ˆ x (( h -1) × m )+ k ) for any ( h, k ) ∈ [ [ H ] ] × [ [ m ] ] , ˆ x ( h,m +1) := ˆ x ( h +1 , 1) for h ∈ [ [ H -1] ] , and ˆ x ( H,m +1) := ˆ x ( T +1) . Similarly,

Next, we will use the following key observation, which lower bounds the sum of the players' (external) regrets under strong convexity-concavity.

Lemma A.18. Suppose that f : X ×Y → R is a µ -strongly convex-concave function with respect to ∥ · ∥ 2 . Then, for any Nash equilibrium z ⋆ = ( x ⋆ , y ⋆ ) ∈ Z ,

Proof. First, by µ -strong convexity of f ( x , · ) , we have that for any time t ∈ [ [ m ] ] ,

Similarly, by µ -strong concavity of f ( · , y ) , we have that for any time t ∈ [ [ m ] ] ,

Further, for any Nash equilibrium ( x ⋆ , y ⋆ ) ∈ Z it holds that f ( x ( t ) , y ⋆ ) ≥ f ( x ( t ) , y ( t ) ) ≥ f ( x ⋆ , y ( t ) ) . Combining this fact with (43) and (44) and summing over all t ∈ [ [ m ] ] gives the statement.

In turn, this readily implies the following lower bound for the dynamic regret.

Lemma A.19. Suppose that f ( h ) : X × Y → R is a µ -strongly convex-concave function with respect to ∥ · ∥ 2 , for any h ∈ [ [ H ] ] . Consider a sequence s ( T ) z = (( x ( t,⋆ ) , y ( t,⋆ ) )) 1 ≤ t ≤ T , so that x ( t,⋆ ) = x ( h,⋆ ) and y ( t,⋆ ) = y ( h,⋆ ) for any t ∈ [ [ T ] ] such that ⌊ ( t -1) /m ⌋ = h ∈ [ [ H ] ] . If ( x ( h,⋆ ) , y ( h,⋆ ) ) ∈ Z is a Nash equilibrium of f ( h ) ,

where z ( h,k ) := z (( h -1) × m )+ k ) for any ( h, k ) ∈ [ [ H ] ] × [ [ m ] ] .

We next combine this with the following monotonicity property of OGD : If z ⋆ is a Nash equilibrium, ∥ ˆ z ( t ) -z ⋆ ∥ 2 is a decreasing function in t [48, Proposition C.10]. This leads to the following refinement of Lemma A.19.

Lemma A.20. Under the assumptions of Lemma A.19, if additionally η ≤ 1 2 µ ,

where (45) uses that 1 4 η ≥ µ 2 along with Young's inequality and triangle inequality, and (46) follows from [48, Proposition C.10].

Armed with this important lemma, we are ready to establish our main result (Theorem 3.5), the detailed version of which is given below. We first recall that the function f : X × Y → R is said to be L -smooth if ∥ F ( z ) -F ( z ′ ) ∥ 2 ≤ L ∥ z -z ′ ∥ 2 , where F ( z ) := ( ∇ x f ( x , y ) , -∇ y f ( x , y )) .

Theorem A.21 (Detailed version of Theorem 3.5) . Let f ( h ) : X ×Y be a µ -strongly convex-concave and L -smooth function, for h ∈ [ [ H ] ] . Suppose that both players employ OGD with learning rate η = min { 1 8 L , 1 2 µ } for T repetitions, where T = m × H and m ≥ 2 ηµ . Then,

Proof. Combining Lemma A.20 with (41) and (42),

for a sequence of Nash equilibria ( z ( h,⋆ ) ) 1 ≤ h ≤ H , where we used the notation u ( t ) z := ( u ( t ) x , u ( t ) y ) and m ( t ) z := ( m ( t ) x , m ( t ) y ) . Now we bound the first term of the right-hand side above as

where we used the fact that m ≥ 2 ηµ and ˆ z ( h,m +1) = ˆ z ( h +1 , 1) , for h ∈ [ [ H -1] ] . Hence, continuing from above,

since ∥ ˆ z ( h +1 , 1) -z ( h +1 ,⋆ ) ∥ 2 2 ≤ 2 ∥ ˆ z ( h +1 , 1) -z ( h,⋆ ) ∥ 2 2 + 2 ∥ z ( h,⋆ ) -z ( h +1 ,⋆ ) ∥ 2 2 , by the triangle inequality and Young's inequality. Moreover, for t ≥ 2 ,

by L -smoothness. As a result,

and the claimed bound on the second-order path length follows. Finally, the second claim of the theorem follows from Claim A.10 using convexity-concavity, analogously to Theorem 3.3.

We conclude this subsection by pointing out an improved variation-dependent regret bound, which follows directly from Theorem A.21 ( cf . Corollary A.12).

Corollary A.22. In the setup of Theorem A.21, the maximum of the two players' regrets, max { Reg ( T ) L ,x , Reg ( T ) L ,y } , can be upper bounded by

Thus, setting the learning rate optimally implies that Reg ( T ) L ,x , Reg ( T ) L ,y = O (√ 1 + S ( H ) NE + V ( H ) ∇ f ) .

A.3 Proofs from Section 3.3

In this subsection, we provide the proofs from Section 3.3.

A.3.1 Potential games

We first characterize the behavior of (online) gradient descent ( GD ) in time-varying potential games; we recall that GD is equivalent to OGD under the prediction m ( t ) x = 0 for all t . Below we give the formal definition of a potential game.

̸

Definition A.23 (Potential game) . An n -player game admits a potential if there exists a function Φ : × n i =1 X i → R such that for any player i ∈ [ [ n ] ] , any joint strategy profile x -i ∈ × i ′ = i X i ′ , and any pair of strategies x i , x ′ i ∈ X i ,

The key ingredient in the proof of Theorem 3.7 is the following bound on the second-order path length of the dynamics.

Proposition A.24. Suppose that each player employs GD with a sufficiently small learning rate η > 0 and initialization ( x (1) 1 , . . . , x (1) n ) ∈ × n i =1 X i in a sequence of time-varying potential games. Then,

This bound can be derived from [2, Theorem 4.3]. Now, we note that if Φ (1) = Φ (2) = · · · = Φ ( T ) , the right-hand side of (47) telescops, thereby implying that the second-order path-length is bounded. More generally, the right-hand side of (47) can be upper bounded by

where Φ max is an upper bound on | Φ ( t ) ( · ) | for any t ∈ [ [ T ] ] , and V ( T ) Φ is the variation measure of the potential functions we introduced in Section 3.3. Namely, V ( T ) Φ := ∑ T -1 t =1 d (Φ ( t ) , Φ ( t +1) ) , where d : (Φ , Φ ′ ) ↦→ max z ∈ × n i =1 X i (Φ( z ) -Φ ′ ( z )) . Furthermore, similarly to Claim A.10, we know that the Nash equilibrium gap in the t -th potential game can be bounded in terms of ∑ n i =1 ∥ x ( t +1) i -x ( t ) i ∥ 2 . As a result, combining this property with Proposition A.24 and (48) establishes Theorem 3.7, the statement of which is recalled below.

Theorem 3.7. Suppose that each player employs (online) GD with a sufficiently small learning rate in a sequence of time-varying potential games. Then, ∑ T t =1 ( EQGAP ( t ) ( z ( t ) ) ) 2 = O (Φ max + V ( T ) Φ ) , where Φ max is such that | Φ ( t ) ( · ) | ≤ Φ max.

A.3.2 General-sum games

We next turn out attention to general-sum multi-player games in normal form using the well-known bilinear formulation of correlated equilibria (Section 3.3). More precisely, this zero-sum game is played between a mediator , who selects strategies from the set of correlated distributions Ξ := ∆( × n i =1 A i ) , and the n players. Per the usual description of correlated equilibria, a mediator recommends to each player i ∈ [ [ n ] ] an action in A i , and then that player selects an action that could be different from the one recommended by the mediator. As a result, each pure strategy of Player i corresponds to a different mapping from A i to A i . Player i wants to maximize the deviation benefit, while the mediator wants to minimize it. We let ¯ X i be the set of (mixed) strategies of Player i . The identity mapping d i : A i ∋ a i ↦→ a i will be referred to as the direct (or obedient) strategy, as it prescribes following the recommendation of the mediator. As a result, this game can be naturally cast as the bilinear saddle-point problem in the form of (4). By the existence of correlated equilibria [50], it follows that there exists a mediator strategy µ ⋆ ∈ Ξ such that ¯ x i ⊤ A ⊤ i µ ⋆ ≤ 0 , for any player i ∈ [ [ n ] ] and ¯ x i ∈ ¯ X i ; in words, there is no benefit for any player to deviate from the recommendation of the mediator.

Now, to establish Property 3.8, let us first define the regret of any player i ∈ [ [ n ] ] as

where ¯ x ⋆ i ∈ ¯ X i , so that ∑ n i =1 Reg ( T ) i is easily seen to be equal to the regret of Player max in (4). Further, the dynamic regret of the mediator-Player min in (4)-can be expressed as

The key idea in the proof of Property 3.8 is that the time-invariant direct strategy ( d 1 , . . . , d n ) suffices to guarantee Property 3.1 as long as the mediator is selecting a sequence of CE.

Property 3.8. Suppose that Ξ ∋ µ ( t,⋆ ) is a correlated equilibrium of the game at any time t ∈ [ [ T ] ] . Then, DReg ( T ) µ ( µ (1 ,⋆ ) , . . . , µ ( T,⋆ ) ) + ∑ n i =1 Reg ( T ) i ≥ 0 .

Proof. We have that

Now for any correlated equilibrium µ ( t,⋆ ) of the t -th game, we have that ⟨ µ ( t,⋆ ) , A ( t ) i ¯ x ( t ) i ⟩ ≤ 0 for any player i ∈ [ [ n ] ] , ¯ x i ∈ ¯ X i , and time t ∈ [ [ T ] ] , which in turn implies that -∑ T t =1 ⟨ µ ( t,⋆ ) , ∑ n i =1 A ( t ) i ¯ x ( t ) i ⟩ ≥ 0 . Moreover, ∑ n i =1 max ¯ x ⋆ i ∈ ¯ X i ∑ T t =1 ⟨ ¯ x ⋆ i , ( A ( t ) i ) ⊤ µ ( t ) ⟩ ≥ ∑ n i =1 ∑ T t =1 ⟨ d i , ( A ( t ) i ) ⊤ µ ( t ) ⟩ = 0 , by definition of the direct strategy d i ∈ ¯ X i . This concludes the proof.

Property 3.8 in conjunction with the argument of Theorem 3.3 readily imply Theorem 3.9. We should note here that the variation measure V ( T ) A := ∑ n i =1 ∑ T -1 t =1 ∥ A ( t +1) i -A ( t ) i ∥ 2 2 is also immediately bounded in terms of the second-order variation of the sum of the players' utility tensors. It is also worth noting that the description of the bilinear formulation (4) grows exponentially with the number of players; this is unavoidable in explicitly represented (normal-form) games, but it is worth investigating whether more compact representations exist in succinct classes of games, such as multi-player polymatrix.

Next, we provide the main implication of Theorem 3.9 in the meta-learning setting, which is similar to the meta-learning guarantee of Proposition A.14 we established earlier in two-player zero-sum games. Below, we denote by Ξ ( h,⋆ ) the set of correlated equilibria of the h -th game in the meta-learning sequence.

Corollary A.25 (Meta-learning in general games) . Suppose that each player employs OGD in (4) with a suitable learning rate η > 0 and the prediction of (37) in a meta-learning general-sum problem with H ∈ N games, each repeated for m ∈ N consecutive iterations. Then, for an average game,

iterations suffice so that the mediator reaches an ϵ -approximate correlated equilibrium, where

In particular, in the meta-learning regime, H ≫ 1 , the iteration-complexity bound (49) is dominated by the (algorithm-independent) similarity metric of the correlated equilibria V ( H ) CE H . Corollary A.25 establishes significant gains when V ( H ) CE H ≪ 1 .

Finally, we conclude this subsection by providing a variation-dependent regret bound in general-sum multi-player games. To do so, we combine Corollary A.12 with Theorem 3.9, leading to the following guarantee.

Corollary A.26 (Regret in general-sum games) . In the setup of Theorem 3.9, for any player i ∈ [ [ n ] ] .

In particular, if one selects optimally the learning rate, Corollary A.26 implies that the individual regret of each player is bounded by O (√ 1 + V ( T ) CE + V ( T ) A ) . Wenote again that there are techniques that would allow (nearly) recovering such regret guarantees without having to know the variation measures in advance [93].

A.4 Proofs from Section 3.4

Finally, in this subsection we present the proofs omitted from Section 3.4. We begin with Proposition 3.10, the statement of which is recalled below. We first recall that a regularizer ϕ x , 1 -strongly convex with respect to a norm ∥ · ∥ , is said to be G -smooth if ∥∇ ϕ x ( x ) -∇ ϕ x ( x ′ ) ∥ ∗ ≤ G ∥ x -x ′ ∥ , for all x , x ′ .

Proposition 3.10. Suppose that both players in a (static) two-player zero-sum game employ OMD with a smooth regularizer. Then, DReg ( T ) x , DReg ( T ) y = O ( √ T ) .

Proof. First, using Claim A.10, it follows that the dynamic regret DReg ( T ) x of Player x up to time T can be bounded as

where G > 0 is the smoothness parameter of the regularizer, and η > 0 is the learning rate. We further know that ∑ T t =1 ( ∥ x ( t ) -ˆ x ( t ) ∥ 2 + ∥ x ( t ) -ˆ x ( t +1) ∥ 2 ) = O (1) for any instance of OMD in a twoplayer zero-sum game [2], which in turn implies that ∑ T t =1 ( ∥ x ( t ) -ˆ x ( t ) ∥ + ∥ x ( t ) -ˆ x ( t +1) ∥ ) = O ( √ T ) by Cauchy-Schwarz. Thus, combining with (50) we have shown that DReg ( T ) x = O ( √ T ) . Similar reasoning yields that DReg ( T ) y = O ( √ T ) , concluding the proof.

Let us next make a certain refinement of Observation 3.11. Staying on (static) bilinear saddle-point problems, we consider the unconstrained setting where X = R d x and Y = R d y , and we focus on the following update rule for τ ∈ N .

where x (0) , x ( -1) ∈ X and y (0) , y ( -1) ∈ Y . This is an instance of optimistic follow the regularized leader ( OFTRL ) [78], although our discussion here also covers the corresponding OMD variant. Then, summing (51) for τ = 1 , 2 , . . . , t we have

in turn implying that

Above, we define ¯ x ( t ) := 1 t ∑ t τ =1 x ( τ ) and ¯ y ( t ) := 1 t ∑ t τ =1 y ( τ ) . That is, the update rule (52) describes the evolution of the time average of (51). As a result, this implies that (52) converges in a last-iterate sense with a T -1 rate, simply because OFTRL incurs O (1) regret [78]. This is conceptually interesting as it suggests a simple way to analyze the convergence of the last iterate solely through a regret-based analysis. In fact, (52) is a variant of the so-called Halpern's iteration [32]-a classical technique in optimization-that incorporates optimism; a similar update rule was recently shown to exhibit a new form of acceleration by Yoon and Ryu [90], and has engendered considerable interest ever since. Returning to Observation 3.11, one can use (52) to guarantee O (log T ) dynamic regret in the standard feedback model, although it is not clear how to extend this argument in the constrained setting ( cf. [14]).

General-sum games We next extend our scope beyond bilinear saddle-point problems. We first observe that obtaining sublinear dynamic regret is precluded in general-sum games. In particular, we note that the computational-hardness result below (Proposition 3.12) holds beyond the online learning setting. It should be stressed that, at least in normal-form games, without imposing computational or memory restrictions there are trivial online algorithms that guarantee even O (1) dynamic regret by first exploring the payoff tensors and then computing a Nash equilibrium [26]. We suspect that under the memory limitations imposed by Daskalakis et al. [26] there could be unconditional information-theoretic lower bounds, but that is left for future work.

Proposition 3.12. Any polynomial-time algorithm incurs ∑ n i =1 DReg ( T ) i ≥ CT for any polynomial T ∈ N , even if n = 2 and C > 0 is an absolute constant, unless ETH for PPAD is false [73].

Proof. We will make use of the fact that computing a Nash equilibrium in two-player (normalform) games to a sufficiently small accuracy ϵ = O (1) requires superpolynomial time, unless the exponential-time hypothesis for PPAD fails [73]. Indeed, suppose that there exist polynomial-time algorithms that always guarantee that ∑ n i =1 DReg ( T ) i ≤ ϵT for some polynomial T ∈ N , where n := 2 . Then, this implies that there exists a time t ∈ [ [ T ] ] such that

which in turn implies that ( x ( t ) 1 , x ( t ) 2 ) is an ϵ -approximate Nash equilibrium. Further, such a time t ∈ [ [ T ] ] can be identified in polynomial time since T ≤ poly ( |A 1 | , |A 2 | ) . This concludes the proof.

Wealso note the following computational hardness result, which rests on the more standard complexity assumption that PPAD is not contained in P . As in Proposition 3.12, below we tacitly assume that the underlying algorithm is deterministic.

Proposition A.27. Unless PPAD ⊆ P , no polynomial-time algorithm guarantees ∑ n i =1 DReg ( T ) i ≤ poly ( |A 1 | , |A 2 | ) T 1 -ω for all T ∈ N , where ω ∈ (0 , 1] is an absolute constant, even if n = 2 .

The proof follows directly from the PPAD -hardness result of Chen et al. [19]. Of course, if we do not operate in the online learning model, it is computationally trivial to come up with algorithms that guarantee that one of the two players will have 0 dynamic regret by simply best responding to the strategy of the other player.

Finally, we provide the proof of Theorem 3.13, the detailed version of which is provided below.

Theorem A.28 (Detailed version of Theorem 3.13) . Consider an n -player game such that ∥∇ x i u i ( z ) -∇ x i u i ( z ′ ) ∥ 2 ≤ L ∥ z -z ′ ∥ 2 , where z , z ′ ∈ × n i =1 X i , for any player i ∈ [ [ n ] ] . Then, if all players employ OGD with learning rate η > 0 it holds that

̸

Proof. First, applying Lemma A.1 subject to the constraint that ∑ T -1 t =1 ✶ { x ( t +1 ,⋆ ) i = x ( t,⋆ ) i } ≤ K -1 gives that for any Player i ∈ [ [ n ] ] ,

Further, by L -smoothness we have that

for any t ∈ [ [ T -1] ] , where ( x ( t ) 1 , . . . , x ( t ) n ) = z ( t ) ∈ × n i =1 X i is the joint strategy profile at time t . Thus, summing (53) over all i ∈ [ [ n ] ] and taking η ≤ 1 2 L √ n implies that ∑ n i =1 K -DReg ( T ) i ≤ 2 K -1 2 η ∑ n i =1 D 2 X i + η ∑ n i =1 ∥ u (1) i ∥ 2 2 , yielding the first part of the statement since D 2 Z = ∑ n i =1 D 2 X i , where we recall the notation Z := × n i =1 X i . The second part follows directly from (53) using the stability property of OGD : ∥ x ( t +1) i -x ( t ) i ∥ 2 = O ( η ) , for any time t ∈ [ [ T -1] ] .

Remark A.29 . It is not hard to show that Item 2 above can be improved to O K,T ( K ) using clairvoyant mirror descent ( CMD ) [68], but under a stronger feedback model. In particular, using the (squared) Euclidean regularizer it is direct to show (see [39, 65]) that the dynamic regret of CMD is bounded solely by the first-order variation of the sequence of comparators, which suffices for this claim. It is open whether similar results can be obtained in the standard feedback model.

B Experimental examples

Finally, although the focus of this paper is theoretical, in this section we provide some illustrative experimental examples. In particular, Appendix B.1 contains experiments on time-varying potential games, while Appendix B.2 focuses on time-varying (two-player) zero-sum games. For simplicity, we will be assuming that each game is represented in normal form.

B.1 Time-varying potential games

Here we consider time-varying 2 -player identical-interest games. We point out that such games are potential games (recall Definition A.23), and as such, they are indeed amenable to our theory in Section 3.3.

In our first experiment, we first sampled two matrices A , P ∈ R d x × d y , where d x = d y = 1000 . Then, we defined each payoff matrix as A ( t ) := A ( t -1) + P t -α for t ≥ 1 , where A (0) := A . Here, α > 0 is a parameter that controls the variation of the payoff matrices. In this time-varying setup, we let each player employ (online) GD with learning rate η := 0 . 1 . The results obtained under different random initializations of matrices A and P are illustrated in Figure 1.

Next, we operate in the same time-varying setup but each player is now employing multiplicative weights update ( MWU ), instead of gradient descent, with η := 0 . 1 . As shown in Figure 2, while the cumulative equilibrium gap is much larger compared to using GD (Figure 1), the dynamics still appear to be approaching equilibria, although our theory does not cover MWU . We suspect that theoretical results such as Theorem 3.7 should hold for MWU as well, but that has been left for future work.

In our third experiment for identical-interest games, we again first sampled two matrices A , P ∈ R d x × d y , where d x = d y = 1000 . Then, we defined A ( t ) := A ( t -1) + ϵ P for t ≥ 1 , where A (0) := A . Here, ϵ > 0 is the parameter intended to capture the variation of the payoff matrices. The results obtained under different random initializations of A and P are illustrated in Figure 3. As an aside, it is worth pointing out that this particular setting can be thought of as a game in which the variation in the payoff matrices is controlled by another learning agent. In particular, our theoretical results could be helpful for characterizing the convergence properties of two-timescale learning algorithms, in which the deviation of the game is controlled by a player constrained to be updating its strategies with a much smaller learning rate.

B.2 Time-varying zero-sum games

We next conduct experiments on time-varying bilinear saddle-point problems when players are employing OGD . Such problems were studied extensively earlier in Section 3.1 from a theoretical standpoint.

First, we sampled two matrices A , P ∈ R d x × d y , where d x = d y = 10 ; here we consider lowerdimensional payoff matrices compared to the experiments in Appendix B.1 for convenience in the graphical illustrations. Then, we defined each payoff matrix as A ( t ) := A ( t -1) + P t -α for t ≥ 1 , where A (1) := A . The results obtained under different random initializations are illustrated in Figure 4.

0

50

100

150

200

0

2

4

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

0

.

0

0

.

2

0

.

4

0

.

6

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

50

100

150

200

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

0

1

2

3

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

0

.

0

0

.

2

0

.

4

0

.

6

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

50

100

150

200

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

0

1

2

3

4

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

50

100

150

200

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

Iteration (t)

0

2

4

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

Iteration (t)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

50

100

150

200

Iteration (t)

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

0

100

200

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

0

2

4

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

20

40

60

80

100

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

0

100

200

300

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

0

2

4

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

20

40

60

80

100

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

0

100

200

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

0

2

4

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

20

40

60

80

100

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

50

100

150

200

Iterations

0

100

200

300

∑

t

τ

=1

(EG

(

τ

)

)

2

0

50

100

150

200

Iterations

0

2

4

EG

(

t

)

α

= 0.5

α

= 0.2

α

= 0.1

0

20

40

60

80

100

Iterations

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

100

200

300

400

500

0

.

0

0

.

5

1

.

0

∑

t

τ

=1

(EG

(

τ

)

)

2

0

100

200

300

400

500

0

.

0

0

.

1

0

.

2

EG

(

t

)

/epsilon1

= 0.001

/epsilon1

= 0.01

/epsilon1

= 0.1

0

50

100

150

200

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

100

200

300

400

500

0

.

0

0

.

5

1

.

0

∑

t

τ

=1

(EG

(

τ

)

)

2

0

100

200

300

400

500

0

.

0

0

.

1

0

.

2

0

.

3

EG

(

t

)

/epsilon1

= 0.001

/epsilon1

= 0.01

/epsilon1

= 0.1

0

50

100

150

200

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

100

200

300

400

500

0

1

2

∑

t

τ

=1

(EG

(

τ

)

)

2

0

100

200

300

400

500

0

.

0

0

.

1

0

.

2

0

.

3

EG

(

t

)

/epsilon1

= 0.001

/epsilon1

= 0.01

/epsilon1

= 0.1

0

50

100

150

200

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

100

200

300

400

500

Iterations

0

1

2

3

∑

t

τ

=1

(EG

(

τ

)

)

2

0

100

200

300

400

500

Iterations

0

.

0

0

.

1

0

.

2

EG

(

t

)

/epsilon1

= 0.001

/epsilon1

= 0.01

/epsilon1

= 0.1

0

50

100

150

200

Iterations

0

2

4

6

max(Reg

(

t

)

x

(

t

)

y

)

0

200

400

600

800

1000

0

10000

20000

30000

∑

t

τ

=1

(EG

(

τ

)

)

2

0

200

400

600

800

1000

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

EG

(

t

)

α

= 2

α

= 1

α

= 0.7

0

200

400

600

800

1000

0

5

10

15

20

max(Reg

(

t

)

x

(

t

)

y

)

0

200

400

600

800

1000

0

2500

5000

7500

10000

∑

t

τ

=1

(EG

(

τ

)

)

2

0

200

400

600

800

1000

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

EG

(

t

)

α

= 2

α

= 1

α

= 0.7

0

200

400

600

800

1000

0

10

20

30

40

max(Reg

(

t

)

x

(

t

)

y

)

0

200

400

600

800

1000

0

20000

40000

∑

t

τ

=1

(EG

(

τ

)

)

2

0

200

400

600

800

1000

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

EG

(

t

)

α

= 2

α

= 1

α

= 0.7

0

200

400

600

800

1000

0

10

20

30

max(Reg

(

t

)

x

(

t

)

y

)

0

200

400

600

800

1000

Iteration (t)

0

2500

5000

7500

10000

∑

t

τ

=1

(EG

(

τ

)

)

2

0

200

400

600

800

1000

Iteration (t)

0

.

0

0

.

2

0

.

4

0

.

6

EG

(

t

)

α

= 2

α

= 1

α

= 0.7

0

200

400

600

800

1000

Iteration (t)

0

10

20

max(Reg

(

t

)

x

(

t

)

y

)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of online learning in time-varying games under different setups. Specifically, the authors consider the case where all the players apply optimistic gradient descent (OGD) algorithm with a certain choice of learning rate. The main results that in the two-player bilinear game setup, the sum of squared duality gap is bounded by $O(1+V_{\\epsilon-NE}+V_A)$, recovering the best-known result in the stationary game setup. This result is based on the bounded second-order path length of the learning dynamic and the important observation that sum of dynamic regret with respect to (approximated) Nash is (almost) non-negative. Next, they extend this result to the strongly convex-concave game with multiple steps and obtain similar duality gap guarantees. The author also consider the potential game and general-sum game setup with guarantees on the duality gap and CEgap bound respectively. Experiments are also done to support their theoretical results.\n\nSTRENGTHS:\n- The problem considered in this paper is important and the authors show that the classic OGD algorithm achieves desirable average duality-gap and other equilibrium-related gap bound with provable guarantees.\n- The authors also do experiments on time-varying potential games and time-varying zero-sum games to verify their obtained NEgap bound.\n\nWEAKNESSES:\nThe main concern is the novelty of this paper. Specifically, this paper shares similarity to the results in [60] in many perspectives, although the authors explain in many places in the paper that how their results are different from [60].\n- One of the main lemma Lemma A.1 is the same as the DRVU property shown in [60].\n- Property A.3 is very similar to Eq.(32) shown in [60].\n- Example shown in Proposition 3.2 is almost the same as the example shown in Appendix C, case 2 in [60].\n- From the technical perspective, I feel like the analysis is very similar to the one shown in [60]. While [60] does not consider the approximated NE path-length, it is not hard to extend their analysis to the approximated NE path-length by replacing P_T with \\epsilon-NE path-length +T\\epsilon. Also, the boundedness of the second-order dynamic is also shown in Lemma 16/18 in [60].\n- In addition, as mentioned by the authors, there are parameter-tuning issues in achieving better individual regret guarantees in bilinear and  strongly convex-concave games, which is also handled by the meta-base structure proposed in [60].\n\nThe authors also derive results for general-sum games and potential games. Given Property 3.8 that the sum of regret with respect to CE is positive, it seems that the average CEgap bound is also not very hard to obtain given the bounded second-order dynamic of OGD.\n\nQUESTIONS:\n- As mentioned in the above section, can the authors explain more in detail on the main technical difficulties and challenges compared to the analysis shown in [60]. \n- In Property 3.1, the authors argue that this extends the exact NE path-length in [60] to the approximated NE path-length. In [60], they also provide a bound with respect to W_T, which is the variance of the game matrix. Can the authors explain more on the comparison between the epsilon-NE path-length and W_T? Is there a case where both the exact NE path-length and the W_T are large but the epsilon-NE path-length is very small?\n\nLIMITATIONS:\nSee Weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors consider no-regret learning in multiagent games where the underlying game varies across different rounds. They study several classes of games and various learning algorithms that the agents can use. Naturally, the results they obtained are parametrized by variation measures of the underlying game that the agents participate in. To be more precise:\n* For time varying zero-sum games, they focus on the setting where both of the agents are using optimistic gradient descent (OGD),  which is a variant of gradient descent that puts a bias on more recent rounds of the game. Interestingly, they show that almost all iterates of OGD are approximate Nash equilibria provided that some variation measures related to changes in the set of approximate equilibria of the games and the underlying payoff matrices are $o(T)$.\n* Then, they consider sequences of games where the games in each round are strongly convex-concave. For this class of games, they are able to show a similar result as above, but under weaker variation conditions for the underlying sequence of games.\n* Finally, they consider time-varying general-sum games. Naturally, since Nash equilibria are not tractable in this setting, they consider convergence to correlated equilibria. They prove similar results as above, but now the variation measure they use is related to the set of correlated equilibria of the game.\nTheir results have implications to other settings as well such as meta-learning and dynamic regret guarantees in static games.\n\nSTRENGTHS:\n* The paper studies a very natural problem and provides strong results under various settings of interest, which also have implications in other settings, as I mentioned in the summary.\n* For the most part, the paper is easy to follow and the authors have done a good job placing their work in the literature. \n* The authors are not trying to oversell the proof technique, which is heavily inspired by prior works, but uses some natural and clever modifications. For example, instead of letting the variation measure to depend on variations of the set of exact Nash equilibria which would make the problem very difficult to handle since this set is very sensitive to any changes of the payoff matrices, the authors consider variations of the set of approximate Nash equilibria, which behaves much nicer. Since the results are strong and general, I don't think that the authors should be penalized for the fact that the proof techniques are not very novel.\n\nWEAKNESSES:\n* Some parts of the paper might be a bit hard to follow for non-experts, especially in Section 1.1. For example, the MVI property and the RVU bound were not defined. I think the authors could make the transition to this section a bit smoother, although I understand that the space limitations are making it trickier.\n* Even though the variation measures the authors use are intuitive and it makes sense that the regret should scale with these quantities, there are no lower bounds to show the extent to which these results are optimal.\n\n\nSome minor comments:\n* In Proposition 3.10, it might be useful to state which dynamic benchmark you consider for their dynamic regret bound.\n* With this bibliography style it is a bit hard to keep track of the references, although I understand that it saves some valuable space.\n\nQUESTIONS:\n* What are the technical challenges to generalize the results to the bandit or some other partial feedback model?\n* Would there be any benefit if you considered a variational measure wrt approximate correlated equilibria in general sum games instead of exact correlated equilibria?\n* To what extent do you think that the results are tight?\n* Do the results for two-player zero-sum games generalize to multiagent zero-sum polymatrix games? I don't see any inherent obstacles to do that using your approach, but I might be missing something.\n* Another class of general sum games that is tractable in the single-shot setting are games in which the underlying matrices are rank-1. Is there any hope to obtain similar results for this class of games? I would imagine that the techniques would need to be substantially different from your approach.\n* In Theorem 3.3 (and similar results) if the parameter $L$ is not known does the usual guess-and-double trick work to get the bound?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning dynamics in games that change over time. This is a similar setting to [60], but while [60] focuses on regret guarantees, this paper focuses on iterate convergence to NE.\n\nThe main result states that for bilinear zero-sum games, running optimistic OGD guarantees that,\n$$\n\\sum_{t=1}^T EqGap_t = O(V_{NE-\\epsilon}^T + V_A^T),\n$$\nwhere $EqGap_t$ is the NE gap (i.e., the difference between the player utility and best response), $V_{NE-\\epsilon}^T$ is the variation of $\\epsilon$-approximate NEs of the games $+\\epsilon T$ (in fact they allow different $\\epsilon$ for each $T$), and $V_A^T$ measures the variation of the game matrices. $V_{NE-\\epsilon}^T$ can be much smaller compared to the variation of the exact NEs.\n\nThe paper continues by providing variation-dependent bound for strongly convex games. Next, they provide a bound on the sum of NE gaps for general sum-potential games (which depends on some notion of variation of the potential function), as well as bound on the sum of CE gaps in general games. Finally, they present results for *dynamic* regret in static games.\n\nSTRENGTHS:\nThe paper provides a set of interesting results. These include,\n- Various results on the sum of NE gaps. Specifically, I find the notion of $V_{NE-\\epsilon}^T$ very elegant, and indeed, it seems like a much more reasonable notion for characterizing the complexity of time-varying-games instance.\n- Bounds on the dynamic regret in static games - even though this is a basic question, according to the authors these are the first results that show $\\sqrt{T}$ dynamic regret in games (they also show $\\log T$ dynamic regret under a stronger feedback model)\n\nWEAKNESSES:\n- The main text lacks proofs/proof sketches. So it is impossible to understand the main ideas and techniques, even at a high level, without diving into the full technical proofs in the appendix (+ it makes it hard to evaluate the technical contribution of the paper).\n\n- In several places, it is quite hard to follow the text. Specifically, section 1.1 is rather technical, given that it is part of the intro. In addition, the section on general games that starts in line 287 was not sufficiently clear to me. For example, the authors mention that *\"there exist matrices\n$A_1, . . . , A_n$, with each matrix $A_i$ depending solely on the utility of Player i...\"* but what are exactly these matrices represent? What does the value of the optimization problem in (3) represents? Why does *\"incorporating the 0 vector will be useful\"*? and why does there exist $\\mu^\\star$ that satisfy the condition in line 300?\n\n- Lack of comparison to previous work: what is the result of [30] and how is it compars to your result in \"meta-learning\"? How does Corollary 3.4(2) compares to the result in [60] (except for the difference between $V_{NE-\\epsilon}^T$ and $V_{NE}^T$)\n\nQUESTIONS:\nWhat is the difference between the sum NE gaps (Corollary 3.4) and dynamic regret (line 342)?\n\nSee additional questions in the Weaknesses part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies optimistic gradient descent for time-varying games.  Authors prove convergence bounds for zero sum games involving the first order variation of approximate nash equilibrium, which can be significantly tigher than variation bounds involving exact nash equilibria and second order bounds in payoff matrices. The paper also includes refined convergence bounds involving second-order variation for strongly-convex-concave games.  The results also have implications for meta-learning, where games are repeated many times.\n\nThe authors also extend results to time-varying general-sum multiplayer games with correlated equilibria, extending exisiting meta-learning similarity measures to general sum games and proving new single-player regret bounds. Techniques are applied to static games, improving our understanding of dynamic regret.\n\nSTRENGTHS:\nNonstationarity is an important and challenging area of study for learning in games, and is under-explored.\n\nBounds involving approximate nash equilibrium variation can be significantly tighter than existing bounds involving variation in exact nash equilibria.\n\nThe results for general sum games solving two independent open problems.\n\nIdeas provide improved understanding of dynamic regret for static games, including both positive and negative result.\n\nThe paper is well written, including a variety of results while still providing technical exposition on insights behind the proofs and contextualizing the result.\n\nWEAKNESSES:\nObservation 3.11 requires two point feedback, so it's less clear this is a significant improvement.\n\nThe paper could probably be a bit more self contained.  The paper borrows ideas from [60], like a dynamic RVU bound but it is hard to follow without additional context.\n\nQUESTIONS:\nTypo 88: accelerates-> accelerated\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}