Title: No-regret Learning in Harmonic Games: Extrapolation in the Face of Conflicting Interests
Venue: NeurIPS 2024
Accepted: yes
Keywords: learning in games

no-regret algorithms

FTRL

harmonic games

Poincaré recurrence
Abstract: The long-run behavior of multi-agent online learning -- and, in particular, no-regret learning -- is relatively well-understood in potential games, where players have common interests. By contrast, in general harmonic games -- the strategic complement of potential games, where players have competing interests -- very little is known outside the narrow subclass of $2$-player zero-sum games with a fully-mixed equilibrium. Our paper seeks to partially fill this gap by focusing on the full class of (generalized) harmonic games and examining the convergence properties of "follow-the-regularized-leader" (FTRL), the most widely studied class of no-regret learning schemes. As a first result, we show that the continuous-time dynamics of FTRL are Poincaré recurrent, i.e., they return arbitrarily close to their starting point infinitely often, and hence fail to converge. In discrete time, the standard, "vanilla" implementation of FTRL may lead to even worse outcomes, eventually trapping the players in a perpetual cycle of best-responses. However, if FTRL is augmented with a suitable extrapolation step -- which includes as special cases the optimistic and mirror-prox variants of FTRL -- we show that learning converges to a Nash equilibrium from any initial condition, and all players are guaranteed at most $\mathcal{O}(1)$ regret. These results provide an in-depth understanding of no-regret learning in harmonic games, nesting prior work on $2$-player zero-sum games, and showing at a high level that potential and harmonic games are complementary not only from the strategic but also from the dynamic viewpoint.

No-Regret Learning in Harmonic Games: Extrapolation in the Face of Conflicting Interests

Abstract

The long-run behavior of multi-agent learning - and, in particular, no-regret learning - is relatively well-understood in potential games, where players have aligned interests. By contrast, in harmonic games - the strategic counterpart of potential games, where players have conflicting interests - very little is known outside the narrow subclass of 2 -player zero-sum games with a fully-mixed equilibrium. Our paper seeks to partially fill this gap by focusing on the full class of (generalized) harmonic games and examining the convergence properties of follow-the-regularizedleader (FTRL), the most widely studied class of no-regret learning schemes. As a first result, we show that the continuous-time dynamics of FTRL are Poincaré recurrent , that is, they return arbitrarily close to their starting point infinitely often, and hence fail to converge. In discrete time, the standard, 'vanilla' implementation of FTRL may lead to even worse outcomes, eventually trapping the players in a perpetual cycle of best-responses. However, if FTRL is augmented with a suitable extrapolation step - which includes as special cases the optimistic and mirror-prox variants of FTRL - we show that learning converges to a Nash equilibrium from any initial condition, and all players are guaranteed at most O ( 1 ) regret. These results provide an in-depth understanding of no-regret learning in harmonic games, nesting prior work on 2 -player zero-sum games, and showing at a high level that harmonic games are the canonical complement of potential games, not only from a strategic, but also from a dynamic viewpoint.

1 Introduction

The question of 'as if' rationality - that is, whether selfishly-minded, myopic agents may learn to behave ' as if ' they were fully rational - has been one of the cornerstones of non-cooperative game theory, and for good reason. Especially in modern applications of game theory to machine learning and data science - from online ad auctions to recommender systems and multi-agent reinforcement

Georgios Piliouras

learning - the standard postulates of rationality (knowledge of the game, capacity to compute an equilibrium, flawless execution of equilibrium strategies, common knowledge of rationality, etc.) are almost never met in practice; as a result, game-theoretic predictions that rely on these assumptions are likewise put into question. By contrast, given the ease of implementing and deploying cheap, computationally efficient learning algorithms and policies at a large scale, it is often more logical to turn to the policy being deployed as the object of interest. The aim is then to understand its long-run behavior - and, in particular, whether it ultimately leads to equilibrium.

A major obstacle in this approach is the complexity of computing a Nash equilibrium, a problem which is known to be complete for PPAD - and hence intractable - by the seminal work of Daskalakis et al. [12]. This result implies that it is not plausible to expect any algorithm to converge to Nash equilibrium in all games (at least, not in a reasonable amount of time), so it dovetails naturally with the impossibility results of Hart & Mas-Colell [22, 23] who showed that there are no uncoupled learning dynamics that converge to Nash equilibrium in all games. On that account, it is natural to ask in which classes of games we can expect a learning algorithm to converge, in which classes we cannot, and under what conditions.

Perhaps the most well-behaved class of games in terms of learning is the class of potential games [44, 55], where players have common interests - not necessarily driving them to play the same strategy, but in the sense that externalities are symmetric and aligned along a common objective (the potential of the game). In this class of games, the behavior of learning dynamics - and, in particular, no-regret learning [8, 13, 19, 27, 28, 36, 40, 43, 59] - are relatively well understood, and there is a wide range of equilibrium convergence results, from continuous to discrete time, and even with bandit, payoff-based feedback [24, 25, 55].

By contrast, in the presence of conflicting interests, the situation can be quite different. In two-player zero-sum games with a fully-mixed equilibrium - such as Matching Pennies - the continuous-time dynamics of no-regret, regularized learning are recurrent in the sense of Poincaré - that is, the induced trajectory of play returns arbitrarily close to where it started infinitely many times [41, 48]. In discrete time, the situation becomes more complicated: the vanilla version of follow-the-regularizedleader (FTRL) - the most widely studied family of no-regret algorithms - is no longer recurrent, but it diverges away from equilibrium in the same class of games [18, 42]. On the other hand, if players employ an optimistic / extra-gradient variant of FTRL, the induced trajectory of play converges to equilibrium [15, 42] and, under certain conditions, it is even possible to show that it converges at a geometric rate [62].

At the same time, zero-sum games may also admit a potential function, so it is not possible to predict the outcome of a learning process based on where it stands along the potential / zero-sum axis. The non-trivial intersection of these classes means that potential and zero-sum games are not complementary, and this, not only from a strategic, but also from a dynamic viewpoint. Instead, the true strategic complement of potential games is the class of harmonic games . This class was first considered by Candogan et al. [6], who established a remarkable decomposition result: Every game in normal form can be decomposed as the sum of a potential game and a harmonic game, and this decomposition is unique up to affine transformations that do not alter the equilibrium outcomes of the game. In particular, the class of potential and harmonic games intersect trivially (up to strategic equivalence), and all two-player zero-sum games with an interior equilibrium are harmonic, thus lending credence to the fact that it is harmonic games, not zero-sum games, that correctly capture the notion of conflicting interests in this context.

This raises the question:

What is the behavior of no-regret algorithms and dynamics in harmonic games?

Except for a very recent paper by Legacci et al. [35] (which we discuss below), almost nothing is known on this question. With this backdrop, our contributions can be summarized as follows:

Starting with a continuous-time model of no-regret learning, we show that all FTRL dynamics are Poincaré recurrent in all harmonic games. This generalizes and extends the recent result of Legacci et al. [35] for the replicator dynamics in uniform harmonic games (a subclass of harmonic games in which the uniform distribution is always a Nash equilibrium). 2

In discrete-time models of learning, the standard implementation of FTRL cannot be expected to converge (since it fails to do so in Matching Pennies). To correct this behavior, we consider a flexible algorithmic template, inspired by Azizian et al. [3] and dubbed extrapolated FTRL (FTRL+), which augments FTRL with a forward-looking, extrapolation step (including as special cases the optimistic and extra-step variants of FTRL, cf. Section 4). We then establish the following results:

(a) Under extrapolated FTRL, players are guaranteed constant individual regret (so, as a consequence, the players' empirical frequency of play converges to coarse correlated equilibrium at a rate of O ( 1 / 𝑇 ) ). 3 This should be contrasted with the results of [13, 14] who showed that players can achieve polylogarithmic regret in any game (finite or convex).

(b) The induced trajectory of play converges to Nash equilibrium from any initial condition.

Our results aim to provide an in-depth understanding of no-regret learning in harmonic games, nesting prior work on 2 -player zero-sum games - from Poincaré recurrence [41, 48] to constant regret [27] and convergence under optimistic / extra-gradient schemes [11, 15, 18, 42, 62]. In partiucular, at a high level, our results show that harmonic games are the canonical complement of potential games, not only from a strategic, but also from a dynamic, learning viewpoint.

2 Preliminaries

2.1. Preliminaries on finite games. Throughout the sequel, we will work with finite games in normal form . Formally, such games consist of ( i ) a finite set of players 𝑖 ∈ N ≡ { 1 , . . . , 𝑁 } ; ( ii ) a finite set of actions A 𝑖 per player 𝑖 ∈ N ; and ( iii ) an ensemble of payoff functions 𝑢 𝑖 : Q 𝑗 A 𝑗 → ℝ , each determining the reward 𝑢 𝑖 ( 𝛼 ) of player 𝑖 ∈ N in a given action profile 𝛼 = ( 𝛼 1 , . . . , 𝛼 𝑁 ) . Putting everything together, we will write A : = Q 𝑖 A 𝑖 for the game's action space and Γ ≡ Γ ( N , A , 𝑢 ) for the game with primitives as above.

During play, each player selects an action according to some mixed strategy , that is, a probability distribution 𝑥 𝑖 over A 𝑖 which assigns probability 𝑥 𝑖 𝛼 𝑖 to 𝛼 𝑖 ∈ A 𝑖 . In a slight abuse of notation, if 𝑥 𝑖 assigns all probability mass to some action 𝛼 𝑖 ∈ A 𝑖 (that is, 𝑥 𝑖 𝛼 𝑖 = 1 ), we will identify 𝑥 𝑖 with 𝛼 𝑖 and we will call it pure . We will also write X 𝑖 : = Δ ( A 𝑖 ) ⊆ ℝ A 𝑖 for the mixed strategy space of player 𝑖 , 𝑥 = ( 𝑥 1 , . . . , 𝑥 𝑁 ) for the strategy profile collecting the strategies of all players, and X : = Q 𝑖 X 𝑖 for the game's strategy space .

The mixed payoff of player 𝑖 under a mixed strategy profile 𝑥 ∈ X may then be written as

where 𝑥 𝛼 : = Q 𝑖 𝑥 𝑖 𝛼 𝑖 denotes the joint probability of 𝛼 = ( 𝛼 1 , . . . , 𝛼 𝑁 ) ∈ A under 𝑥 ∈ X , and, in standard game-theoretic notation, we write ( 𝑥 𝑖 ; 𝑥 -𝑖 ) = ( 𝑥 1 , . . . , 𝑥 𝑖 , . . . , 𝑥 𝑁 ) for the profile where player 𝑖 plays 𝑥 𝑖 ∈ X 𝑖 against the strategy 𝑥 -𝑖 ∈ X -𝑖 : = Q 𝑗 ≠ 𝑖 X 𝑗 of all other players. We also respectively define the individual payoff field of player 𝑖 and the game's payoff field as

so 𝑢 𝑖 ( 𝑥 ) = P 𝛼 𝑖 ∈ A 𝑖 𝑣 𝑖 𝛼 𝑖 ( 𝑥 ) 𝑥 𝑖 𝛼 𝑖 ≡ ⟨ 𝑣 𝑖 ( 𝑥 ) , 𝑥 𝑖 ⟩ , where ⟨· , ·⟩ is the standard duality pairing on ℝ A 𝑖 . By multilinearity, each player's individual payoff field is Lipschitz continuous on X , and we will write 𝐺 𝑖 for its Lipschitz modulus, that is

Remark. In the above and throughout, ∥·∥ denotes an ambient norm on ℝ A 𝑖 (usually the 𝐿 1 norm), and ∥·∥∗ is the corresponding dual norm (usually the 𝐿 ∞ norm). To simplify notation, we will not carry the player index 𝑖 in ∥·∥ , and we will instead rely on the context to resolve any ambiguities.

In terms of solution concepts, we will focus almost exclusively on the notion of a Nash equilibrium (NE), i.e., a strategy profile 𝑥 ∗ ∈ X that is unilaterally stable in the sense that

Equivalently, (NE) can be expressed in terms of the game's payoff field as a variational inequality of the form

Thus, writing supp ( 𝑥 ∗ 𝑖 ) = { 𝛼 𝑖 ∈ A 𝑖 : 𝑥 𝑖 𝛼 𝑖 > 0 } for the support of 𝑥 ∗ 𝑖 , it follows that 𝑥 ∗ is a Nash equilibrium if and only if 𝑢 𝑖 ( 𝛼 𝑖 ; 𝑥 ∗ -𝑖 ) ≥ 𝑢 𝑖 ( 𝛽 𝑖 ; 𝑥 ∗ -𝑖 ) for all 𝛼 𝑖 ∈ supp ( 𝑥 ∗ 𝑖 ) and all 𝛽 𝑖 ∈ A 𝑖 , 𝑖 ∈ N . We will use all this freely in the rest of our paper.

2.2. Harmonic games. Our main focus in what follows will be the class of harmonic games , first introduced by Candogan et al. [6] as a game-theoretic model for strategic situations with conflicting, anti-aligned interests. Specifically, as was shown by Candogan et al. [6] - and, in a more general setting, by Abdou et al. [1] - every game in normal form can be decomposed as the sum of a potential game and a harmonic game, and this decomposition is unique up to affine transformations that do not alter the equilibrium outcomes of the game. 4 In this decomposition, the potential component of a game captures multi-agent strategic interactions with common interests, whereas the harmonic component covers interactions with conflicting interests. 5

Formally, adapting the more general setup by Abdou et al. [1], we have the following definition:

Definition 1. A finite game Γ ≡ Γ ( N , A , 𝑢 ) is said to be harmonic when it admits a harmonic measure , i.e., a collection of weights 𝜇 𝑖 𝛼 𝑖 ∈ ( 0 , ∞) , 𝛼 𝑖 ∈ A 𝑖 , 𝑖 ∈ N , such that

In particular, if Γ is harmonic relative to the uniform measure 𝜇 𝑖 𝛼 𝑖 = 1 , 𝛼 𝑖 ∈ A 𝑖 , 𝑖 ∈ N , we will say that Γ is a uniform harmonic game (UHG).

Remark. With regard to terminology, Candogan et al. [6] call 'harmonic games' what we call 'uniform harmonic games', and Abdou et al. [1] call ' 𝜇 -harmonic games' what we call 'harmonic games'. 6 We use this convention because it simultaneously simplifies notation and terminology while capturing all relevant strategic features of the game; for a detailed discussion, see Appendix A. To avoid needless repetition, and unless there is a danger of confusion, when we say that Γ is harmonic, we will write 𝜇 𝑖 for the corresponding measure, and we will write 𝑚 𝑖 = | 𝜇 𝑖 | = P 𝛽 𝑖 ∈ A 𝑖 𝜇 𝑖𝛽 𝑖 for the total mass of 𝜇 𝑖 . ⋄

Broadly speaking, in harmonic games, for any player considering a deviation toward a specific pure strategy profile, there exist other players with an incentive to deviate away from said profile. In this regard, harmonic games can be seen as the strategic complement of potential games, where player interests are aligned and sequences of unilateral best responses generate a finite improvement path that terminates at a pure Nash equilibrium [44]. By contrast, except for trivial cases (like the zero game) harmonic games do not admit pure Nash equilibria, and they possess non-terminating best-response paths. For all these reasons, harmonic games can be considered as 'orthogonal' to potential games, in a sense made precise by the decomposition results of Candogan et al. [6] and Abdou et al. [1].

It is of course natural to ask what is the relation between harmonic games and zero-sum games. Games belonging to the latter class - such as Matching Pennies and Rock-Paper-Scissors - have long been used as prototypical examples of strategic conflict; at the same time, there are zero-sum games that are also potential (and even possess strict equilibria), so the potential / zero-sum distinction does not capture the whole picture. As a matter of fact, it is not a coincidence that the textbook examples of zero-sum games admit fully-mixed Nash equilibria: as we discuss in Appendix A, two-player zero-sum games with a fully mixed Nash equilibrium are harmonic, so the existing results for such games are, in a sense, more closely attuned to their harmonic character.

3 Continuous-time analysis: Poincaré recurrence

The most basic rationality postulate in the context of online learning is the minimization of a player's (external) regret, i.e., the difference between a player's cumulative payoff and that of the player's best possible strategy in hindsight. In more detail, assuming for the moment that play evolves in continuous time, the regret of player 𝑖 ∈ N relative to a sequence of play 𝑥 ( 𝑡 ) ∈ X is defined as

and we say that the player has no regret under 𝑥 ( 𝑡 ) if Reg 𝑖 ( 𝑇 ) = 𝑜 ( 𝑇 ) as 𝑇 →∞ .

The most widely used scheme for attaining no regret is the family of policies known as follow-theregularized-leader (FTRL) [57, 58]. At a high level, the idea behind FTRL is that, at all times 𝑡 ≥ 0 , each player 𝑖 ∈ N plays a mixed strategy 𝑥 𝑖 ( 𝑡 ) ∈ X 𝑖 that maximizes the player's cumulative payoff up to time 𝑡 minus a certain regularization penalty. In our continuous-time setting, this gives rise to the FTRL dynamics

or, more compactly,

where ℎ 𝑖 : X 𝑖 → ℝ is a convex penalty function known as the regularizer of the method, 𝑄 𝑖 denotes the regularized choice map of player 𝑖 , and 𝑄 = ( 𝑄 1 , . . . , 𝑄 𝑁 ) denotes the profile thereof. Formally, writing Y 𝑖 ≡ ℝ 𝐴 𝑖 for the payoff space of player 𝑖 ∈ N - that is, the space of all possible payoff vectors 𝑣 𝑖 of player 𝑖 - the regularized choice map 𝑄 𝑖 : Y 𝑖 → X 𝑖 is defined as

In essence, 𝑄 𝑖 is a 'soft' version of the arg max correspondence 𝑦 𝑖 ↦→ arg max 𝑥 𝑖 ∈ X 𝑖 ⟨ 𝑦 𝑖 , 𝑥 𝑖 ⟩ , suitably regularized by a penalty term intended to incentivize exploration. For technical reasons, we will also assume that each ℎ 𝑖 is strongly convex , i.e.,

for some 𝐾 𝑖 > 0 (commonly referred to as the strong convexity modulus of ℎ 𝑖 ), and for all 𝑥 𝑖 , 𝑥 ′ 𝑖 ∈ X , 𝑡 ∈ [ 0 , 1 ] . In plain words, this simply means that ℎ 𝑖 has 'enough curvature' in the sense that it can be bounded from below by a (positive) quadratic function which agrees with ℎ 𝑖 to first order.

The go-to example of this setup is the entropic regularizer

which yields the so-called logit choice map

By Pinsker's inequality, the entropic regularizer is 1 -strongly convex relative to the 𝐿 1 -norm on X 𝑖 [57], and by a standard calculation [37, 54], the induced sytem (FTRL-D) boils down to the replicator dynamics of Taylor & Jonker [60]. Some other standard examples of (FTRL-D) include the Euclidean projection dynamics of Friedman [17] when ℎ 𝑖 ( 𝑥 𝑖 ) = ( 1 / 2 )∥ 𝑥 𝑖 ∥ 2 2 , the 𝑞 -replicator dynamics [21, 38], etc. To streamline our presentation, we defer a detailed discussion of these examples to Appendix C, and we proceed below to state the main regret guarantee of (FTRL-D), originally due to [33]:

Theorem 1. Under (FTRL-D) , each player's regret is bounded as Reg 𝑖 ( 𝑇 ) ≤ 𝐻 𝑖 : = max ℎ 𝑖 -min ℎ 𝑖 .

Theorem 1 showcases the strong no-regret properties of (FTRL-D): it is not possible to guarantee less than constant, O ( 1 ) regret, so (FTRL-D) is optimal in this regard. In turn, by standard results [47], Theorem 1 implies further that the players' (correlated) empirical frequencies 𝑧 𝛼 1 ,..., 𝛼 𝑁 ( 𝑡 ) : = ( 1 / 𝑡 ) ∫ 𝑡 0 Q 𝑖 𝑥 𝑖 𝛼 𝑖 ( 𝜏 ) 𝑑𝜏 converge to the game's set of coarse correlated equilibria (CCE) at a rate of O ( 1 / 𝑡 ) .

Importantly, this result makes no assumptions about the underlying game, but it does not carry the same predictive power in all games: for one thing, a game's set of CCE may include highly non-rationalizable outcomes (such as dominated strategies and the like) [61]; for another, the timeaveraging that is inherent in the definition of empirical distributions may conceal a wide range of non-convergence phenomena, from cycles to chaos [48, 56]. On that account, the day-to-day behavior of (FTRL-D) in harmonic games cannot be understood from Theorem 1 alone, and requires a closer, more in-depth look.

Our first result below provides such a lense and shows that (FTRL-D) is almost-periodic in harmonic games, a property known as Poincaré recurrence.

Theorem 2. Suppose Γ is harmonic. Then almost every orbit 𝑥 ( 𝑡 ) of (FTRL-D) returns arbitrarily close to its starting point infinitely often: specifically, for ( Lebesgue ) almost every initial condition 𝑥 ( 0 ) = 𝑄 ( 𝑦 ( 0 )) ∈ X , there exists an increasing sequence of times 𝑡 𝑛 ↑ ∞ such that 𝑥 ( 𝑡 𝑛 ) → 𝑥 ( 0 ) .

An immediate consequence of Theorem 2 is that no-regret learning under (FTRL-D) fails to converge in any harmonic game; in particular, since the orbits of (FTRL-D) eventually return to (almost) where they started, it is debatable if the players have learned anything at all, despite the fact that they incur at most constant regret. This cyclic, non-convergent landscape is the polar opposite of the long-run behavior of (FTRL-D) in potential games, where the dynamics are known to converge globally [24]. Thus, in addition to the strategic viewpoint of the previous section, Theorem 2 shows that harmonic games are orthogonal to potential games also from a dynamic viewpoint.

Theorem 2 also provides a far-reaching generalization of existing results on Poincaré recurrence in (possibly networked) two-player zero-sum games with an interior equilibrium [41] to general-sum, 𝑁 -player games. Combined with our previous remark, and given that the zero-sum property is not as meaningful for 𝑁 players as it is for two, 7 the class of harmonic games can be seen as the more natural 𝑁 -player generalization of two-player zero-sum games from a learning viewpoint.

To the best of our knowledge, the only comparable result to Theorem 2 in the literature is the very recent paper of Legacci et al. [35] who showed that the replicator dynamics - a special case of (FTRL-D) - are Poincaré recurrent in uniform harmonic games, that is, in harmonic games where the uniform distribution is a Nash equilibrium, cf. (A.1) and the discussion surrounding Definition 1. In this regard, Theorem 2 extends the recent results of Legacci et al. [35] along two axes: ( i ) it applies to the entire class of FTRL dynamics (not only the replicator dynamics); and ( ii ) it applies to the entire class of harmonic games (and not only uniformly harmonic games).

In terms of techniques, Legacci et al. [35] obtained their result through a surprising connection between a certain Riemannian metric underlying the replicator dynamics and the defining relation of uniformly harmonic games. This relation no longer holds for different instances of (FTRL-D) or for non-uniform harmonic games, so the techniques of [35] cannot be extended - and, in fact, Legacci et al. [35] stated this generalization as an open problem. Our techniques instead rely on the fact that the orbits 𝑦 ( 𝑡 ) of (FTRL-D) comprise a volume-preserving flow in the game's payoff space Y ≡ Q 𝑖 Y 𝑖 (though not necessarily on X ), and then deriving a suitable constant of motion. In the case of the logit map (9), this constant of motion can be written as

where 𝜇 = ( 𝜇 𝑖 𝛼 𝑖 ) 𝛼 𝑖 ∈ A 𝑖 ,𝑖 ∈ N is the harmonic measure on X defining Γ . In the more general case, the construction of a constant of motion for (FTRL-D) involves a characterization of harmonic games in terms of a 'strategic center', which we carry out in detail in Appendix C.

4 Discrete-time analysis: Convergence and constant regret via extrapolation

We now proceed to examine the regret and convergence properties of regularized learning algorithms in harmonic games. Starting with the standard, vanilla implementation of FTRL, we reproduce a well-known observation that FTRL spirals out to a non-terminating cycle of best-responses in Matching Pennies (which is a harmonic game). Subsequently, to correct this non-convergent behavior, we examine a flexible algorithmic template, which we call extrapolated FTRL (FTRL+), and which includes as special cases the optimistic and extra-gradient versions of FTRL.

4.1. Vanilla implementation of FTRL. Building on the discussion of the previous section, the standard implementation of FTRL in discrete time for 𝑛 = 1 , 2 , . . . is

or, in more compact, iterative notation

where, as per (6), the map 𝑄 𝑖 : Y 𝑖 → X 𝑖 denotes the regularized choice map of player 𝑖 ∈ N , 𝜆 𝑖 is a player-specific regularization weight parameter, and 𝜂 𝑖 = 1 / 𝜆 𝑖 represents the learning rate of player 𝑖 . Apart from their obvious differences - discrete vs. continuous time - a salient point that sets (FTRL) apart from (FTRL-D) is the inclusion of the parameter 𝜂 𝑖 ; this parameter is necessary to control the algorithm's behavior, and we will discuss it in detail in the sequel.

As mentioned in the introduction, a major shortfall of (FTRL) - and one of the main reasons for the increased popularity of optimistic / extra-gradient methods - is that it may spiral away from Nash equilibrium, even in simple 2 × 2 games with a unique equilibrium. The standard example of this behavior is Matching Pennies, a two-player zero-sum game with a fully-mixed equilibrium which is also uniformly harmonic, so the trajectories of (FTRL-D) are Poincaré recurrent (and, in fact, periodic). In more detail, this game can be compactly represented by the payoff field 𝑣 ( 𝑥 1 , 𝑥 2 ) = ( 4 𝑥 2 -2 , 2 -4 𝑥 1 ) for 𝑥 1 , 𝑥 2 ∈ [ 0 , 1 ] , and its unique Nash equilibrium is 𝑥 ∗ = ( 1 / 2 , 1 / 2 ) . Thus, if we run (FTRL) with a Euclidean reqularizer - that is, ℎ 𝑖 ( 𝑥 𝑖 ) = 𝑥 2 𝑖 / 2 for 𝑖 = 1 , 2 - and the same learning rate 𝜂 for both players, a straightforward calculation shows that the distance 𝐷 𝑛 = ( 𝑥 1 ,𝑛 -𝑥 ∗ 1 ) 2 / 2 + ( 𝑥 2 ,𝑛 -𝑥 ∗ 2 ) 2 / 2 between 𝑥 𝑛 and 𝑥 ∗ evolves as

as long as 𝑥 𝑛 + 𝜂𝑣 ( 𝑥 𝑛 ) ∈ X . In other words, the distance of the iterates of (FTRL) from the game's equilibrium grows at a geometric rate until 𝑥 𝑛 reaches the boundary of X and is ultimately trapped in a non-terminating cycle of best responses, cf. Fig. 1. In this regard, the rationality properties of (FTRL) are even worse than those of (FTRL-D) because the game's equilibrium is now repelling .

4.2. Extrapolated FTRL. To mitigate this undesirable, divergent behavior of (FTRL), a standard approach in the literature is the inclusion of a forward-looking, ' extrapolation step '. Instead of updating the algorithm's 'base state' 𝑥 𝑛 directly, players first move to an interim 'leading state' 𝑥 𝑛 + 1 / 2 using payoff information from 𝑥 𝑛 (this is the extrapolation step); subsequently, players update 𝑥 𝑛 using payoff information from the leading state 𝑥 𝑛 + 1 / 2 , and the process repeats. In this way, players attempt to anticipate their payoff landscape and, in so doing, to take a more informed update step at each iteration.

The seed of this idea goes back to Korpelevich [32] and Popov [49] in the context of solving monotone variational inequality problems, and it has since percolated to a wide array of ' extra-gradient ' or ' optimistic ' methods, such as the mirror-prox algorithm of Nemirovski [45], the dual extrapolation variant of Nesterov [46], the optimistic mirror descent algorithm of Chiang et al. [9] and Rakhlin & Sridharan [50], and many others. Given the different operational envelope of each of these methods, we consider below an integrated algorithmic template, which we call extrapolated FTRL (FTRL+), and which is sufficiently flexible to account for a broad range of these schemes.

Formally, the proposed algorithmic blueprint unfolds in two phases as follows:

𝑎 )

𝑏 )

In the above, 𝜂 𝑖 > 0 is the learning rate of player 𝑖 , 𝑥 𝑛 and 𝑥 𝑛 + 1 / 2 denote respectively the method's base and leading states at stage 𝑛 = 1 , 2 , . . . , and ˆ 𝑣 𝑖,𝑛 and ˆ 𝑣 𝑖,𝑛 + 1 / 2 are sequences of 'black-box' payoff models at 𝑥 𝑛 and 𝑥 𝑛 + 1 / 2 respectively.

Specifically, following Azizian et al. [3], we will assume throughout that

i.e., players always update the base state 𝑥 𝑛 using payoff information from the leading state 𝑥 𝑛 + 1 / 2 . By contrast, the leading state 𝑥 𝑛 + 1 / 2 can be generated in many different ways, depending on the targeted update structure. In this regard, we will consider the linear model

where the player-specific coefficients 𝑎 𝑖 , 𝑏 𝑖 ≥ 0 satisfy 𝑎 𝑖 + 𝑏 𝑖 ≤ 1 and represent a mix of past and present payoff information. In this way, depending on the values of 𝑎 𝑖 and 𝑏 𝑖 , we obtain the following prototypical regularized learning methods as special cases of (FTRL+):

a ) FTRL: if 𝑎 𝑖 = 𝑏 𝑖 = 0 for all 𝑖 ∈ N , players essentially forego any look-ahead efforts, so we get

In turn, this gives 𝑥 𝑛 + 1 / 2 = 𝑥 𝑛 , i.e., (FTRL+) regresses to (FTRL).

b ) Extra-Step FTRL: if 𝑎 𝑖 = 1 and 𝑏 𝑖 = 0 for all 𝑖 ∈ N , we have

i.e., players use payoff information from their current state to generate the leading state 𝑥 𝑛 + 1 / 2 . This update structure requires two payoff queries per iteration and its origins can be traced back to the work of Korpelevich [32]. Specifically, depending on the choice of ℎ 𝑖 , it is essentially equivalent to the mirror-prox [45] and dual extrapolation [46] algorithms, it contains as a special case the forward-looking algorithm of [15, 42], etc.

c ) Optimistic FTRL: if 𝑎 𝑖 = 0 and 𝑏 𝑖 = 1 for all 𝑖 ∈ N , we have

i.e., players reuse the latest available payoff information instead of making a fresh query at 𝑥 𝑛 (so the algorithm only requires one payoff query per iteration). In this way, (FTRL+) recovers the optimistic algorithms of [9, 26, 50, 59], the OMW update scheme of [11, 59] when 𝑄 = Λ , etc.

Clearly, the list above is not exhaustive: many other configurations are possible, e.g., with different players using different parameter settings for 𝑎 𝑖 and 𝑏 𝑖 , depending on the information they have at hand and any other individual considerations. To avoid needlessly complicating the analysis, our only standing assumption will be that 𝑎 𝑖 + 𝑏 𝑖 > 0 for all 𝑖 ∈ N (since, otherwise, the benefits of the extrapolation step would vanish). In particular, by rescaling the players' learning rates if needed, we will normalize 𝑎 𝑖 and 𝑏 𝑖 to 𝑎 𝑖 + 𝑏 𝑖 = 1 , leading to the convex model

for some arbitrarily chosen ensemble of player-specific extrapolation coefficients 𝜆 𝑖 ∈ [ 0 , 1 ] , 𝑖 ∈ N . Remark. To simplify the presentation of our results, we will assume throughout the rest of our paper that (FTRL+) is initialized with 𝑦 1 = 𝑦 1 / 2 = 0 .

4.3. Analysis & results. With all this in hand, we are finally in a position to state our main results for (FTRL+) in harmonic games. We begin by showing that (FTRL+) achieves order-optimal regret:

Theorem 3. Suppose that each player in a harmonic game Γ is following (FTRL+) with learning rate 𝜂 𝑖 ≤ 𝑚 𝑖 𝐾 𝑖 [ 2 ( 𝑁 + 2 ) max 𝑗 𝑚 𝑗 𝐺 𝑗 ] -1 and payoff models as per (13a) and (15) . Then the individual regret of each player 𝑖 ∈ N is bounded as

where 𝐻 𝑖 = max ℎ 𝑖 -min ℎ 𝑖 , and 𝐺 𝑖 is the Lipschitz modulus of 𝑣 𝑖 .

Even though Theorem 3 invites a natural comparison with the constant regret bound of Theorem 1, the continuous- and discrete-time settings are fundamentally different, so any conclusions drawn from such a comparison would be specious. Indeed, constant regret guarantees in the spirit of (16) are particularly rare in the context of discrete-time algorithms, and as far as we are aware, similar bounds have only been established for optimistic methods in variationally stable and two-player zero-sum games [27]; other than that - and always to the best of our knowledge - the tightest regret bounds available for general games (finite or convex) seem to be (poly)logarithmic [13, 14]. In this regard, just like the recurrence result of Theorem 2, the O ( 1 ) regret bound of Theorem 3 represents a significant extension of existing results on zero-sum games (and polylogarithmic regret in general games), and suggests that, from a learning viewpoint, harmonic games are the most natural generalization of two-player zero-sum games to a general 𝑁 -player context. We defer the proof of Theorem 3 to Appendix D.

0.0

0.2

0.4

0.6

0.8

1.0

Strategy first player

0.0

0.2

0.4

0.6

0.8

1.0

Strategy second player

(-2.0)

(-2.0)

(-1.0)

(-1.0)

(0.0)

(0.0)

(1.0)

(1.0)

(2.0)

(2.0)

(-29,23,-8)

(-6,-9,-6)

（-6,-9,-6)

(24,0,0)

(0,4,5)

（-29,23,-8)

（0,4,5)

(24,0,0)

(2,-3,4)

(7,-15,-8)

(7,-15,-8)

(2,-3,4)

(1,-10,1)

(7,2,-6)

(7,2, 6)

(1, 10, 1)

As an immediate corollary of the above, we conclude that, under (FTRL+), the empirical frequencies of play 𝑧 𝛼,𝑛 : = ( 1 / 𝑛 ) P 𝑛 𝑘 = 1 𝑥 𝛼,𝑘 , 𝛼 ∈ A , converge to the game's set of CCE at a rate of O ( 1 / 𝑛 ) . This rate is, again, optimal, but as we discussed in Section 3, it offers little information in games where the marginalization of CCE does not lead to Nash equilibrium - and, in general 𝑁 -player harmonic games, there is little hope that it would. In addition, even when the marginalization of CCE is Nash, the actual trajectory of play may - and, in fact, often does - behave very differently from the time-averaged frequency of play.

Despite these hurdles, we show below that (FTRL+) does converge to Nash equilibrium. To state this result formally, we will focus on the case where each player's regularizer is smooth in the sense that

for all 𝑥 𝑖 ∈ im 𝑄 𝑖 and all 𝑥 ′ 𝑖 ∈ X 𝑖 . 8 Our prototypical examples - the entropic and Euclidean regularizers - both satisfy this mild requirement, as do all regularizers of the form ℎ 𝑖 ( 𝑥 𝑖 ) = P 𝛼 𝑖 ∈ A 𝑖 𝜃 𝑖 ( 𝑥 𝑖 𝛼 𝑖 ) for some smooth convex function 𝜃 𝑖 : [ 0 , 1 ] → ℝ . We then have the following convergence result:

Theorem 4. Suppose that each player in a harmonic game Γ follows (FTRL+) with learning rate 𝜂 𝑖 ≤ 𝑚 𝑖 𝐾 𝑖 [ 2 ( 𝑁 + 2 ) max 𝑗 𝑚 𝑗 𝐺 𝑗 ] -1 and payoff models as per (13a) and (15) . Then 𝑥 𝑛 converges to the set of Nash equilibria of Γ .

To the best of our knowledge, Theorem 4 is the first result of its kind for harmonic games - and, in that regard, it is somewhat unexpected. To be sure, two-player zero-sum games with a fully-mixed equilibrium exhibit a comparable pattern: FTRL is Poincaré recurrent in continuous time, its vanilla discretization is unstable, and its optimistic / forward-looking implementation is convergent. However, the convex-concave structure of min-max games which enables this analysis is completely absent in harmonic games, so it is less clear what to expect in this case (where even the set of Nash equilibria is non-convex, cf. Fig. 1). By this token, the convergence of (FTRL+) in harmonic games is a property that one could optimistically hope for, but not one that can be taken for granted.

From a technical standpoint, the proof of Theorems 3 and 4 involves two concurrent challenges:

Deriving a Lyapunov function with a 'sufficient descent' property for all harmonic games.

Providing an integrated analysis for all possible update structures in (FTRL+).

With regard to the first point, our analysis hinges on the 'energy function'

In the above, 𝑝 ∈ X is a benchmark strategy profile acting as a 'reference point' for the analysis while

denotes the Fenchel coupling associated to the regularizer ℎ 𝑖 of player 𝑖 ∈ N , and represents a 'primal-dual' measure of divergence between 𝑝 𝑖 ∈ X 𝑖 and 𝑦 𝑖 ∈ Y 𝑖 (for an in-depth discussion, see Appendices B and D). Then, letting 𝐸 𝑛 = 𝐸 ( 𝑝, 𝑦 𝑛 ) , the heavy lifting for our analysis is provided by the 'template inequality'

A first important consequence of (20) is that the sequences 𝐴 𝑛 = ∥ 𝑥 𝑛 + 1 -𝑥 𝑛 + 1 / 2 ∥ 2 and 𝐵 𝑛 = ∥ 𝑥 𝑛 + 1 / 2 -𝑥 𝑛 ∥ 2 are both summable: this requires a repeated use of the Fenchel-Young inequality, and an instantiation of 𝑝 to the strategic center 𝑞 of Γ ; we detail the relevant arguments in Appendices A and D. Then, by establishing a similar template inequality for each player 𝑖 ∈ N , we are able to bound the players' individual regret by the same upper bound that we derived for P 𝑛 𝐴 𝑛 and P 𝑛 𝐵 𝑛 , and which is (up to certain secondary factors) the bound (16).

For the convergence to Nash equilibrium, the summability argument above also plays a crucial role. First, by a standard result on numerical sequences, the summability of 𝐴 𝑛 and 𝐵 𝑛 coupled with the template inequality (20) implies that the energy 𝐸 𝑛 of the algorithm relative to the game's strategic center converges to some limit value 𝐸 ∞ . In turn, this implies that the score sequence 𝑦 𝑛 is bounded up to a multiple of the vector ( 1 , . . . , 1 ) , which corresponds to a constant payoff shift in the underlying game. Then, by focusing on convergent subsequences of 𝑦 𝑛 and the optimality condition resulting from the definition of 𝑄 , we are able to show that any limit point of 𝑣 ( 𝑥 𝑛 ) satisfies the variational characterization (VI) of Nash equilibria, from which our claim follows.

5 Concluding remarks

Our results suggest that the long-run behavior of no-regret algorithms and dynamics in harmonic games is a very rich topic, and one which opens the door to an entirely new class of games where positive convergence results can be obtained. We find this particularly appealing, not only because harmonic games comprise the strategic complement of potential games, but also because they go beyond standard problems with a convex structure - for instance, even their equilibrium set is non-convex. As such, the fact that it is possible to obtain optimal regret guarantees and positive equilibrium convergence results in this setting is very promising for future work on the topic.

In terms of open questions, it would be important to examine the rate of convergence of (FTRL+) to equilibrium. Even though (FTRL+) has order-optimal regret bounds, this only helps in establishing a convergence rate to the game's set of coarse correlated equilibria; for Nash equilibria, earlier work by Golowich et al. [19] and some more recent results by Cai et al. [5] and Gorbunov et al. [20] have shed some light on the convergence of constrained Euclidean optimistic methods, but the technology therein does not extend to non-monotone, non-Euclidean problems. Inspired by Wei et al. [62], we conjecture that the convergence rate of (FTRL+) in harmonic games is linear: this is based on the observation that any harmonic game admits a fully-mixed Nash equilibrium, and the weighted sum in the definition of a harmonic game looks formally similar to the condition needed to establish metric subregularity in [62]; however, a proof would likely require different techniques.

Another important research direction has to do with the information available to the players. A first open question here concerns the case where players do not have access to full information on their mixed payoff vectors, but can only observe their pure payoffs - either in a 'what if', counterfactual manner, or in the form of bandit, payoff-based feedback. In a similar manner, the algorithms presented here are not adaptive, in the sense that the players' step-size policy has to satisfy a certain bound that depends on correctly estimating some of the game's parameters. Obtaining an adaptive version of (FTRL+) which, in the spirit of Rakhlin & Sridharan [50] and Hsieh et al. [27, 28, 29], remains convergent and attains order-optimal regret in both adversarial and game-theoretic settings without any pre-play tuning is also an ambitious question for future research.

Acknowledgments and Disclosure of Funding

This research was supported in part by the French National Research Agency (ANR) in the framework of the PEPR IA FOUNDRY project (ANR-23-PEIA-0003), the 'Investissements d'avenir program' (ANR-15-IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), MIAI@Grenoble Alpes (ANR-19-P3IA-0003), the project IRGA2024-SPICE-G7H-IRG24E90, and NSF grant CCF2212233. PM is also with the Archimedes Research Unit/Athena RC & NKUA and acknowledges financial support by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.

References

Abdou, J., Pnevmatikos, N., Scarsini, M., and Venel, X. Decomposition of games: Some strategic considerations. Mathematics of Operations Research , 47(1):176-208, February 2022.

Arnold, V. I. Mathematical Methods of Classical Mechanics . Springer-Verlag, 1989.

Azizian, W., Iutzeler, F., Malick, J., and Mertikopoulos, P. The rate of convergence of Bregman proximal methods: Local geometry vs. regularity vs. sharpness. SIAM Journal on Optimization , 34(3):2440-2471, September 2024.

Bertsekas, D. P. Convex optimization algorithms . Athena Scientific, 2015.

Cai, Y., Oikonomou, A., and Zheng, W. Tight last-iterate convergence of the extragradient and the optimistic gradient descent-ascent algorithm for constrained monotone variational inequalities. 2022.

Candogan, O., Menache, I., Ozdaglar, A., and Parrilo, P. A. Flows and decompositions of games: Harmonic and potential games. Mathematics of Operations Research , 36(3):474-503, 2011.

Chen, G. and Teboulle, M. Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM Journal on Optimization , 3(3):538-543, August 1993.

Chen, X. and Peng, B. Hedging in games: Faster convergence of external and swap regrets. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 18990-18999. Curran Associates, Inc., 2020.

Chiang, C.-K., Yang, T., Lee, C.-J., Mahdavi, M., Lu, C.-J., Jin, R., and Zhu, S. Online optimization with gradual variations. In COLT '12: Proceedings of the 25th Annual Conference on Learning Theory , 2012.

Combettes, P. L. Quasi-Fejérian analysis of some optimization algorithms. In Butnariu, D., Censor, Y., and Reich, S. (eds.), Inherently Parallel Algorithms in Feasibility and Optimization and Their Applications , pp. 115-152. Elsevier, New York, NY, USA, 2001.

Daskalakis, C. and Panageas, I. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In ITCS '19: Proceedings of the 10th Conference on Innovations in Theoretical Computer Science , 2019.

Daskalakis, C., Goldberg, P. W., and Papadimitriou, C. H. The complexity of computing a Nash equilibrium. Communications of the ACM , 52(2):89-97, 2009.

Daskalakis, C. C., Fishelson, M., and Golowich, N. Near-optimal no-regret learning in general games. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems , 2021.

Farina, G., Anagnostides, I., Luo, H., Lee, C.-W., Kroer, C., and Sandholm, T. Near-optimal no-regret learning dynamics for general convex games. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems , 2022.

Fasoulakis, M., Markakis, E., Pantazis, Y., and Varsos, C. Forward looking best-response multiplicative weights update methods for bilinear zero-sum games. In Camps-Valls, G., Ruiz, F. J. R., and Valera, I. (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pp. 11096-11117. PMLR, 28-30 Mar 2022.

Flokas, L., Vlatakis-Gkaragkounis, E. V., Lianeas, T., Mertikopoulos, P., and Piliouras, G. No-regret learning and mixed Nash equilibria: They do not mix. In NeurIPS '20: Proceedings of the 34th International Conference on Neural Information Processing Systems , 2020.

Friedman, D. Evolutionary games in economics. Econometrica , 59(3):637-666, 1991.

Gidel, G., Berard, H., Vignoud, G., Vincent, P., and Lacoste-Julien, S. A variational inequality perspective on generative adversarial networks. In ICLR '19: Proceedings of the 2019 International Conference on Learning Representations , 2019.

Golowich, N., Pattathil, S., and Daskalakis, C. Tight last-iterate convergence rates for no-regret learning in multi-player games. In NeurIPS '20: Proceedings of the 34th International Conference on Neural Information Processing Systems , 2020.

Gorbunov, E., Taylor, A., and Gidel, G. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems , 2022.

Harper, M. Escort evolutionary game theory. Physica D: Nonlinear Phenomena , 240(18):1411-1415, September 2011.

Hart, S. and Mas-Colell, A. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68(5):1127-1150, September 2000.

Hart, S. and Mas-Colell, A. Stochastic uncoupled dynamics and Nash equilibrium. Games and Economic Behavior , 57:286-303, 2006.

Héliou, A., Cohen, J., and Mertikopoulos, P. Learning with bandit feedback in potential games. In NIPS '17: Proceedings of the 31st International Conference on Neural Information Processing Systems , 2017.

Hofbauer, J. and Sigmund, K. Evolutionary Games and Population Dynamics . Cambridge University Press, Cambridge, UK, 1998.

Hsieh, Y.-G., Iutzeler, F., Malick, J., and Mertikopoulos, P. On the convergence of single-call stochastic extra-gradient methods. In NeurIPS '19: Proceedings of the 33rd International Conference on Neural Information Processing Systems , pp. 6936-6946, 2019.

Hsieh, Y.-G., Antonakopoulos, K., and Mertikopoulos, P. Adaptive learning in continuous games: Optimal regret bounds and convergence to Nash equilibrium. In COLT '21: Proceedings of the 34th Annual Conference on Learning Theory , 2021.

Hsieh, Y.-G., Antonakopoulos, K., Cevher, V., and Mertikopoulos, P. No-regret learning in games with noisy feedback: Faster rates and adaptivity via learning rate separation. In NeurIPS '22: Proceedings of the 36th International Conference on Neural Information Processing Systems , 2022.

Hsieh, Y.-G., Iutzeler, F., Malick, J., and Mertikopoulos, P. Multi-agent online optimization with delays: Asynchronicity, adaptivity, and optimism. Journal of Machine Learning Research , 23(78):1-49, May 2022.

Jiang, X., Lim, L.-H., Yao, Y ., and Ye, Y . Statistical ranking and combinatorial Hodge theory. Mathematical Programming , 127(1):203-244, 2011.

Juditsky, A., Nemirovski, A. S., and Tauvel, C. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

Korpelevich, G. M. The extragradient method for finding saddle points and other problems. Èkonom. i Mat. Metody , 12:747-756, 1976.

Kwon, J. and Mertikopoulos, P. A continuous-time approach to online optimization. Journal of Dynamics and Games , 4(2):125-148, April 2017.

Lee, J. M. Introduction to Smooth Manifolds . Number 218 in Graduate Texts in Mathematics. SpringerVerlag, New York, NY, 2 edition, 2003.

Legacci, D., Mertikopoulos, P., and Pradelski, B. S. R. A geometric decomposition of finite games: Convergence vs. recurrence under exponential weights. https://arxiv.org/abs/2405.07224 , 2024.

Lin, T., Zhou, Z., Mertikopoulos, P., and Jordan, M. I. Finite-time last-iterate convergence for multi-agent learning in games. In ICML '20: Proceedings of the 37th International Conference on Machine Learning , 2020.

Mertikopoulos, P. and Moustakas, A. L. The emergence of rational behavior in the presence of stochastic perturbations. The Annals of Applied Probability , 20(4):1359-1388, July 2010.

Mertikopoulos, P. and Sandholm, W. H. Learning in games via reinforcement and regularization. Mathematics of Operations Research , 41(4):1297-1324, November 2016.

Mertikopoulos, P. and Sandholm, W. H. Riemannian game dynamics. Journal of Economic Theory , 177: 315-364, September 2018.

Mertikopoulos, P. and Zhou, Z. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173(1-2):465-507, January 2019.

Mertikopoulos, P., Papadimitriou, C. H., and Piliouras, G. Cycles in adversarial regularized learning. In SODA '18: Proceedings of the 29th annual ACM-SIAM Symposium on Discrete Algorithms , 2018.

Mertikopoulos, P., Lecouat, B., Zenati, H., Foo, C.-S., Chandrasekhar, V., and Piliouras, G. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In ICLR '19: Proceedings of the 2019 International Conference on Learning Representations , 2019.

Mertikopoulos, P., Hsieh, Y.-P., and Cevher, V. A unified stochastic approximation framework for learning in games. Mathematical Programming , 203:559-609, January 2024.

Monderer, D. and Shapley, L. S. Potential games. Games and Economic Behavior , 14(1):124 - 143, 1996.

Nemirovski, A. S. Prox-method with rate of convergence 𝑂 ( 1 / 𝑡 ) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Nesterov, Y. Dual extrapolation and its applications to solving variational inequalities and related problems. Mathematical Programming , 109(2):319-344, 2007.

Nisan, N., Roughgarden, T., Tardos, É., and Vazirani, V. V. (eds.). Algorithmic Game Theory . Cambridge University Press, 2007.

Piliouras, G. and Shamma, J. S. Optimization despite chaos: Convex relaxations to complex limit sets via Poincaré recurrence. In SODA '14: Proceedings of the 25th annual ACM-SIAM Symposium on Discrete Algorithms , 2014.

Popov, L. D. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR , 28(5):845-848, 1980.

Rakhlin, A. and Sridharan, K. Optimization, learning, and games with predictable sequences. In NIPS '13: Proceedings of the 27th International Conference on Neural Information Processing Systems , 2013.

Robinson, C. Dynamical Systems: Stability, Symbolic Dynamics, and Chaos . CRC Press, November 1998.

Rockafellar, R. T. Convex Analysis . Princeton University Press, Princeton, NJ, 1970.

Rockafellar, R. T. and Wets, R. J. B. Variational Analysis , volume 317 of A Series of Comprehensive Studies in Mathematics . Springer-Verlag, Berlin, 1998.

Rustichini, A. Optimal properties of stimulus-response learning models. Games and Economic Behavior , 29(1-2):244-273, 1999.

Sandholm, W. H. Potential games with continuous player sets. Journal of Economic Theory , 97:81-108, 2001.

Sato, Y., Akiyama, E., and Farmer, J. D. Chaos in learning a simple two-person game. Proceedings of the National Academy of Sciences of the USA , 99(7):4748-4751, April 2002.

Shalev-Shwartz, S. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107-194, 2011.

Shalev-Shwartz, S. and Singer, Y. Convex repeated games and Fenchel duality. In NIPS' 06: Proceedings of the 19th Annual Conference on Neural Information Processing Systems , pp. 1265-1272. MIT Press, 2006.

Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. Fast convergence of regularized learning in games. In NIPS '15: Proceedings of the 29th International Conference on Neural Information Processing Systems , pp. 2989-2997, 2015.

Taylor, P. D. and Jonker, L. B. Evolutionary stable strategies and game dynamics. Mathematical Biosciences , 40(1-2):145-156, 1978.

Viossat, Y. and Zapechelnyuk, A. No-regret dynamics and fictitious play. Journal of Economic Theory , 148(2):825-842, March 2013.

Wei, C.-Y., Lee, C.-W., Zhang, M., and Luo, H. Linear last-iterate convergence in constrained saddle-point optimization. In ICLR '21: Proceedings of the 2021 International Conference on Learning Representations , 2021.

Appendices

A Harmonic Games

The class of uniform harmonic games (UHGs) introduced by Candogan et al. [6] provides a gametheoretic framework for modeling strategic situations with conflicting, anti-aligned interests. 9 Broadly speaking, the characterizing property of uniform harmonic games is the following: for any player considering a deviation towards a specific pure strategy profile, there exist other players who are motivated to deviate away from that profile.

Given a finite game Γ = Γ ( N , A , 𝑢 ) , this is formalized by the condition that, for all 𝛼 ∈ A ,

From a strategic viewpoint, uniform harmonic games complement potential games: Candogan et al. [6] showed that any finite game can be uniquely decomposed into the sum of a potential game and a uniform harmonic game, up to linear transformations of the payoff functions that do not change the strategic structure of the game.

Since their introduction, harmonic games have generated a substantial body of literature; for a brief survey, we refer the reader to [35].

A.1. Harmonic games, measures and comeasures. The class of uniform harmonic games exhibits intriguing, yet restrictive, properties. Notably, a UHG always admits the uniformly mixed strategy as

a NE, and it generally possesses a continuum of Nash equilibria [6]. Additionally, the framework of UHGs and the decomposition proposed by Candogan et al. [6] are incompatible with common game-theoretical transformations, such as the duplication of strategies or rescaling of payoffs [1]. To address the above limitations, Abdou et al. [1] extended the definition of harmonic games by the introduction of two parameters: a measure , that is a positive weight each player assigns to each of their own strategy; and a comeasure , that is a positive weight each player assigns to each of the other players' action profiles.

Definition A.1. Let Γ ( N , A , 𝑢 ) be a finite game. A player measure 𝜇 𝑖 is a function 𝜇 𝑖 : A 𝑖 → ℝ ++ ; a player co-measure 𝛾 𝑖 is a function 𝛾 𝑖 : A -𝑖 → ℝ ++ . Correspondingly, a collection 𝜇 = { 𝜇 𝑖 } 𝑖 ∈ N (resp. 𝛾 = { 𝛾 𝑖 } 𝑖 ∈ N ) of player measures (resp. comeasures) is called game measure (resp. game comeasure ). If 𝜇 𝑖 is a player measure, we will write | 𝜇 𝑖 | : = P 𝛼 𝑖 𝜇 𝑖 𝛼 𝑖 . Finally, a probability measure is a game measure 𝜇 such that | 𝜇 𝑖 | = 1 for all 𝑖 ∈ N ; a uniform measure is a game measure 𝜇 such that 𝜇 𝑖 𝛼 𝑖 = 1 for all 𝑖 ∈ N , 𝛼 𝑖 ∈ A 𝑖 ; and a uniform comeasure is a game comeasure 𝛾 such that 𝛾 𝑖 𝛼 -𝑖 = 1 for all 𝑖 ∈ N , 𝛼 -𝑖 ∈ A -𝑖 .

With these notions in place, Abdou et al. [1] define a finite game Γ to be ( 𝜇, 𝛾 ) -harmonic if there exist a game measure 𝜇 and a game comeasure 𝛾 such that, for all 𝛼 ∈ A ,

In this work, we focus solely on harmonic games with uniform comeasure . As discussed after Definition 1 in the main body of the article, this choice comes without loss of generality: the game comeasure in Eq. (A.2) can be absorbed by a payoff rescaling to give a game that is still harmonic, and preference equivalent to the original game - in a sense that we make precise in the next section.

A.2. Preference equivalence between harmonic games. The strategic structure of a game is preserved under monotonic transformations of the utility functions, since the set of pure Nash equilibria of a game is an ordinal object - it depends only on the signs of unilateral payoff differences, and not on their absolute values. For this reason, two games Γ ( N , A , 𝑢 ) and Γ ′ ( N , A , 𝑢 ′ ) are called preference-equivalent (PE) if for all 𝛼, 𝛽 ∈ A and all 𝑖 ∈ N , we have

Two games are strategically equivalent (SE) - and we write Γ ∼ Γ ′ - if they have the same unilateral payoff differences, that is if

for all 𝛼, 𝛽 ∈ A and all 𝑖 ∈ N ; strategically equivalent games are clearly preference-equivalent.

Lemma A.2. Let Γ 𝜇,𝛾 = Γ 𝜇,𝛾 ( N , A , 𝑢 ) be a harmonic game in the sense of Eq. (A.2) . Then the game ( N , A , 𝑢 ′ ) with 𝑢 ′ 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) = 𝛾 𝑖 𝛼 -𝑖 𝑢 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) is preference-equivalent to the game Γ 𝜇,𝛾 , and it is harmonic in the sense of Eq. (A.2) with measure 𝜇 and uniform comeasure.

Proof. Let 𝑢 ′′ 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) = 𝜇 𝑖 𝛼 𝑖 𝛾 𝑖 𝛼 -𝑖 𝑢 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) . Then replacing above, for all 𝛼 ∈ A ,

Let 𝑢 ′ 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) = 𝑢 ′′ 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) 𝜇 𝑖 𝛼 𝑖 = 𝛾 𝑖 𝛼 -𝑖 𝑢 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) . The game 𝑢 ′ is preference-equivalent to 𝑢 , and

for all 𝛼 ∈ A , so 𝑢 ′ is harmonic in the sense of A.2 with measure 𝜇 and uniform comeasure. ■

In the proof above we perform the intermediate step 𝑢 → 𝑢 ′′ rather than defining directly 𝑢 → 𝑢 ′ to stress the difference between rescaling the payoffs of a game by a game measure 𝜇 and by a game comeasure 𝛾 . The game with payoffs 𝑢 ′ = 𝛾𝑢 (the meaning of this notation made precise in the proof above) is preference-equivalent to the game with payoffs 𝑢 , i.e., rescaling the payoffs by a comeasure does not change the strategic structure of the game. On the other hand, the game with

payoffs 𝑢 ′′ = 𝜇𝑢 ′ - again, the meaning made precise in the proof - is not PE to the game with payoffs 𝑢 ′ : rescaling the payoffs by a measure can change the preferences of the players, and leads to a game with intrinsically different strategic structure.

Lemma A.2 motivates our choice to focus in this work on harmonic games with arbitrary measures and uniform comeasures, and to adopt (HG) from Definition 1 to characterize harmonic games: a harmonic game (HG) Γ 𝜇 = Γ 𝜇 ( N , A , 𝑢 ) is a finite game ( N , A , 𝑢 ) with a game measure 𝜇 such that (HG) holds, i.e., P 𝑖 ∈ N P 𝛽 𝑖 ∈ A 𝑖 𝜇 𝑖𝛽 𝑖 [ 𝑢 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) -𝑢 𝑖 ( 𝛽 𝑖 ; 𝛼 -𝑖 )] = 0 for all 𝛼 ∈ A .

A.3. Mixed characterization of harmonic games. The defining property (HG) allows for an equivalent characterization of harmonic games in terms of their mixed payoffs:

Lemma A.3. A finite game Γ = Γ ( N , A , 𝑢 ) is harmonic with measure 𝜇 if and only if

Proof. Given a finite game Γ = Γ ( N , A , 𝑢 ) and a game measure 𝜇 , let 𝐹 𝑖 : A → ℝ be defined by 𝐹 𝑖 ( 𝛼 ) = P 𝛽 𝑖 ∈ A 𝑖 𝜇 𝑖𝛽 𝑖 [ 𝑢 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) -𝑢 𝑖 ( 𝛽 𝑖 ; 𝛼 -𝑖 )] . By definition, Γ is a 𝜇 -harmonic game if and only if 𝐹 ( 𝛼 ) : = P 𝑖 ∈ N 𝐹 𝑖 ( 𝛼 ) = 0 for all 𝛼 ∈ A . Denote (with slight abuse of notation) by 𝐹 : X → ℝ the multilinear extension of 𝐹 : A → ℝ , i.e., 𝐹 ( 𝑥 ) = P 𝛼 𝑥 𝛼 𝐹 ( 𝛼 ) , with 𝑥 𝛼 : = Q 𝑖 𝑥 𝑖 𝛼 𝑖 . Now, 𝐹 ( 𝛼 ) = 0 for all 𝛼 ∈ A if and only if 𝐹 ( 𝑥 ) = 0 for all 𝑥 ∈ X , which is the case if and only if

from which we conclude by factoring out the | 𝜇 𝑖 | terms.

■

Remark. The first equality in the second line holds true for harmonic games with uniform comeasure 𝛾 𝑖 𝛼 -𝑖 = 1 , since 𝛾 𝑖 𝛼 -𝑖 ≠ 1 terms would couple with the corresponding 𝑥 -𝑖 𝛼 -𝑖 terms in the sum.

The above result can be reformulated as follows:

Proposition A.4. A finite game Γ = Γ ( N , A , 𝑢 ) is harmonic if and only if it admits a strategic center ( 𝑚, 𝑞 ) , viz. if there exist (i ) a vector 𝑚 ∈ ℝ 𝑁 ++ and (ii ) a fully mixed strategy 𝑞 ∈ X such that

This expression is intriguing: it suggest that a game is harmonic precisely if there exists a fully mixed strategy 𝑞 such that, for all 𝑥 ∈ X , the payoff vector 𝑣 ( 𝑥 ) is perpendicular (with respect to a 𝑚 -weighted inner product) to 𝑥 -𝑞 ; cf. Example A.1 and Fig. 2. The striking dynamical consequences of this 'circular' strategic structure - hinted at in Fig. 2, showing a periodic orbit of FTRL in continuous time - are captured precisely by Theorem 2 in the main text.

Proof of Proposition A.4. Let Γ 𝜇 = Γ 𝜇 ( N , A , 𝑢 ) be harmonic; then by Lemma A.3 that there exist a strategic center ( 𝑚, 𝑞 ) given by 𝑚 𝑖 : = | 𝜇 𝑖 | and 𝑞 𝑖 : = 𝜇 𝑖 /| 𝜇 𝑖 | with 𝑖 ∈ N . Conversely let Γ = Γ ( N , A , 𝑢 ) admit a strategic center ( 𝑚, 𝑞 ) ; then by the same argument Γ is harmonic with 𝜇 𝑖 : = 𝑚 𝑖 𝑞 𝑖 for all 𝑖 ∈ N . ■

An immediate corollary is the following:

Corollary A.5. If a finite game Γ admits a strategic center ( 𝑚, 𝑞 ) , then 𝑞 is a Nash equilibrium.

Proof. By Proposition A.4 if Γ admits a strategic center ( 𝑚, 𝑞 ) then it is 𝜇 -harmonic with 𝜇 𝑖 = 𝑚 𝑖 𝑞 𝑖 for all 𝑖 ∈ N ; and ( 𝜇 𝑖 /| 𝜇 𝑖 |) 𝑖 ∈ N is always a NE for 𝜇 -harmonic games [1, Theorem 1]. ■

Remark. The converse does not hold: a fully mixed Nash equilibrium is not necessarily a strategic center. If it were, a game would be harmonic precisely if it admitted a fully mixed NE, which is not the case - think for example of coordination or anti-coordination games, that admit a fully mixed Nash equilibrium and are not harmonic.

Prob. player 1 assigns to N in {A, N}

Prob. player 2 assigns to N in {D, N}

(A,N) : u = (2, -4)

(N,N) : u = (0, 0)

(A,D) : u = (-3, 1)

(N,D) : u = (0, -1)

Example A.1 (A harmonic game: Siege) . Consider the following 2 × 2 game: an army (the row player) must choose between Attacking a fortress (pure strategy 𝐴 ) and Not attacking (pure strategy 𝑁 ). Simultaneously, the fortress (the column player) decides whether to activate its Defenses (pure strategy 𝐷 ) or Not (pure strategy 𝑁 ). Engaging in either action (the attack or the defense) incurs a preparation cost of 𝑐 > 0 . The army gains 𝑎 𝑠 > 𝑐 if it attacks an undefended fortress, but suffers a loss of 𝑎 𝑓 > 0 if it attacks and encounters defenses (the subscripts 𝑠 and 𝑓 standing respectively for 'successful' and 'failed'). Conversely, the fortress benefits by 𝑑 𝑠 > 0 if it is defended against an attack, while it incurs a loss of 𝑑 𝑓 > 0 if attacked without defenses; defeating the attacking army is worth the preparation cost for the fortress, namely 𝑑 𝑠 -𝑐 > -𝑑 𝑓 . This scenario is captured by the following payoff matrix, specialized on the right to the case 𝑐 = 1 , 𝑎 𝑠 = 3 , 𝑎 𝑓 = 2 , 𝑑 𝑠 = 2 , 𝑑 𝑓 = 4 :

To determine if the game is harmonic, look for a solution of the linear system

subject to the constraints 𝜇 𝑖 𝛼 𝑖 > 0 for all 𝑖 ∈ N , 𝛼 𝑖 ∈ A 𝑖 . For a fixed payoff function 𝑢 , this is a system of Q 𝑗 ∈ N 𝐴 𝑗 linear equations (one for each 𝛼 ∈ A ) in the P 𝑗 ∈ N 𝐴 𝑗 variables GLYPH<0> ( 𝜇 𝑖 𝛼 𝑖 ) 𝛼 𝑖 ∈ A 𝑖 GLYPH<1> 𝑖 ∈ N , where 𝐴 𝑖 is the number of pure actions of player 𝑖 ∈ N . With 𝑢 given by (A.6) - left,

is a feasible solution of (HG) for any 𝜆 > 0 , so the game is harmonic with a 1 -dimensional set of measures. The corresponding strategic center ( 𝑚, 𝑞 ) with 𝑚 𝑖 = P 𝛼 𝑖 𝜇 𝑖 𝛼 𝑖 , 𝑞 𝑖 = 𝜇 𝑖 / 𝑚 𝑖 , 𝑖 ∈ { 1 , 2 } is

As a sanity check, compute the payoff field and verify that (HG-center) holds true in the specialized case (A.6) - right. Denoting the mixed strategies of players 1 and 2 respectively by 𝑥 ∈ Δ ({ 𝐴, 𝑁 }) and 𝑦 ∈ Δ ({ 𝐷, 𝑁 }) , the payoff fields are 𝑣 1 ( 𝑥, 𝑦 ) = (-3 𝑦 𝐷 + 2 𝑦 𝑁 , 0 ) , 𝑣 2 ( 𝑥, 𝑦 ) =

( 𝑥 𝐴 -𝑥 𝑁 , -4 𝑥 𝐴 ) . Choosing 𝜆 = 3 the strategic center gives weights 𝑚 = ( 6 , 5 ) and Nash equilibrium 𝑞 = [( 1 / 6 , 5 / 6 ) , ( 2 / 5 , 3 / 5 )] . Condition (HG-center) boils down to 6 ⟨ 𝑣 1 , 𝑥 -𝑞 1 ⟩ + 5 ⟨ 𝑣 2 , 𝑦 -𝑞 2 ⟩ = 0 , which one readily verifies to hold true by replacing the expressions above and recalling that 𝑥 𝐴 + 𝑥 𝑁 = 1 = 𝑦 𝐷 + 𝑦 𝑁 . Fig. 2 illustrates the situation: each payoff vector 𝑣 ( 𝑥 ) (black arrows) is perpendicular (with respect to a weighted inner product) to the vector 𝑥 -𝑞 (dotted segment) between the evaluation point 𝑥 of the payoff field and the fully mixed Nash equilibrium 𝑞 (red point). ⋄

A.4. Harmonic and zero-sum games. Candogan et al. [6]'s uniform harmonic games, defined by Eq. (A.1), are precisely the harmonic games with uniform measure, which makes uniform harmonic games a strict subset of the set of HGs. Importantly, HGs include another archetypal class of perfectcompetition games: as we show in this section, two-player zero-sum games (2ZSGs) with an interior NE 𝑥 ∗ are harmonic with (probability) measure 𝜇 = 𝑥 ∗ .

To show this, we will need the following definition and lemma:

Definition A.6 (Non-strategic game) . A finite normal form game Γ = Γ ( N , A , 𝑘 ) is called nonstrategic if the payoff of each player does not depend on their own choice, viz. if 𝑘 𝑖 ( 𝛼 𝑖 , 𝛼 -𝑖 ) = 𝑘 𝑖 ( 𝛽 𝑖 , 𝛼 -𝑖 ) for all 𝑖 ∈ N , 𝛼 ∈ A , 𝛽 𝑖 ∈ A 𝑖 .

Lemma A.7. Two finite games Γ ( N , A , 𝑢 ) , Γ ′ ( N , A , 𝑢 ′ ) are strategically equivalent in the sense of Eq. (A.4) if and only if their difference is a non-strategic game.

Proof. Let Γ -Γ ′ be non-strategic; then 𝑘 : = 𝑢 ′ -𝑢 fulfills the condition of Definition A.6, which shows that 𝑢 and 𝑢 ′ fulfill Eq. (A.4). Conversely let Γ and Γ ′ be strategically equivalent; set 𝑘 : = 𝑢 ′ -𝑢 and rearrange the terms in Eq. (A.4) to immediately conclude that 𝑘 is a non-strategic game. ■

Proposition A.8. Let Γ 𝜇 = Γ 𝜇 ( N , A , 𝑢 ) be a harmonic game. If the measure 𝜇 fulfills | 𝜇 𝑖 | = | 𝜇 𝑗 | for all 𝑖, 𝑗 ∈ N then Γ 𝜇 is strategically equivalent to a zero-sum game.

Proof. Recall that | 𝜇 𝑖 | ≡ P 𝛼 𝑖 𝜇 𝑖 𝛼 𝑖 . Under the assumption | 𝜇 𝑖 | = | 𝜇 𝑗 | for all 𝑖, 𝑗 ∈ N , let 𝑐 : = | 𝜇 𝑖 | for any 𝑖 ∈ N . By (HG), the payoff 𝑢 of Γ 𝜇 in this case fulfills P 𝑖 ∈ N [ 𝑢 𝑖 ( 𝛼 ) -𝑘 𝑖 ( 𝛼 )] = 0 for all 𝛼 ∈ A , with 𝑘 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) : = 𝑐 -1 P 𝛽 𝑖 𝜇 𝑖𝛽 𝑖 𝑢 𝑖 ( 𝛽 𝑖 , 𝛼 -𝑖 ) . Set 𝑢 ′ 𝑖 : = 𝑢 𝑖 -𝑘 𝑖 . By definition 𝑢 ′ is a zero-sum game; furthermore, the difference between 𝑢 𝑖 and 𝑢 ′ 𝑖 is non-strategic, since 𝑘 𝑖 ( 𝛼 𝑖 ; 𝛼 -𝑖 ) does not depend on 𝛼 𝑖 . Thus 𝑢 𝑖 and 𝑢 ′ 𝑖 are strategically equivalent by Lemma A.7. ■

In particular we have the following:

Corollary A.9. Let Γ 𝜇 = Γ 𝜇 ( N , A , 𝑢 ) be a harmonic game. If the measure 𝜇 is a probability measure, then Γ 𝜇 is strategically equivalent to a zero-sum game.

The converse holds true only in the case of two-player games:

Proposition A.10. Every two-player zero-sum game with an interior Nash equilibrium 𝑥 ∗ is harmonic, with (probability) measure 𝜇 = 𝑥 ∗ .

Proof. Let Γ = Γ ( N , A , 𝑢 ) be a two-player zero-sum game with interior Nash equilibrium 𝑥 ∗ . If we show that

then we can conclude by Lemma A.3 that Γ is harmonic with measure 𝑥 ∗ . Eq. (A.9) holds indeed true: | 𝑥 ∗ 𝑖 | = 1 for all 𝑖 ∈ N , and it is well known [41, 42] that two-player zero-sum games with an interior equilibrium 𝑥 ∗ fulfill P 𝑖 ∈ N ⟨ 𝑣 𝑖 ( 𝑥 ) , 𝑥 𝑖 -𝑥 ∗ 𝑖 ⟩ = 0 for all 𝑥 ∈ X , so we are done. ■

Harmonic games thus encompass and substantially generalize two prototypical classes of games with anti-aligned incentives, serving as an ideal complement to the class of potential games. This is made precise in [1]: building on the work of Candogan et al. [6], Abdou et al. [1] showed that, for any choice of game measure 𝜇 , every finite game can be uniquely decomposed into the sum of a potential and a 𝜇 -harmonic game, up to strategic equivalence.

This establishes harmonic games as the natural complement of potential games from a strategic perspective; Theorem 2 in the main text shows that this holds true from a dynamic perspective as well.

B Basic properties of regularizers and the induced choice maps

In this appendix, we collect a number of properties concerning regularizers and the associated choice maps. To avoid carrying around the player index 𝑖 ∈ N , we state all our results for a generic convex subset C of some real vector space V . The desired properties for FTRL will then be obtained by specializing C to X 𝑖 or X and V to ℝ A 𝑖 or Q 𝑗 ℝ A 𝑗 , depending on the context.

B.1. Preliminary definitions. To begin, let V be a 𝑑 -dimensional normed space with norm ∥·∥ . In what follows, we will write Y : = V ∗ for the dual space of V , ⟨ 𝑦, 𝑥 ⟩ for the canonical pairing between 𝑥 ∈ V and 𝑦 ∈ V ∗ , and ∥ 𝑦 ∥∗ = max {⟨ 𝑦, 𝑥 ⟩ : ∥ 𝑥 ∥ ≤ 1 } for the induced dual norm on Y . Following standard conventions in convex analysis, functions will be allowed to take values in the extended real line ℝ ∪ {∞} , and if 𝑓 : V → ℝ ∪ {∞} is a convex function on V , we will denote its effective domain as

In addition, assuming dom 𝑓 ≠ ∅ , the subdifferential of 𝑓 at 𝑥 is defined as

and we denote the domain of subdifferentiability of 𝑓 as

Finally, to ease notation, a convex function 𝑓 : C → ℝ will be identified with the extended-real-valued function ¯ 𝑓 : V → ℝ ∪ {∞} that agrees with 𝑓 on C and is identically equal to ∞ on V \ C .

With all this in hand, let C be a closed convex subset of V , and let ℎ : C → ℝ be a 𝐾 -strongly convex regularizer on C , that is,

By standard arguments in convex analysis, this readily implies that

where

denotes the one-sided directional derivative of ℎ at 𝑥 along the direction of 𝑥 ′ -𝑥 . To proceed, we will need the following basic objects:

The convex conjugate ℎ ∗ : Y → ℝ of ℎ :

The regularized choice map - or mirror map -𝑄 : Y → X induced by ℎ :

The associated Fenchel coupling 𝐹 : X × Y → ℝ of ℎ :

Remark. The terminology 'Fenchel coupling' is due to [38, 40], which we follow closely in terms of notation and conventions.

The proposition below provides some basic properties concerning the first two objects above:

Proposition B.1. Let ℎ be a 𝐾 -strongly convex regularizer on C . Then:

( a ) 𝑄 is single-valued on Y ; in particular, for all 𝑥 ∈ dom 𝜕ℎ and all 𝑦 ∈ Y , we have:

( b ) The image im 𝑄 of 𝑄 satisfies ri C ⊆ im 𝑄 = dom 𝜕ℎ ⊆ C .

( c ) The convex conjugate ℎ ∗ : Y → ℝ of ℎ is differentiable and

( d ) 𝑄 is ( 1 / 𝐾 ) -Lipschitz continuous, that is,

( e ) Fix some 𝑦 ∈ Y and set 𝑥 = 𝑄 ( 𝑦 ) . Then, for all 𝑥 ′ ∈ X we have:

In particular, if 𝜕ℎ admits a continuous selection ∇ ℎ : dom 𝜕ℎ → Y , we have

or, equivalently, where

denotes the polar cone to C at 𝑥 .

Proof. These properties are fairly well known (except possibly the last one), so we only provide a quick proof or a precise pointer to the literature.

( a ) The maximum in (B.8) is attained for all 𝑦 ∈ V ∗ and is unique because ℎ is strongly convex. Furthermore, 𝑥 solves (B.8) if and only if 𝑦 -𝜕ℎ ( 𝑥 ) ∋ 0 , i.e., if and only if 𝑦 ∈ 𝜕ℎ ( 𝑥 ) .

( b ) By (B.10), we readily get im 𝑄 = dom 𝜕ℎ . Consequently, the rest of our claim follows from standard results in convex analysis, see e.g., Rockafellar [52, Chap. 26].

( c ) The equality 𝑄 = ∇ ℎ ∗ follows immediately from Danskin's theorem, see e.g., Bertsekas [4, Proposition 5.4.8, Appendix B].

( d ) See Rockafellar & Wets [53, Theorem 12.60(b)].

( e ) Since 𝑦 ∈ 𝜕ℎ ( 𝑥 ) by (B.10), we readily get that

Hence, by rearranging and taking the limit 𝜃 → 0 + , 10 we conclude that

as claimed. Finally, for our last assertion, let 𝑧 = 𝑥 ′ -𝑥 and set

so 𝜙 ( 𝜃 ) ≥ 𝐾𝜃 2 ∥ 𝑧 ∥ 2 / 2 ≥ 0 for all 𝜃 ∈ [ 0 , 1 ] . By construction, it is straightforward to verify that the function 𝜓 ( 𝜃 ) = ⟨∇ ℎ ( 𝑥 + 𝜃𝑧 ) -𝑦, 𝑧 ⟩ is a selection of subgradients of 𝜙 , i.e.,

Since 𝜓 is in addition continuous (because ∇ ℎ is), it follows that 𝜙 ′ ( 𝜃 ) = 𝜓 ( 𝜃 ) for all 𝜃 ∈ [ 0 , 1 ] by a well-known characterization of the one-sided derivatives of convex functions, cf. Rockafellar [52, Theorem 24.2]. Hence, with 𝜙 convex and 𝜙 ( 𝜃 ) ≥ 𝜙 ( 0 ) for all 𝜃 ∈ [ 0 , 1 ] , we conclude that ⟨∇ ℎ ( 𝑥 ) -𝑦, 𝑧 ⟩ = 𝜓 ( 0 ) = 𝜙 ′ ( 0 ) ≥ 0 , and our proof is complete. ■

The next proposition collects some basic properties of the Fenchel coupling.

Proposition B.2. Let ℎ be a 𝐾 -strongly convex regularizer on C . Then, for all 𝑝 ∈ X and all 𝑦, 𝑦 ′ ∈ Y , we have:

( 𝑎 ) 𝐹 ( 𝑝, 𝑦 ) ≥ 0 with equality if and only if 𝑝 = 𝑄 ( 𝑦 ) . (B.21a)

Proof. These properties are also fairly standard, but we provide a quick proof for completeness.

( a ) By the Fenchel-Young inequality, we have ℎ ( 𝑝 ) + ℎ ∗ ( 𝑦 ) ≥ ⟨ 𝑦, 𝑝 ⟩ for all 𝑝 ∈ X , 𝑦 ∈ Y , with equality if and only if 𝑦 ∈ 𝜕ℎ ( 𝑝 ) . Our claim then follows from (B.10).

( b ) Let 𝑥 = 𝑄 ( 𝑦 ) so 𝑦 ∈ 𝜕ℎ ( 𝑥 ) by (B.10). Then, by the definition of 𝐹 , we have

and our proof is complete.

■

In view of Proposition B.2, 𝐹 ( 𝑝, 𝑦 ) can be seen a 'primal-dual' measure of divergence between 𝑝 ∈ X and 𝑦 ∈ Y , and the alternate expression (19) is straightforward. This observation will play a major role in the sequel.

B.2. Basic lemmas. Moving forward, we note that the various update steps in (FTRL+) can be written as

for some 𝑦, 𝑤 ∈ Y . With this in mind, we proceed below to state a series of basic lemmas for the Fenchel coupling before and after an update of the form (B.22). These results are not new, cf. [31, 40, 42] and references therein; however, the assumptions used to derive them vary significantly in the literature, so we provide detailed proofs for completeness.

All of the results that follow below are stated for a 𝐾 -strongly convex regularizer on C . The first result is a primal-dual version of the so-called 'three-point identity' for mirror descent [7]:

Lemma B.1. Fix some 𝑝 ∈ X , 𝑦 ∈ Y , and let 𝑥 = 𝑄 ( 𝑦 ) . Then, for all 𝑦 + ∈ Y , we have:

Proof. By definition, we have:

Thus, subtracting (B.24b) and (B.24c) from (B.24a), and rearranging, we get

Our assertion then follows by recalling that 𝑥 = 𝑄 ( 𝑦 ) , so ℎ ( 𝑥 ) + ℎ ∗ ( 𝑦 ) = ⟨ 𝑦, 𝑥 ⟩ . ■

The next result we present concerns the Fenchel coupling before and after a direct update step; similar results exist in the literature, but we again provide a proof for completeness.

Lemma B.2. Fix some 𝑝 ∈ X and 𝑦, 𝑤 ∈ Y . Then, letting 𝑥 = 𝑄 ( 𝑦 ) , 𝑦 + = 𝑦 + 𝑤 , and 𝑥 + = 𝑄 ( 𝑦 + ) as per (B.22) , we have:

Proof. By the three-point identity (B.23), we have

so our first claim follows by rearranging. For our second claim, simply note that

so our claim follows from Proposition B.2.

■

The last result we present here is sometimes referred to as a 'four-point lemma', and concerns the Fenchel coupling before and after an extrapolation step:

Lemma B.3. Fix some 𝑝 ∈ X and 𝑦, 𝑤 1 , 𝑤 2 ∈ Y . Then, letting 𝑥 = 𝑄 ( 𝑦 ) , 𝑦 + 𝑖 = 𝑦 + 𝑤 𝑖 , and 𝑥 + 𝑖 = 𝑄 ( 𝑦 + 𝑖 ) , 𝑖 = 1 , 2 , as per (B.22) , we have:

so (B.29a) follows by writing ⟨ 𝑤 2 , 𝑥 + 2 -𝑝 ⟩ = ⟨ 𝑤 2 , 𝑥 + 1 -𝑝 ⟩ + ⟨ 𝑤 2 , 𝑥 + 2 -𝑥 + 1 ⟩ , and (B.29b) follows from the three-point identity (B.23) for the Fenchel coupling. Finally, for (B.29c), the Fenchel-Young inequality in Peter-Paul form yields

and our claim follows again by invoking Proposition B.2 to write

and then substituting the result in (B.31)

■

Lemmas B.2 and B.3 will be responsible for most of the heavy lifting to derive a Lyapunov function for (FTRL+). We discuss the relevant details in Appendix D.

We conclude this section with a variational characterization of the abstract update (B.22) in the case where 𝜕ℎ of ℎ admits a continuous selection - or, alternatively, ℎ is smooth in the sense of (17).

Lemma B.4. Fix some 𝑦, 𝑦 + ∈ Y , and let 𝑥 + = 𝑄 ( 𝑦 + ) . Then, for all 𝑝 ∈ X , we have

Proof. Invoking (B.14) in Proposition B.1 with 𝑦 ← 𝑦 + , 𝑥 ← 𝑥 + , and 𝑥 ′ ← 𝑝 , we get

Our claim then follows by subtracting ⟨ 𝑦, 𝑝 -𝑥 + ⟩ from both sides of the above.

■

C Continuous-time analysis

C.1. Dynamical systems notions. To fix notation, we recall here some basics from the theory of dynamical systems, roughly following [2, 51]. In this section, M is an open subset of a Euclidean space of dimension 𝑑 .

We consider a system of ordinary differential equations (ODEs) of the form

where 𝑥 ( 𝑡 ) is a curve in M defined on an open interval I ⊆ ℝ (that without loss of generality we assume to contain 0 ), and 𝑋 : M → ℝ 𝑑 is a smooth function. The function 𝑋 is called vector field because it assigns a vector 𝑋 ( 𝑥 ) to each point 𝑥 in M , and (DS) is called dynamical system generated by 𝑋 .

Given 𝑥 0 ∈ M , an orbit with initial condition 𝑥 0 is a solution 𝑥 ( 𝑡 ) of (DS) with 𝑥 ( 0 ) = 𝑥 0 . The flow generated by 𝑋 is the smooth function Θ : I × M → M such that Θ 0 ( 𝑥 0 ) = 𝑥 0 for all 𝑥 0 ∈ M and 𝑑 𝑑𝑡 Θ 𝑡 ( 𝑥 ) = 𝑋 ( Θ 𝑡 ( 𝑥 )) for all 𝑡 ∈ I . In words, Θ 𝑡 ( 𝑥 0 ) is the orbit 𝑥 ( 𝑡 ) with initial condition 𝑥 0 ; the existence and uniqueness of this function is guaranteed by the existence and uniqueness theorem of solutions of ordinary differential equations.

A flow Θ is called volume-preserving if vol ( Θ 𝑡 ( U )) = vol ( U ) for any (Lebesgue) measurable subset U ⊆ M and all 𝑡 ∈ I . Liouville's theorem gives a sufficient condition for a flow to be volume-preserving based on the divergence of its generating field: 11

Theorem (Liouville) . If div 𝑋 ≡ 0 then the flow generated by 𝑋 is volume-preserving.

Volume-preserving flows are closely related to recurrent dynamical patterns. A point 𝑥 ∈ M is said to be recurrent under (DS) if, for every neighborhood U of 𝑥 ∈ M , there exists an increasing sequence of time 𝑡 𝑛 ↑ ∞ such that Θ 𝑡 𝑛 ( 𝑥 ) is defined and falls in U for all 𝑛 . Moreover, (DS) is said to be Poincaré recurrent if almost every point 𝑥 ∈ M is recurrent. The celebrated Poincaré recurrence theorem gives a sufficient condition for a dynamical system to be Poincaré recurrent:

Theorem (Poincaré) . Let 𝑋 be a smooth vector field on M . If the flow induced by 𝑋 is volumepreserving and all the orbits of (DS) are bounded, then (DS) is Poincaré recurrent.

C.2. Basic properties of FTRL. In this section we survey some of the properties of the followthe-regularized-leader learning scheme in a continuous-time, multi-agent setting, in line with the presentations of [16, 38, 41]. For ease of reference we recall here some of the notions introduced in Appendix B and in Sections 2 and 3 from the main body of the paper.

Let Γ = Γ ( N , A , 𝑢 ) be a finite normal form game, and let 𝑣 denote its payoff field. The game's strategy space is X = Q 𝑗 ∈ N Δ ( A 𝑗 ) ⊆ V : = Q 𝑗 ℝ A 𝑗 , and the game's payoff space is Y : = V ∗ . The payoff field is a map 𝑣 : V → Y that evaluated at a strategy 𝑥 ∈ X acts linearly on any 𝑥 ′ ∈ X by

Assume now that Γ is played continuously over time. As discussed in Section 3, the main idea behind the follow-the-regularized-leader learning scheme is that, at any given time 𝑡 ≥ 0 , each player 𝑖 ∈ N tracks their cumulative payoff up to time 𝑡 and plays a 'regularized' best response strategy in light of this information. Concretely, given a cumulative payoff vector 𝑦 𝑖 ( 𝑡 ) ∈ Y 𝑖 , each player 𝑖 ∈ N selects this optimal strategy 𝑥 𝑖 ( 𝑡 ) ∈ X 𝑖 by means of a regularized best response map 𝑄 𝑖 : Y 𝑖 → X 𝑖 , a singlevalued analogue of the best-response correspondence 𝑦 𝑖 ↦→ arg max 𝑥 𝑖 ∈ X 𝑖 ⟨ 𝑦 𝑖 , 𝑥 𝑖 ⟩ . A standard way [57] of obtaining such map is to introduce a regularizer function ℎ 𝑖 : X 𝑖 → ℝ that is ( i ) continuous on X 𝑖 , ( ii ) smooth on ri X 𝑖 , the relative interior of X 𝑖 , and ( iii ) strongly convex on X 𝑖 (as per Eq. (B.4)); and to consider the induced choice map 𝑄 𝑖 : Y 𝑖 → X 𝑖 defined by

By Proposition B.1, 𝑄 𝑖 is well-defined and Lipschitz continuous, and it coincides with the differential ∇ ℎ ∗ 𝑖 of ℎ ∗ 𝑖 : Y 𝑖 → ℝ , the convex conjugate of ℎ 𝑖 .

In a continuous time setting, this regularized learning scheme translates into the following implicit equations of motion, which govern the evolution of the cumulative payoff 𝑦 ( 𝑡 ) ∈ Y and of the mixed strategy profile 𝑥 ( 𝑡 ) ∈ X as the players attempt to maximize their payoff over time:

for all 𝑖 ∈ N , 𝛼 𝑖 ∈ A 𝑖 . A straightforward computation shows that this is equivalent to Eq. (5) from Section 3 in the main text, that governs the evolution of the mixed strategy 𝑥 ( 𝑡 ) ∈ X :

Importantly, Eq. (C.2) can be cast in the form (DS) of a dynamical system in the game's payoff space. For each 𝑖 ∈ N , differentiation with respect to 𝑡 yields

and by aggregating the player indices we obtain the system of ODEs

where 𝑌 : = 𝑣 ◦ 𝑄 : Y → Y is a continuous vector field on Y ; cf. Fig. 3.

Existence and uniqueness of a global solution 𝑦 ( 𝑡 ) ∈ Y of Eq. (C.3) for any initial condition 𝑦 ( 0 ) ∈ Y are guaranteed by standard arguments [38, Prop. 3.1]; in line with the terminology of the previous section we will refer to such a solution as a dual orbit .

C.3. Constant of motion for harmonic games. The following result shows that FTRL in harmonic games admits a constant of motion.

Proposition C.1. Let Γ = Γ ( N , A , 𝑢 ) be a finite game and consider a vector 𝑚 ∈ ℝ 𝑁 ++ and a fully mixed strategy 𝑞 ∈ X . Then the weighted Fenchel coupling 𝐹 𝑚,𝑞 : Y → ℝ defined by

is a constant of motion under (FTRL-D) if and only if Γ is harmonic with strategic center ( 𝑚, 𝑞 ) .

Proof. Let 𝑦 ( 𝑡 ) be a dual orbit. Then by chain rule

where the second equality holds by (FTRL-D) and Eq. (B.11). Then, by the characterization of harmonic games in terms of a strategic center (HG-center), the time derivative of the weighted Fenchel coupling vanishes identically along a dual orbit of (FTRL-D) precisely if the underlying game is harmonic. ■

The existence of this constant of motion is fundamental for proving Theorem 2, i.e., the Poincaré recurrence of continuous-time FTRL in harmonic games. With this key element established, the remainder of this appendix closely follows the proof technique described by [41] for the analogous result in the context of two-player zero-sum games.

C.4. FTRL in the space of payoff differences. For any initial condition 𝑦 ( 0 ) ∈ Y , a dual orbit of (FTRL-D) induces a curve 𝑥 ( 𝑡 ) = 𝑄 ( 𝑦 ( 𝑡 )) in the game's strategy space X which solves Eq. (5) for all 𝑡 ≥ 0 ; in the following we will refer to such curve as trajectory of play . Crucially, a trajectory of play is in general not the global solution of a dynamical system / 𝑥 = 𝑋 ( 𝑥 ) for some vector field 𝑋 : X → X in the game's strategy space. The reason for this is that the map 𝑄 : Y → X is not necessarily invertible, so there is in general no way to identify a unique a vector field 𝑋 on X that is related to the vector field 𝑌 on Y via 𝑄 .

for all 𝛼 𝑖 ∈ A 𝑖 .

Related vector fields and induced dynamical systems. The concept of vector fields related by a smooth map is standard in differential geometry (e.g., [34, p. 181]). Let M , M ′ be open subsets of Euclidean space: given a vector field 𝑌 on M and a smooth map 𝐹 : M → M ′ , a vector field 𝑋 on M ′ is called 𝐹 -related to 𝑌 if, for all 𝑦 ∈ M , ( Jac 𝐹 ) 𝑦 · 𝑌 ( 𝑦 ) = 𝑋 ( 𝑥 ) , with 𝑥 = 𝐹 ( 𝑦 ) . Here Jac 𝐹 is the Jacobian matrix of 𝐹 , and · represents matrix-vector multiplication. If 𝐹 is invertible then such vector field exists always and is unique; else, it might exist and not be unique, or not exist at all.

Vector fields that are related via a smooth map are useful inasmuch as they generate 'compatible' dynamical systems:

Lemma C.2. Let 𝐹 : M → M ′ be a smooth map between open subsets of Euclidean spaces, and let / 𝑦 = 𝑌 ( 𝑦 ) be a dynamical system on M . Let 𝑦 ( 𝑡 ) be an orbit with initial condition 𝑦 0 ∈ M , and consider the curve on M ′ defined by 𝑥 ( 𝑡 ) : = 𝐹 ( 𝑦 ( 𝑡 )) . If there exists a vector field 𝑋 on M ′ that is 𝐹 -related to 𝑌 , then the curve 𝑥 ( 𝑡 ) is an orbit of the dynamical system / 𝑥 = 𝑋 ( 𝑥 ) with initial condition 𝑥 0 = 𝐹 ( 𝑦 0 ) .

Proof. By chain rule,

where the last equality follows by the assumption that 𝑋 is 𝐹 -related to 𝑌

. ■

In the following, if 𝐹 : M → M ′ is a smooth function between open subsets of Euclidean spaces, and 𝑌, 𝑋 are vector fields fulfilling the assumptions of Lemma C.2, we say that the dynamical system / 𝑦 = 𝑌 ( 𝑦 ) on M induces the dynamical system / 𝑥 = 𝑋 ( 𝑥 ) on M ′ via 𝐹 .

FTRL induced in the space of payoff differences. The choice map 𝑄 : Y → X is in general not smooth, and neither injective nor surjective [16, Sec.3], so it generally does not allow to induce the dynamical system (C.3) from the game's payoff space Y to the game's strategy space X . 12 In other words, the learning process (FTRL-D) in a finite game gives rise to a dynamical system in the game's payoff space Y , to which the theorems presented in Appendix C.1 can in principle be applied; however, it can be shown that the orbits of Eq. (C.3) in Y are not bounded, preventing the application of Poincaré's theorem. Furthermore, the dual orbits do not convey direct information on the day-to-day behavior of the players, due to the lack of invertibility of the choice map.

Conversely, the objects of interest from a dynamical, learning viewpoint - that is, the trajectories of play in the game's strategy space X - present technical difficulties and do not easily fit the dynamical systems framework depicted in Appendix C.1. In the following we show how these difficulties can be circumvented by analyzing the dynamics induced by (FTRL-D) in yet a third space Z , that arises by taking the differences between payoffs - rather than their absolute values - as the objects of study.

To make this precise, given the game Γ = Γ ( N , A , 𝑢 ) fix a benchmark strategy ˆ 𝛼 𝑖 ∈ A 𝑖 for every player 𝑖 ∈ N , and consider the hyperplane Z 𝑖 : = { 𝑧 𝑖 ∈ ℝ 𝐴 𝑖 : 𝑧 𝑖 ˆ 𝛼 𝑖 = 0 } ⊂ ℝ 𝐴 𝑖 . Clearly, Z 𝑖 GLYPH<27> ℝ 𝐴 𝑖 -1 . Each player's strategy space Y 𝑖 = ℝ 𝐴 𝑖 can be mapped onto Z 𝑖 by the linear operator

Π 𝑖 is clearly smooth, and a standard check shows that Π 𝑖 is surjective and not injective: ker Π 𝑖 = { 𝑦 𝑖 : 𝑦 𝑖 𝛼 𝑖 = 𝑦 𝑖𝛽 𝑖 for all 𝛼 𝑖 , 𝛽 𝑖 ∈ A 𝑖 } is the 1 -dimensional linear subspace spanned by the vector 1 𝑖 = ( 1 , . . . , 1 ) ∈ Y 𝑖 ; and Π -1 ( 𝑧 𝑖 ) = 𝑧 𝑖 + ker Π 𝑖 for any 𝑧 𝑖 ∈ Z 𝑖 . In particular, for all 𝑦 𝑖 , 𝑦 ′ 𝑖 ∈ Y 𝑖 , we have that Π 𝑖 ( 𝑦 𝑖 ) = Π 𝑖 ( 𝑦 ′ 𝑖 ) if and only if 𝑦 𝑖 -𝑦 ′ 𝑖 is proportional to 1 𝑖 .

Since every 𝑧 𝑖 ∈ Z 𝑖 is the image of some payoff 𝑦 𝑖 via Π 𝑖 , the space Z : = Q 𝑗 Z 𝑗 is called the game's payoff-difference space ; we will denote by Π the product map Π ≡ Q 𝑗 Π 𝑗 , i.e., (cf. Fig. 3)

Lemma C.3. The choice map 𝑄 : Y → X is invariant on the level sets of Π .

Proof. Let 𝑦, 𝑦 ′ ∈ Y . By the discussion above, Π ( 𝑦 ) = Π ( 𝑦 ′ ) iff 𝑦 ′ 𝑖 -𝑦 𝑖 = 𝜆 1 𝑖 for some 𝜆 ∈ ℝ . Then for each 𝑖 ∈ N ,

Proposition C.4. The dynamical system (C.3) in the game's payoff space Y induces a dynamical system

in the game's payoff-difference space Z via the map (C.8) .

Proof. By the discussion in the previous section (and in particular Lemma C.2), if we exhibit a vector field 𝑍 on Z that is Π -related to 𝑌 , then our proof is complete. Thus we look for a vector field 𝑍 such that, for all 𝑦 ∈ Y ,

with 𝑧 = Π ( 𝑦 ) . By Eq. (C.7), ( Jac Π 𝑖 ) 𝛼 𝑖 𝛽 𝑖 = 𝛿 𝛼 𝑖 𝛽 𝑖 -𝛿 ˆ 𝛼 𝑖 𝛽 𝑖 . Since 𝑌 = 𝑣 ◦ 𝑄 , the sought-after vector field 𝑍 must fulfill, for all 𝑦 ∈ Y and all 𝛼 𝑖 ∈ A 𝑖 ,

with 𝑧 = Π ( 𝑦 ) . For each 𝑖 ∈ N define now (cf. Fig. 3)

for any 𝑦 𝑖 ∈ Π -1 𝑖 ( 𝑧 𝑖 ) , and denote by ˆ 𝑄 : Z → X the induced product map. Such map exists since Π 𝑖 is surjective, and is well-defined by Lemma C.3. It follows that the vector field on Z defined by

for all 𝑖 ∈ N , 𝑧 𝑖 ∈ Z 𝑖 , 𝛼 𝑖 ∈ A 𝑖 fulfills Eq. (C.11), and is hence Π -related to 𝑌 . ■

This result shows that, for every dual orbit 𝑦 ( 𝑡 ) of Eq. (C.3) with initial condition 𝑦 0 ∈ Y , the curve 𝑧 ( 𝑡 ) = Π ( 𝑦 ( 𝑡 )) is an orbit of the dynamical system (C.9) in Z with initial condition Π ( 𝑦 0 ) . To conclude this section we give a result implying that if the weighted Fenchel coupling (C.4) is a constant of motion constant then the solution orbits of (C.9) in Z are bounded.

Lemma C.5. For any 𝑖 ∈ N , let 𝑦 𝑖,𝑛 be a sequence in Y 𝑖 , and let 𝑝 𝑖 be a point in the relative interior of X 𝑖 . If sup 𝑛 | ℎ ∗ 𝑖 ( 𝑦 𝑖,𝑛 ) - ⟨ 𝑦 𝑖,𝑛 , 𝑝 𝑖 ⟩| < ∞ , then also the score differences remain bounded, i.e., | 𝑦 𝑖 𝛼 𝑖 ,𝑛 -𝑦 𝑖𝛽 𝑖 ,𝑛 | < ∞ for all 𝛼 𝑖 , 𝛽 𝑖 ∈ A 𝑖 and all 𝑛 .

Proof. See [41, Appendix D].

■

Lemma C.6. If the weighted Fenchel coupling (C.4) is a constant of motion under (FTRL-D) for some fully mixed 𝑝 ∈ X then the orbits of / 𝑧 = 𝑍 ( 𝑧 ) as in Eq. (C.9) are bounded in Z .

Proof. Assume that 𝐹 𝑚,𝑞 ( 𝑦 ) = P 𝑖 𝑚 𝑖 𝐹 𝑖 ( 𝑝 𝑖 , 𝑦 𝑖 ) = P 𝑖 𝑚 𝑖 GLYPH<0> ℎ 𝑖 ( 𝑝 𝑖 ) + ℎ ∗ 𝑖 ( 𝑦 𝑖 ) - ⟨ 𝑝 𝑖 , 𝑦 𝑖 ⟩ GLYPH<1> is a constant of motion for (FTRL-D) for some fully mixed 𝑝 ∈ X and some 𝑚 ∈ ℝ 𝑁 ++ . Let 𝑦 ( 𝑡 ) be an orbit of (FTRL-D) in Y , and let 𝑦 𝑖,𝑛 : = 𝑦 𝑖 ( 𝑡 𝑛 ) for any sequence of time 𝑡 𝑛 . Let furthermore 𝐹 𝑖,𝑛 : = ℎ ∗ 𝑖 ( 𝑦 𝑖,𝑛 ) - ⟨ 𝑝 𝑖 , 𝑦 𝑖,𝑛 ⟩ . Then sup 𝑛 | 𝐹 𝑖,𝑛 | < ∞ . By Lemma C.5, this implies that | 𝑧 𝑖 𝛼 𝑖 ,𝑛 | < ∞ for all 𝛼 𝑖 ∈ A 𝑖 , all 𝑖 ∈ N , and all 𝑛 . ■

C.5. Recurrence of FTRL in harmonic games. We now have all the ingredients to prove that almost every trajectory of play 𝑥 ( 𝑡 ) of (FTRL-D) in harmonic games returns arbitrarily close to its starting point infinitely often.

Theorem 2. Suppose Γ is harmonic. Then almost every orbit 𝑥 ( 𝑡 ) of (FTRL-D) returns arbitrarily close to its starting point infinitely often: specifically, for ( Lebesgue ) almost every initial condition 𝑥 ( 0 ) = 𝑄 ( 𝑦 ( 0 )) ∈ X , there exists an increasing sequence of times 𝑡 𝑛 ↑ ∞ such that 𝑥 ( 𝑡 𝑛 ) → 𝑥 ( 0 ) .

Proof of Theorem 2. The proof relies on the following steps:

the vector field 𝑍 defined in Eq. (C.13) has vanishing divergence, so its induced flow is volumepreserving in Z by Liouville's theorem;

the orbits of the dynamical system / 𝑧 = 𝑍 ( 𝑧 ) of Eq. (C.9) are bounded in Z since the weighted Fenchel coupling (C.4) is a constant of motion for FTRL in harmonic games;

the dynamical system / 𝑧 = 𝑍 ( 𝑧 ) is recurrent in Z by Poincaré theorem;

by continuity of Eq. (C.12), almost every trajectory of play 𝑥 ( 𝑡 ) of (FTRL-D) with initial condition in the image of ˆ 𝑄 returns arbitrarily close to its starting point infinitely often.

Indeed, div 𝑍 ( 𝑧 ) = P 𝑖 P 𝛼 𝑖 𝜕 𝜕𝑧 𝑖 𝛼 𝑖 (( 𝑣 𝑖 𝛼 𝑖 -𝑣 𝑖 ˆ 𝛼 𝑖 ) ◦ ˆ 𝑄 𝑖 ( 𝑧 𝑖 )) . For the first term, by chain rule,

since 𝜕𝑣 𝑖 𝛼 𝑖 𝜕𝑥 𝑖𝛽 𝑖 ≡ 0 by multilinearity of the payoff functions. The second term yields identical result with ˆ 𝛼 𝑖 ← 𝛼 𝑖 , so we conclude that div 𝑍 = 0 . By Lemma C.6, the invariance of the weighted Fenchel coupling under (FTRL-D) implies that the payoff differences 𝑧 𝑖 𝛼 𝑖 ( 𝑡 ) = 𝑦 𝑖 𝛼 𝑖 ( 𝑡 ) -𝑧 𝑖 ˆ 𝛼 𝑖 ( 𝑡 ) remain bounded for all 𝑡 ∈ [ 0 , ∞) . So, by Poincaré theorem, the dynamical system / 𝑧 = 𝑍 ( 𝑧 ) is Poincaré recurrent, i.e., there exists a sequence of time 𝑡 𝑛 ↑ ∞ such that lim 𝑛 →∞ 𝑧 ( 𝑡 𝑛 ) = 𝑧 0 for almost every 𝑧 0 ∈ Z . By continuity of (C.12), almost every trajectory of play 𝑥 ( 𝑡 ) = 𝑄 ( 𝑦 ( 𝑡 )) = ˆ 𝑄 ( 𝑧 ( 𝑡 )) with 𝑥 0 ∈ im ˆ 𝑄 fulfills lim 𝑛 →∞ 𝑥 ( 𝑡 𝑛 ) = 𝑥 0 , which concludes our proof by noting that the image of ˆ 𝑄 is the same as the image of 𝑄 . ■

D Discrete-time analysis

In this appendix, our aim is to provide the full proofs for the discrete-time guarantees of (FTRL+), as presented in Section 4. Our analysis hinges on a series of energy functions and associated template inequalities, which we introduce in the next section.

D.1. Lyapunov functions and template inequalities for (FTRL+) . The main building block of our analysis is a suitable Lyapunov function for the discrete-time algorithmic template (FTRL+). Motivated by the continuous-time analysis of Appendix C, we begin by considering the player-specific Fenchel couplings

induced by the regularizer ℎ 𝑖 of player 𝑖 ∈ N .

Suppose now that the game is harmonic relative to some measure 𝜇 = ( 𝜇 1 , . . . , 𝜇 𝑁 ) , let 𝑚 𝑖 = P 𝛼 𝑖 ∈ A 𝑖 𝜇 𝑖 𝛼 𝑖 denote the mass of 𝜇 𝑖 , and assume further that each player is running (FTRL+) with learning rate 𝜂 𝑖 > 0 . Our analysis will hinge on the energy function

which, as we show below, satisfies the following template inequality:

Proposition D.1. Suppose that each player is running (FTRL+) with learning rate 𝜂 𝑖 > 0 in a harmonic game as above. Then, for all 𝑝 𝑖 ∈ X 𝑖 , 𝑖 ∈ N , and all 𝑛 = 1 , 2 , . . . , the algorithm's energy 𝐸 𝑛 : = 𝐸 ( 𝑝, 𝑦 𝑛 ) enjoys the iterative bound:

Proof. We begin by applying the bound (B.29b) of Lemma B.3 with the array of substitutions

We then get

and hence, by (B.29b):

Accordingly, with 𝐸 𝑛 = 𝐸 ( 𝑝, 𝑦 𝑛 ) , the bound (D.2) follows by multiplying both sides by 𝑚 𝑖 / 𝜂 𝑖 and summing over 𝑖 ∈ N . ■

Thanks to Proposition D.1, we are now in a position to state and prove the following summability guarantee for (FTRL+).

Proposition D.2. Suppose that each player in a harmonic game Γ with harmonic measure 𝜇 is following (FTRL+) with learning rate 𝜂 𝑖 ≤ 𝑚 𝑖 𝐾 𝑖 [ 2 ( 𝑁 + 2 ) max 𝑗 𝑚 𝑗 𝐺 𝑗 ] -1 . Then, for all 𝑇 , we have:

In particular, the sequences 𝐴 𝑛 : = ∥ 𝑥 𝑛 + 1 / 2 -𝑥 𝑛 ∥ 2 and 𝐵 𝑛 : = ∥ 𝑥 𝑛 + 1 -𝑥 𝑛 + 1 / 2 ∥ 2 are both summable.

Proof. By reshuffling the terms of the template inequality (D.2), we get

We now proceed to bound each term of (D.6) individually, paying no heed to make the resulting bounds as tight as possible.

Bounding (D.6a) . By the Fenchel-Young inequality, we have:

Bounding (D.6b) . Again, by the Fenchel-Young inequality, we obtain:

Bounding (D.6c) . Finally, by the lower bound on the Fenchel coupling of Proposition B.2, we get:

Thus, by folding Eqs. (D.7)-(D.9) back into (D.6), we obtain the bound

Now, if we instantiate (D.10) to 𝑝 ← 𝑞 where 𝑞 is the strategic center of Γ , its left-hand side (LHS) will vanish by (HG-center). Hence, summing over all 𝑛 = 1 , 2 , . . . , 𝑇 , (D.10) ultimately yields

so (D.11) becomes

where we used our initialization convention 𝑥 1 = 𝑥 1 / 2 and the fact that the third line of (D.11) is negative. We thus get

from which our assertion follows immediately.

■

D.2. Proof of Theorem 3. We are now in a position to prove the regret guarantees of (FTRL+), which we restate below for convenience.

Theorem 3. Suppose that each player in a harmonic game Γ is following (FTRL+) with learning rate 𝜂 𝑖 ≤ 𝑚 𝑖 𝐾 𝑖 [ 2 ( 𝑁 + 2 ) max 𝑗 𝑚 𝑗 𝐺 𝑗 ] -1 and payoff models as per (13a) and (15) . Then the individual regret of each player 𝑖 ∈ N is bounded as

where 𝐻 𝑖 = max ℎ 𝑖 -min ℎ 𝑖 , and 𝐺 𝑖 is the Lipschitz modulus of 𝑣 𝑖 .

Proof. By a minor reshuffling of terms in (D.4), we readily get

and hence, by a repeated application of the Fenchel-Young inequality in its Peter-Paul form:

Hence, by using the Lipschitz continuity of 𝑣 𝑖 , we finally get

Now, by our step-size assumption, we readily obtain

Thus, summing over 𝑛 = 1 , 2 , . . . , 𝑇 , and keeping in mind that our assumptions for 𝜂 𝑖 also give 𝐺 𝑖 < 𝐾 𝑖 /( 2 𝜂 𝑖 ) , we finally get

where we used the fact that 𝐹 𝑖 ( 𝑝 𝑖 , 0 ) = ℎ ( 𝑝 ) -min ℎ 𝑖 ≤ max ℎ 𝑖 -min ℎ 𝑖 = : 𝐻 𝑖 . Our result then follows by invoking (D.5) and using the fact that 𝑚 𝑖 𝐺 𝑖 ≤ max 𝑗 𝑚 𝑗 𝐺 𝑗 for all 𝑖 ∈ N . ■

D.3. Proof of Theorem 4. With all this in hand, we are finally in a position to prove our main equilibrium convergence result for (FTRL+). For convenience, we restate the relevant theorem below.

Theorem 4. Suppose that each player in a harmonic game Γ follows (FTRL+) with learning rate 𝜂 𝑖 ≤ 𝑚 𝑖 𝐾 𝑖 [ 2 ( 𝑁 + 2 ) max 𝑗 𝑚 𝑗 𝐺 𝑗 ] -1 and payoff models as per (13a) and (15) . Then 𝑥 𝑛 converges to the set of Nash equilibria of Γ .

Proof. Our proof proceeds in a series of steps, as detailed below.

Step 1: Convergence of energy levels . We begin by showing that the energy 𝐸 𝑛 ≡ 𝐸 ( 𝑞, 𝑦 𝑛 ) of (FTRL+) relative to the game's harmonic center converges to some finite value 𝐸 ∞ . That this is so follows from a well-known property of quasi-Fejér sequences [10, Lemma 3.1], whose proof we reproduce below for completeness.

Indeed, by Eq. (D.10) and Proposition D.2, we have

with 𝜀 𝑛 , 𝑛 = 1 , 2 , . . . summable. Letting 𝐸 ′ 𝑛 = 𝐸 𝑛 + P ∞ 𝑘 = 𝑛 𝜀 𝑘 , we further get

by (D.19), so 𝐸 ′ 𝑛 converges. Since 𝜀 𝑛 is summable, it follows that 𝐸 𝑛 also converges, as claimed. ⋄

Step 2: Boundedness of score differences . We now proceed to show that the normalized score differences 𝑧 𝑛 = Π ( 𝑦 𝑛 ) where Π is the normalization operator (C.8) are bounded. Indeed, by the definition of 𝐸 𝑛 = 𝐸 ( 𝑞, 𝑦 𝑛 ) = P 𝑖 ∈ N ( 𝑚 𝑖 / 𝜂 𝑖 ) 𝐹 𝑖 ( 𝑞 𝑖 , 𝑦 𝑖,𝑛 ) , it follows that sup 𝑛 𝐹 𝑖 ( 𝑞 𝑖 , 𝑦 𝑖,𝑛 ) < ∞ for all 𝑖 ∈ N . Thus, by Lemma C.5, we conclude that each component of 𝑧 𝑛 is bounded, so 𝑧 is itself bounded. ⋄

Step 3: Convergent subsequences of (FTRL+). We now observe that (FTRL+) enjoys the following series of properties:

The sequence 𝑧 𝑛 = Π ( 𝑦 𝑛 ) admits a subsequence 𝑧 𝑛 𝑘 that converges to some limit point 𝑧 ∞ ∈ Z (a consequence of the fact that 𝑧 𝑛 is bounded, see above).

In turn, this implies that the subsequence 𝑥 𝑛 𝑘 = 𝑄 ( 𝑦 𝑛 𝑘 ) = ˆ 𝑄 ( 𝑧 𝑛 𝑘 ) converges to some 𝑥 ∞ ∈ X .

Since the sequences 𝐴 𝑛 = ∥ 𝑥 𝑛 + 1 / 2 -𝑥 𝑛 ∥ 2 and 𝐵 𝑛 = ∥ 𝑥 𝑛 + 1 -𝑥 𝑛 + 1 / 2 ∥ 2 are both summable (by Proposition D.2), we further have lim 𝑘 →∞ 𝑥 𝑛 𝑘 + 1 / 2 = 𝑥 ∞ and, more generally, by a straightforward induction:

Likewise, for the sequence of payoff signals ˆ 𝑣 𝑛 , we have

Step 4: Variational characterization of limit points . We now proceed to show that 𝑣 ∞ : = 𝑣 ( 𝑥 ∞) belongs to the polar cone PC ( 𝑥 ∞) = { 𝑤 ∈ Y : ⟨ 𝑤, 𝑥 -𝑥 ∞⟩ ≤ 0 for all 𝑥 ∈ X } to X at 𝑥 ∞ . To do so, suppose that (FTRL+) performs 𝑟 steps from 𝑛 𝑘 so

where, to ease notation, we have made the simplifying assumption that 𝜂 𝑖 = 𝜂 for all 𝑖 ∈ N . 13 Then, by invoking Lemma B.4 with 𝑦 ← 𝑦 𝑛 𝑘 and 𝑦 + ← 𝑦 𝑛 𝑘 + 𝑟 , we obtain

where, in the second line, we have used the fact that ⟨ 𝑦, 𝑥 ′ -𝑥 ⟩ = ⟨ Π ( 𝑦 ) , 𝑥 ′ -𝑥 ⟩ for all 𝑥, 𝑥 ′ ∈ X and all 𝑦 ∈ Y . Thus, letting 𝑘 →∞ , we get from Step 3 and the continuity of ∇ ℎ that

for all 𝑟 = 1 , 2 , . . . and all 𝑥 ∈ X . 14 Since 𝑟 can be chosen arbitrarily, we must have ⟨ 𝑣 ( 𝑥 ∞) , 𝑥 -𝑥 ∞⟩ ≤ 0 for all 𝑥 ∈ X . Hence, by the variational characterization (VI) of Nash equilibria, we conclude that 𝑥 ∞ must be itself a Nash equilibrium of Γ , and our proof is complete. ■

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction follows the same logic and mostly also order as the technical results of the article.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss exact assumptions throughout. Moreover, in the concluding remarks we focus on limitations that are also natural avenues for future research.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We wrote the article with highest attention to rigor and mathematical detail. As such we provide all assumptions for each result and use unambigious notation and language.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our simulations are merely illustrative and as such are not critical for the main results of the paper. However, we fully disclose how they were performed.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not release the code at this point as we don't have time to anonymize it. We describe in detail how simulations are performed and will make the code available at a later stage.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips. cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all details for the simulations.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All simulations were performed on standard laptops and as such do not carry any particular computational burden.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss in the introduction the use of learning in games.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper looks at how no-regret learning algorithms behave in harmonic games, which model situations where players have conflicting interests. This is different from the often-studied potential games where interests are shared. The authors show that in continuous time, FTRL dynamics are Poincaré recurrent and do not converge. In discrete time, FTRL can cause never-ending cycles of best responses. But, by adding an extrapolation step (FTRL+), they show that learning can reach a Nash equilibrium from any starting point, ensuring constant regret and giving important insights into harmonic games.

STRENGTHS:
The paper stands out for its novel exploration of no-regret learning in harmonic games, which are less studied compared to potential games. The use of FTRL+ to ensure convergence in these games is a creative extension of existing algorithms. Furthermore, the the result on Poincare recurrence in Theorem 2 is insightful and interesting. The statements and proofs of the results are clear and rigorous, and the figured and presentation aid clarity.

WEAKNESSES:
Empirical results beyond the matching pennies example could strengthen the work. The proofs are somewhat dense and challenging to follow. A stronger differentiation from the work by Legacci et al. would be beneficial.

QUESTIONS:
How do the choices of parameters $\lambda_i$ and $\eta_i$ affect the performance of the FTRL+ algorithm? How might one select these parameters for different types of harmonic games? Could you expand on the types of harmonic games tested in your experiments? Are there benchmarks the algorithm could be compared against?

LIMITATIONS:
Limitations are sufficiently covered given the theoretical nature of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the behavior of no-regret dynamics, and in particular follow the regularized leader (FTRL), in harmonic games--the strategic counterpart of potential games. They establish the following main results: i) the continuous-time version of FTRL is Poincare recurrent; ii) an extrapolated version of FTRL (FTRL+), which includes the optimistic and mirror-prox variant, converges to a Nash equilibrium; iii) under FTRL+ each player attains $O(1)$ regret.\n\nSTRENGTHS:\nThe paper provides a clear and compelling motivation for investigating no-regret dynamics in harmonic games. In particular, the decomposition of Candogan et al. shows that, together with potential games, harmonic games constitute the basic building blocks of any game. Further, harmonic games generalized zero-sum games with a fully-mixed equilibrium, the latter class having received extensive attention in the literature. The results obtained in the paper make a concrete step towards better understanding no-regret dynamics in harmonic games, which I believe is an important contribution and will be welcomed by the learning in games community at NeurIPS. I also believe that the paper may lead to interesting follow-up work on harmonic games. Overall, the main message of the paper is compelling.\n\nThe writing of the paper is also exemplary and of really high quality. The main body does a great job at providing high-level sketches of the main technical ideas, and the appendix is also very well organized and polished. I did not detect any notable issues with the soundness of the claims.\n\nWEAKNESSES:\nRegarding the significance of the results, it can be argued that some of the results are somewhat incremental based on existing results. The fact that FTRL is Poincare recurrent is perhaps not that surprising conceptually given the recent paper of Legacci et al. which shows that for the special case of replicator dynamics; I do understand though that the extension requires some new ideas, as the authors discuss. Moreover, the fact that FTRL+ attains constant regret is also not particularly surprising. To put that result into better context, I would recommend citing the paper of Daskalakis, Fishelson and Golowich (NeurIPS 2021) that shows $O(\\log^4 T)$ regret for any game; as far as I understand, the only improvement pertains to logarithmic factors in $T$, which is perhaps not very significant. But I do not believe that the points above constitute basis for rejection.\n\nQUESTIONS:\nNo further questions.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-agent no-regret learning dynamics in general harmonic games, which is the strategic complement of potential games. The paper's main contributions are the convergence properties of the family of \"Follow-the-regularized-leader\" (FTRL) algorithms and its variants in general harmonic games, significantly extending previous results in two-player zero-sum games with fully mixed equilibrium and uniformly harmonic games. Specifically, the main results are (1) the continuous-time dynamics of FTRL are Poincaré recurrent, (2) the discrete-time dynamics of FTRL diverges, but FTRL with an extrapolation step (called FTRL+) guarantees $O(1)$ individual regret and global asymptotic last-iterate convergence/point convergence to a Nash equilibrium. These results show that potential and harmonic games are complementary not only from the strategic but also from the dynamic viewpoint.\n\nSTRENGTHS:\n1. This paper is very well-written and easy to follow. I appreciate that the main body clearly explains the main ideas, and the appendix provides rigorous and detailed proof. \n2. The problem of no-regret learning dynamics in games is relevant. This paper contributes to the area by establishing the previously unknown convergence properties of the FTRL dynamics in general harmonic games, including Poincaré recurrence of continuous-time FTRL, last-iterate convergence, and constant regret of FTRL+. The results and techniques in this paper shed light on the further development of learning in harmonic games.\n\nWEAKNESSES:\nI do not see any major weakness in the paper. \n\nMinor comments:\n\nTheorem 3/4: $m_i$ is used to choose the step size but has not been defined? The definition is in the appendix but a pointer should be given in the main body.\n\nLine 314: \"Similar bounds have only been established for optimistic methods in two-player zero-sum games [25]\". [25] established constant regret bounds for all variationally stable games, a class of multi-player games that includes two-player zero-sum games. This should be acknowledged. \n\n[25] Hsieh, Yu-Guan, Kimon Antonakopoulos, and Panayotis Mertikopoulos. \"Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium.\" In Conference on Learning Theory. 2021\n\nQUESTIONS:\n1. $m_i$ is used to choose the step size for convergence of FTRL+. Is there an efficient way to estimate an upper bound of $m_i$?\n2. This paper left the rate of convergence for FTRL+ as an open question. I would like to know if similar results hold for optimistic online mirror descent (OOMD)-type algorithms in harmonic games. I think proving convergence rates for OOMD algorithms, especially OGDA, might be more promising than FTRL+.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe contributions of this paper are two-fold:\n\ni) They prove that continuous FTRL dynamics for general harmonic games are Poincaré recurrent and hence do not converge. Their result generalizes the original result of \n\nMertikopoulos, P., Papadimitriou, C. H., and Piliouras, G., 'Cycles in adversarial regularized learning,' In SODA ’18: Proceedings of the 29th annual ACM-SIAM Symposium on Discrete Algorithms, 2018, \n\nfor zero-sum games with an interior equilibrium (as a trivial example of harmonic games), and the result of \n\nLegacci, Davide, Panayotis Mertikopoulos, and Bary Pradelski, 'A geometric decomposition of finite games: Convergence vs. recurrence under no-regret learning,' arXiv preprint arXiv:2405.07224 (2024), \n\nfor uniform harmonic games to general harmonic games by volume-preserving flow arguments.\n\nii) Moreover, they show that the regret of general harmonic games for the class of Extrapolated FTRL algorithms (discrete-time dynamics) is constant in time ($T$).\n\nSTRENGTHS:\n- The contributions of this paper are solid and interesting. This paper addresses fundamental problems in harmonic games.\n\n- I enjoyed reading the paper. It is well-written, especially the introduction to harmonic games in the appendix and the new viewpoint of Mixed characterization of harmonic games and its role on upper bounding the path length of the no-regret dynamics.\n\nWEAKNESSES:\n- Just some missing citations on no-regret learning for games, e.g.,\n\n-- Daskalakis, Constantinos, Alan Deckelbaum, and Anthony Kim. \"Near-optimal no-regret algorithms for zero-sum games.\" Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms. \\\n-- Chen, Xi, and Binghui Peng. \"Hedging in games: Faster convergence of external and swap regrets.\" Advances in Neural Information Processing Systems 33 (2020). \\\n-- Piliouras, Georgios, Ryann Sim, and Stratis Skoulakis. \"Beyond time-average convergence: Near-optimal uncoupled online learning via clairvoyant multiplicative weights update.\" Advances in Neural Information Processing Systems 35 (2022). \\\n...\n\n\n__Minor Suggestions__\n- Please emphasize in the claims that the regret bounds entailed are constant in $T$ and not potentially in the dimension of the action sets $\\mathcal{A}$, since $H_i$ is not constant in $|\\mathcal{A}_i|$.\n- Regarding the notation for $\\nu_i(x)$, maybe consider changing it to $\\nu_i(x_{-i})$ to improve readability.\n- In line 268, $x_{n + 1}$ instead of $x_{n}$?\n- In line 277, \"and\" before \"which\" seems to be a typo.\n\nQUESTIONS:\nI do not have any particular question (as the results shown in this paper are not unexpected) except that could authors provide some intuition on the conjecture that convergence to Nash Eq. of no-regret learnings for general harmonic games would be linear beyond the work of \n\nWei, C.-Y., et al. Linear last-iterate convergence in constrained saddle-point optimization. In ICLR ’21: Proceedings of the 2021 International Conference on Learning Representations, 2021.\n\n? (Non-asymptotic version of Theorem 4)\n\nLIMITATIONS:\nNot applied.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}