Title: The Equivalence of Dynamic and Strategic Stability under Regularized Learning in Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Regularized learning

dynamic stability

strategic stability

Nash equilibrium
Abstract: In this paper, we examine the long-run behavior of regularized, no-regret learning in finite N-player games. A well-known result in the field states that the empirical frequencies of play under no-regret learning converge to the game’s set of coarse correlated equilibria; however, our understanding of how the players' _actual strategies_ evolve over time is much more limited – and, in many cases, non-existent. This issue is exacerbated further by a series of recent results showing that _only_ strict Nash equilibria are stable and attracting under regularized learning, thus making the relation between learning and _pointwise_ solution concepts particularly elusive. In lieu of this, we take a more general approach and instead seek to characterize the _setwise_ rationality properties of the players' day-to-day trajectory of play. To do so, we focus on one of the most stringent criteria of setwise strategic stability, namely that any unilateral deviation from the set in question incurs a cost to the deviator – a property known as _closedness under better replies_ (club). In so doing, we obtain a remarkable equivalence between strategic and dynamic stability: _a product of pure strategies is closed under better replies if and only if its span is stable and attracting under regularized learning._ In addition, we estimate the rate of convergence to such sets, and we show that methods based on entropic regularization (like the exponential weights algorithm) converge at a geometric rate, while projection-based methods converge within a finite number of iterations, even with bandit, payoff-based feedback.

The Equivalence of Dynamic and Strategic Stability under Regularized Learning in Games

Abstract

In this paper, we examine the long-run behavior of regularized, no-regret learning in finite games. A well-known result in the field states that the empirical frequencies of no-regret play converge to the game's set of coarse correlated equilibria; however, our understanding of how the players' actual strategies evolve over time is much more limited - and, in many cases, non-existent. This issue is exacerbated by a series of recent results showing that only strict Nash equilibria are stable and attracting under regularized learning, thus making the relation between learning and pointwise solution concepts particularly elusive. In lieu of this, we take a more general approach and instead seek to characterize the setwise rationality properties of the players' day-to-day play. To that end, we focus on one of the most stringent criteria of setwise strategic stability, namely that any unilateral deviation from the set in question incurs a cost to the deviator - a property known as closedness under better replies (club). In so doing, we obtain a far-reaching equivalence between strategic and dynamic stability: a product of pure strategies is closed under better replies if and only if its span is stable and attracting under regularized learning. In addition, we estimate the rate of convergence to such sets, and we show that methods based on entropic regularization (like the exponential weights algorithm) converge at a geometric rate, while projection-based methods converge within a finite number of iterations, even with bandit, payoff-based feedback.

1 Introduction

Background. The question of whether players can learn to emulate rational behavior through repeated interactions has been one of the mainstays of non-cooperative game theory, and it has recently gained increased momentum owing to a surge of breakthrough applications to machine learning and data science, from online ad auctions to multi-agent reinforcement learning. Informally, this question can be stated as follows:

If each player follows an iterative procedure aiming to increase their individual payoff, does the players' long-run behavior converge to a rationally admissible state?

A natural setting for studying this question is to assume that each player is following a no-regret algorithm, i.e., a policy which is asymptotically as good against a given sequence of payoff functions as the best fixed strategy in hindsight. In this framework, the link between learning and rationality is provided by a folk result which states that, under no-regret learning, the empirical frequency of play converges to the game's set of coarse correlated equilibria (CCE) - also known as the game's Hannan set [26]. This result has been of seminal importance to the field because no-regret play can be achieved via a wide class of 'regularized learning' policies, as exemplified by the 'followthe-regularized-leader' (FTRL) family of algorithms [45, 46] and its variants - optimistic methods [15, 28, 41, 42, 47], HEDGE / EXP3 [4, 6, 10, 11], implicitly normalized forecasters [1, 3], etc.

All these policies have (at least) one thing in common: they seek to provide the tightest possible guarantees for each player's individual regret, thus accelerating convergence to the game's Hannan set. As such, in games where the marginalization of coarse correlated equilibria coincides with the game's Nash equilibria (like two-player zero-sum games), we obtain a positive equilibrium convergence guarantee: the long-run empirical frequency of play evolves 'as if' the players were rational to begin with - i.e., as if they had full knowledge of the game, common knowledge of rationality, the ability to communicate this knowledge, etc.

On the other hand, in many other contexts, the marginals of Hannan-consistent correlated strategies may fail even the weakest axioms of rational behavior and rationalizability (such as the elimination of strictly dominated strategies). In particular, a well-known example of Viossat & Zapechelnyuk [49] (which we discuss in detail in Section 4) shows that it is possible to have negative regret for all time, but still employ only strictly dominated strategies throughout the entire horizon of play.

The reason for this disconnect is that no-regret play has significant predictive power for the empirical frequency of play - that is, the long-run empirical distribution of pure strategy profiles - but much less so for the players' day-to-day sequence of play - i.e., the evolution of the players' actual mixed strategies over time. In particular, even when the marginalization of the Hannan set is Nash, the actual trajectory of play may - and, in fact, often does - diverge away from the game's set of equilibria [16, 24, 34-36] or exhibits chaotic, unpredictable oscillations [12, 38]. Thus, especially in the context of regularized learning - where players learn independently from one another, with no common correlating device - the blanket guarantee of no-regret play may quickly become irrelevant, and even misleading, providing the veneer of rational behavior but not the substance.

Motivated by the above, our paper seeks to understand the rationality properties of the players' actual sequence of play under regularized learning, as encoded by the following question:

Which sets of mixed strategies are stable and attracting under regularized learning? Are these sets robust to strategic deviations? And, if so, is the converse also true?

Our contributions in the context of related work. This question has attracted significant interest in the literature, especially in its pointwise version, namely: Which mixed strategy profiles are stable and attracting under regularized learning? Are the dynamics' stable states robust to unilateral deviations? And, if so, are these the only stable states of regularized learning?

In the related setting of population games, the answer to this question is sometimes referred to as the 'folk theorem of evolutionary game theory' [14, 27, 51]. Somewhat informally, this theorem states that, under the replicator dynamics (the continuous-time analogue of the exponential / multiplicative weights algorithm, itself an archetypal regularized learning method), the following is true for all games: only Nash equilibria are (Lyapunov) stable, and a state is stable and attracting under the replicator dynamics if and only it is a strict Nash equilibrium of the underlying game [27, 51]. 1

In the context of regularized learning, [13, 20, 33] showed that a similar equivalence holds for the dynamics of FTRL in continuous time: a state is stable and attracting under the FTRL dynamics if and only if it is a strict Nash equilibrium. Subsequently, Giannou et al. [22, 23] extended this equivalence to an entire class of regularized learning schemes, with different types of feedback and/or update structures - from optimistic methods to algorithms run with bandit, payoff-based information. In all these cases, the same principle emerges: under regularized learning, a state is asymptotically stable and attracting if and only if it is a strict Nash equilibrium .

This is an important pointwise prediction but it does not cover cases where regularized learning algorithms do not converge to a point, but to a set (such as a limit cycle or other non-trivial attractor). In this case, the very definition of strategic stability is an intricate affair, and there are several definitions that come into play [7, 17, 21, 43]. The first such notion that we consider is that of 'resilience to strategic deviations', namely that every unilateral deviation from the set under study is deterred by some other element thereof. Our first contribution in this direction is a universal guarantee to the effect that, with probability 1 , in any game, and from any initial condition, the long-run limit of any regularized learning algorithm is a resilient set.

This result is significant in its universality, but the notion of resilience is not sufficiently strong to disallow irrational behavior - and, in fact, it is subject to similar shortcomings as Hannan consistency.

To account for this deficiency, we turn to a much more stringent criterion of setwise strategic stability, that of closedness under better replies (club). This notion, originally due to Ritzberger & Weibull [43], states that any deviation from a product of pure strategies is costly, and it is one of the strictest setwise refinements in game theory. In particular, it refines the notion of closedness under rational behavior (curb) [7], and it satisfies all the seminal strategic stability requirements of Kohlberg & Mertens [30], including robustness to strategic payoff perturbations. 2

In this general context, we show that regularized learning enjoys a striking relation with club sets: A product of pure strategies is closed under better replies if and only if its span is stable and attracting under regularized learning. In fact, we show that this equivalence can be refined to sets that are minimally closed under better replies (in the sense that they do not contain a strictly smaller closed under better replies (club) set): a product of puer strategies is minimally club (m-club) if and only if its span is irreducibly stable and attracting (in that it does not contain a smaller asymptotically stable span of strategies). Finally, we also estimate the rate of convergence to club sets, and we establish convergence at a geometric rate for entropically regularized methods - like HEDGE and EXP3 - and in a finite number of iterations under projection-based methods.

In light of the above, our results can be seen both as a far-reaching setwise generalization of the folk theorem of evolutionary game theory, as well as a bona fide algorithmic analogue of a precursor result for the replicator dynamics, originally due to Ritzberger & Weibull [43]. Importantly, our analysis covers several different update structures - 'vanilla' regularized methods, but also their optimistic variants - as well as a wide range of information models - from full payoff information to bandit, payoff-based feedback.

2 Preliminaries

We start by recalling some basics from game theory, roughly following the classical treatise of Fudenberg & Tirole [21]. First, a finite game in normal form consists of ( i ) a finite set of players 𝑖 ∈ N ≡ { 1 , . . . , 𝑁 } ; ( ii ) a finite set of actions - or pure strategies -A 𝑖 per player 𝑖 ∈ N ; and ( iii ) an ensemble of payoff functions 𝑢 𝑖 : Q 𝑗 A 𝑗 → ℝ , each determining the reward 𝑢 𝑖 ( 𝛼 ) of player 𝑖 ∈ N in a given action profile 𝛼 = ( 𝛼 1 , . . . , 𝛼 𝑁 ) . Collectively, we will write A = Q 𝑗 A 𝑗 for the game's action space and Γ ≡ Γ ( N , A , 𝑢 ) for the game with primitives as above.

During play, each player 𝑖 ∈ N may randomize their choice of action by playing a mixed strategy , i.e., a probability distribution 𝑥 𝑖 ∈ X 𝑖 : = Δ ( A 𝑖 ) over A 𝑖 that selects 𝛼 𝑖 ∈ A 𝑖 with probability 𝑥 𝑖 𝛼 𝑖 . To lighten notation, we identify 𝛼 𝑖 ∈ A 𝑖 with the mixed strategy that assigns all weight to 𝛼 𝑖 (thus justifying the terminology 'pure strategies'). Then, writing 𝑥 = ( 𝑥 𝑖 ) 𝑖 ∈ N for the players' strategy profile and X = Q 𝑖 X 𝑖 for the game's strategy space , the players' payoff functions may be extended to all of X by setting

where, in a slight abuse of notation, we write 𝑥 𝛼 for the joint probability of playing 𝛼 ∈ A under 𝑥 , i.e., 𝑥 𝛼 = Q 𝑖 𝑥 𝑖 𝛼 𝑖 . This randomized framework will be referred to as the mixed extension of Γ and we will denote it by Δ ( Γ ) .

For concision, we will also write ( 𝑥 𝑖 ; 𝑥 -𝑖 ) = ( 𝑥 1 , . . . , 𝑥 𝑖 , . . . , 𝑥 𝑁 ) for the strategy profile where player 𝑖 plays 𝑥 𝑖 ∈ X 𝑖 against the strategy profile 𝑥 -𝑖 ∈ Q 𝑗 ≠ 𝑖 X 𝑗 of all other players (and likewise for pure strategies). In this notation, we also define each player's mixed payoff vector as

so the payoff to player 𝑖 ∈ N under 𝑥 ∈ X becomes

Moving forward, the best-response correspondence of player 𝑖 ∈ N is defined as the set-valued mapping br 𝑖 : X ⇒ X 𝑖 given by

Extending this over all players, we will write br = Q 𝑖 br 𝑖 for the product correspondence br ( 𝑥 ) = br 1 ( 𝑥 ) × · · · × br 𝑁 ( 𝑥 ) , and we will say that 𝑥 ∗ ∈ X is a Nash equilibrium (NE) if 𝑥 ∗ ∈ br ( 𝑥 ∗ ) .

Equivalently, given that 𝑢 𝑖 ( 𝑥 ′ 𝑖 ; 𝑥 -𝑖 ) is linear in 𝑥 ′ 𝑖 , we conclude that 𝑥 ∗ is a Nash equilibrium if and only if

As a final point of note, if 𝑥 ∗ is a Nash equilibrium where each player has a unique best response - that is, br 𝑖 ( 𝑥 ∗ ) = { 𝑥 ∗ 𝑖 } for all 𝑖 ∈ N - we will say that 𝑥 ∗ is strict because, in this case, 𝑢 𝑖 ( 𝑥 ∗ ) > 𝑢 𝑖 ( 𝑥 𝑖 ; 𝑥 ∗ -𝑖 ) for all 𝑥 𝑖 ≠ 𝑥 ∗ 𝑖 , 𝑖 ∈ N . An immediate consequence of this is that strict equilibria are pure , i.e., each 𝑥 ∗ 𝑖 is a pure strategy. Among Nash equilibria, strict equilibria are the only ones that are 'structurally robust' (in the sense that they remain invariant to small perturbations of the underlying game), so they play a particularly important role in game theory.

3 Regularized learning in games

Throughout our paper, we will consider iterative decision processes that unfold as follows:

At each stage 𝑡 = 1 , 2 , . . . , every participating agent selects an action.

Agents receive a reward determined by their chosen actions and their individual payoff functions.

Based on this reward (or other feedback), the agents update their strategies and the process repeats.

In this online setting, a crucial requirement is the minimization of the players' regret , i.e., the difference between a player's cumulative payoff over time and the player's best possible strategy in hindsight. Formally, if the players' actions at each epoch 𝑡 = 1 , 2 , . . . are collectively drawn by the probability distribution 𝑧 𝑡 ∈ Δ ( A ) , the regret of each player 𝑖 ∈ N is defined as

and we will say that player 𝑖 has no regret if Reg 𝑖 ( 𝑇 ) = 𝑜 ( 𝑇 ) .

One of the most widely used policies to achieve no-regret play is the so-called 'follow-the-regularizedleader' (FTRL) family of algorithms and its variants [45, 46]. To motivate the analysis to come, we begin with an archetypal FTRL method, the exponential / multiplicative weights algorithm, also known as HEDGE [4, 5, 11].

3.1. A gentle start. We begin our discussion with a 'stimulus-response' approach in the spirit of Erev & Roth [19]: First, at each stage 𝑡 = 1 , 2 , . . . , every player 𝑖 ∈ N employs a mixed strategy 𝑥 𝑖,𝑡 ∈ X 𝑖 to select an action 𝛼 𝑖,𝑡 ∈ A 𝑖 . Subsequently, to measure the performance of their pure strategies over time, each player further maintains a score variable which is updated recursively as

In words, 𝑦 𝑖 𝛼 𝑖 ,𝑡 simply tracks the cumulative payoff of the pure strategy 𝛼 𝑖 ∈ A 𝑖 up to time 𝑡 (inclusive). 3 As such, this score can be treated as a propensity to play a given pure strategy at any given stage: the strategies 𝛼 𝑖 ∈ A 𝑖 with the highest propensity scores 𝑦 𝑖 𝛼 𝑖 ,𝑡 + 1 should be played with higher probability at stage 𝑡 + 1 .

The most widely used instantiation of this stimulus-response mechanism is the logit choice rule

which means that each player selects an action with probability that is exponentially proportional to its score. In this way, we obtain the exponential / multiplicative weights - or HEDGE - algorithm

where 𝛾 𝑡 is the algorithm's 'learning rate'. For an appetizer to the literature on (HEDGE), see [2, 10, 11, 31, 45] and references therein.

The rest of the methods we discuss below will vary some - or even all - of the components of (HEDGE): the information used to update the players' propensity scores, the way that propensity scores are mapped to mixed strategies, and/or even the way that pure actions are selected. However, all of the methods under study will be characterized by the same 'stimulus-response' reinforcement mechanism: actions that seem to be performing better over time are employed with higher probability, up to some 'regularization' that incentivizes exploration of underperforming actions.

3.2. The regularized learning template. In the rest of our paper, we will work with an abstract regularized learning (RL) template which builds on the same stimulus-response principle as (HEDGE), while allowing us to simultaneously consider different types of feedback, strategy sampling policies, update structures, etc. To lighten notation below, we will drop the player index 𝑖 ∈ N when the meaning can be inferred from the context; also, to stress the distinction between 'strategy-like' and 'payoff-like' variables, we will write throughout Y 𝑖 : = ℝ A 𝑖 and Y : = Q 𝑖 Y 𝑖 for the game's ' payoff space ', in direct analogy to X 𝑖 and X = Q 𝑖 X 𝑖 for the game's strategy space .

With all this in hand, consider the following general class of regularized learning methods:

Aggregate payoff information (stimulus):

Update choice probabilities (response):

In tune with (HEDGE), the various elements of (RL) are defined as follows:

𝑋 𝑖,𝑡 ∈ X 𝑖 denotes the mixed strategy of player 𝑖 at time 𝑡 = 1 , 2 , . . .

𝑌 𝑖,𝑡 ∈ Y 𝑖 is a 'score vector' that measures the performance of the player's actions over time.

𝑄 𝑖 : Y 𝑖 → X 𝑖 is a 'regularized choice map' that maps score vectors to choice probabilities.

ˆ 𝑣 𝑖,𝑡 is a surrogate / approximation of the mixed payoff vector 𝑣 𝑖 ( 𝑋 𝑡 ) of player 𝑖 at time 𝑡 .

𝛾 𝑡 > 0 is a step-size / sensitivity parameter of the form 𝛾 𝑡 ∝ 1 / 𝑡 ℓ 𝛾 for some ℓ 𝛾 ∈ [ 0 , 1 ] .

In words, at each stage of the process, every player 𝑖 ∈ N observes - or otherwise estimates - a proxy ˆ 𝑣 𝑖,𝑡 of their individual payoff vector; subsequently, players augment their actions' scores based on this information, they select a mixed strategy via the regularized choice map 𝑄 𝑖 , and the process repeats. To streamline our presentation, we discuss in detail the precise definition of ˆ 𝑣 and 𝑄 in Sections 3.3 and 3.4 below, and we present a series of examples of (RL) in Section 3.5 right after.

3.3. Aggregating payoff information. As noted above, the main idea of regularized learning is to track the players' payoff vector 𝑣 ( 𝑋 𝑡 ) . Importantly, there are several different modeling choices that can be made here: players may have direct access to their payoff vectors (in the full information setting), or some noisy approximation obtained by an inner randomization of the algorithm (e.g., when they receive information on their pure actions); they may have to recreate their payoff vectors altogether (as in the bandit setting), or their estimates may be based on a strategy other than the one they actually played (as in the case of optimistic algorithms).

In all cases, we will represent the surrogate payoff vector ˆ 𝑣 𝑡 as

where 𝑏 𝑡 = 𝔼 [ ˆ 𝑣 𝑡 | F 𝑡 ] -𝑣 ( 𝑋 𝑡 ) and 𝑈 𝑡 = ˆ 𝑣 𝑡 -𝔼 [ ˆ 𝑣 𝑡 | F 𝑡 ] respectively denote the offset and the random error of ˆ 𝑣 𝑡 relative to 𝑣 ( 𝑋 𝑡 ) . To streamline our presentation, we will also assume that ∥ 𝑏 𝑡 ∥ = O ( 1 / 𝑡 ℓ 𝑏 ) and ∥ 𝑈 𝑡 ∥ = O ( 𝑡 ℓ 𝜎 ) for some ℓ 𝑏 , ℓ 𝜎 ≥ 0 ; we discuss the specifics of these bounds later in the paper.

3.4. From scores to strategies. Regarding the 'scores-to-strategies' step of (RL), we will follow the classical approach of Shalev-Shwartz [45] and assume that each player is employing a regularized choice map of the general form

In the above, the regularizer ℎ 𝑖 : X 𝑖 → ℝ acts as a penalty that smooths out the 'hard' argmax correspondence 𝑦 𝑖 ↦→ arg max 𝑥 𝑖 ∈ X 𝑖 ⟨ 𝑦 𝑖 , 𝑥 𝑖 ⟩ . Accordingly, instead of following the 'leader' (i.e., playing the strategy with the highest propensity score), players follow the 'regularized leader' - that is, they allow for a certain degree of uncertainty in their choice of strategy [10, 33, 45, 46].

To ease notation, we will work with kernelized regularizers of the form ℎ 𝑖 ( 𝑥 𝑖 ) = P 𝛼 𝑖 ∈ A 𝑖 𝜃 ( 𝑥 𝑖 𝛼 𝑖 ) for some continuous function 𝜃 : [ 0 , 1 ] → ℝ with inf 𝑧 ∈( 0 , 1 ] 𝜃 ′′ ( 𝑧 ) > 0 . We will also say that the players' regularizers are steep if lim 𝑧 → 0 + 𝜃 ′ ( 𝑧 ) = -∞ , and non-steep otherwise.

Example 3.1. A standard family of kernelized regularizers is given by 𝜃 ( 𝑧 ) = 𝑧 𝜌 /[ 𝜌 ( 𝜌 -1 )] for 𝜌 ∈ ( 0 , 1 ) ∪ ( 1 , 2 ] and 𝜃 ( 𝑧 ) = 𝑧 log 𝑧 for 𝜌 = 1 [10, 31, 33, 53]. This family includes:

For 𝜌 = 2 , the quadratic regularizer 𝜃 ( 𝑧 ) = 𝑧 2 / 2 , which yields the Euclidean projection map

For 𝜌 = 1 , the entropic regularizer 𝜃 ( 𝑧 ) = 𝑧 log 𝑧 , which yields the logit choice map (7). √

For 𝜌 = 1 / 2 , the fractional power regularizer 𝜃 ( 𝑧 ) = -4 𝑧 that underlies the TSALLIS-INF algorithm of [1, 53] (see also Section 3.5 below). ♦

3.5. Specific algorithms. We now proceed to discuss some archetypal examples of (RL).

Algorithm 1 (Follow the regularized leader) . The standard 'follow-the-regularized-leader' (FTRL) method of Shalev-Shwartz & Singer [46] is obtained when players observe their full payoff vectors, that is, ˆ 𝑣 𝑖,𝑡 = 𝑣 𝑖 ( 𝑋 𝑡 ) . In this case, (RL) boils down to the deterministic update rule

or, more explicitly

For a detailed discussion of (FTRL), see [10, 31, 45]. We only note here that, as a special case, when (FTRL) is run with the logit choice setup of Eq. (7), a standard calculation yields the exponential / multiplicative weights (HEDGE) [4, 32, 45, 50]. ♦

Algorithm 2 (Optimistic FTRL) . A notable variant of FTRL - originally due to Popov [40] and subsequently popularized by Rakhlin & Sridharan [41, 42] - is the so-called optimistic FTRL method. This scheme employs an 'optimistic' correction intended to anticipate future steps, and it updates as

with 𝑋 𝑖,𝑡 = 𝑄 𝑖 ( 𝑌 𝑖,𝑡 ) . As a special case, if (Opt-FTRL) is run with the logit choice map (7), we obtain the familiar update rule known as optimistic multiplicative weights (OMW) [15, 41, 42, 47].

Compared to (FTRL), the gain vector ˆ 𝑣 𝑡 = 2 𝑣 ( 𝑋 𝑡 ) -𝑣 ( 𝑋 𝑡 -1 ) of (Opt-FTRL) has offset 𝑏 𝑡 = 𝑣 ( 𝑋 𝑡 ) -𝑣 ( 𝑋 𝑡 -1 ) relative to 𝑣 ( 𝑋 𝑡 ) . Thus, even though (Opt-FTRL) assumes full access to the players' mixed payoff vectors, it uses this information differently than (FTRL): in particular, the offset of (Opt-FTRL) is non-zero by design , not because of some systematic error in the payoff measurement process. ♦

Now, up to this point, we have not detailed how players might observe their full, mixed payoff vectors. This assumption simplifies the analysis immensely, but it is not realistic in applications to e.g., online advertising and network science, where players may only be able to observe their realized payoffs, and have no information about the strategies of other players or actions they did not play. On that account, we describe below a range of payoff-based policies where players estimate their counterfactual, 'what-if' payoffs indirectly .

The most common way to achieve this is via the importance-weighted estimator

where 𝑥 ∈ X is the players' strategy profile, and ˆ 𝛼 ∈ A is drawn according to 𝑥 . This estimator is at the heart of the online learning literature [10, 11, 31, 45] and it leads to the following methods:

Algorithm 3 (Bandit FTRL) . Plugging (IWE) directly into (RL) yields the bandit FTRL policy

where (IWE) is sampled at the mixed strategy profile

for some 'explicit exploration' parameter 𝛿 𝑡 ∝ 1 / 𝑡 ℓ 𝛿 , ℓ 𝛿 > 0 , which specifies the mix between 𝑋 𝑖,𝑡 and the uniform distribution unif A 𝑖 on A 𝑖 . As we discuss in the sequel, this combination of (IWE) with the explicit exploration mechanism (11) means that the surrogate payoff vector ˆ 𝑣 𝑡 = IWE ( ˆ 𝑋 𝑡 ) used to update (B-FTRL) has offset and noise bounded respectively as 𝑏 𝑡 = O ( 𝛿 𝑡 ) and 𝑈 𝑡 = O ( 1 / 𝛿 𝑡 ) .

Two special cases of (B-FTRL) that have attracted significant attention in the literature are:

The exponential weights algorithm for exploration and exploitation (EXP3) [6, 11, 31], obtained by running (B-FTRL) with the logit choice map (7).

The Tsallis implicitly normalized forecaster (TSALLIS-INF) [1, 3, 52, 53] that was proposed as a more efficient alternative to EXP3, and which updates as

i.e., as (B-FTRL) with the fractional power regularizer 𝜃 ( 𝑧 ) = -4 √ 𝑧 of Example 3.1. ♦

For illustration purposes, we provide some more examples of (RL) in Appendix B.

4 First results: resilience to strategic deviations

We are now in a position to begin our analysis of the rationality properties of the players' longrun behavior under (RL). To that end, we should first note that no-regret play may still lead to counterintuitive and highly non-rationalizable outcomes, e.g., with all players selecting dominated strategies for all time. The example below is adapted from Viossat & Zapechelnyuk [49].

In this game, 𝐵 and 𝐷 are strictly dominated for both players by their stronger 'twins' ( 𝐴 and 𝐶 respectively). However, it is easy to check that if both players choose between ( 𝐵, 𝐵 ) and ( 𝐷, 𝐷 ) with probability 1 / 2 each, the resulting distribution of play 𝑧 ∈ Δ ( A ) satisfies 𝑢 𝑖 ( 𝛼 𝑖 ; 𝑧 -𝑖 ) -𝑢 𝑖 ( 𝑧 ) ≤ -1 / 6 for all 𝛼 𝑖 ∈ { 𝐴, 𝐵, 𝐶, 𝐷 } , 𝑖 = 1 , 2 . As a result, the players' regret under 𝑧 𝑡 ≡ 𝑧 is negative , even though both players play strictly dominated strategies at all times. ♦

The example above shows unequivocally that

No-regret play does not suffice to exclude non-rationalizable outcomes.

In addition, Example 4.1 also shows that predictions based on correlated play are not always appropriate for describing the players' behavior under (RL): the end-state of any regularized learning algorithm will be a closed connected set of mixed strategies, so it is not possible to play only ( 𝐵, 𝐵 ) or ( 𝐷, 𝐷 ) in the long run. We are thus led to the following natural questions: What are the rationality properties of long-run play under (RL) ? Is the players' behavior robust to strategic deviations?

To study these questions formally, we will focus on the limit set L ( 𝑋 ) of 𝑋 𝑡 under (RL), viz.

In words, L ( 𝑋 ) is the set of limit points of 𝑋 𝑡 or, equivalently, the smallest subset of X to which 𝑋 𝑡 converges. Clearly, the simplest instance of a limit set is when L ( 𝑋 ) is a singleton, i.e., when 𝑋 𝑡 converges to a point. This case has attracted significant interest in the literature: for example, if L ( 𝑋 ) = { 𝑥 ∗ } then, for certain special cases of (RL), it is known that 𝑥 ∗ is a Nash equilibrium of Γ [34]. However, beyond this relatively simple regime, the structure of the limit sets of (RL) could be arbitrarily complicated and their rationality properties are not well-understood.

With this in mind, as a first attempt to study whether the long-run behavior of (RL) is 'robust to strategic deviations', we will consider the following notion of resilience to strategic deviations :

Definition 1. A closed subset S of X is resilient to strategic deviations - or simply resilient - if, for every deviation 𝑥 𝑖 ∈ X 𝑖 of every player 𝑖 ∈ N , we have

Informally, S is resilient if every unilateral deviation from S is deterred by some (possibly different) element thereof. In particular, if S is a singleton, we immediately recover the definition of a Nash equilibrium; beyond this case however, other examples include the set of undominated strategies of a game, the support face of the equilibria of two-player zero-sum games, etc. Importantly, as we show below, the limit sets of (RL) are resilient in all games:

Theorem 1. Let 𝑋 𝑡 , 𝑡 = 1 , 2 , . . . , be the sequence of play generated by (RL) with step-size / gain parameters ℓ 𝛾 > 2 ℓ 𝜎 and ℓ 𝑏 > 0 . Then, with probability 1 , the limit set L ( 𝑋 ) of 𝑋 𝑡 is resilient.

Proof sketch. The proof of Theorem 1 boils down to two interleaved arguments that we detail in Appendix C. The first hinges on showing that, if ℙ ( L ( 𝑋 ) = S ) > 0 for some non-random S ⊆ X , S must be resilient. This is argued by contradiction: if 𝑝 𝑖 ∈ X 𝑖 is a unilateral deviation violating Definition 1, we must also have lim inf 𝑡 →∞[ 𝑢 𝑖 ( 𝑝 𝑖 ; 𝑋 -𝑖,𝑡 ) -𝑢 𝑖 ( 𝑋 𝑡 )] > 0 with positive probability. However, the existence of a strategy that consistently outperforms 𝑋 𝑡 runs contrary to the fact that strategies that (RL) selects against underperforming strategies. We make this intuition precise via an energy argument that leverages a series of results from martingale limit theory (which is where the requirements for 𝛾 𝑡 , 𝑏 𝑡 and 𝑈 𝑡 come in). Then, to get the stronger statement that the random set L ( 𝑋 ) is resilient w.p. 1 , we show that the above remains true if 𝑝 𝑖 is replaced by a deviation 𝑞 𝑖 which is close enough to 𝑝 𝑖 and has rational entries. Since there is a countable number of such profiles, we can use a union bound on an enumeration of the rationals to isolate a deviation witnessing the negation of Definition 1; our claim then follows by applying our argument for non-random sets. ■

Theorem 1 is our first universal guarantee for (RL), so some remarks are in order. First, we should point out that the requirements ℓ 𝑏 > 0 and 2 ℓ 𝜎 < ℓ 𝛾 are a priori implicit because they depend on the offset and magnitude statistics of the feedback sequence ˆ 𝑣 𝑡 . However, in most learning algorithms, these quantities are under the explicit control of the players: for example, as we show in Appendix B, Algorithm 2 has ℓ 𝑏 = ℓ 𝛾 while, for Algorithm 3, we have ℓ 𝑏 = ℓ 𝜎 = ℓ 𝛿 . In this way, when instantiated to Algorithms 1-3 (and special cases thereof), Theorem 1 yields the following corollary:

Corollary 1. Suppose that Algorithms 1-3 are run with ℓ 𝛾 ∈ ( 0 , 1 ] and, for Algorithm 3, ℓ 𝛿 ∈ ( 0 , ℓ 𝛾 / 2 ) . Then, with probability 1 , the limit set L ( 𝑋 ) of 𝑋 𝑡 is resilient.

Now, since Theorem 1 applies to all games, it would seem to provide a universally positive answer to whether (RL) is robsut to strategic deviations. However, this is not so: a direct calculation shows that the face of X that is spanned by the dominated strategies ( 𝐵, 𝐵 ) and ( 𝐷, 𝐷 ) of Example 4.1 is resilient, so Theorem 1 cannot exclude convergence to a set where dominated strategies survive. Thus, just like no-regret play, the notion of resilience does not suffice by itself to capture the idea of rational behavior. This is because, albeit natural, resilience is too lax to provide a meaningful link between robustness to unilateral deviations - a game-theoretic requirement - and stability under regularized learning - a dynamic requirement. We address this question in detail in the next section.

5 A characterization of strategic stability under regularized learning

Similar to the set of pure strategies that arise from no-regret play, the main limitation of resilience is that a payoff-improving deviation may be countered by an action profile where the deviator also switched to a different strategy; in other words, resilience is not a self-enforcing barrier to deviations. In view of this, we will focus below on a much more stringent criterion of strategic stability, namely that any deviation from the set in question incurs a cost to the deviating agent.

Club sets. To make all this precise, define the better-reply correspondence of player 𝑖 ∈ N as

and write btr = Q 𝑖 btr 𝑖 for the product correspondence btr ( 𝑥 ) = btr 1 ( 𝑥 ) × · · · × btr 𝑁 ( 𝑥 ) . [In words, btr 𝑖 assigns to each 𝑥 ∈ X those strategies of player 𝑖 that are (weakly) better against 𝑥 than 𝑥 𝑖 .] In addition, given a product of pure strategies C = Q 𝑖 ∈ N C 𝑖 with C 𝑖 ⊆ A 𝑖 for all 𝑖 ∈ N , let S = Δ ( C ) denote the span of C , and let P ( X ) denote the collection of all such sets. We then say that S ∈ P ( X ) is closed under better replies - a club set for short - if it is closed under btr , i.e., btr ( S ) ⊆ S ; finally, S is said to be minimally club (m-club) if it does not admit a proper club subset.

Of course, the entire strategy space X is closed under better replies so, a priori, club sets could also contain dominated strategies and / or other non-rationalizable outcomes. By contrast, minimal club sets are much more rigid in their relation to rational behavior because any unilateral deviation from an m-club set is costly , and m-club sets are minimal in this regard. On that account, m-club sets can be seen as the closest setwise analogue to strict Nash equilibria.

This analogy is accentuated further by the following properties of m-club sets, all due to Ritzberger &Weibull [43], who introduced the concept:

Every game admits an m-club set; and if this set is a singleton, then it is a strict Nash equilibrium.

Any m-club set S is fixed under better replies, that is, btr ( S ) = S (implying in turn that S cannot contain any dominated strategies, including iteratively dominated ones).

Any m-club set S contains an essential equilibrium component , i.e., a component of Nash equilibria such that every small perturbation of the game admits a nearby equilibrium; in addition, this component has full support on S , i.e., it employs all pure strategy profiles that lie in S . 4

Going back to our online learning setting, the above leads to the following natural set of questions:

Are club sets ( minimal or not ) stable under the dynamics of regularized learning? Are they attracting? And, if so, are they the only such sets?

Any answer to these questions - positive or negative - would be an important step in delineating the relation between strategic stability (in the above sense) and dynamic stability under (RL). To that end, we start by formalizing some notions of dynamic stability that will be central in the sequel:

Definition 2. Fix some subset S of X and a tolerance level 𝜖 > 0 . We then say that S is:

Stochastically stable if, for every neighborhood U of S in X , there exists a neighborhood U 1 of S such that

Stochastically attracting if there exists a neighborhood U 1 of S such that

Stochastically asymptotically stable if it is stochastically stable and attracting.

Irreducibly stable if S is stochastically asymptotically stable and it does not admit a strictly smaller stochastically asymptotically subset S ′ with supp ( S ′ ) ⊊ supp ( S ) .

With all this in hand, our main result below provides a sharp characterization of strategic stability in the context of regularized learning:

Theorem 2. Fix some set S ∈ P ( X ) and suppose that (RL) is run with a steep regularizer and step-size / gain parameters ℓ 𝛾 ∈ [ 0 , 1 ] , ℓ 𝑏 > 0 , and ℓ 𝜎 < 1 / 2 . Then:

S is stochastically asymptotically stable under (RL) if and only if it is a club set.

S is irreducibly stable under (RL) if and only if it is an m-club set.

In addition, we also get the following convergence rate estimates for club sets:

Theorem 3. Let S ∈ P ( X ) be a club set, and let 𝑋 𝑡 , 𝑡 = 1 , 2 , . . . , be the sequence of play generated by (RL) with parameters ℓ 𝛾 ∈ [ 0 , 1 ] , ℓ 𝑏 > 0 , and ℓ 𝜎 < 1 / 2 . Then, for all 𝜖 > 0 , there exists an ( open, unbounded ) initialization domain D ⊆ Y such that, with probability at least 1 -𝜖 , we have

where 𝐶, 𝑐 1 , 𝑐 2 are constants ( 𝐶, 𝑐 2 > 0 ), and the rate function 𝜑 is given by 𝜑 ( 𝑧 ) = ( 𝜃 ′ ) -1 ( 𝑧 ) if 𝑧 > lim 𝑧 → 0 + 𝜃 ′ ( 𝑧 ) , and 𝜑 ( 𝑧 ) = 0 otherwise.

Specifically, if we instantiate Theorem 3 to Algorithms 1-3, we get the explicit estimates:

Corollary 2. Suppose that Algorithms 1-3 are run with ℓ 𝛾 ∈ [ 0 , 1 ] and, for Algorithm 3, ℓ 𝛿 ∈ ( 0 , 1 / 2 ) . Then, with notation as in Theorem 3, 𝑋 𝑡 converges to S at a rate of

 for positive constants 𝐶, 𝑐 > 0 . In particular, the projection-based variants of Algorithms 1-3 converge to m-club sets in a finite number of steps.

PureCoordination

CoordinationwithaSpectator

MatchingPennieswithaTwist

MatchingPennieswithOutsideOption

Proof sketch. The proof of Theorems 2 and 3 is quite involved so we defer it to Appendix D. At a high level, it hinges on constructing a family of 'primal-dual' energy functions, one per pure deviation from the set S under study. If unilateral deviations from S incur a cost to the deviator (that is, if S is club), these energy functions can be 'bundled together' to produce a suitable Lyapunov-like function for S . In more detail, the minimization of each individual energy function implies that the score variable 𝑌 𝑡 of (RL) diverges along an 'astral direction' in the payoff space Y - i.e., it escapes to infinity along the interior of a certain convex cone of Y [18]. Because this minimization occurs at infinity, the aggregation of offsets and random errors in (RL) affords some extra 'wiggle room' in our martingale analysis, so we are able to show that 𝑋 𝑡 = 𝑄 ( 𝑌 𝑡 ) remains close to S under a much wider range of parameters compared to Theorem 1. Then, a series of convex analysis arguments in the spirit of [33] coupled with the definition of 𝑄 allows us to show that the escape of 𝑌 𝑡 along the intersection of all these cones implies convergence to S at the specified rate.

On the converse side, if an asymptotically stable set is not club, we can find a non-costly (and possibly profitable) deviation 𝑧 from S which is selected against by (RL). However, this extinction runs contrary to the reinforcement of better replies under (RL), an argument which can be made precise by applying the martingale law of large numbers to ⟨ 𝑌 𝑡 , 𝑧 ⟩ [25]. The irreducible stability of m-club sets then follows by invoking this criterion reductively for any potentially stable subset S ′ of S . ■

Discussion and remarks. Theorems 2 and 3 are our main results linking dynamic and strategic stability, so we conclude with a series of remarks. First, we should note that Theorem 2 can be summed up as follows: a product of pure strategies is ( minimally ) closed under better replies if and only if its span is ( irreducibly ) stable under regularized learning. Importantly, this equivalence is based solely on the game's payoff data: it does not depend on the specific choices underlying (RL), including the choice map employed by each player, whether some players are using an optimistic adjustment or not, if they have access to their full payoff vectors, etc. As such, this equivalence provides a crisp operational criterion for identifying which pure strategy combinations ultimately persist under regularized learning - and, via Theorem 3, how fast this identification takes place.

In this light, Theorem 2 essentially states that the only robust prediction that can be made for the outcome of a regularized learning process is (minimal) closedness under better replies. This interpretation has significant cutting power for the emergence of rational behavior. To begin, in terms of equilibrium play, it effortlessly implies that a pure strategy profile is stochastically asymptotically stable under (RL) if and only if it is a strict Nash equilibrium. A version of this equivalence was only recently proved in [20] and [22] (in continuous and discrete time respectively), so Theorem 2 can be seen as a far-reaching generalization of these recent results. More to the point, since every m-club set S contains an essential equilibrium component that is fully supported in S , Theorem 2 also provides an important link between dynamic and structural stability: if an equilibrium - or a component of equilibria - is not robust to perturbations of the underlying game, it cannot be robustly identified by a regularized learning process (and vice versa). This remark is of particular importance for extensive-form games as such games often have non-generic equilibrium components that cannot be treated otherwise by the existing theory.

Finally, we should stress that Theorems 2 and 3 guarantee convergence even with a constant step-size. Together with the finite-time convergence guarantees of Corollary 2 for projection-based methods, this feature is a testament to the robustness of club sets as, in the presence of uncertainty, convergence almost always requires a vanishing step-size which can slow convergence down to a crawl. We find this robust convergence landscape particularly intriguing for future research on the topic.

Acknowledgments and Disclosure of Funding

PM is also with the Archimedes Research Unit - Athena RC - Department of Mathematics, National &Kapodistrian University of Athens. This work has been partially supported by the French National Research Agency (ANR) in the framework of the 'Investissements d'avenir' program (ANR-15IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), MIAI@Grenoble Alpes (ANR-19P3IA-0003), and project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program

References

Abernethy, J., Lee, C., and Tewari, A. Fighting bandits with a new kind of smoothness. In NIPS '15: Proceedings of the 29th International Conference on Neural Information Processing Systems , 2015.

Arora, S., Hazan, E., and Kale, S. The multiplicative weights update method: A meta-algorithm and applications. Theory of Computing , 8(1):121-164, 2012.

Audibert, J.-Y. and Bubeck, S. Regret bounds and minimax policies under partial monitoring. Journal of Machine Learning Research , 11:2635-2686, 2010.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of the 36th Annual Symposium on Foundations of Computer Science , 1995.

Auer, P., Cesa-Bianchi, N., and Fischer, P. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47:235-256, 2002.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.

Basu, K. and Weibull, J. W. Strategy subsets closed under rational behavior. Economics Letters , 36: 141-146, 1991.

Bauschke, H. H., Borwein, J. M., and Combettes, P. L. Bregman monotone optimization algorithms. SIAM Journal on Control and Optimization , 42(2):596-636, 2003.

Benaïm, M. Dynamics of stochastic approximation algorithms. In Azéma, J., Émery, M., Ledoux, M., and Yor, M. (eds.), Séminaire de Probabilités XXXIII , volume 1709 of Lecture Notes in Mathematics , pp. 1-68. Springer Berlin Heidelberg, 1999.

Bubeck, S. and Cesa-Bianchi, N. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning , 5(1):1-122, 2012.

Cesa-Bianchi, N. and Lugosi, G. Prediction, Learning, and Games . Cambridge University Press, 2006.

Chotibut, T., Falniowski, F., Misiurewicz, M., and Piliouras, G. Family of chaotic maps from game theory. Dynamical Systems , 2020.

Coucheney, P., Gaujal, B., and Mertikopoulos, P. Penalty-regulated dynamics and robust learning procedures in games. Mathematics of Operations Research , 40(3):611-633, August 2015.

Cressman, R. Evolutionary Dynamics and Extensive Form Games . The MIT Press, 2003.

Daskalakis, C. and Panageas, I. Last-iterate convergence: Zero-sum games and constrained min-max optimization. In ITCS '19: Proceedings of the 10th Conference on Innovations in Theoretical Computer Science , 2019.

Daskalakis, C., Ilyas, A., Syrgkanis, V., and Zeng, H. Training GANs with optimism. In ICLR '18: Proceedings of the 2018 International Conference on Learning Representations , 2018.

Demichelis, S. and Ritzberger, K. From evolutionary to strategic stability. Journal of Economic Theory , 113:51-75, 2003.

Dudík, M., Schapire, R. E., and Telgarsky, M. Convex analysis at infinity: An introduction to astral space. https://arxiv.org/abs/2205.03260, 2022.

Erev, I. and Roth, A. E. Predicting how people play games: Reinforcement learning in experimental games with unique, mixed strategy equilibria. American Economic Review , 88:848-881, 1998.

Flokas, L., Vlatakis-Gkaragkounis, E. V., Lianeas, T., Mertikopoulos, P., and Piliouras, G. No-regret learning and mixed Nash equilibria: They do not mix. In NeurIPS '20: Proceedings of the 34th International Conference on Neural Information Processing Systems , 2020.

Fudenberg, D. and Tirole, J. Game Theory . The MIT Press, 1991.

Giannou, A., Vlatakis-Gkaragkounis, E. V., and Mertikopoulos, P. Survival of the strictest: Stable and unstable equilibria under regularized learning with partial information. In COLT '21: Proceedings of the 34th Annual Conference on Learning Theory , 2021.

Giannou, A., Vlatakis-Gkaragkounis, E. V., and Mertikopoulos, P. The convergence rate of regularized learning in games: From bandits and uncertainty to optimism and beyond. In NeurIPS '21: Proceedings of the 35th International Conference on Neural Information Processing Systems , 2021.

Gidel, G., Berard, H., Vignoud, G., Vincent, P., and Lacoste-Julien, S. A variational inequality perspective on generative adversarial networks. In ICLR '19: Proceedings of the 2019 International Conference on Learning Representations , 2019.

Hall, P. and Heyde, C. C. Martingale Limit Theory and Its Application . Probability and Mathematical Statistics. Academic Press, New York, 1980.

Hannan, J. Approximation to Bayes risk in repeated play. In Dresher, M., Tucker, A. W., and Wolfe, P. (eds.), Contributions to the Theory of Games, Volume III , volume 39 of Annals of Mathematics Studies , pp. 97-139. Princeton University Press, Princeton, NJ, 1957.

Hofbauer, J. and Sigmund, K. Evolutionary game dynamics. Bulletin of the American Mathematical Society , 40(4):479-519, July 2003.

Hsieh, Y.-G., Iutzeler, F., Malick, J., and Mertikopoulos, P. On the convergence of single-call stochastic extra-gradient methods. In NeurIPS '19: Proceedings of the 33rd International Conference on Neural Information Processing Systems , pp. 6936-6946, 2019.

Juditsky, A., Nemirovski, A. S., and Tauvel, C. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

Kohlberg, E. and Mertens, J.-F. On the strategic stability of equilibria. Econometrica , 54(5):1003-1037, September 1986.

Lattimore, T. and Szepesvári, C. Bandit Algorithms . Cambridge University Press, Cambridge, UK, 2020.

Littlestone, N. and Warmuth, M. K. The weighted majority algorithm. Information and Computation , 108 (2):212-261, 1994.

Mertikopoulos, P. and Sandholm, W. H. Learning in games via reinforcement and regularization. Mathematics of Operations Research , 41(4):1297-1324, November 2016.

Mertikopoulos, P. and Zhou, Z. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173(1-2):465-507, January 2019.

Mertikopoulos, P., Papadimitriou, C. H., and Piliouras, G. Cycles in adversarial regularized learning. In SODA '18: Proceedings of the 29th annual ACM-SIAM Symposium on Discrete Algorithms , 2018.

Mertikopoulos, P., Lecouat, B., Zenati, H., Foo, C.-S., Chandrasekhar, V., and Piliouras, G. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In ICLR '19: Proceedings of the 2019 International Conference on Learning Representations , 2019.

Nemirovski, A. S. Prox-method with rate of convergence 𝑂 ( 1 / 𝑡 ) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Palaiopanos, G., Panageas, I., and Piliouras, G. Multiplicative weights update with constant step-size in congestion games: Convergence, limit cycles and chaos. In NIPS '17: Proceedings of the 31st International Conference on Neural Information Processing Systems , 2017.

Piliouras, G., Sim, R., and Skoulakis, S. Optimal no-regret learning in general games: Bounded regret with unbounded step-sizes via clairvoyant mwu. https://arxiv.org/abs/2111.14737 , 2021.

Popov, L. D. A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical Notes of the Academy of Sciences of the USSR , 28(5):845-848, 1980.

Rakhlin, A. and Sridharan, K. Online learning with predictable sequences. In COLT '13: Proceedings of the 26th Annual Conference on Learning Theory , 2013.

Rakhlin, A. and Sridharan, K. Optimization, learning, and games with predictable sequences. In NIPS '13: Proceedings of the 27th International Conference on Neural Information Processing Systems , 2013.

Ritzberger, K. and Weibull, J. W. Evolutionary selection in normal-form games. Econometrica , 63(6): 1371-99, November 1995.

Rockafellar, R. T. and Wets, R. J. B. Variational Analysis , volume 317 of A Series of Comprehensive Studies in Mathematics . Springer-Verlag, Berlin, 1998.

Shalev-Shwartz, S. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107-194, 2011.

Shalev-Shwartz, S. and Singer, Y. Convex repeated games and Fenchel duality. In NIPS' 06: Proceedings of the 19th Annual Conference on Neural Information Processing Systems , pp. 1265-1272. MIT Press, 2006.

Syrgkanis, V., Agarwal, A., Luo, H., and Schapire, R. E. Fast convergence of regularized learning in games. In NIPS '15: Proceedings of the 29th International Conference on Neural Information Processing Systems , pp. 2989-2997, 2015.

van Damme, E. Stability and perfection of Nash equilibria . Springer-Verlag, Berlin, 1987.

Viossat, Y. and Zapechelnyuk, A. No-regret dynamics and fictitious play. Journal of Economic Theory , 148(2):825-842, March 2013.

Vovk, V. G. Aggregating strategies. In COLT '90: Proceedings of the 3rd Workshop on Computational Learning Theory , pp. 371-383, 1990.

Weibull, J. W. Evolutionary Game Theory . MIT Press, Cambridge, MA, 1995.

Zimmert, J. and Seldin, Y. An optimal algorithm for stochastic and adversarial bandits. In AISTATS '19: Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , 2019.

Zimmert, J. and Seldin, Y. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

A Auxiliary results

In this appendix we collect some basic properties of the regularized choice maps and some results from probability theory that will be useful in the sequel.

A.1. Regularized choice maps and their properties. Thoughout this appendix, we will suppress the player index 𝑖 ∈ N , and we will follow standard conventions in convex analysis [44] that treat ℎ as an extended-real-valued function ℎ : V → ℝ ∪ {∞} with ℎ ( 𝑥 ) = ∞ for all 𝑥 ∈ V \ X . With this in mind, the subdifferential of a ℎ at 𝑥 ∈ X is defined as

where Y denotes here the algebraic dual V ∗ of V . Accordingly, the domain of subdifferentiability of ℎ is dom 𝜕ℎ : = { 𝑥 ∈ dom ℎ : 𝜕ℎ ≠ ∅} , and the convex conjugate of ℎ is defined as

for all 𝑦 ∈ Y . We then have the following basic results.

Lemma A.1. Let ℎ be a regularizer on X , and let 𝑄 : Y → X be the induced choice map. Then:

𝑄 is single-valued, and, for all 𝑥 ∈ X , 𝑦 ∈ Y , we have 𝑥 = 𝑄 ( 𝑦 ) ⇐⇒ 𝑦 ∈ 𝜕ℎ ( 𝑥 ) .

For all 𝑥 ∈ ri X , we have 𝜕ℎ ( 𝑥 ) = {( 𝜃 ′ ( 𝑥 𝛼 ) + 𝜇 ) 𝛼 ∈ A : 𝜇 ∈ ℝ } .

For all 𝑦 ∈ Y , we have 𝑄 ( 𝑦 ) = ∇ ℎ ∗ ( 𝑦 ) .

𝑄 is ( 1 / 𝐾 ) -Lipschitz continuous with 𝐾 : = inf ( 0 , 1 ] 𝜃 ′′ ( 𝑧 ) . In particular, as a special case, the logit choice map Λ is 1 -Lipschitz continuous in the ( 𝐿 1 , 𝐿 ∞ ) pair of norms on Y and X respectively.

If 𝑦 𝛼 -𝑦 𝛼 ′ →-∞ for some 𝛼 ′ ≠ 𝛼 , then 𝑄 𝛼 ( 𝑦 ) → 0 .

Remark. Some of the properties presented in Lemma A.1 are well known in the literature on regularized learning methods (see e.g., [33] and references therein), but we provide a proof of the entire lemma for completeness. ♦

Proof of Lemma A.1. For the first property of 𝑄 , note that the maximum in (9) is attained for all 𝑦 ∈ Y because ℎ is lower-semicontinuous (l.s.c.) and strongly convex. Furthermore, 𝑥 solves (9) if and only if 𝑦 -𝜕ℎ ( 𝑥 ) ∋ 0 , i.e., if and only if 𝑦 ∈ 𝜕ℎ ( 𝑥 ) .

For our second claim, if 𝑥 ∈ ri ( X ) , the first-order stationarity conditions for the convex problem (9) that defines 𝑄 become

because the inequality constraints 𝑥 𝛼 ≥ 0 are all inactive (recall that 𝑥 ∈ ri ( X ) by assumption). Now, by the first part of the theorem we have 𝑥 = 𝑄 ( 𝑦 ) if and only if 𝑦 ∈ 𝜕ℎ ( 𝑥 ) , so we conclude that 𝜕ℎ ( 𝑥 ) = {( 𝜃 ′ ( 𝑥 𝛼 ) + 𝜇 ) 𝛼 ∈ A : 𝜇 ∈ ℝ } , as claimed.

For the fourth item, the expression 𝑄 = ∇ ℎ ∗ is an immediate consequence of Danskin's theorem, while the Lipschitz continuity of 𝑄 follows from standard results, see e.g., [44, Theorem 12.60(b)].

For our last claim, let 𝑦 𝑡 be a sequence in Y such that 𝑦 𝛼,𝑡 -𝑦 𝛼 ′ ,𝑡 →-∞ and let 𝑥 𝑡 = 𝑄 ( 𝑦 𝑡 ) . Then, by descending to a subsequence if necessary, assume there exists some 𝜀 > 0 such that 𝑥 𝛼,𝑡 ≥ 𝜀 > 0 for all 𝑡 . Then, by the defining relation 𝑄 ( 𝑦 ) = arg max {⟨ 𝑦, 𝑥 ⟩ -ℎ ( 𝑥 )} of 𝑄 , we have:

for all 𝑥 ′ ∈ X . Therefore, taking 𝑥 ′ 𝑡 = 𝑥 𝑡 + 𝜀 ( 𝑒 𝛼 ′ -𝑒 𝛼 ) , we readily obtain

which contradicts our original assumption that 𝑦 𝛼,𝑡 -𝑦 𝛼 ′ ,𝑡 →-∞ . With X compact, the above shows that 𝑥 ∗ 𝛼 = 0 for any limit point 𝑥 ∗ of 𝑥 𝑡 , i.e. 𝑄 𝛼 ( 𝑦 𝑡 ) → 0 . ■

The second collection of results concerns the Fenchel coupling , an energy function that was first introduced in [33, 34] and is defined as follows:

This coupling will play a major role in the proofs of Theorem 1, so we prove two of its most basic properties below.

Lemma A.2. For all 𝑝 ∈ X and all 𝑦, 𝑦 ′ ∈ Y , we have:

In particular, if ℎ ( 0 ) = 0 , we have

Proof of Lemma A.2. By the strong convexity of ℎ relative to ∥·∥ (cf. Lemma A.1), we have

leading to the bound

for all 𝑡 ∈ ( 0 , 1 ] . The bound (A.7a) then follows by letting 𝑡 → 0 + in (A.10).

For our second claim, we have

where the inequality in the second line follows from the fact that ℎ ∗ is ( 1 / 𝐾 ) -strongly smooth [44, Theorem 12.60(e)]. ■

A.2. Basic results from probability theory. We conclude this appendix with some useful results from probability theory that we will use freely throughout the sequel. For a complete treatment, we refer the reader to Hall & Heyde [25].

Lemma A.3 (Azuma-Hoeffding inequality) . Let 𝑀 𝑡 ∈ ℝ , 𝑡 = 1 , 2 , . . . , be a martingale with ∥ 𝑀 𝑡 -𝑀 𝑡 -1 ∥∞ ≤ 𝜎 𝑡 ( a.s. ) . Then, for all 𝜂 > 0 , we have

Lemma A.4 (Kolmogorov's inequality) . Let 𝑍 𝑡 ∈ ℝ , 𝑡 = 1 , 2 , . . . , be a martingale difference sequence that is bounded in 𝐿 2 . Then:

Lemma A.5 (Doob's maximal inequality) . Let 𝑍 𝑡 ∈ ℝ , 𝑡 = 1 , 2 , . . . , be a martingale difference sequence that is bounded in 𝐿 𝑝 for some 𝑝 ≥ 1 . Then

Lemma A.6 (Burkholder-Davis-Gundy inequality) . Let 𝑍 𝑡 , 𝑡 = 1 , 2 , . . . , be a martingale difference sequence in ℝ 𝑛 . Then, for all 𝑝 > 1 , there exist constants 𝑐 𝑝 , 𝐶 𝑝 that depend only on 𝑝 and are such that

Lemma A.7 (Robbins-Siegmund) . Let F 𝑡 , 𝑡 = 1 , 2 , . . . , be a filtration on a complete probability space ( Ω , F , ℙ ) , and suppose that the sequences 𝑋 𝑡 , 𝐿 𝑡 and 𝐾 𝑡 F 𝑡 -measurable, nonnegative, and such that

Then, 𝑋 𝑡 converges to some random variable 𝑋 ∞ with probability 1 on the event GLYPH<8>P ∞ 𝑡 = 1 𝐿 𝑡 < ∞ and P ∞ 𝑡 = 1 𝐾 𝑡 < ∞ GLYPH<9> .

B Specific algorithms and their properties

B.1. Known algorithms as special cases of (RL) . To complement our analysis in the main part of our paper, we detail below how Algorithms 1-3 can be recast in the general framework of (RL). To lighten notation, we will assume that 𝑏 𝑡 , 𝑈 𝑡 and ˆ 𝑣 𝑡 are respectively bounded as

and we will set

so we can take 𝑀 𝑡 = 𝐺 + 𝐵 𝑡 + 𝜎 𝑡 in (B.1). We will also make free use of the fact that 𝑣 is Lipschitz continuous on X , and we will write 𝐿 for its Lipschitz modulus in the ( 𝐿 1 , 𝐿 ∞ ) pair of norms on X and Y respectively, viz.

We now proceed to establish the required bounds for Algorithms 1-3:

Algorithm 1. Since ˆ 𝑣 𝑡 = 𝑣 ( 𝑋 𝑡 ) , we readily get 𝑏 𝑡 = 𝑈 𝑡 = 0 by definition, so Algorithm 1 fits the scheme (RL) for free with ℓ 𝑏 = ∞ , ℓ 𝜎 = 0 . ♦

Algorithm 2. For the case of (Opt-FTRL), we have ˆ 𝑣 𝑡 = 2 𝑣 ( 𝑋 𝑡 ) -𝑣 ( 𝑋 𝑡 -1 ) so 𝑏 𝑡 = 𝑣 ( 𝑋 𝑡 ) -𝑣 ( 𝑋 𝑡 -1 ) , which is F 𝑡 -measurable. We thus get

Moreover, given that ˆ 𝑣 is F 𝑡 -measurable, we readily get 𝑈 𝑡 = 0 .

♦

Algorithm 3. Since ˆ 𝛼 𝑡 is sampled according to ˆ 𝑋 𝑡 = ( 1 -𝛿 𝑡 ) 𝑋 𝑖,𝑡 + 𝛿 𝑡 unif A 𝑖 (cf. Eq. (11) in Section 3), we readily obtain 𝔼 [ ˆ 𝑣 𝑖,𝑡 | F 𝑡 ] = 𝑣 𝑖 ( ˆ 𝑋 𝑡 ) , and hence, by (B.3), we get

Moreover, since ˆ 𝑋 𝑖 𝛼 𝑖 ,𝑡 ≥ 𝛿 𝑡 / 𝐴 𝑖 , it follows that ∥ ˆ 𝑣 𝑡 ∥∞ = O ( 1 / 𝛿 𝑡 ) = O ( 𝑡 ℓ 𝛿 ) . ♦

For comparison purposes, we illustrate the algorithms' behavior in a simple 2 × 2 × 2 game in Fig. 2 in Appendix E.

B.2. Further algorithms and illustrations. To demonstrate the breadth of (RL) as an algorithmic template, we provide below some more examples of algorithms from the game-theoretic literature that can be recast as special cases thereof (see also Table 1 for a recap).

Algorithm 4 (Mirror-prox) . Aprogenitor of (Opt-FTRL) is the so-called mirror-prox (MP) algorithm [29, 37], which updates as:

The main difference between (MP) and (Opt-FTRL) is that the former utilizes two surrogate gain vectors per iteration - meaning in particular that the interim, leading state ˜ 𝑋 𝑡 is generated with payoff information from 𝑋 𝑡 , not ˜ 𝑋 𝑡 -1 . This method has been used extensively in the literature for solving variational inequalities and two-player, zero-sum games, cf. Juditsky et al. [29] and references therein.

A calculation similar to that for (Opt-FTRL) shows that Algorithm 4 has 𝐵 𝑡 = O ( 1 / 𝑡 ℓ 𝛾 ) and 𝜎 𝑡 = 0 because the algorithm has no further randomization. ♦

Algorithm 5 (Clairvoyant multiplicative weights) . A recent variant of the exponential / multiplicative weights (HEDGE) algorithm is the so-called clairvoyant multiplicative weights (CMW) algorithm [39]

The main difference between (CMW) and (HEDGE) is that the proxy payoff vector ˆ 𝑣 𝑡 in (CMW) is based on the future state 𝑋 𝑡 + 1 and not the current state 𝑋 𝑡 . To perform this 'clairvoyant' update, the players of the game must coordinate to solve an implicit fixed point problem, so (CMW) is only meaningful when one has access to the payoff function 𝑣 (·) . In this regard, (CMW) can be seen as a Bregman proximal point method in the general spirit of Bauschke et al. [8].

To cast (CMW) as an instance of the generalized template (RL), simply note that the sequence of input signals is given by ˆ 𝑣 𝑡 = 𝑣 ( 𝑋 𝑡 + 1 ) , so 𝑈 𝑡 = 0 and 𝑏 𝑡 = 𝑣 ( 𝑋 𝑡 + 1 ) -𝑣 ( 𝑋 𝑡 ) = O ( 𝛾 𝑡 ) = O ( 1 / 𝑡 ℓ 𝛾 ) . ♦

C Proof of Theorem 1

Our main goal in this appendix will be to prove Theorem 1 on the resilience properties of (RL). For convenience, we restate below the relevant result for ease of reference:

Theorem 1. Let 𝑋 𝑡 , 𝑡 = 1 , 2 , . . . , be the sequence of play generated by (RL) with step-size / gain parameters ℓ 𝛾 > 2 ℓ 𝜎 and ℓ 𝑏 > 0 . Then, with probability 1 , the limit set L ( 𝑋 ) of 𝑋 𝑡 is resilient.

Proof. Our proof that L ( 𝑋 ) is resilient hinges on an energy-based technique that we will employ repeatedly in other parts of our analysis. To begin, introduce a player-strategy deviation pair ( 𝑖, 𝑧 𝑖 ) , and say that a set is resilient to ( 𝑖, 𝑧 𝑖 ) if there exists an element of the set, say 𝑥 ∗ , which counters said deviation, i.e., such that 𝑢 𝑖 ( 𝑥 ∗ ) ≥ 𝑢 𝑖 ( 𝑧 𝑖 ; 𝑥 ∗ -𝑖 ) . In this specific case, our proof proceeds by contradiction, namely by assuming that, with positive probability, L ( 𝑋 ) is not resilient to ( 𝑖, 𝑧 𝑖 ) . The main steps of our proof unfold as follows:

Step 1. Assume that L ( 𝑋 ) is not resilient to ( 𝑖, 𝑧 𝑖 ) with positive probability. Then there exists 𝑐, 𝜖, 𝑡 0 > 0 such that

Proof of Step 1. The function 𝑓 : 𝑥 ∈ X ↦→ 𝑢 𝑖 ( 𝑧 𝑖 ; 𝑥 -𝑖 ) -𝑢 𝑖 ( 𝑥 ) is continuous and X is compact, so there is a definite function 𝜂 ≡ 𝜂 ( 𝛿 ) such that if ∥ 𝑥 -𝑥 ′ ∥ ≤ 𝜂 ( 𝛿 ) , then | 𝑓 ( 𝑥 ) -𝑓 ( 𝑥 ′ )| ≤ 𝛿 . Now, by assumption, {∀ 𝑥 ∗ ∈ L ( 𝑋 ) , 𝑢 𝑖 ( 𝑧 𝑖 ; 𝑥 ∗ -𝑖 ) > 𝑢 𝑖 ( 𝑥 ∗ )} is of positive probability. We thus get

for some 𝑐 > 0 in (C.2c), and where (C.2a) is because L ( 𝑋 ) is closed - hence compact - almost surely. Therefore, by definition of 𝜂 (·) ,

Now, let 𝑡 0 such that ℙ {∀ 𝑡 ≥ 𝑡 0 , dist ( 𝑋 𝑡 , L ( 𝑋 )) ≤ 𝜂 ( 𝑐 )} > 1 -𝜖 2 . Then by construction, we get

■

and our proof is complete.

Intuitively, the existence of an action that consistently outperforms 𝑋 𝑡 runs contrary to the behavior that one would expect from any regularized learning algorithm. We will proceed to make this intuition precise below by means of an energy argument. To that end, consider the Fenchel coupling

Then, by Lemma A.2 in Appendix A, we readily get that

where, in obvious notation, we are identifying 𝑧 𝑖 ∈ A 𝑖 with the corresponding vertex 𝑒 𝑧 𝑖 of X 𝑖 = Δ ( A 𝑖 ) . To proceed, the main idea will be to relate 𝛾 𝑡 ⟨ ˆ 𝑣 𝑖,𝑡 , 𝑧 𝑖 -𝑋 𝑖,𝑡 ⟩ to its 'perfect' counterpart 𝛾 𝑡 ⟨ 𝑣 𝑖 ( 𝑋 𝑡 ) , 𝑧 𝑖 -𝑋 𝑖,𝑡 ⟩ . We formalize this below.

Step 2. If L ( 𝑋 ) is not resilient to ( 𝑖, 𝑧 𝑖 ) , there exists 𝑡 1 ≥ 𝑡 0 such that, with probability 𝜀 ′ / 2 > 0 , and for all 𝑡 ≥ 𝑡 1 , we have

Proof of Step 2. With probability 𝜀 ′ and for all 𝑡 ≥ 𝑡 0 , we have

The combination of Eqs. (C.5) and (C.8) then provides the following upper bound of 𝐹 𝑡 + 1 :

We are thus left to show is that 𝑐 P 𝑡 𝑠 = 𝑡 0 𝛾 𝑠 is the dominant term above. To do so, we proceed to examine each term individually:

Second-order term: We first deal with the second-order term P 𝑡 𝑠 = 𝑡 0 ∥ ˆ 𝑣 𝑠,𝑖 ∥ 2 ∞ 2 𝜅 ℎ 𝛾 2 𝑠 . By expanding the ∥ ˆ 𝑣 𝑠,𝑖 ∥ 2 ∞ , we readily get

However, by our assumptions on the parameters of (RL), we readily get

so we conclude that by the Stolz-Cesàro theorem.

Bias term: By far the most immediate, the bias term 𝐸 𝑏,𝑡 is bounded as

Noise term: Finally, the noise term 𝐸 𝑈,𝑡 is bounded by means of the Azuma-Hoeffding inequality, cf. Lemma A.3 in Appendix A. Specifically, with probability at least 1 -𝜀 ′ / 2 , we have

for all 𝑡 ≥ 𝑡 0 . To proceed, note that a second application of the Stolz-Cesàro theorem yields P 𝑡 𝑠 = 𝑡 0 𝜎 2 𝑠 𝛾 2 𝑠 = o ( P 𝑡 𝑠 = 𝑡 0 𝛾 𝑠 ) and, moreover, note that log ( 4 𝑡 2 / 𝜀 ′ ) = O ( P 𝑡 𝑠 = 𝑡 0 𝛾 𝑠 ) . Taking square roots and multiplying then yields that

with probability at least 1 -𝜀 ′ / 2 .

We are now in a position to establish the bound Eq. (C.6). Indeed, putting Eqs. (C.13), (C.14) and (C.16) together, we readily infer that there exists 𝑡 1 ≥ 𝑡 0 such that, with probability at least 1 -𝜀 ′ / 2 , we have

for all 𝑡 ≥ 𝑡 1 . This proves Eq. (C.6) and concludes our proof.

■

Summarizing the above, we have shown that, with probability at least 1 -𝜀 ′ / 2 , we have

Since 𝐹 is nonnegative (by Lemma A.2), we have established that the event where L ( 𝑋 ) is not resilient to ( 𝑖, 𝑧 𝑖 ) is an event of probability zero. However, since there are uncountably many strategic deviations, the proof is not yet complete; the last step involves an approximation by deviations with rational entries.

Step 3. L ( 𝑋 ) is almost-surely resilient.

Proof of Step 3. The key point of the proof is the observation that a closed set is resilient if and only if it is rationally resilient, i.e., it nullifies all rational deviations 𝑧 𝑖 ∈ X 𝑖 ∩ ℚ A 𝑖 (which are countably many). Indeed, if L ( 𝑋 ) is not resilient with positive probability, then, likewise, L ( 𝑋 ) will not be rationally resilient with positive probability either. Because there are countably many rational deviations, there must be a rational strategic deviation ( 𝑖, 𝑧 𝑖 ) (with 𝑧 𝑖 ∈ X 𝑖 ∩ ℚ A 𝑖 ) to which L ( 𝑋 ) is not resilient. This comes in contradiction with the conclusions of Step 2. ■

This concludes the last required step, so the proof of Theorem 1 is now complete.

■

D Proof of Theorems 2 and 3

In this last appendix, our goal is to prove our characterization of club sets, namely:

Theorem 2. Fix some set S ∈ P ( X ) and suppose that (RL) is run with a steep regularizer and step-size / gain parameters ℓ 𝛾 ∈ [ 0 , 1 ] , ℓ 𝑏 > 0 , and ℓ 𝜎 < 1 / 2 . Then:

S is stochastically asymptotically stable under (RL) if and only if it is a club set.

S is irreducibly stable under (RL) if and only if it is an m-club set.

Theorem 3. Let S ∈ P ( X ) be a club set, and let 𝑋 𝑡 , 𝑡 = 1 , 2 , . . . , be the sequence of play generated by (RL) with parameters ℓ 𝛾 ∈ [ 0 , 1 ] , ℓ 𝑏 > 0 , and ℓ 𝜎 < 1 / 2 . Then, for all 𝜖 > 0 , there exists an ( open, unbounded ) initialization domain D ⊆ Y such that, with probability at least 1 -𝜖 , we have

where 𝐶, 𝑐 1 , 𝑐 2 are constants ( 𝐶, 𝑐 2 > 0 ), and the rate function 𝜑 is given by 𝜑 ( 𝑧 ) = ( 𝜃 ′ ) -1 ( 𝑧 ) if 𝑧 > lim 𝑧 → 0 + 𝜃 ′ ( 𝑧 ) , and 𝜑 ( 𝑧 ) = 0 otherwise.

Our proof strategy will be to construct a sheaf of 'linearized' energy functions which, when bundled together, yield a suitable Lyapunov-like function for S . To do so, let C = Q 𝑖 C 𝑖 denote the support of S (cf. the definition of club sets), and let

and denote the set of all pure strategic deviations from S . Then, our ensemble of candidate energy functions will be given by

The motivation for this definition is given by the following lemma.

Lemma D.1. Suppose that the sequence 𝑦 𝑡 ∈ V ∗ , 𝑡 = 1 , 2 , . . . , has 𝐸 𝑧 ( 𝑦 𝑡 ) →-∞ for all 𝑧 ∈ Z as 𝑡 →∞ . Then the sequence 𝑥 𝑡 = 𝑄 ( 𝑦 𝑡 ) converges to S as 𝑡 →∞ .

Proof. Let 𝑧 = 𝑒 𝑖 𝛼 ′ 𝑖 -𝑒 𝑖 𝛼 𝑖 for some 𝑖 ∈ N , 𝛼 𝑖 ∈ C 𝑖 , and 𝛼 ′ 𝑖 ∈ A 𝑖 \ C 𝑖 . Since 𝐸 𝑧 ( 𝑦 𝑡 ) → -∞ by assumption, we get 𝑦 𝑖 𝛼 ′ 𝑖 ,𝑡 -𝑦 𝑖 𝛼 𝑖 ,𝑡 →-∞ and hence, by Lemma A.1, we conclude that 𝑄 𝑖 𝛼 ′ 𝑖 ( 𝑥 𝑡 ) → 0 as 𝑡 →∞ . In turn, given that this holds for all 𝑖 ∈ N and all 𝛼 ′ 𝑖 ∈ A 𝑖 \ C 𝑖 , we conclude that 𝑥 𝑡 = 𝑄 ( 𝑦 𝑡 ) converges to S . ■

In view of the above, we will focus on showing that 𝐸 𝑧 ( 𝑌 𝑡 ) → -∞ for all 𝑧 ∈ Z . As a first step, we establish a basic template inequality for the evolution of 𝐸 𝑧 under (RL).

Lemma D.2. Fix some 𝑧 ∈ Z and let 𝐸 𝑡 : = 𝐸 𝑧 ( 𝑌 𝑡 ) . Then, for all 𝑡 = 1 , 2 , . . . , we have

where the error terms 𝜉 𝑡 and 𝜓 𝑡 are given by

Proof. Simply set 𝑦 ← 𝑌 𝑡 + 1 in 𝐸 𝑧 ( 𝑦 ) , invoke the definition of the update 𝑌 𝑡 ← 𝑌 𝑡 + 1 in (RL), and note that | ⟨ 𝑏 𝑡 , 𝑧 ⟩| ≤ ∥ 𝑧 ∥∥ 𝑏 𝑡 ∥∞ ≤ 2 𝐵 𝑡 by the definition of Z . ■

The key take-away from (D.4) is that, if 𝑋 𝑡 is close to S and 𝛼 𝑖 ∈ C 𝑖 , 𝛼 ′ 𝑖 ∈ A 𝑖 \ C 𝑖 , we will have

by the continuity of 𝑢 𝑖 and the assumption that S is a club set. More concretely, by the definition of the better-reply correspondence, we have

and hence, by continuity, there exists a neighborhood B of S such that

In other words, as long as 𝑋 𝑡 is sufficiently close to S , (D.4) exhibits a consistent negative drift pushing 𝐸 𝑡 towards -∞ .

To exploit this 'dynamic consistency' property of S , it will be convenient to introduce the family of sets

As we show below, these sets are mapped under 𝑄 to neighborhoods of S , so they are particularly well-suited to serve as initialization domains for (RL). This is encoded in the following properties:

Lemma D.3. Let 𝑥 = 𝑄 ( 𝑦 ) for some 𝑦 ∈ V ∗ . Then, for all 𝛼 𝑖 , 𝛼 ′ 𝑖 , 𝑖 ∈ N , we have

with 𝜑 defined as per Theorem 3, i.e.,



Corollary D.1. For all 𝛿 > 0 there exists some 𝜖 𝛿 ∈ ℝ such that, for all 𝜖 > 𝜖 𝛿 and all 𝑦 ∈ D 𝜖 , we have

Proof of Lemma D.3. Suppressing the player index for simplicity, the first-order stationarity conditions for the convex problem (9) readily give

where 𝜇 is the Lagrange multiplier for the equality constraint P 𝛼 𝑥 𝛼 = 1 , and 𝜈 𝛼 is the complementary slackness multiplier of the inquality constraint 𝑥 𝛼 ≥ 0 (so 𝜈 𝛼 = 0 whenever 𝑥 𝛼 > 0 ). Thus, rewriting (D.13) for some 𝛼 ∈ A , we get

and hence where we used the fact that 𝜈 𝛼 ≥ 0 . Now, if 𝜃 ′ ( 1 - ) + 𝑦 𝛼 ′ -𝑦 𝛼 < 𝜃 ′ ( 0 + ) and 𝑥 𝛼 ′ > 0 (so 𝜈 𝛼 ′ = 0 ), we will have 𝜃 ′ ( 𝑥 𝛼 ′ ) < 𝜃 ′ ( 0 + ) , a contradiction. This shows that 𝑥 𝛼 ′ = 0 if 𝜃 ′ ( 1 - ) + 𝑦 𝛼 ′ -𝑦 𝛼 < 𝜃 ′ ( 0 + ) , so (D.10) is satisfied in this case. Otherwise, if 𝑥 𝛼 ′ > 0 , we must have 𝜈 𝛼 ′ = 0 by complementary slackness, so (D.10) follows by applying the second branch of (D.11) to (D.15). ■

The above provides us with a fairly good handle on the local geometric and dynamic properties of S . On the flip side however, the various error terms in (D.5) may be positive, so 𝐸 𝑡 may fail to be decreasing and 𝑋 𝑡 may drift away from S . On that account, it will be convenient to introduce the aggregate error processes

Intuitively, the aggregates (D.16) measure the total effect of each error term in (D.4), so we will establish a first series of results under the following general requirements:

1. Subleading error growth:

where 𝜏 𝑡 = P 𝑡 𝑠 = 1 𝛾 𝑠 and both limits are to be interpreted in the almost sure sense.

2. Drift dominance:

for some 𝐶 > 0 and 𝛼 ∈ [ 0 , 1 ) .

In a nutshell, (Sub) posits that the aggregate error processes I 𝑡 and II 𝑡 of (D.16) are subleading relative to the long-run drift of (D.4), while (Dom) goes a step further and asks that said errors are asymptotically dominated by the drift in (D.4). Accordingly, under these implicit error control conditions, we obtain the interim convergence result below:

Proposition D.1. Let S be a club set, fix some confidence threshold 𝜂 > 0 , and let 𝑋 𝑡 = 𝑄 ( 𝑌 𝑡 ) be the sequence of play generated by (RL) . If (Sub) and (Dom) hold, there exists an unbounded initialization domain D ⊆ V ∗ such that

Proof of Proposition D.1. Fix some 𝑧 ∈ Z , let 𝐸 𝑡 = 𝐸 𝑧 ( 𝑌 𝑡 ) , and pick 𝛼 ∈ [ 0 , 1 ) so that (Dom) holds for some 𝐶 > 0 . In addition, set 𝑐 = -sup 𝑥 ∈ B ⟨ 𝑣 ( 𝑥 ) , 𝑧 ⟩ > 0 , let 𝑡 0 = inf { 𝑡 : 𝑐𝜏 𝑡 > 𝐶𝜏 𝛼 𝑡 } , and write Δ 𝐸 = max 𝑡 { 𝐶𝜏 𝛼 𝑡 -𝑐𝜏 𝑡 } . Then, if 𝑌 1 is initialized in D ← D ( 𝜖 + Δ 𝐸 ) where 𝜖 is such that D ( 𝜖 ) ⊆ B , we will have 𝑌 𝑡 ∈ D ( 𝜖 ) for all 𝑡 . Indeed, this being trivially the case for 𝑡 = 1 , assume it to be the case for all 𝑠 = 1 , 2 , . . . , 𝑡 . Then, by (D.4) and our inductive hypothesis, we get

i.e., 𝐸 𝑡 + 1 ∈ D ( 𝜖 ) , as claimed.

Now, since 𝐸 𝑡 ∈ D ( 𝜖 ) for all 𝑡 , we conclude that

Thus, if (Sub) holds, we readily get 𝐸 𝑡 → -∞ with probability 1 on the event that (Dom.I) and (Dom.II) both hold. This implies that 𝐸 𝑡 →-∞ , and since 𝑧 ∈ Z above is arbitrary, we conclude that 𝑋 𝑡 → S with probability at least 1 -2 𝜂 , as claimed. ■

We are now in a position to prove Theorem 2.

Proof of Theorem 2. Our proof will hinge on showing that (Sub) and (Dom) hold under the stated step-size and sampling parameter schedules. Our claim will then follow by a direct application of Proposition D.1 and a reduction to a suitable subface of X .

First, regarding (Sub), the law of large numbers for martingale difference sequences [25, Theorem 2.18] shows that I 𝑡 / 𝜏 𝑡 → 0 with probability 1 on the event GLYPH<8>P 𝑡 𝛾 2 𝑡 𝔼 [ 𝜉 2 𝑡 | F 𝑡 ]/ 𝜏 2 𝑡 < ∞ GLYPH<9> . However

so, in turn, we get

given that ℓ 𝜎 < 1 / 2 . This establishes (Sub.I); the remaining requirement (Sub.II) follows trivially by noting that P 𝑡 𝑠 = 1 𝛾 𝑠 𝐵 𝑠 GLYPH<14>P 𝑡 𝑠 = 1 𝛾 𝑠 → 0 if and only if 𝐵 𝑡 → 0 , which is immediate from the theorem's assumptions.

Second, regarding (Dom), since 𝐵 𝑡 is deterministic and 𝐵 𝑡 = O ( 1 / 𝑡 ℓ 𝑏 ) for some ℓ 𝑏 > 0 , it is always possible to find 𝐶 > 0 and 𝛼 ∈ ( 0 , 1 ) so that (Dom.II) holds. We are thus left to establish (Dom.I). To that end, let I ∗ 𝑡 = sup 1 ≤ 𝑠 ≤ 𝑡 | I 𝑡 | and set 𝑃 𝑡 : = ℙ GLYPH<0> I ∗ 𝑡 > 𝐶𝜏 𝛼 𝑡 / 2 GLYPH<1> so

where 𝑐 𝑞 is a positive constant depending only on 𝐶 and 𝑞 , and we used Kolmogorov's inequality (Lemma A.4) in the first step and the Burkholder-Davis-Gundy inequality (Lemma A.6) in the second.

To proceed, we will require the following variant of Hölder's inequality [9, p. 15]:

valid for all 𝑎 𝑠 , 𝑏 𝑠 ≥ 0 and all 𝜌 > 1 , 𝜆 ∈ [ 0 , 1 ) . Then, substituting 𝑎 𝑠 ← 𝛾 2 𝑠 , 𝑏 𝑠 ← ∥ 𝑈 𝑠 ∥ 2 ∞ , 𝜌 ← 𝑞 / 2 and 𝜆 ← 1 / 2 -1 / 𝑞 , (D.24) gives

We now consider two cases, depending on whether the numerator of (D.26) is summable or not.

Case 1: ℓ 𝛾 ( 1 + 𝑞 / 2 ) ≥ 1 + 𝑞ℓ 𝜎 . In this case, the numerator of (D.26) is summable under the theorem's assumptions, so the fraction in (D.26) behaves as O ( 1 / 𝑡 ( 1 -ℓ 𝛾 ) ( 1 +( 𝛼 -1 / 2 ) 𝑞 ) ) .

Case 2: ℓ 𝛾 ( 1 + 𝑞 / 2 ) < 1 + 𝑞ℓ 𝜎 . In this case, the numerator of (D.26) is not summable under the theorem's assumptions, so the fraction in (D.26) behaves as O GLYPH<0> 𝑡 1 -ℓ 𝛾 ( 1 + 𝑞 / 2 )+ 𝑞ℓ 𝜎 GLYPH<14> 𝑡 ( 1 -ℓ 𝛾 ) ( 1 +( 𝛼 -1 / 2 ) 𝑞 ) GLYPH<1> .

Thus, working out the various exponents, a tedious - but otherwise straightforward - calculation shows that there exists some 𝛼 ∈ ( 0 , 1 ) such that 𝑃 𝑡 is summable as long as ℓ 𝜎 < 1 / 2 -1 / 𝑞 and 0 ≤ ℓ 𝛾 < 𝑞 /( 2 + 𝑞 ) . Hence, if 𝛾 is sufficiently small relative to 𝜂 , we conclude that

Finally, if ℓ 𝛾 > 1 / 2 + ℓ 𝜎 , (Dom.I) is a straightforward consequence of (D.24) for 𝑞 = 2 .

With all this in hand, the final steps of our proof proceed as follows:

Closedness = ⇒ Stability. Our assertion follows by invoking Proposition D.1.

■

Stability = ⇒ Closedness. Suppose that S is not club. Then there exists some pure strategy 𝛼 ∈ C and some deviation 𝛼 ′ ∉ C such that the deviation from 𝛼 to 𝛼 ′ is not costly to the deviating player. Thus, if we consider the restriction of the game to the face spanned by 𝛼 and 𝛼 ′ (a single-player game with two strategies), the corresponding score difference will be

By our standing assumptions for 𝑏 𝑡 and 𝑈 𝑡 (and Doob's martingale convergence theorem for the latter), both P 𝑠 = 1 𝛾 𝑠 𝑏 𝑠 and P 𝑠 = 1 𝛾 𝑠 𝑈 𝑠 will be bounded from below by some (a.s.) finite random variable 𝐴 0 . Since 𝜃 is steep, it follows that, with probability 1 , lim inf 𝑡 →∞( 𝑦 𝛼,𝑡 ) > 0 , so C cannot be stable. ■

Minimality = ⇒ Irreducible Stability. Suppose that S is m-club. Then, by our previous claim, S is stochastically asymptotically stable. If S contains a proper subface S ′ ⊊ S that is also stochastically asymptotically stable, S ′ must be club by the converse implication of the first part of the theorem. However, in that case, S would not be m-club, a contradiction which proves our claim. ■

Irreducible Stability = ⇒ Minimality. For our last claim, assume that S is irreducibly stable. By the first part of our theorem, this implies that S is club. Then, if it so happens that S is not m-club, it would contain a proper club subface S ′ ⊊ S ; by the first part of our theorem, this set would be itself stochastically asymptotically stable, in contradiction to the irreducibility assumption. This shows that S is m-club and concludes our proof. ■

We are only left to establish the convergence rate estimate of Theorem 3.

Long-run behavior of Hedge

Long-runbehavior of OMW

1.0

1.0

0.5

0.5

0.0

0.0

0.0

0'0

0.0

0.0

0.5

0.5

S°0

0.5

1.01.0

1.01.0

Long-runbehavior of EXP3

Long-run behavior of Tsallis-INF

1.0

1.0

0.5

0.5

0.0

0.0

0.0

0'0

0.0

0.0

0.5

0.5

0.5

0.5

1.01.0

1.01.0

Proof of Theorem 3. Going back to (D.21) and invoking Lemma D.3 shows that there exist constants 𝑐 1 > 0 and 𝑐 2 ∈ ℝ such that, for all 𝛼 𝑖 ∈ A 𝑖 \ C 𝑖 , 𝑖 ∈ N , we have

with probability 1 on the events of (Dom). We thus get

■

and our proof is complete.

As for the rate estimates of Corollary 2, the proof boils down to a simple derivation of the corresponding rate functions:

Proof of Corollary 2. By a straightforward calculation, we have:

If 𝜃 ( 𝑧 ) = 𝑧 log 𝑧 then 𝜑 ( 𝑧 ) = exp ( 1 + 𝑧 ) . √

If 𝜃 ( 𝑧 ) = -4 𝑧 then 𝜑 ( 𝑧 ) = 4 / 𝑧 2 .

If 𝜃 ( 𝑧 ) = 𝑧 2 / 2 then 𝜑 ( 𝑧 ) = [ 𝑧 ] 1 0 .

Our claims then follow immediatly from the rate estimate (17) of Theorem 2. ■

E Numerical experiments

In all our experiments, we ran the EXP3 variant of bandit FTRL (B-FTRL) (cf. Algorithm 3) with step-size and sampling radius parameters 𝛾 𝑡 = 0 . 2 × 𝑡 -1 / 2 and 𝛿 𝑡 = 0 . 1 × 𝑡 -0 . 15 respectively. The algorithm was run for 𝑇 = 10 4 iterations and, to reduce graphical clutter, we plotted only every third point of each trajectory. Trajectories have been colored throughout with darker hues indicating later times (e.g., light blue indicates that the trajectory is closer in time to its starting point, darker shades of blue indicate proximity to the termination time). The algorithm's initial conditions were taken from a uniform initialization grid of the form 𝑦 1 ∈ {-1 , 0 , 1 } 3 and perturbed by a uniform random number in [-0 . 1 , -0 . 1 ] to avoid non-generic initializations.

In general, the two defining elements of (RL) are a ) the regularizer of the method; and b ) the feedback available to the players. From our experiments, we conclude that methods with Euclidean regularization tend to have faster identification rates (i.e., converge to the support of an equilibrium / club set faster), but they are more 'extreme' than methods with an 'entropy-like' regularizer (in the sense that players tend to play pure strategies more often). As for the feedback available to the players, payoff-based methods tend to have higher variance (and hence a slower rate of convergence) relative to methods with full information; otherwise, from a qualitative viewpoint, there are no perceptible differences in their limiting behavior.

Finally, optimistic / extra-oracle methods with full information exhibit better convergence properties in two-player zero-sum games (relative to standard FTRL policies); however, this is a fragile advantage

that evaporates in the presence of noise and/or uncertainty (in which case "vanilla" and "optimistic" methods are essentially indistinguishable). We illustrate these findings in Fig. 2.

Regarding Fig. 1, the payoffs of the chosen games were normalized to [-1 , 1 ] and players are assumed to choose between two actions labeled ' 𝑂 ' and ' 1 '. The specific tableaus are shown in the table below, next to the respective portrait (all taken from Fig. 1.

UTILITIES

Player I

Player II

Player III

I, II, III

(

0

,

0

,

0

)

(

0

,

0

,

1

)

(

0

,

1

,

0

)

(

0

,

1

,

1

)

(

1

,

0

,

0

)

(

1

,

0

,

1

)

(

1

,

1

,

0

)

(

1

,

1

,

1

)

1

0

0

1

0

1

1

0

1

0

0

1

0

1

1

0

1

0

0

1

0

1

1

0

UTILITIES

Player I

Player II

Player III

I, II, III

(

0

,

0

,

0

)

(

0

,

0

,

1

)

(

0

,

1

,

0

)

(

0

,

1

,

1

)

(

1

,

0

,

0

)

(

1

,

0

,

1

)

(

1

,

1

,

0

)

(

1

,

1

,

1

)

1

1

0

0

0

0

1

1

1

1

0

0

0

0

1

1

0

0

0

0

0

0

0

0

UTILITIES

Player I

Player II

Player III

I, II, III

(

0

,

0

,

0

)

(

0

,

0

,

1

)

(

0

,

1

,

0

)

(

0

,

1

,

1

)

(

1

,

0

,

0

)

(

1

,

0

,

1

)

(

1

,

1

,

0

)

(

1

,

1

,

1

)

0

0

0

0

0.1

0.1

0.1

0.1

1

0

0

1

0

1

1

0

0

1

1

0

1

0

0

1

UTILITIES

Player I

Player II

Player III

I, II, III

(

0

,

0

,

0

)

(

0

,

0

,

1

)

(

0

,

1

,

0

)

(

0

,

1

,

1

)

(

1

,

0

,

0

)

(

1

,

0

,

1

)

(

1

,

1

,

0

)

(

1

,

1

,

1

)

-1

1

-1

1

1

-1

1

-1

1

1

-1

-1

-1

1

1

-1

1

-1

-1

1

-1

1

1

-1

PureCoordination

CoordinationwithaSpectator

MatchingPennies with aTwist

MatchingPennies with Outside Option

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the long run behavior of no-regret learning, and introduces the notion of resilience to strategic deviations as a metric with which to characterize no-regret learning algorithms. Moreover, to further strengthen their results the paper utilizes the idea of setwise strategic stability (m-club). In particular, the paper shows a nice connection between regularized learning and club sets. Finally, they estimate convergence rates to club sets, showing geometric rate for entropic regularizers and finite iterations for projection-based methods.\n\nSTRENGTHS:\nI think this paper is a conceptual step forward in the area of regularized learning dynamics. A major point of contention in many prior works in learning in games has been the disconnect between desirable equilibria and equilibria that are actually converged to by regularized learning. The results given in the paper are fairly broad, and the characterization of club sets as an alternative solution concept for regularized learning is (as far as I can tell) quite watertight and intuitive.\n\nWEAKNESSES:\nI only have very minor complaints with the paper, namely the section on regularized learning that introduces a few different algorithms under the RL umbrella. I feel this section is quite long and the notation is unnecessarily heavy, most of these could be in the appendix. I would have preferred to see more space allocated to either an extended proof sketch for Thms 2 & 3, or more experimental details.\n\nQUESTIONS:\nA question regarding the actual convergence properties of regularized learners to club sets in comparison to classical solution concepts:  how do the payoffs of these points compare to the Nash equilibrium values? Do you see any interesting behaviors of note between standard FTRL dynamics and other RL dynamics that are significantly different in your experiments?\n\nAlso, is there any connection between club sets and the chain recurrent set of the dynamics? Of course the paper is focused on discrete time regularized learning, but do you have any intuition as to whether club sets could be useful to construct/converge to chain recurrent sets in the continuous-time setting?\n\nLIMITATIONS:\nThe limitations and scope of the proposed ideas are discussed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper shows the convergence of regularized learning in games to sets satisfying a property called closeness under better replies. Moreover, convergence rates are derived even with bandit feedback.

STRENGTHS:
The pointwise behavior of regularized learning in games has gained lots of attention recently, and the paper considers a fundamental question in this field. The authors provide complete and general answers to the questions.

WEAKNESSES:
The paper is super notation-heavy and considers a very general setting in terms of both algorithms and feedback models. It would be much easier to follow if the authors could first provide some basic or toy examples and then generalize. Minor: a corrupted reference at the beginning of Line 267.

QUESTIONS:
While the questions (line 59-60) are important and the results are mathematically sound, what do they imply to regularized learning, in particular, for practitioners? For example, from prior works, we already know that a day-to-day strategy can be arbitrarily bad so it is probably better to do some averaging. Does this paper extend our understanding in this direction?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

comments: SUMMARY:
The paper deals with a fundamental question about long-term behavior of regularized learning algorithms in finite player finite action static games. They prove an interesting equivalence between the set of strategies which are closed under better replies with that of stochastically stable and attracting fixed points of wide class of regularized learning algorithm popularly studied in literature. Furthermore, they also study the rate of convergence of regularized learning algorithms to this set.

STRENGTHS:
The paper studies a very fundamental equivalence relation between payoff structure of a static game with that of limit sets of regularized learning algorithms. This result is very strong result which enhances our understanding of learning in static games. Particuarly, this paper brings the notion of better replies from economic literature and presents a deep connection with asymptotic properties of learning algorithm.  

The clarity of presentation of this paper is very good. 

The literature survey is also up to the mark to the best of my knowledge.

WEAKNESSES:
The paper has no major weakness in my view.

QUESTIONS:
-- Is there any characterization on size of m-club set given certain regularity structure of game. This will provide more predictive power about the asymptotic behavior of common learning dynamics

LIMITATIONS:
Some typographical errors 

1. In supplementary material, line 267 has ?? 
2. In equation C.13 the right hand side is missing.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05