SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Victor Boone Panayotis Mertikopoulos
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France {victor.boone, panayotis.mertikopoulos}@univ-grenoble-alpes.fr
In this paper, we examine the long-run behavior of regularized, no-regret learning in finite games. A well-known result in the field states that the empirical frequencies of no-regret play converge to the game's set of coarse correlated equilibria; however, our understanding of how the players' actual strategies evolve over time is much more limited – and, in many cases, non-existent. This issue is exacerbated by a series of recent results showing that only strict Nash equilibria are stable and attracting under regularized learning, thus making the relation between learning and pointwise solution concepts particularly elusive. In lieu of this, we take a more general approach and instead seek to characterize the setwise rationality properties of the players' day-to-day play. To that end, we focus on one of the most stringent criteria of setwise strategic stability, namely that any unilateral deviation from the set in question incurs a cost to the deviator – a property known as closedness under better replies (club). In so doing, we obtain a far-reaching equivalence between strategic and dynamic stability: a product of pure strategies is closed under better replies if and only if its span is stable and attracting under regularized learning. In addition, we estimate the rate of convergence to such sets, and we show that methods based on entropic regularization (like the exponential weights algorithm) converge at a geometric rate, while projection-based methods converge within a finite number of iterations, even with bandit, payoff-based feedback.
Background. The question of whether players can learn to emulate rational behavior through repeated interactions has been one of the mainstays of non-cooperative game theory, and it has recently gained increased momentum owing to a surge of breakthrough applications to machine learning and data science, from online ad auctions to multi-agent reinforcement learning. Informally, this question can be stated as follows:
If each player follows an iterative procedure aiming to increase their individual payoff, does the players' long-run behavior converge to a rationally admissible state?
A natural setting for studying this question is to assume that each player is following a no-regret algorithm, i.e., a policy which is asymptotically as good against a given sequence of payoff functions as the best fixed strategy in hindsight. In this framework, the link between learning and rationality is provided by a folk result which states that, under no-regret learning, the empirical frequency of play converges to the game's set of coarse correlated equilibria (CCE) – also known as the game's Hannan set [26]. This result has been of seminal importance to the field because no-regret play can be achieved via a wide class of "regularized learning" policies, as exemplified by the "follow-the-regularized-leader" (FTRL) family of algorithms [45, 46] and its variants – optimistic methods [15, 28, 41, 42, 47], HEDGE/EXP3 [4, 6, 10, 11], implicitly normalized forecasters [1, 3], etc.
All these policies have (at least) one thing in common: they seek to provide the tightest possible guarantees for each player's individual regret, thus accelerating convergence to the game's Hannan set. As such, in games where the marginalization of coarse correlated equilibria coincides with the game's Nash equilibria (like two-player zero-sum games), we obtain a positive equilibrium convergence guarantee: the long-run empirical frequency of play evolves "as if" the players were rational to begin with - i.e., as if they had full knowledge of the game, common knowledge of rationality, the ability to communicate this knowledge, etc.
On the other hand, in many other contexts, the marginals of Hannan-consistent correlated strategies may fail even the weakest axioms of rational behavior and rationalizability (such as the elimination of strictly dominated strategies). In particular, a well-known example of Viossat & Zapechelnyuk [49] (which we discuss in detail in Section 4) shows that it is possible to have negative regret for all time, but still employ only strictly dominated strategies throughout the entire horizon of play.
The reason for this disconnect is that no-regret play has significant predictive power for the empirical frequency of play – that is, the long-run empirical distribution of pure strategy profiles – but much less so for the players' day-to-day sequence of play – i.e., the evolution of the players' actual mixed strategies over time. In particular, even when the marginalization of the Hannan set is Nash, the actual trajectory of play may – and, in fact, often does – diverge away from the game's set of equilibria [16, 24, 34–36] or exhibits chaotic, unpredictable oscillations [12, 38]. Thus, especially in the context of regularized learning – where players learn independently from one another, with no common correlating device – the blanket guarantee of no-regret play may quickly become irrelevant, and even misleading, providing the veneer of rational behavior but not the substance.
Motivated by the above, our paper seeks to understand the rationality properties of the players' actual sequence of play under regularized learning, as encoded by the following question:
Which sets of mixed strategies are stable and attracting under regularized learning? Are these sets robust to strategic deviations? And, if so, is the converse also true?
Our contributions in the context of related work. This question has attracted significant interest in the literature, especially in its pointwise version, namely: Which mixed strategy profiles are stable and attracting under regularized learning? Are the dynamics' stable states robust to unilateral deviations? And, if so, are these the only stable states of regularized learning?
In the related setting of population games, the answer to this question is sometimes referred to as the "folk theorem of evolutionary game theory" [14, 27, 51]. Somewhat informally, this theorem states that, under the replicator dynamics (the continuous-time analogue of the exponential/multiplicative weights algorithm, itself an archetypal regularized learning method), the following is true for all games: only Nash equilibria are (Lyapunov) stable, and a state is stable and attracting under the replicator dynamics if and only if it is a strict Nash equilibrium of the underlying game [27, 51].<sup>1</sup>
In the context of regularized learning, [13, 20, 33] showed that a similar equivalence holds for the dynamics of FTRL in continuous time: a state is stable and attracting under the FTRL dynamics if and only if it is a strict Nash equilibrium. Subsequently, Giannou et al. [22, 23] extended this equivalence to an entire class of regularized learning schemes, with different types of feedback and/or update structures – from optimistic methods to algorithms run with bandit, payoff-based information. In all these cases, the same principle emerges: under regularized learning, a state is asymptotically stable and attracting if and only if it is a strict Nash equilibrium.
This is an important pointwise prediction but it does not cover cases where regularized learning algorithms do not converge to a point, but to a set (such as a limit cycle or other non-trivial attractor). In this case, the very definition of strategic stability is an intricate affair, and there are several definitions that come into play [7, 17, 21, 43]. The first such notion that we consider is that of "resilience to strategic deviations", namely that every unilateral deviation from the set under study is deterred by some other element thereof. Our first contribution in this direction is a universal guarantee to the effect that, with probability 1, in any game, and from any initial condition, the long-run limit of any regularized learning algorithm is a resilient set.
This result is significant in its universality, but the notion of resilience is not sufficiently strong to disallow irrational behavior – and, in fact, it is subject to similar shortcomings as Hannan consistency.
1Strictness here means that each player has a unique best response at equilibrium.
To account for this deficiency, we turn to a much more stringent criterion of setwise strategic stability, that of closedness under better replies (club). This notion, originally due to Ritzberger & Weibull [43], states that any deviation from a product of pure strategies is costly, and it is one of the strictest setwise refinements in game theory. In particular, it refines the notion of closedness under rational behavior (curb) [7], and it satisfies all the seminal strategic stability requirements of Kohlberg & Mertens [30], including robustness to strategic payoff perturbations.[2]
In this general context, we show that regularized learning enjoys a striking relation with club sets: A product of pure strategies is closed under better replies if and only if its span is stable and attracting under regularized learning. In fact, we show that this equivalence can be refined to sets that are minimally closed under better replies (in the sense that they do not contain a strictly smaller closed under better replies (club) set): a product of puer strategies is minimally club (m-club) if and only if its span is irreducibly stable and attracting (in that it does not contain a smaller asymptotically stable span of strategies). Finally, we also estimate the rate of convergence to club sets, and we establish convergence at a geometric rate for entropically regularized methods – like HEDGE and EXP3 – and in a finite number of iterations under projection-based methods.
In light of the above, our results can be seen both as a far-reaching setwise generalization of the folk theorem of evolutionary game theory, as well as a bona fide algorithmic analogue of a precursor result for the replicator dynamics, originally due to Ritzberger & Weibull [43]. Importantly, our analysis covers several different update structures – “vanilla” regularized methods, but also their optimistic variants – as well as a wide range of information models – from full payoff information to bandit, payoff-based feedback.
We start by recalling some basics from game theory, roughly following the classical treatise of Fudenberg & Tirole [21]. First, a finite game in normal form consists of  $(i)$  a finite set of players  $i\in \mathcal{N}\equiv \{1,\ldots ,N\}$ ;  $(ii)$  a finite set of actions - or pure strategies -  $\mathcal{A}_i$  per player  $i\in \mathcal{N}$ ; and  $(iii)$  an ensemble of payoff functions  $u_{i}\colon \prod_{j}\mathcal{A}_{j}\to \mathbb{R}$ , each determining the reward  $u_{i}(\alpha)$  of player  $i\in \mathcal{N}$  in a given action profile  $\alpha = (\alpha_{1},\dots,\alpha_{N})$ . Collectively, we will write  $\mathcal{A} = \prod_{j}\mathcal{A}_{j}$  for the game's action space and  $\Gamma \equiv \Gamma (\mathcal{N},\mathcal{A},u)$  for the game with primitives as above.
During play, each player  $i \in \mathcal{N}$  may randomize their choice of action by playing a mixed strategy, i.e., a probability distribution  $x_i \in \mathcal{X}_i \coloneqq \Delta(\mathcal{A}_i)$  over  $\mathcal{A}_i$  that selects  $\alpha_i \in \mathcal{A}_i$  with probability  $x_{i\alpha_i}$ . To lighten notation, we identify  $\alpha_i \in \mathcal{A}_i$  with the mixed strategy that assigns all weight to  $\alpha_i$  (thus justifying the terminology "pure strategies"). Then, writing  $x = (x_i)_{i \in \mathcal{N}}$  for the players' strategy profile and  $\mathcal{X} = \prod_i \mathcal{X}_i$  for the game's strategy space, the players' payoff functions may be extended to all of  $\mathcal{X}$  by setting
$$
u _ {i} (x) := \mathbb {E} _ {\alpha \sim x} \left[ u _ {i} (\alpha) \right] = \sum_ {\alpha \in \mathcal {A}} u _ {i} (\alpha) x _ {\alpha} \tag {1}
$$
where, in a slight abuse of notation, we write  $x_{\alpha}$  for the joint probability of playing  $\alpha \in \mathcal{A}$  under  $x$ , i.e.,  $x_{\alpha} = \prod_{i} x_{i\alpha_i}$ . This randomized framework will be referred to as the mixed extension of  $\Gamma$  and we will denote it by  $\Delta(\Gamma)$ .
For concision, we will also write  $(x_{i};x_{-i}) = (x_{1},\ldots ,x_{i},\ldots ,x_{N})$  for the strategy profile where player  $i$  plays  $x_{i}\in \mathcal{X}_{i}$  against the strategy profile  $x_{-i}\in \prod_{j\neq i}\mathcal{X}_j$  of all other players (and likewise for pure strategies). In this notation, we also define each player's mixed payoff vector as
$$
v _ {i} (x) = \left(u _ {i} \left(\alpha_ {i}; x _ {- i}\right)\right) _ {\alpha_ {i} \in \mathcal {A} _ {i}} \tag {2}
$$
so the payoff to player  $i\in \mathcal{N}$  under  $x\in \mathcal{X}$  becomes
$$
u _ {i} (x) = \sum_ {\alpha_ {i} \in \mathcal {A} _ {i}} u _ {i} (\alpha_ {i}; x _ {- i}) x _ {i \alpha_ {i}} = \langle v _ {i} (x), x _ {i} \rangle . \tag {3}
$$
Moving forward, the best-response correspondence of player  $i \in \mathcal{N}$  is defined as the set-valued mapping  $\mathsf{br}_i \colon \mathcal{X} \Rightarrow \mathcal{X}_i$  given by
$$
\operatorname {b r} _ {i} (x) = \arg \max  _ {x _ {i} ^ {\prime} \in \mathcal {X} _ {i}} u _ {i} \left(x _ {i} ^ {\prime}; x _ {- i}\right) \quad \text {f o r a l l} x \in \mathcal {X}. \tag {4}
$$
Extending this over all players, we will write  $\mathsf{br} = \prod_{i}\mathsf{br}_{i}$  for the product correspondence  $\mathsf{br}(x) = \mathsf{br}_1(x)\times \dots \times \mathsf{br}_N(x)$ , and we will say that  $x^{*}\in \mathcal{X}$  is a Nash equilibrium (NE) if  $x^{*}\in \mathsf{br}(x^{*})$ .
2Roughly speaking, robustness to strategic payoff perturbations means that the set under study remains stable even if the payoffs of the game are subject to small - but possibly adversarial - perturbations.
Equivalently, given that  $u_{i}(x_{i}^{\prime};x_{-i})$  is linear in  $x_{i}^{\prime}$ , we conclude that  $x^{*}$  is a Nash equilibrium if and only if
$$
u _ {i} \left(x ^ {*}\right) \geq u _ {i} \left(\alpha_ {i}; x _ {- i} ^ {*}\right) \quad \text {f o r a l l} \alpha_ {i} \in \mathcal {A} _ {i} \text {a n d a l l} i \in \mathcal {N}. \tag {NE}
$$
As a final point of note, if  $x^{*}$  is a Nash equilibrium where each player has a unique best response – that is,  $\mathsf{br}_i(x^*) = \{x_i^*\}$  for all  $i \in \mathcal{N}$  – we will say that  $x^{*}$  is strict because, in this case,  $u_{i}(x^{*}) > u_{i}(x_{i};x_{-i}^{*})$  for all  $x_{i} \neq x_{i}^{*}, i \in \mathcal{N}$ . An immediate consequence of this is that strict equilibria are pure, i.e., each  $x_{i}^{*}$  is a pure strategy. Among Nash equilibria, strict equilibria are the only ones that are “structurally robust” (in the sense that they remain invariant to small perturbations of the underlying game), so they play a particularly important role in game theory.
3 Regularized learning in games
Throughout our paper, we will consider iterative decision processes that unfold as follows:
1. At each stage  $t = 1,2,\ldots$ , every participating agent selects an action.
2. Agents receive a reward determined by their chosen actions and their individual payoff functions.
3. Based on this reward (or other feedback), the agents update their strategies and the process repeats.
In this online setting, a crucial requirement is the minimization of the players' regret, i.e., the difference between a player's cumulative payoff over time and the player's best possible strategy in hindsight. Formally, if the players' actions at each epoch  $t = 1,2,\ldots$  are collectively drawn by the probability distribution  $z_{t}\in \Delta (\mathcal{A})$ , the regret of each player  $i\in \mathcal{N}$  is defined as
$$
\operatorname {R e g} _ {i} (T) = \max  _ {\alpha_ {i} \in \mathcal {A} _ {i}} \sum_ {t = 1} ^ {T} \left[ u _ {i} \left(\alpha_ {i}; z _ {- i, t}\right) - u _ {i} \left(z _ {t}\right) \right], \tag {5}
$$
and we will say that player  $i$  has no regret if  $\operatorname{Reg}_i(T) = o(T)$ .
One of the most widely used policies to achieve no-regret play is the so-called "follow-the-regularized-leader" (FTRL) family of algorithms and its variants [45, 46]. To motivate the analysis to come, we begin with an archetypal FTRL method, the exponential/multiplicative weights algorithm, also known as HEDGE [4, 5, 11].
3.1. A gentle start. We begin our discussion with a "stimulus-response" approach in the spirit of Erev & Roth [19]: First, at each stage  $t = 1,2,\ldots$ , every player  $i\in \mathcal{N}$  employs a mixed strategy  $x_{i,t}\in \mathcal{X}_i$  to select an action  $\alpha_{i,t}\in \mathcal{A}_i$ . Subsequently, to measure the performance of their pure strategies over time, each player further maintains a score variable which is updated recursively as
$$
y _ {i \alpha_ {i}, t + 1} = y _ {i \alpha_ {i}, t} + u _ {i} \left(\alpha_ {i}; \alpha_ {- i, t}\right) \quad \text {f o r a l l} \alpha_ {i} \in \mathcal {A} _ {i}. \tag {6}
$$
In words,  $y_{i\alpha_i,t}$  simply tracks the cumulative payoff of the pure strategy  $\alpha_i \in \mathcal{A}_i$  up to time  $t$  (inclusive). As such, this score can be treated as a propensity to play a given pure strategy at any given stage: the strategies  $\alpha_i \in \mathcal{A}_i$  with the highest propensity scores  $y_{i\alpha_i,t+1}$  should be played with higher probability at stage  $t+1$ .
The most widely used instantiation of this stimulus-response mechanism is the logit choice rule
$$
\Lambda_ {i} \left(y _ {i}\right) \equiv \frac {\left(\exp \left(y _ {i \alpha_ {i}}\right)\right) _ {\alpha_ {i} \in \mathcal {A} _ {i}}}{\sum_ {\alpha_ {i} \in \mathcal {A} _ {i}} \exp \left(y _ {i \alpha_ {i}}\right)} \tag {7}
$$
which means that each player selects an action with probability that is exponentially proportional to its score. In this way, we obtain the exponential/multiplicative weights – or HEDGE – algorithm
$$
y _ {i, t + 1} = y _ {i, t} + \gamma_ {t} v _ {i} \left(\alpha_ {t}\right) \quad x _ {i, t + 1} = \Lambda_ {i} \left(y _ {i, t + 1}\right) \quad \alpha_ {i, t + 1} \sim x _ {i, t + 1} \tag {HEDGE}
$$
where  $\gamma_{t}$  is the algorithm's "learning rate". For an appetizer to the literature on (HEDGE), see [2, 10, 11, 31, 45] and references therein.
The rest of the methods we discuss below will vary some – or even all – of the components of (HEDGE): the information used to update the players' propensity scores, the way that propensity scores are mapped to mixed strategies, and/or even the way that pure actions are selected. However, all of the methods under study will be characterized by the same "stimulus-response" reinforcement mechanism: actions that seem to be performing better over time are employed with higher probability, up to some "regularization" that incentivizes exploration of underperforming actions.
3Of course, updating these scores requires the knowledge of the "what if" pure payoffs  $u_{i}(\alpha_{i};\alpha_{-i,t})$  at each stage  $t$ , but we assume for the moment that this information is available (we will relax this assumption later on).
3.2. The regularized learning template. In the rest of our paper, we will work with an abstract regularized learning (RL) template which builds on the same stimulus-response principle as (HEDGE), while allowing us to simultaneously consider different types of feedback, strategy sampling policies, update structures, etc. To lighten notation below, we will drop the player index  $i \in \mathcal{N}$  when the meaning can be inferred from the context; also, to stress the distinction between "strategy-like" and "payoff-like" variables, we will write throughout  $\mathcal{Y}_i \coloneqq \mathbb{R}^{\mathcal{A}_i}$  and  $\mathcal{Y} \coloneqq \prod_i \mathcal{Y}_i$  for the game's "payoff space", in direct analogy to  $\mathcal{X}_i$  and  $\mathcal{X} = \prod_i \mathcal{X}_i$  for the game's strategy space.
With all this in hand, consider the following general class of regularized learning methods:
Aggregate payoff information (stimulus):  $Y_{i,t + 1} = Y_{i,t} + \gamma_t\hat{v}_{i,t}$
Update choice probabilities (response):  $X_{i,t + 1} = Q_i(Y_{i,t + 1})$
In tune with (HEDGE), the various elements of (RL) are defined as follows:
1.  $X_{i,t}\in \mathcal{X}_i$  denotes the mixed strategy of player  $i$  at time  $t = 1,2,\ldots$
2.  $Y_{i,t} \in \mathcal{V}_i$  is a "score vector" that measures the performance of the player's actions over time.
3.  $Q_{i}\colon \mathcal{Y}_{i}\to \mathcal{X}_{i}$  is a "regularized choice map" that maps score vectors to choice probabilities.
4.  $\hat{v}_{i,t}$  is a surrogate/ approximation of the mixed payoff vector  $v_{i}(X_{t})$  of player  $i$  at time  $t$ .
5.  $\gamma_{t} > 0$  is a step-size/sensitivity parameter of the form  $\gamma_t\propto 1 / t^{\ell_\gamma}$  for some  $\ell_{\gamma}\in [0,1]$
In words, at each stage of the process, every player  $i \in \mathcal{N}$  observes - or otherwise estimates - a proxy  $\hat{v}_{i,t}$  of their individual payoff vector; subsequently, players augment their actions' scores based on this information, they select a mixed strategy via the regularized choice map  $Q_{i}$ , and the process repeats. To streamline our presentation, we discuss in detail the precise definition of  $\hat{v}$  and  $Q$  in Sections 3.3 and 3.4 below, and we present a series of examples of (RL) in Section 3.5 right after.
3.3. Aggregating payoff information. As noted above, the main idea of regularized learning is to track the players' payoff vector  $v(X_{t})$ . Importantly, there are several different modeling choices that can be made here: players may have direct access to their payoff vectors (in the full information setting), or some noisy approximation obtained by an inner randomization of the algorithm (e.g., when they receive information on their pure actions); they may have to recreate their payoff vectors altogether (as in the bandit setting), or their estimates may be based on a strategy other than the one they actually played (as in the case of optimistic algorithms).
In all cases, we will represent the surrogate payoff vector  $\hat{v}_t$  as
$$
\hat {v} _ {t} = v \left(X _ {t}\right) + U _ {t} + b _ {t} \tag {8}
$$
where  $b_{t} = \mathbb{E}[\hat{v}_{t} \mid \mathcal{F}_{t}] - v(X_{t})$  and  $U_{t} = \hat{v}_{t} - \mathbb{E}[\hat{v}_{t} \mid \mathcal{F}_{t}]$  respectively denote the offset and the random error of  $\hat{v}_{t}$  relative to  $v(X_{t})$ . To streamline our presentation, we will also assume that  $\| b_{t} \| = \mathcal{O}(1 / t^{\ell_{b}})$  and  $\| U_{t} \| = \mathcal{O}(t^{\ell_{\sigma}})$  for some  $\ell_{b}, \ell_{\sigma} \geq 0$ ; we discuss the specifics of these bounds later in the paper.
3.4. From scores to strategies. Regarding the "scores-to-strategies" step of (RL), we will follow the classical approach of Shalev-Shwartz [45] and assume that each player is employing a regularized choice map of the general form
$$
Q _ {i} \left(y _ {i}\right) = \arg \max  _ {x _ {i} \in \mathcal {X} _ {i}} \left\{\left\langle y _ {i}, x _ {i} \right\rangle - h _ {i} \left(x _ {i}\right) \right\} \quad \text {f o r a l l} y _ {i} \in \mathcal {Y} _ {i}. \tag {9}
$$
In the above, the regularizer  $h_i \colon \mathcal{X}_i \to \mathbb{R}$  acts as a penalty that smooths out the "hard" argmax correspondence  $y_i \mapsto \arg \max_{x_i \in \mathcal{X}_i} \langle y_i, x_i \rangle$ . Accordingly, instead of following the "leader" (i.e., playing the strategy with the highest propensity score), players follow the "regularized leader" – that is, they allow for a certain degree of uncertainty in their choice of strategy [10, 33, 45, 46].
To ease notation, we will work with kernelized regularizers of the form  $h_i(x_i) = \sum_{\alpha_i \in \mathcal{A}_i} \theta(x_{i\alpha_i})$  for some continuous function  $\theta: [0,1] \to \mathbb{R}$  with  $\inf_{z \in (0,1]} \theta''(z) > 0$ . We will also say that the players' regularizers are steep if  $\lim_{z \to 0^+} \theta'(z) = -\infty$ , and non-steep otherwise.