SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gabriele Farina MIT
Charilaos Pipis MIT
chpipis@mit.edu
No-regret learners seek to minimize the difference between the loss they cumulated through the actions they played, and the loss they would have cumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. The size of the set of transformations considered by the learner determines a natural notion of rationality. As the set of transformations each learner considers grows, the strategies played by the learners recover more complex game-theoretic equilibria, including correlated equilibria in normal-form games and extensive-form correlated equilibria in extensive-form games. At the extreme, a no-swap-regret agent is one that minimizes regret against the set of all functions from the set of strategies to itself. While it is known that the no-swap-regret condition can be attained efficiently in nonsequential (normal-form) games, understanding what is the strongest notion of rationality that can be attained efficiently in the worst case in sequential (extensive-form) games is a longstanding open problem. In this paper we provide a positive result, by showing that it is possible, in any sequential game, to retain polynomial-time (in the game tree size) iterations while achieving sublinear regret with respect to all linear transformations of the mixed strategy space, a notion called no-linear-swap regret. This notion of hindsight rationality is as strong as no-swap-regret in nonsequential games, and stronger than no-trigger-regret in sequential games—thereby proving the existence of a subset of extensive-form correlated equilibria robust to linear deviations, which we call linear-deviation correlated equilibria, that can be approached efficiently.
The framework of regret minimization provides algorithms that players can use to gradually improve their strategies in a repeated game, enabling learning strong strategies even when facing unknown and potentially adversarial opponents. One of the appealing properties of no-regret learning algorithms is that they are uncoupled, meaning that each player refines their strategy based on their own payoff function, and on other players' strategies, but not on the payoff functions of other players. Nonetheless, despite their uncoupled nature and focus on local optimization of each player's utility, it is one of the most celebrated results in the theory of learning in games that in many cases, when all players are learning using these algorithms, the empirical play recovers appropriate notions of equilibrium—a global notion of game-theoretic optimality. Strategies constructed via no-regret learning algorithms (or approximations thereof) have been key components in constructing human-level and even superhuman AI agents in a variety of adversarial games, including Poker [Moravčík et al., 2017, Brown and Sandholm, 2018, 2019], Stratego [Perolat et al., 2022], and Diplomacy [Bakhtin et al., 2023].
In regret minimization, each learning agent seeks to minimize the difference between the loss (opposite of reward) they accumulated through the actions they played, and the loss they would have
accumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. The size of the set of transformation functions considered by the learning agent determines a natural notion of rationality of the agent. Already when the agents seeks to learn strategies that cumulate low regret against constant strategy transformations only—a notion of regret called external regret—the average play of the agents converges to a Nash equilibrium in two-player constant-sum games, and to a coarse correlated equilibrium in general-sum multiplayer games. As the sets of transformations the each agent considers grows, more complex equilibria can be achieved, including correlated equilibria in normal-form games (Foster and Vohra [1997], Fudenberg and Levine [1995, 1999], Hart and Mas-Colell [2000, 2001]; see also the monograph by Fudenberg and Levine [1998]) and extensive-form correlated equilibria in extensive-form games [Farina et al., 2022]. At the extreme, a maximally hindsight-rational agent is one that minimizes regret against the set of all functions from the strategy space to itself (aka. swap regret). While it is known that maximum hindsight rationality can be attained efficiently in nonsequential (normal-form) games [Stoltz and Lugosi, 2007, Blum and Mansour, 2007], it is a major open problem to determine whether the same applies to sequential (i.e., extensive-form) games, and more generally what is the strongest notion of rationality that can be attained efficiently in the worst case in the latter setting.
In this paper, we provide a positive result in that direction, by showing that hindsight rationality can be achieved efficiently in general imperfect-information extensive-form games when one restricts to the set of all linear transformations of the mixed strategy space—a notion called linear-swap regret, and that coincides with swap regret in normal-form games. In order to establish the result, we introduce several intermediate results related to the geometry of sequence-form strategies in extensive-form games. In particular, a crucial result is given in Theorem 3.1, which shows that the set of linear functions  $\mathcal{M}_{\mathcal{Q}\to \mathcal{P}}$  from the sequence-form strategy set  $\mathcal{Q}$  of a player in an extensive-form game to a generic convex polytope  $\mathcal{P}$  can be captured using only polynomially many linear constraints in the size of the game tree and the number of linear constraints that define  $\mathcal{P}$ . Applying the result to the special case  $\mathcal{P} = \mathcal{Q}$ , we are then able to conclude that the polytope of linear transformations  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$  from the sequence-form strategy set to itself can be captured by polynomially many linear constraints in the size of the game tree, and the norm of any element is polynomially bounded. The polynomial characterization and bound for  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$  is used in conjunction with an idea of Gordon et al. [2008] to construct a no-linear-swap-regret minimizer for the set of strategies  $\mathcal{Q}$  starting from two primitives: i) a no-external-regret algorithm for the set of transformations  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$ , and ii) an algorithm to compute a fixed point strategy for any transformation in  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$ . In both cases, the polynomial representation of  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$  established through Theorem 3.1 plays a fundamental role. It allows, on the one hand, to satisfy requirement ii) using linear programming. On the other hand, it enables us to construct a no-external-regret algorithm that outputs transformations in  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$  with polynomial-time iterations, by leveraging the known properties of online projected gradient descent, exploiting the tractability of projecting onto polynomially-representable polytopes.
Finally, in the last section of the paper we turn our attention away from hindsight rationality to focus instead on the properties of the equilibria that our no-linear-swap-regret dynamics recover in extensive-form games. The average play of no-linear-swap-regret players converges to a set of equilibria that we coin linear-deviation correlated equilibria (LCEs). LCEs form a superset of correlated equilibria and a subset of extensive-form correlated equilibria in extensive-form games. In Section 4 we show that these inclusions are in general strict, and provide additional results about the complexity of computing a welfare-maximizing LCE.
Related work As mentioned in the introduction, the existence of uncoupled no-regret dynamics leading to correlated equilibrium (CE) in multiplayer normal-form games is a celebrated result dating back to at least the work by Foster and Vohra [1997]. That work inspired researchers to seek uncoupled learning procedures in other settings as well. For example, Stoltz and Lugosi [2007] studies learning dynamics leading to CE in games with an infinite (but compact) action set, while Kakade et al. [2003] focuses on graphical games. In more recent years, a growing effort has been spent towards understanding the relationships between no-regret learning and equilibria in imperfect-information extensive-form games, the settings on which we focus. Extensive-form games pose additional challenges when compared to normal-form games, due to their sequential nature and presence of imperfect information. While efficient no-external-regret learning dynamics for extensive-form games are known (including the popular CFR algorithm [Zinkevich et al., 2008]), as of today not much is known about no-swap-regret and the complexity of learning CE in extensive-form games.
In recent work, Anagnostides et al. [2023] construct trigger-regret dynamics that converge to EFCE at a rate of  $O\left(\frac{\log T}{T}\right)$ , whereas our regret dynamics converge to linear-deviation correlated equilibria at a slower rate of  $O\left(\frac{1}{\sqrt{T}}\right)$ . While the aforementioned paper proposes a general methodology that applies to CEs in normal-form games and EFCE/EFCCE in sequential games, the authors' construction fundamentally relies on being able to express the fixed points computed by the algorithm as (linear combinations of) rational functions with positive coefficients of the deviation matrices. For EFCE and EFCCE this fundamentally follows from the fact that the fixed points can be computed inductively, solving for stationary distributions of local Markov chains at each decision point. The no-linear-swap regret algorithm proposed in our paper does not offer such a local characterization of the fixed point and thus, we cannot immediately transfer the improved regret bounds to our case.
The closest notion to CE that is known to be efficiently computable in extensive-form games is extensive-form correlated equilibrium (EFCE) [von Stengel and Forges, 2008, Huang and von Stengel, 2008]. The question of whether the set of EFCE could be approached via uncoupled no-regret dynamics with polynomial-time iterations in the size of the extensive-form games was recently settled in the positive [Farina et al., 2022, Celli et al., 2020]. In particular, Farina et al. [2022] show that EFCE arises from the average play of no-trigger-regret algorithms, where trigger deviations are a particular subset of linear transformations of the sequence-form strategy polytope  $\mathcal{Q}$  of each player. Since this paper focuses on learning dynamics that guarantee sublinear regret with respect to any linear transformation of  $\mathcal{Q}$ , it follows immediately that the dynamics presented in this paper recover EFCE as a special case.
The concept of linear-swap-regret minimization has been considered before in the context of Bayesian games. Mansour et al. [2022] study a setting where a no-regret learner competes in a two-player Bayesian game with a rational utility maximizer, that is a strictly more powerful opponent than a learner. Under this setting, it can be shown that in every round the optimizer is guaranteed to obtain at least the Bayesian Stackelberg value of the game. Then they proceed to prove that minimizing linear-swap regret is necessary if we want to cap the optimizer's performance at the Stackelberg value, while minimizing polytope-swap regret (a generalization of swap regret for Bayesian games, and strictly stronger than linear-swap) is sufficient to cap the optimizer's performance. Hence, these results highlight the importance of developing learning algorithms under stronger notions of rationality, as is our aim in this paper. Furthermore, these results provide evidence that constructing a no-linear-swap regret learner, as is our goal here, can present benefits when compared to other less rational learners. In a concurrent paper, Fujii [2023] defines the notion of untruthful swap regret for Bayesian games and proves that, for Bayesian games, it is equivalent to the linear-swap regret which is of interest in our paper.
Bayesian games can be considered as a special case of extensive-form games, where a chance node initially selects one of the possible types  $\Theta$  for each player. Thus, our algorithm minimizing linear-swap regret in extensive-form games also minimizes linear-swap regret in Bayesian games. However, we remark that our regret bound depends polynomially on the number of player types  $|\Theta|$  as they are part of the game tree representation, while Fujii [2023] has devised an algorithm for Bayesian games, whose regret only depends on  $\log |\Theta|$ .
Finally, we also mention that nonlinear deviations have been explored in extensive-form games, though we are not aware of notable large sets for which polynomial-time no-regret dynamics can be devised. Specifically, we point to the work by Morrill et al. [2021], which defines the notion of "behavioral deviations". These deviations are nonlinear with respect to the sequence-form representation of strategies in extensive-form games. The authors categorize several known or novel types of restricted behavioral deviations into a Deviation Landscape that highlights the relations between them. Even though both the linear deviations, we consider in this paper, and the behavioral deviations seem to constitute rich measures of rationality, none of them contains the other and thus, linear deviations do not fit into the Deviation Landscape of Morrill et al. [2021] (see also Remark E.2).
We recall the standard model of extensive-form games, as well as the framework of learning in games.
2.1 Extensive-Form Games
While normal-form games (NFGs) correspond to nonsequential interactions, such as Rock-Paper-Scissors, where players simultaneously pick one action and then receive a payoff based on what others picked, extensive-form games (EFGs) model games that are played on a game tree. They capture both sequential and simultaneous moves, as well as private information and are therefore a very general and expressive model of games, capturing chess, go, poker, sequential auctions, and many other settings as well. We now recall basic properties and notation for EFGs.
Game tree In an  $n$ -player extensive-form game, each node in the game tree is associated with exactly one player from the set  $\{1, \dots, n\} \cup \{c\}$ , where the special player  $c$  called the chance player—is used to model random stochastic outcomes, such as rolling a die or drawing cards from a deck. Edges leaving from a node represent actions that a player can take at that node. To model private information, the game tree is supplemented with an information partition, defined as a partition of nodes into sets called information sets. Each node belongs to exactly one information set, and each information set is a nonempty set of tree nodes for the same Player  $i$ . An information set for Player  $i$  denotes a collection of nodes that Player  $i$  cannot distinguish among, given what she has observed so far. (We remark that all nodes in a same information set must have the same set of available actions, or the player would distinguish the nodes). The set of all information sets of Player  $i$  is denoted  $\mathcal{I}_i$ . In this paper, we will only consider perfect-recall games, that is, games in which the information sets are arranged in accordance with the fact that no player forgets what the player knew earlier.
Sequence-form strategies Since nodes belonging to the same information set for a player are indistinguishable to that player, the player must play the same strategy at each of the nodes. Hence, a strategy for a player is exactly a mapping from an information set to a distribution over actions. In other words, it is the information sets, and not the game tree nodes, that capture the decision points of the player. We can then represent a strategy for a generic player  $i$  as a vector indexed by each valid information set-action pair  $(j,a)$ . Any such valid pair is called a sequence of the player; the set of all sequences is denoted as  $\Sigma_{i} := \{(j,a) : j \in \mathcal{I}_{i}, a \in \mathcal{A}_{j}\} \cup \{\varnothing\}$ , where the special element  $\varnothing$  is called empty sequence. Given an information set  $j \in \mathcal{I}_{i}$ , we denote by  $p_{j}$  the parent sequence of  $j$ , defined as the last pair  $(j,a) \in \Sigma_{i}$  encountered on the path from the root to any node  $v \in j$ ; if no such pair exists we let  $p_{j} = \emptyset$ . Finally, we denote by  $\mathcal{C}_{\sigma}$  the children of sequence  $\sigma \in \Sigma_{i}$ , defined as the information sets  $j \in \mathcal{I}_{i}$  for which  $p_{j} = \sigma$ . Sequences  $\sigma$  for which  $\mathcal{C}_{\sigma}$  is an empty set are called terminal; the set of all terminal sequences is denoted  $\Sigma_{i}^{\perp}$ .
Example 2.1. Consider the tree-form decision process faced by Player 1 in the small game of Figure 1 (Left). The decision process has four decision nodes  $\mathcal{J}_1 = \{\mathsf{A},\mathsf{B},\mathsf{C},\mathsf{D}\}$  and nine sequences including the empty sequence  $\varnothing$ . For decision node  $D$ , the parent sequence is  $p_D = A2$ ; for  $B$  and  $C$  it is  $p_B = A1$ ; for  $A$  it is the empty sequence  $p_A = \emptyset$ .
Sequence-form constraints:
$$
\left\{ \begin{array} {c}\pmb {x} [ \varnothing ] = 1, \\ \pmb {x} [ \mathsf {A} 1 ] + \pmb {x} [ \mathsf {A} 2 ] = \pmb {x} [ \varnothing ], \\ \pmb {x} [ \mathsf {B} 3 ] + \pmb {x} [ \mathsf {B} 4 ] = \pmb {x} [ \mathsf {A} 1 ], \\ \pmb {x} [ \mathsf {C} 5 ] + \pmb {x} [ \mathsf {C} 6 ] = \pmb {x} [ \mathsf {A} 1 ], \\ \pmb {x} [ \mathrm {D} 7 ] + \pmb {x} [ \mathrm {D} 8 ] + \pmb {x} [ \mathrm {D} 9 ] = \pmb {x} [ \mathrm {A} 2 ]. \\ \end{array} \right.
$$
Figure 1: (Left) Tree-form decision process considered in the example. Black round nodes belong to Player 1; white round nodes to Player 2. Square white nodes are terminal nodes in the game tree, payoffs are omitted. Gray bags denote information sets. (Right) The constraints that define the sequence-form polytope  $\mathcal{Q}_1$  for Player 1 (besides nonnegativity).
A reduced-normal-form plan for Player  $i$  represents a deterministic strategy for the player as a vector  $\pmb{x} \in \{0,1\}^{\Sigma_i}$  where the entry corresponding to the generic sequence  $\pmb{x}[ja]$  is equal to 1 if the player plays action  $a$  at (the nodes of) information set  $j \in \mathcal{J}_i$ . Information sets that cannot be reached based on the strategy do not have any action select. A crucial property of the reduced-normal-form plan representation of deterministic strategies is the fact that the utility of any player is a multilinear function in the profile of reduced-normal-form plans played by the players. The set of all reduced-
normal-form plans of Player  $i$  is denoted with the symbol  $\Pi_i$ . Typically, the cardinality of  $\Pi_i$  is exponential in the size of the game tree.
The convex hull of the set of reduced-normal-form plans of Player  $i$  is called the sequence-form polytope of the player, and denoted with the symbol  $\mathcal{Q}_i\coloneqq \mathrm{conv}(\Pi_i)$ . It represents the set of all randomized strategies in the game. An important result by Romanovskii [1962], Koller et al. [1996], von Stengel [1996] shows that  $\mathcal{Q}_i$  can be captured by polynomially many constraints in the size of the game tree, as we recall next.
Definition 2.2. The polytope of sequence-form strategies of Player  $i$  is equal to the convex polytope
$$
\mathcal {Q} _ {i} := \left\{\boldsymbol {x} \in \mathbb {R} _ {\geq 0} ^ {\Sigma}: \begin{array}{l l} (1) & \boldsymbol {x} [ \varnothing ] = 1 \\ (2) & \sum_ {a \in \mathcal {A} _ {j}} \boldsymbol {x} [ j a ] = \boldsymbol {x} [ p _ {j} ] \quad \forall j \in \mathcal {J} \end{array} \right\}.
$$
As an example, the constraints that define the sequence-form polytope for Player 1 in the game of Figure 1 (Left) are shown in Figure 1 (Right). The polytope of sequence-form strategies possesses a strong combinatorial structure that enables speeding up several common optimization procedures and will be crucial in developing efficient algorithms to converge to equilibrium.
2.2 Hindsight Rationality and Learning in Games
Games are one of many situations in which a decision-maker has to act in an online manner. For these situations, the most widely used protocol is that of Online Learning (e.g., see Orabona [2022]). Specifically, each learner has a set of actions or behavior they can employ  $\mathcal{X} \subseteq \mathbb{R}^d$  (in extensive-form games, this would typically be the set of reduced-normal-form strategies). At each timestep  $t$  the learner first selects, possibly at random, an element  $x \in \mathcal{X}$ , and then receives a loss (opposite of utility) function  $\ell^{(t)}: \mathcal{X} \mapsto \mathbb{R}$ . Since as we observed the above utilities in EFGs are linear in each player's reduced-normal-form plans, for the rest of the paper we focus on the case in which the loss function  $\ell^{(t)}$  is linear, that is, of the form  $\ell^{(t)}: x \mapsto \langle \ell^{(t)}, x^{(t)} \rangle$ .
A widely adopted objective for the learner is that of ensuring vanishing average regret with high probability. Regret is defined as the difference between the loss the learner cumulated through their choice of behavior, and the loss they would have cumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. In particular, let  $\Phi$  be a desired set of strategy transformations  $\phi : \mathcal{X} \to \mathcal{X}$  that the learner might want to learn not to regret. Then, the learner's  $\Phi$ -regret is defined as the quantity
$$
\Phi \text {- R e g} ^ {(T)} := \max  _ {\phi \in \Phi} \sum_ {t = 1} ^ {T} \left(\langle \boldsymbol {\ell} ^ {(t)}, \boldsymbol {x} ^ {(t)} \rangle - \langle \boldsymbol {\ell} ^ {(t)}, \phi (\boldsymbol {x} ^ {(t)}) \rangle\right)
$$
A no- $\Phi$ -regret algorithm (also known as a  $\Phi$ -regret minimizer) is one that, at all times  $T$ , guarantees with high probability that  $\Phi \cdot \mathrm{Reg}^{(T)} = o(T)$  no matter what is the sequence of losses revealed by the environment. The size of the set  $\Phi$  of strategy transformations defines a natural measure of rationality (sometimes called hindsight rationality) for players, and several choices have been discussed in the literature. Clearly, as  $\Phi$  gets larger, the learner becomes more rational. On the flip side, guaranteeing sublinear regret with respect to all transformations in the chosen set  $\Phi$  might be intractable in general. On one end of the spectrum, perhaps the smallest meaningful choice of  $\Phi$  is the set of all constant transformations  $\Phi^{\mathrm{const}} = \{\phi_{\hat{x}}: x \mapsto \hat{x}\}_{\hat{x} \in \mathcal{X}}$ . In this case,  $\Phi^{\mathrm{const}}$ -regret is also called external regret and has been extensively studied in the field of online convex optimization. On the other end of the spectrum, swap regret corresponds to the setting in which  $\Phi$  is the set of all transformations  $\mathcal{X} \to \mathcal{X}$ . Intermediate, and of central importance in this paper, is the notion of linear-swap regret, which corresponds to the case in which
$$
\Phi := \left\{\boldsymbol {x} \mapsto \mathbf {A} \boldsymbol {x}: \mathbf {A} \in \mathbb {R} ^ {d \times d}, \text {w i t h} \mathbf {A} \boldsymbol {x} \in \mathcal {X} \quad \forall \boldsymbol {x} \in \mathcal {X} \right\} \quad (\text {l i n e a r - s w a p d e v i a t i o n s})
$$
is the set of all linear transformations from  $\mathcal{X}$  to itself.
An important observation is that when all considered deviation functions in  $\Phi$  are linear, an algorithm guaranteeing sublinear  $\Phi$ -regret for the set  $\mathcal{X}$  can be constructed immediately from a deterministic
For the purposes of this paper, the adjective linear refers to the fact that each transformation can be expressed in the form  $\mathbf{x} \mapsto \mathbf{A}\mathbf{x}$  for an appropriate matrix  $\mathbf{A}$ .