Title: Polynomial-Time Linear-Swap Regret Minimization in Imperfect-Information Sequential Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning

algorithmic game theory

extensive form games

correlated equilibrium

swap regret

linear swap regret
Abstract: No-regret learners seek to minimize the difference between the loss they cumulated through the actions they played, and the loss they would have cumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. The size of the set of transformations considered by the learner determines a natural notion of rationality. As the set of transformations each learner considers grows, the strategies played by the learners recover more complex game-theoretic equilibria, including correlated 
equilibria in normal-form games and extensive-form correlated equilibria in extensive-form games. At the extreme, a no-swap-regret agent is one that minimizes regret against the set of all functions from the set of strategies to itself. While it is known that the no-swap-regret condition can be attained efficiently in nonsequential (normal-form) games, understanding what is the strongest notion of rationality that can be attained efficiently in the worst case in sequential (extensive-form) games is a longstanding open problem. In this paper we provide a positive result, by showing that it is possible, in any sequential game, to retain polynomial-time (in the game tree size) iterations while achieving sublinear regret with respect to all linear transformations of the mixed strategy space, a notion called no-linear-swap regret. This notion of hindsight rationality is as strong as no-swap-regret in nonsequential games, and stronger than no-trigger-regret in sequential games—thereby proving the existence of a subset of extensive-form correlated equilibria robust to linear deviations, which we call linear-deviation correlated equilibria, that can be approached efficiently.

Polynomial-Time Linear-Swap Regret Minimization in Imperfect-Information Sequential Games

Abstract

No-regret learners seek to minimize the difference between the loss they cumulated through the actions they played, and the loss they would have cumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. The size of the set of transformations considered by the learner determines a natural notion of rationality. As the set of transformations each learner considers grows, the strategies played by the learners recover more complex game-theoretic equilibria, including correlated equilibria in normal-form games and extensive-form correlated equilibria in extensive-form games. At the extreme, a no-swap-regret agent is one that minimizes regret against the set of all functions from the set of strategies to itself. While it is known that the noswap-regret condition can be attained efficiently in nonsequential (normal-form) games, understanding what is the strongest notion of rationality that can be attained efficiently in the worst case in sequential (extensive-form) games is a longstanding open problem. In this paper we provide a positive result, by showing that it is possible, in any sequential game, to retain polynomial-time (in the game tree size) iterations while achieving sublinear regret with respect to all linear transformations of the mixed strategy space, a notion called no-linear-swap regret . This notion of hindsight rationality is as strong as no-swap-regret in nonsequential games, and stronger than no-trigger-regret in sequential games-thereby proving the existence of a subset of extensive-form correlated equilibria robust to linear deviations, which we call linear-deviation correlated equilibria , that can be approached efficiently.

1 Introduction

The framework of regret minimization provides algorithms that players can use to gradually improve their strategies in a repeated game, enabling learning strong strategies even when facing unknown and potentially adversarial opponents. One of the appealing properties of no-regret learning algorithms is that they are uncoupled , meaning that each player refines their strategy based on their own payoff function, and on other players' strategies, but not on the payoff functions of other players. Nonetheless, despite their uncoupled nature and focus on local optimization of each player's utility, it is one of the most celebrated results in the theory of learning in games that in many cases, when all players are learning using these algorithms, the empirical play recovers appropriate notions of equilibrium -a global notion of game-theoretic optimality. Strategies constructed via no-regret learning algorithms (or approximations thereof) have been key components in constructing humanlevel and even superhuman AI agents in a variety of adversarial games, including Poker [Moravˇ c´ ık et al., 2017, Brown and Sandholm, 2018, 2019], Stratego [Perolat et al., 2022], and Diplomacy [Bakhtin et al., 2023].

In regret minimization, each learning agent seeks to minimize the difference between the loss (opposite of reward) they accumulated through the actions they played, and the loss they would have

accumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. The size of the set of transformation functions considered by the learning agent determines a natural notion of rationality of the agent. Already when the agents seeks to learn strategies that cumulate low regret against constant strategy transformations only-a notion of regret called external regret-the average play of the agents converges to a Nash equilibrium in two-player constant-sum games, and to a coarse correlated equilibrium in general-sum multiplayer games. As the sets of transformations the each agent considers grows, more complex equilibria can be achieved, including correlated equilibria in normal-form games (Foster and Vohra [1997], Fudenberg and Levine [1995, 1999], Hart and Mas-Colell [2000, 2001]; see also the monograph by Fudenberg and Levine [1998]) and extensive-form correlated equilibria in extensive-form games [Farina et al., 2022]. At the extreme, a maximally hindsight-rational agent is one that minimizes regret against the set of all functions from the strategy space to itself (aka. swap regret). While it is known that maximum hindsight rationality can be attained efficiently in nonsequential (normal-form) games [Stoltz and Lugosi, 2007, Blum and Mansour, 2007], it is a major open problem to determine whether the same applies to sequential ( i.e. , extensive-form) games, and more generally what is the strongest notion of rationality that can be attained efficiently in the worst case in the latter setting.

In this paper, we provide a positive result in that direction, by showing that hindsight rationality can be achieved efficiently in general imperfect-information extensive-form games when one restricts to the set of all linear transformations of the mixed strategy space-a notion called linear-swap regret , and that coincides with swap regret in normal-form games. In order to establish the result, we introduce several intermediate results related to the geometry of sequence-form strategies in extensive-form games. In particular, a crucial result is given in Theorem 3.1, which shows that the set of linear functions M Q→P from the sequence-form strategy set Q of a player in an extensive-form game to a generic convex polytope P can be captured using only polynomially many linear constraints in the size of the game tree and the number of linear constraints that define P . Applying the result to the special case P = Q , we are then able to conclude that the the polytope of linear transformations M Q→Q from the sequence-form strategy set to itself can be captured by polynomially many linear constraints in the size of the game tree, and the norm of any element is polynomially bounded. The polynomial characterization and bound for M Q→Q is used in conjunction with an idea of Gordon et al. [2008] to construct a no-linear-swap-regret minimizer for the set of strategies Q starting from two primitives: i) a no-external-regret algorithm for the set of transformations M Q→Q , and ii) an algorithm to compute a fixed point strategy for any transformation in M Q→Q . In both cases, the polynomial representation of M Q→Q established through Theorem 3.1 plays a fundamental role. It allows, on the one hand, to satisfy requirement ii) using linear programming. On the other hand, it enables us to construct a no-external-regret algorithm that outputs transformations in M Q→Q with polynomial-time iterations, by leveraging the known properties of online projected gradient descent, exploiting the tractability of projecting onto polynomially-representable polytopes.

Finally, in the last section of the paper we turn our attention away from hindsight rationality to focus instead on the properties of the equilibria that our no-linear-swap-regret dynamics recover in extensive-form games. The average play of no-linear-swap-regret players converges to a set of equilibria that we coin linear-deviation correlated equilibria (LCEs) . LCEs form a superset of correlated equilibria and a subset of extensive-form correlated equilibria in extensive-form games. In Section 4 we show that these inclusions are in general strict, and provide additional results about the complexity of computing a welfare-maximizing LCE.

Related work As mentioned in the introduction, the existence of uncoupled no-regret dynamics leading to correlated equilibrium (CE) in multiplayer normal-form games is a celebrated result dating back to at least the work by Foster and Vohra [1997]. That work inspired researchers to seek uncoupled learning procedures in other settings as well. For example, Stoltz and Lugosi [2007] studies learning dynamics leading to CE in games with an infinite (but compact) action set, while Kakade et al. [2003] focuses on graphical games. In more recent years, a growing effort has been spent towards understanding the relationships between no-regret learning and equilibria in imperfectinformation extensive-form games, the settings on which we focus. Extensive-form games pose additional challenges when compared to normal-form games, due to their sequential nature and presence of imperfect information. While efficient no-external-regret learning dynamics for extensiveform games are known (including the popular CFR algorithm [Zinkevich et al., 2008]), as of today not much is known about no-swap-regret and the complexity of learning CE in extensive-form games.

In recent work, Anagnostides et al. [2023] construct trigger-regret dynamics that converge to EFCE at a rate of O ( log T T ) , whereas our regret dynamics converge to linear-deviation correlated equilibria at a slower rate of O ( 1 √ T ) . While the aforementioned paper proposes a general methodology that applies to CEs in normal-form games and EFCE/EFCCE in sequential games, the authors' construction fundamentally relies on being able to express the fixed points computed by the algorithm as (linear combinations of) rational functions with positive coefficients of the deviation matrices. For EFCE and EFCCE this fundamentally follows from the fact that the fixed points can be computed inductively, solving for stationary distributions of local Markov chains at each decision point. The no-linear-swap regret algorithm proposed in our paper does not offer such a local characterization of the fixed point and thus, we cannot immediately transfer the improved regret bounds to our case.

The closest notion to CE that is known to be efficiently computable in extensive-form games is extensive-form correlated equilibrium (EFCE) [von Stengel and Forges, 2008, Huang and von Stengel, 2008]. The question of whether the set of EFCE could be approached via uncoupled noregret dynamics with polynomial-time iterations in the size of the extensive-form games was recently settled in the positive [Farina et al., 2022, Celli et al., 2020]. In particular, Farina et al. [2022] show that EFCE arises from the average play of no-trigger-regret algorithms, where trigger deviations are a particular subset of linear transformations of the sequence-form strategy polytope Q of each player. Since this paper focuses on learning dynamics that guarantee sublinear regret with respect to any linear transformation of Q , it follows immediately that the dynamics presented in this paper recover EFCE as a special case.

The concept of linear-swap-regret minimization has been considered before in the context of Bayesian games. Mansour et al. [2022] study a setting where a no-regret learner competes in a two-player Bayesian game with a rational utility maximizer , that is a strictly more powerful opponent than a learner. Under this setting, it can be shown that in every round the optimizer is guaranteed to obtain at least the Bayesian Stackelberg value of the game. Then they proceed to prove that minimizing linear-swap regret is necessary if we want to cap the optimizer's performance at the Stackelberg value, while minimizing polytope-swap regret (a generalization of swap regret for Bayesian games, and strictly stronger than linear-swap) is sufficient to cap the optimizer's performance. Hence, these results highlight the importance of developing learning algorithms under stronger notions of rationality , as is our aim in this paper. Furthermore, these results provide evidence that constructing a no-linear-swap regret learner, as is our goal here, can present benefits when compared to other less rational learners. In a concurrent paper, Fujii [2023] defines the notion of untruthful swap regret for Bayesian games and proves that, for Bayesian games, it is equivalent to the linear-swap regret which is of interest in our paper.

Bayesian games can be considered as a special case of extensive-form games, where a chance node initially selects one of the possible types Θ for each player. Thus, our algorithm minimizing linearswap regret in extensive-form games also minimizes linear-swap regret in Bayesian games. However, we remark that our regret bound depends polynomially on the number of player types | Θ | as they are part of the game tree representation, while Fujii [2023] has devised an algorithm for Bayesian games, whose regret only depends on log | Θ | .

Finally, we also mention that nonlinear deviations have been explored in extensive-form games, though we are not aware of notable large sets for which polynomial-time no-regret dynamics can be devised. Specifically, we point to the work by Morrill et al. [2021], which defines the notion of 'behavioral deviations'. These deviations are nonlinear with respect to the sequence-form representation of strategies in extensive-form games. The authors categorize several known or novel types of restricted behavioral deviations into a Deviation Landscape that highlights the relations between them. Even though both the linear deviations, we consider in this paper, and the behavioral deviations seem to constitute rich measures of rationality, none of them contains the other and thus, linear deviations do not fit into the Deviation Landscape of Morrill et al. [2021] (see also Remark E.2).

2 Preliminaries

We recall the standard model of extensive-form games, as well as the framework of learning in games.

2.1 Extensive-Form Games

While normal-form games (NFGs) correspond to nonsequential interactions, such as Rock-PaperScissors, where players simultaneously pick one action and then receive a payoff based on what others picked, extensive-form games (EFGs) model games that are played on a game tree. They capture both sequential and simultaneous moves, as well as private information and are therefore a very general and expressive model of games, capturing chess, go, poker, sequential auctions, and many other settings as well. We now recall basic properties and notation for EFGs.

Game tree In an n -player extensive-form game, each node in the game tree is associated with exactly one player from the set { 1 , . . . , n } ∪ { c } , where the special player c -called the chance player-is used to model random stochastic outcomes, such as rolling a die or drawing cards from a deck. Edges leaving from a node represent actions that a player can take at that node. To model private information, the game tree is supplemented with an information partition, defined as a partition of nodes into sets called information sets. Each node belongs to exactly one information set, and each information set is a nonempty set of tree nodes for the same Player i . An information set for Player i denotes a collection of nodes that Player i cannot distinguish among, given what she has observed so far. (We remark that all nodes in a same information set must have the same set of available actions, or the player would distinguish the nodes). The set of all information sets of Player i is denoted J i . In this paper, we will only consider perfect-recall games, that is, games in which the information sets are arranged in accordance with the fact that no player forgets what the player knew earlier,.

Sequence-form strategies Since nodes belonging to the same information set for a player are indistinguishable to that player, the player must play the same strategy at each of the nodes. Hence, a strategy for a player is exactly a mapping from an information set to a distribution over actions. In other words, it is the information sets, and not the game tree nodes, that capture the decision points of the player. We can then represent a strategy for a generic player i as a vector indexed by each valid information set-action pair ( j, a ) . Any such valid pair is called a sequence of the player; the set of all sequences is denoted as Σ i := { ( j, a ) : j ∈ J i , a ∈ A j } ∪ { ∅ } , where the special element ∅ is called empty sequence . Given an information set j ∈ J i , we denote by p j the parent sequence of j , defined as the last pair ( j, a ) ∈ Σ i encountered on the path from the root to any node v ∈ j ; if no such pair exists we let p j = ∅ . Finally, we denote by C σ the children of sequence σ ∈ Σ i , defined as the information sets j ∈ J i for which p j = σ . Sequences σ for which C σ is an empty set are called terminal ; the set of all terminal sequences is denoted Σ ⊥ i .

Example 2.1. Consider the tree-form decision process faced by Player 1 in the small game of Figure 1 (Left). The decision process has four decision nodes J 1 = { A , B , C , D } and nine sequences including the empty sequence ∅ . For decision node D , the parent sequence is p D = A2 ; for B and C it is p B = A1 ; for A it is the empty sequence p A = ∅ .

P

Q

A

B

C

D

1 1

1 1 1

1 1

1 1

1 1

1

1 1

1 1 1

2 2

2 2 2

2 2

2 2

2 2

2

2 2

2 2 2

3 3

3 3 3

3 3

3 3

3 3

3

3 3

3 3 3

4 4

4 4 4

4 4

4 4

4 4

4

4 4

4 4 4

5 5

5 5 5

5 5

5 5

5 5

5

5 5

5 5 5

6 6

6 6 6

6 6

6 6

6 6

6

6 6

6 6 6

7 7

7 7 7

7 7

7 7

7 7

7

7 7

7 7

7 8

8 8

8 8

8 8

8 8

8 8

8

8 8

8 8

8 9

9 9

9 9

9 9

9 9

9 9

9

9 9

9 9 9

7 7

7 7 7

7 7

7 7

7 7

7

7 7

7 7

7 8

8 8

8 8

8 8

8 8

8 8

8

8 8

8 8

8 9

9 9

9 9

9 9

9 9

9 9

9

9 9

9 9 9

Sequence-form constraints:

A reduced-normal-form plan for Player i represents a deterministic strategy for the player as a vector x ∈ { 0 , 1 } Σ i where the entry corresponding to the generic sequence x [ ja ] is equal to 1 if the player plays action a at (the nodes of) information set j ∈ J i . Information sets that cannot be reached based on the strategy do not have any action select. A crucial property of the reduced-normal-form plan representation of deterministic strategies is the fact that the utility of any player is a multilinear function in the profile of reduced-normal-form plans played by the players. The set of all reduced-

normal-form plans of Player i is denoted with the symbol Π i . Typically, the cardinality of Π i is exponential in the size of the game tree.

The convex hull of the set of reduced-normal-form plans of Player i is called the sequence-form polytope of the player, and denoted with the symbol Q i := conv (Π i ) . It represents the set of all randomized strategies in the game. An important result by Romanovskii [1962], Koller et al. [1996], von Stengel [1996] shows that Q i can be captured by polynomially many constraints in the size of the game tree, as we recall next.

Definition 2.2. The polytope of sequence-form strategies of Player i is equal to the convex polytope

As an example, the constraints that define the sequence-form polytope for Player 1 in the game of Figure 1 (Left) are shown in Figure 1 (Right). The polytope of sequence-form strategies possesses a strong combinatorial structure that enables speeding up several common optimization procedures and will be crucial in developing efficient algorithms to converge to equilibrium.

2.2 Hindsight Rationality and Learning in Games

Games are one of many situations in which a decision-maker has to act in an online manner. For these situations, the most widely used protocol is that of Online Learning (e.g., see Orabona [2022]). Specifically, each learner has a set of actions or behavior they can employ X ⊆ R d (in extensive-form games, this would typically be the set of reduced-normal-form strategies). At each timestep t the learner first selects, possibly at random, an element x ∈ X , and then receives a loss (opposite of utility) function glyph[lscript] ( t ) : X ↦→ R . Since as we observed the above utilities in EFGs are linear in each player's reduced-normal-form plans, for the rest of the paper we focus on the case in which the loss function glyph[lscript] ( t ) is linear, that is, of the form glyph[lscript] ( t ) : x ↦→〈 glyph[lscript] ( t ) , x ( t ) 〉 .

A widely adopted objective for the learner is that of ensuring vanishing average regret with high probability. Regret is defined as the difference between the loss the learner cumulated through their choice of behavior, and the loss they would have cumulated in hindsight had they consistently modified their behavior according to some strategy transformation function. In particular, let Φ be a desired set of strategy transformations φ : X → X that the learner might want to learn not to regret. Then, the learner's Φ -regret is defined as the quantity

A noΦ -regret algorithm (also known as a Φ -regret minimizer) is one that, at all times T , guarantees with high probability that Φ -Reg ( T ) = o ( T ) no matter what is the sequence of losses revealed by the environment. The size of the set Φ of strategy transformations defines a natural measure of rationality (sometimes called hindsight rationality ) for players, and several choices have been discussed in the literature. Clearly, as Φ gets larger, the learner becomes more rational. On the flip side, guaranteeing sublinear regret with respect to all transformations in the chosen set Φ might be intractable in general. On one end of the spectrum, perhaps the smallest meaningful choice of Φ is the set of all constant transformations Φ const = { φ ˆ x : x ↦→ ˆ x } ˆ x ∈X . In this case, Φ const -regret is also called external regret and has been extensively studied in the field of online convex optimization. On the other end of the spectrum, swap regret corresponds to the setting in which Φ is the set of all transformations X → X . Intermediate, and of central importance in this paper, is the notion of linear-swap regret , which corresponds to the case in which

is the set of all linear transformations from X to itself. 1

An important observation is that when all considered deviation functions in Φ are linear, an algorithm guaranteeing sublinear Φ -regret for the set X can be constructed immediately from a deterministic

noΦ -regret algorithm for X ′ = ∆( X ) by sampling X glyph[owner] x from any x ′ ∈ X ′ so as to guarantee that E [ x ] = x ′ . Since this is exactly the setting we study in this paper, this folklore observation (see also Farina et al. [2022]) enables us to focus on the following problem: does a deterministic noΦ -regret algorithm for the set of sequence-form strategies X = Q i of any player in an extensive-form game, with guaranteed sublinear Φ -regret in the worst case, exist? In this paper we answer the question for the positive.

From regret to equilibrium The setting of Learning in Games refers to the situation in which all players employ a learning algorithm, receiving as loss the negative of the gradient of their own utility evaluated in the strategies output by all the other players. A fascinating aspect of noΦ -regret learning dynamics is that if each player of a game employs a noΦ -regret algorithm, then the empirical frequency of play converges almost surely to the set of Φ -equilibria, which are notions of correlated equilibria, in which the rationality of players is bounded by the size of the set Φ . Formally, for a set Φ of strategy deviations, a Φ -equilibrium is defined as follows.

Definition 2.3. For a n -player extensive-form game G and a set Φ i of deviations for each player, a { Φ i } -equilibrium is a joint distribution µ ∈ ∆(Π 1 ×··· × Π n ) such that for each player i , and every deviation φ ∈ Φ i it holds that

That is, no player i has an incentive to unilaterally deviate from the recommended joint strategy x using any transformation φ ∈ Φ i .

This general framework captures several important notions of equilibrium across a variety of game theoretic models. For example, in both NFGs and EFGs, no-external regret dynamics converge to the set of Coarse Correlated Equilibria. In NFGs, no-swap regret dynamics converge to the set of Correlated Equilibria [Blum and Mansour, 2007]. In EFGs, Farina et al. [2022] recently proved that a specific subset Φ of linear transformations called trigger deviations lead to the set of EFCE.

Reducing Φ -regret to external regret An elegant construction by Gordon et al. [2008] enables constructing noΦ -regret algorithms for a generic set X starting from a no-external-regret algorithm for Φ . We briefly recall the result.

Theorem 2.4 (Gordon et al. [2008]) . Let R be an external regret minimizer having the set of transformations Φ as its action space, and achieving sublinear external regret Reg ( T ) . Additionally, assume that for all φ ∈ Φ there exists a fixed point φ ( x ) = x ∈ X . Then, a Φ -regret minimizer R Φ can be constructed as follows:

To output a strategy x ( t ) at iteration t of R Φ , obtain an output φ ( t ) ∈ Φ of the external regret minimizer R , and return one of its fixed points x ( t ) = φ ( t ) ( x ( t ) ) .

For every linear loss function glyph[lscript] ( t ) received by R Φ , construct the linear function L ( t ) : φ ↦→ glyph[lscript] ( t ) ( φ ( x ( t ) )) and pass it as loss to R .

Let Φ -Reg ( T ) be the Φ -regret of R Φ . Under the previous construction, it holds that

Thus, if R is an external regret minimizer then R Φ is a Φ -regret minimizer.

3 A No-Linear-Swap Regret Algorithm with Polynomial-Time Iterations

In this section, we describe our no-linear-swap-regret algorithm for the set of sequence-form strategies Q of a generic player in any perfect-recall imperfect-information extensive-form game. The algorithm follows the general template for constructing Φ -regret minimizers given by Gordon et al. [2008] and recalled in Theorem 2.4. For this we need two components:

an efficient external regret minimizer for the set M Q→Q of all matrices inducing linear transformations from Q to Q ,

ii) an efficiently computable fixed point oracle for matrices A ∈ M Q→Q , returning x = A x ∈ Q .

The existence of a fixed point, required in ii), is easy to establish by Brouwer's fixed point theorem, since the polytope of sequence-form strategies is compact and convex, and the continuous function x ↦→ A x maps Q to itself by definition. Furthermore, as it will become apparent later in the section, all elements A ∈ M Q→Q have entries in [0 , 1] Σ × Σ . Hence, requirement ii) can be satisfied directly by solving the linear feasibility program { find x : A x = x , x ∈ Q} using any of the known polynomial-time algorithms for linear programming. Establishing requirement i) is where the heart of the matter is, and it is the focus of much of the paper. Here, we give intuition for the main insights that contribute to the algorithm. All proofs are deferred to the appendix.

3.1 The Structure of Linear Transformations of Sequence-Form Strategy Polytopes

The crucial step in our construction is to establish a series of results shedding light on the fundamental geometry of the set M Q→Q of all linear transformations from a sequence-form polytope Q to itself. In fact, our results extend beyond functions from Q to Q to more general functions from Q to a generic compact polytope P := { x ∈ R d : P x = p , x ≥ 0 } for arbitrary P and p . We establish the following characterization theorem, which shows that when the functions are expressed in matrix form, the set M Q→P can be captured by a polynomial number of constraints. The proof is deferred to Appendix B.

Theorem 3.1. Let Q be a sequence-form strategy space and let P be any bounded polytope of the form P := { x ∈ R d : P x = p , x ≥ 0 } ⊆ [0 , γ ] d , where P ∈ R k × d . Then, for any linear function f : Q → P , there exists a matrix A in the polytope

such that f ( x ) = A x for all x ∈ Q . Conversely, any A ∈ M Q→P defines a linear function x ↦→ A x from Q to P , that is, such that A x ∈ P for all x ∈ Q .

The proof operates by induction in several steps. At its core, it exploits the combinatorial structure of sequence-form strategy polytopes, which can be decomposed into sub-problems using a series of Cartesian products and convex hulls. A high-level description of the induction is as follows:

The Base Case corresponds to the case of being at a leaf decision point. In this case, the set of deviations corresponds to all linear transformations from a probability n -simplex into a given polytope P . This set is equivalent to all d × n matrices whose columns are points in P , which can easily be verified formally. This corresponds to constraint (3) in the characterization of M Q→P .

For the Inductive Step , we are at an intermediate decision point j , that is, one for which at least one action leads to further decision points.

-In Lemma B.8 we show that any terminal action a at j leads to a column in the transformation matrix that is necessarily a valid point in the polytope P . This is similar to the base case, and again leads to constraint (3) in the characterization.

-In Lemma B.7, we look at the other case of a non-terminal action a at j . We prove that there always exists an equivalent transformation matrix whose column corresponding to sequence ja is identically 0 (constraint (4)). This allows for the 'crux' of the transformation to happen in the subtrees below ja or equivalently, the subtrees rooted at the children decision points C ja of ja .

-Akey difficulty to conclude the proof is in using the assumption of the inductive step to characterize all such valid transformations. The set of strategies in the subtrees rooted at the children decision points C ja is in general the Cartesian product of the strategies in these subtrees. This explains the need for the fairly technical Proposition B.4, whose goal is to precisely characterize valid transformations of Cartesian products. This leads to constraints (5) and (6) in our final characterization.

We also remark that while the theorem calls for the polytope P to be in the form P = { x ∈ R d : P x = p , x ≥ 0 } , with little work the result can also be extended to handle other representations such as { x ∈ R d : P x ≤ p } . We opted for the form specified in the theorem since it most directly leads to the proof, and since the constraints that define the sequence-form strategy polytope (Definition 2.2) are already in the form of the statement.

In particular, by setting P = Q in Theorem 3.1 (in this case, the dimensions of P will be k = |J | +1 , and d = | Σ | ), we conclude that the set of linear functions from Q to itself is a compact and convex polytope M Q→Q ⊆ [0 , 1] Σ × Σ , defined by O ( | Σ | 2 ) linear constraints. As discussed, this polynomial characterization of M Q→Q is the fundamental insight that enables polynomial-time minimization of linear-swap regret in general extensive-form games.

3.2 Our No-Linear-Swap Regret Algorithm

From here, constructing a no-external-regret algorithm for M Q→Q is relatively straightforward, using standard tools from the rich literature of online learning. For example, in Algorithm 1, we propose a solution employing online projected gradient descent [Gordon, 1999, Zinkevich, 2003].

Combining that no-external-regret algorithm for Φ with the construction by Gordon et al. [2008], we can then establish the following linear-swap regret and iteration complexity bounds for Algorithm 1.

Theorem 3.2 (Informal) . Let Σ denote the set of sequences of the learning player in the extensiveform game, and let η ( t ) = 1 / √ t for all t . Then, for any sequence of loss vectors glyph[lscript] ( t ) ∈ [0 , 1] Σ , Algorithm 1 guarantees linear-swap regret O ( | Σ | 2 √ T ) after any number T of iterations, and runs in O (poly( | Σ | ) log 2 t ) time for each iteration t .

The formal version of the theorem is given in Theorem D.1. It is worth noting that the polynomialsized description of M Q→Q is crucial in establishing the polynomial running time of the algorithm, both in the projection step (5) and in the fixed point computation step (6). We also remark that the choice of online projected gradient descent combined with the ellipsoid method for projections were arbitrary and the useful properties of M Q→Q are retained when using it with any efficient regret minimizer.

4 Linear-Deviation Correlated Equilibrium

As we discussed in the preliminaries, when all players in a game employ noΦ -regret learning algorithms, then the empirical frequency of play converges to the set of Φ -equilibria almost surely. Similarly, when Φ = M Q→Q the players act based on 'no-linear-swap regret' dynamics and converge to a notion of Φ -equilibrium we call linear-deviation correlated equilibrium (LCE). In this section we present some notable properties of the LCE. In particular, we discuss its relation to other already established equilibria, as well as the computational tractability of optimal equilibrium selection.

4.1 Relation to CE and EFCE

The Φ -regret minimization framework, offers a natural way to build a hierarchy of the corresponding Φ -equilibria based on the relationship of the Φ sets of deviations. In particular, if for the sets Φ 1 , Φ 2 it holds that Φ 1 ⊆ Φ 2 , then the set of Φ 2 -equilibria is a subset of the set of Φ 1 -equilibria. Since

glyph[negationslash]

the Correlated Equilibrium is defined using the set of all swap deviations, we conclude that any Φ -equilibrium, including the LCE, is a superset of CE. What is the relationship then of LCE with the extensive-form correlated equilibrium (EFCE)? Farina et al. [2022] showed that the set Φ EFCE inducing EFCE is the set of all 'trigger deviations', which can be expressed as linear transformations of extensive-form strategies. Consequently, the set Φ EFCE is a subset of all linear transformations and thus, it holds that CE ⊆ LCE ⊆ EFCE. In examples E.1 and E.3 of the appendix we show that there exist specific games in which either CE = LCE, or LCE = EFCE. Hence, we conclude that the previous inclusions are strict and it holds CE glyph[subsetnoteql] LCE glyph[subsetnoteql] EFCE.

glyph[negationslash]

For Example E.1 we use a signaling game from von Stengel and Forges [2008] with a known EFCE and we identify a linear transformation that is not captured by the trigger deviations of EFCE. Specifically, it is possible to perform linear transformations on sequences of a subtree based on the strategies on other subtrees of the TFSDP. For Example E.3 we have found a specific game through computational search that has a LCE which is not a normal-form correlated equilibrium. To do that we identify a particular normal-form swap that is non-linear.

Empirical evaluation To further illustrate the separation between no-linear-swap-regret dynamics and no-trigger-regret dynamics, used for EFCE, we provide experimental evidence that minimizing linear-swap-regret also minimizes trigger-regret (Figure 2, left), while minimizing trigger-regret does not minimize linear-swap regret. Specifically, in Figure 2 we compare our no-linear-swap-regret learning dynamics (given in Algorithm 1) to the no-trigger-regret algorithm introduced by Farina et al. [2022]. More details about the implementation of the algorithms is available in Appendix F. In the left plot, we measure on the y-axis the average trigger regret incurred when all players use one or the other dynamics. Since trigger deviations are special cases of linear deviations, as expected, we observe that both dynamics are able to minimize trigger regret. Conversely, in the right plot of Figure 2, the y-axis measures linear-swap-regret. We observe that while our dynamics validate the sublinear regret performance proven in Theorem 3.2, the no-trigger-regret dynamics of Farina et al. [2022] exhibit an erratic behavior that is hardly compatible with a vanishing average regret. This suggests that no-linear-swap-regret is indeed a strictly stronger notion of hindsight rationality.

glyph[integerdivide]eðglyph[integerdivide]e≈

glyph[integerdivide]eðglyph[integerdivide]e≈

glyph[subsetnoteql]glyph[notforces]glyph[notforcesextra]

glyph[integerdivide]glyph[Gmir]glyph[multicloseleft]glyph[daleth]∼glyph[supersetnoteql]

dglyph[archrightdown]glyph[multicloseleft]glyph[Gmir]glyph[greaterdot]glyph[beth]c∼

dglyph[archrightdown]glyph[multicloseleft]glyph[Gmir]glyph[greaterdot]glyph[beth]c∼

glyph[notsatisfies]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[nottriangeqlleft]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[notforces]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[notforces]glyph[nottriangeqlleft]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[notsatisfies]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

I≈eglyph[integerdivide]glyph[Gmir]≈glyph[beth]glyph[multicloseright]glyph[multicloseleft]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]glyph[notforces]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]glyph[notsatisfies]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]glyph[notforcesextra]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]glyph[nottriangeqlright]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]glyph[nottriangeqlleft]

≈glyph[integerdivide]glyph[beth]ððeglyph[integerdivide]

Tglyph[equalorsimilar]glyph[integerdivide]eeglyph[notbar]glyph[barshort]glyph[lessdot]glyph[Gmir]glyph[archrightdown]eglyph[integerdivide] Kglyph[approxorequal]glyph[equalorsimilar]glyph[multicloseleft] glyph[barshort]glyph[multicloseright]glyph[daleth]eglyph[integerdivide]

glyph[subsetnoteql]glyph[notforces]glyph[notforcesextra]

glyph[integerdivide]glyph[Gmir]glyph[multicloseleft]glyph[daleth]∼glyph[supersetnoteql] Tglyph[equalorsimilar]glyph[integerdivide]eeglyph[notbar]glyph[barshort]glyph[lessdot]glyph[Gmir]glyph[archrightdown]eglyph[integerdivide] Kglyph[approxorequal]glyph[equalorsimilar]glyph[multicloseleft] glyph[barshort]glyph[multicloseright]glyph[daleth]eglyph[integerdivide]

Nglyph[multicloseright]glyph[notbar]glyph[lessdot]glyph[beth]glyph[multicloseleft]eglyph[Gmir]glyph[integerdivide] glyph[notbar]∼glyph[precedesorequal]glyph[Gmir]glyph[barshort]glyph[notbar]glyph[integerdivide]eðglyph[integerdivide]e≈

dglyph[archrightdown]glyph[multicloseleft]glyph[Gmir]glyph[greaterdot]glyph[beth]c∼

Nglyph[multicloseright]glyph[notbar]≈glyph[integerdivide]glyph[beth]ððeglyph[integerdivide]glyph[notbar]glyph[integerdivide]eðglyph[integerdivide]e≈

dglyph[archrightdown]glyph[multicloseleft]glyph[Gmir]glyph[greaterdot]glyph[beth]c∼

glyph[nottriangeqlleft]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[notforces]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

glyph[notforces]glyph[nottriangeqlleft]glyph[notturnstile]glyph[notturnstile]glyph[notturnstile]

I≈eglyph[integerdivide]glyph[Gmir]≈glyph[beth]glyph[multicloseright]glyph[multicloseleft]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notturnstile]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notforces]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notsatisfies]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[notforcesextra]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[nottriangeqlright]

glyph[notturnstile]glyph[notshortbar]glyph[notturnstile]glyph[nottriangeqlleft]

glyph[lessdot]glyph[beth]glyph[multicloseleft]eglyph[Gmir]glyph[integerdivide] glyph[notbar]∼glyph[precedesorequal]glyph[Gmir]glyph[barshort]

Nglyph[multicloseright]glyph[notbar]glyph[lessdot]glyph[beth]glyph[multicloseleft]eglyph[Gmir]glyph[integerdivide] glyph[notbar]∼glyph[precedesorequal]glyph[Gmir]glyph[barshort]glyph[notbar]glyph[integerdivide]eðglyph[integerdivide]e≈

Nglyph[multicloseright]glyph[notbar]≈glyph[integerdivide]glyph[beth]ððeglyph[integerdivide]glyph[notbar]glyph[integerdivide]eðglyph[integerdivide]e≈

4.2 Hardness of Maximizing Social Welfare

In many cases we are interested in knowing whether it is possible to select an Equilibrium with maximum Social Welfare. Let MAXPAY-LCE be the problem of finding an LCE in EFGs that maximizes the sum (or any linear combination) of all player's utilities. Below, we prove that we cannot efficiently solve MAXPAY-LCE, unless P=NP, even for 2 players if chance moves are allowed, and even for 3 players otherwise. We follow the structure of the same hardness proof for the problem MAXPAY-CE of finding an optimal CE in EFGs. Specifically, von Stengel and Forges [2008] use a reduction from SAT to prove that deciding whether MAXPAY-CE can attain the maximum value is NP-hard even for 2 players. To do that, they devise a way to map any SAT instance into a polynomially large game tree in which the root is the chance player, the second level corresponds to one player, and

the third level corresponds to the other player. The utilities for both players are exactly the same, thus the players will have to coordinate to maximize their payoff irrespective of the linear combination of utilities we aim to maximize.

Theorem 4.1. For two-player, perfect-recall extensive-form games with chance moves, the problem MAXPAY-LCE is not solvable in polynomial time, unless P=NP.

Remark 4.2. The problem retains its hardness if we remove the chance node and add a third player instead. As showed in von Stengel and Forges [2008], in that case we can always build a polynomially-sized game tree that forces the third player to act as a chance node.

5 Conclusions and Future Work

In this paper we have shown the existence of uncoupled no-linear-swap regret dynamics with polynomial-time iteration complexity in the game tree size in any extensive-form game. This significantly extends prior results related to extensive-form correlated equilibria, and begets learning agents that learn not to regret a significantly larger set of strategy transformations than what was known to be possible before. A crucial technical contribution we made to establish our result, and which might be of independent interest, is providing a polynomial characterization of the set of all linear transformations from a sequence-form strategy polytope to itself. Specifically, we showed that such a set of transformations can be expressed as a convex polytope with a polynomial number of linear constraints, by leveraging the rich combinatorial structure of the sequence-form strategies. Moreover, these no-linear-swap regret dynamics converge to linear-deviation correlated equilibria in extensive-form games, which are a novel type of equilibria that lies strictly between normal-form and extensive-form correlated equilibria.

These new results leave open a few interesting future research directions. Even though we know that there exist polynomial-time uncoupled dynamics converging to linear-deviation correlated equilibrium, we conjecture that it is also possible to obtain an efficient centralized algorithm similar to the Ellipsoid Against Hope for computing EFCE in extensive-form games by Huang and von Stengel [2008]. Additionally, it is an intriguing question to understand whether a no-linear-swap regret algorithm exists that achieves O (log T ) regret per-player, as is the case for no-trigger regret [Anagnostides et al., 2023]. Furthermore, it would be interesting to further explore problems of equilibrium selection related to LCE, possibly by devising suitable Fixed-Parameter Algorithms in the spirit of Zhang et al. [2022]. Finally, the problem of understanding what is the most hindsight rational type of deviations based on which we can construct efficient regret minimizers in extensive-form games remains a major open question.

References

Matej Moravˇ c´ ık, Martin Schmid, Neil Burch, Viliam Lis´ y, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , 356(6337):pp. 508-513, 2017.

Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Science , 365(6456): 885-890, 2019. doi: 10.1126/science.aay2400.

Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Marc Lanctot, Jean-Baptiste Lespiau, Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalie Beauguerlange, Remi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science , 378(6623):990-996, 2022.

Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, and Noam Brown. Mastering the game of no-press Diplomacy via human-regularized reinforcement learning and planning. In International Conference on Learning Representations (ICLR) , 2023.

Dean P. Foster and Rakesh V. Vohra. Calibrated learning and correlated equilibrium. Games and Economic Behavior , 21(1):40-55, 1997.

Drew Fudenberg and David K Levine. Consistency and cautious fictitious play. Journal of Economic Dynamics and Control , 19(5-7):1065-1089, 1995.

Drew Fudenberg and David K Levine. Conditional universal consistency. Games and Economic Behavior , 29(1-2):104-130, 1999.

Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68(5):1127-1150, 2000.

Sergiu Hart and Andreu Mas-Colell. A general class of adaptive strategies. Journal of Economic Theory , 98(1):26-54, 2001.

Drew Fudenberg and David K Levine. The theory of learning in games , volume 2. MIT press, 1998.

Gabriele Farina, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Simple uncoupled no-regret learning dynamics for extensive-form correlated equilibrium. Journal of the ACM , 69(6), 2022.

Gilles Stoltz and G´ abor Lugosi. Learning correlated equilibria in games with compact sets of strategies. Games and Economic Behavior , 59(1):187-208, 2007.

Avrim Blum and Yishay Mansour. From external to internal regret. J. Mach. Learn. Res. , 8, 2007.

Geoffrey J Gordon, Amy Greenwald, and Casey Marks. No-regret learning in convex games. In International Conference on Machine learning , pages 360-367, 2008.

Sham Kakade, Michael Kearns, John Langford, and Luis Ortiz. Correlated equilibria in graphical games. In Proceedings of the 4th ACM Conference on Electronic Commerce , pages 42-47, 2003.

Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in Neural Information Processing Systems , pages 1729-1736, 2008.

Ioannis Anagnostides, Gabriele Farina, and Tuomas Sandholm. Near-optimal Φ -regret learning in extensive-form games. In International Conference on Machine Learning (ICML) , 2023.

von Stengel and F. Forges. Extensive-form correlated equilibrium: Definition and computational complexity. Mathematics of Operations Research , 33(4):1002-1022, 2008.

Wan Huang and Bernhard von Stengel. Computing an extensive-form correlated equilibrium in polynomial time. In International Workshop on Internet and Network Economics , pages 506-513. Springer, 2008.

Andrea Celli, Alberto Marchesi, Gabriele Farina, and Nicola Gatti. No-regret learning dynamics for extensive-form correlated equilibrium. In Advances in Neural Information Processing Systems , volume 33, 2020.

Yishay Mansour, Mehryar Mohri, Jon Schneider, and Balasubramanian Sivan. Strategizing against Learners in Bayesian games. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 5221-5252. PMLR, 02-05 Jul 2022.

Kaito Fujii. Bayes correlated equilibria and no-regret dynamics, 2023.

Dustin Morrill, Ryan D'Orazio, Marc Lanctot, James R. Wright, Michael Bowling, and Amy R. Greenwald. Efficient deviation types and learning for hindsight rationality in extensive-form games. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 7818-7828. PMLR, 2021.

Romanovskii. Reduction of a Game with Complete Memory to a Matrix Game. Soviet Mathematics , 3, 1962.

Daphne Koller, Nimrod Megiddo, and Bernhard von Stengel. Efficient computation of equilibria for extensive two-person games. Games and Economic Behavior , 14(2):247-259, 1996.

Bernhard von Stengel. Efficient computation of behavior strategies. Games and Economic Behavior , 14(2):220-246, 1996.

Francesco Orabona. A Modern Introduction to Online Learning, 2022.

Geoffrey Gordon. Regret bounds for prediction problems. In Proceedings of 12th Annual Conference on Computational Learning Theory (COLT '99) , pages 29 - 40, July 1999.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning , ICML'03, page 928-935. AAAI Press, 2003.

Brian Hu Zhang, Gabriele Farina, Andrea Celli, and Tuomas Sandholm. Optimal correlated equilibria in general-sum extensive-form games: Fixed-parameter algorithms, hardness, and two-sided column-generation. In David M. Pennock, Ilya Segal, and Sven Seuken, editors, EC '22: The 23rd ACM Conference on Economics and Computation, Boulder, CO, USA, July 11 - 15, 2022 , pages 1119-1120. ACM, 2022.

Nisheeth K. Vishnoi. Algorithms for Convex Optimization . Cambridge University Press, 2021.

Bernd G¨ artner and Jir´ ı Matousek. Approximation Algorithms and Semidefinite Programming . Springer Publishing Company, Incorporated, 2014.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www. gurobi.com .

W. Kuhn. A Simplified Two-Person Poker. In H. W. Kuhn and A. W. Tucker, editors, Contributions to the Theory of Games , volume 1 of Annals of Mathematics Studies, 24 , pages 97-103. Princeton University Press, Princeton, New Jersey, 1950.

A Additional Extensive-Form Game Notation

In the proofs, we will make use of the following symbols and notation.

Furthermore, when the subscript referring to players can be inferred or is irrelevant (that is, the quantities refer to a generic player), then we might skip it.

As hinted by some of the rows in the above table, we will sometimes find it important to consider partial strategies that only specify behavior at a decision node j and all of its descendants j ′ glyph[follows] j . We make this formal through the following definition.

Definition A.1. The set of sequence-form strategies for the subtree rooted at j , denoted Q glyph[followsequal] j , is the set of all vectors x ∈ R Σ glyph[followsequal] j ≥ 0 such that probability-mass-conservation constraints hold at decision node j and all of its descendants j ′ glyph[follows] j , specifically

Finally, we define the symbol Σ ⊥ i to be the set of all terminal sequences for Player i . Thus, the set Σ i \ Σ ⊥ i would give us all the non-terminal sequences of that player.

Access to coordinates By definition, sequence-form strategies are vectors indexed by sequences. To access the coordinate corresponding to sequence σ , we will use the notation x [ σ ] . Occasionally, we will need to extract a subvector corresponding to all sequences that are successor of an information set j , that is, all sequences σ glyph[followsequal] j . For that, we use the notation x [ glyph[followsequal] j ] .

Remark on the structure of sequence-form strategies We further remark the following known fact about the structure of sequence-form strategies. Intuitively, it crystallizes the idea that sequenceform strategies encode product of probabilities of actions on the path from the root to any decision point. The proof follows directly from the definitions.

Lemma A.2. Let j ∈ J i be an information set for a generic player. Then, given any sequence-form strategy x ∈ Q glyph[followsequal] j , action a ∈ A j , and child information set j ′ ∈ C ja , there exists a sequence-form strategy x glyph[followsequal] j ′ ∈ Q glyph[followsequal] j ′ such that

B Proof of the Characterization Theorem (Theorem 3.1)

In this section we prove the central result of this paper, the characterization given in Theorem 3.1 of linear functions from the sequence-form strategy polytope Q to the generic polytope P := { x ∈ R d : P x = p , x ≥ 0 } , where P ∈ R k × d and p ∈ R k .

We will prove the characterization theorem by induction on the structure of the extensive-form strategy polytope. To do so, it will be useful to introduce a few additional objects and notations. We do so in the next subsection.

B.1 Additional Objects and Notation Used in the Proof

First, we introduce a parametric version of the polytope P , where the right-hand side vector is made variable.

Definition B.1. Given any b ∈ R k , we will denote with P ( b ) the polytope

Furthermore, we introduce the equivalence relation ∼ = D to indicate that two matrices induce the same linear function when restricted to domain D .

Definition B.2. Given two matrices A , B of the same dimension, we write A ∼ = D B if A x = B x for all x ∈ D . Similarly, given two sets U , V of matrices we write U ∼ = D V to mean that for any A ∈ U there exists B ∈ V with A ∼ = D B , and vice versa.

Additionally, we introduce a symbol to denote the set of all matrices that induce linear functions from a set U to a set V .

Definition B.3. Given any sets U , V we denote with L U→V the set of all matrices that induce linear transformations from U to V , that is,

Finally, we remark that for a matrix A whose columns are indexed using sequences σ ∈ Σ , we represent its columns as A ( σ ) . Furthermore, for sequence-form strategies x ∈ Q , we use x [ σ ] to represent their entries, and x [Σ glyph[followsequal] j ] to represent a vector consisting only of the entries corresponding to sequences σ ∈ Σ glyph[followsequal] j .

B.2 A Key Tool: Linear Transformations of Cartesian Products

We are now ready to introduce the following Proposition, which will play an important role in the proof of Theorem 3.1.

Proposition B.4. Let U 1 , . . . , U m be sets, with 0 / ∈ aff U i 2 for all i = 1 , . . . , m . Furthermore, for any i = 1 , . . . , m and any b i ∈ R k , let M U i →P ( b i ) be such that M U i →P ( b i ) ∼ = U i L U i →P ( b i ) . Then, for all b ∈ R k ,

Proof. We prove the result by showing the two directions of the inclusion separately.

( ⊇ ) First, we show that for any b ∈ R k , any matrix A = [ A 1 | · · · | A m ] that belongs to the set on the right-hand side of (14) induces a linear transformation from U 1 ×··· × U m to P ( b ) and thus belongs to L ( U 1 ×···×U m ) →P ( b ) . To that end, we note that for any x = ( x 1 , . . . , x m ) ∈ U 1 ×··· × U m ,

where in both cases we used the fact that A i maps any point in U i to a point in P ( b i ) = { y : P y = b i , y ≥ 0 } by (11). Hence A x ∈ P ( b ) for all x ∈ U 1 ×··· × U m , as we wanted to show.

( ⊆ ) We now look at the converse, showing that for any b ∈ R k and matrix B = [ B 1 | · · · | B m ] ∈ L ( U 1 ×···×U m ) →P ( b ) , there exists a matrix A = [ A 1 | · · · | A m ] that satisfies constraints (11)-(13) and such that B x = A x for all x ∈ U 1 ×··· × U m . As a first step, in the next lemma we show that B is always equivalent to another matrix B ′ = [ B ′ 1 | · · · | B ′ m ] ∼ = ( U 1 ×···×U m ) B that satisfies B ′ i x i ≥ 0 for all i = 1 , . . . , m and x i ∈ U i .

Lemma B.5. There exist B ′ 1 , . . . , B ′ n , such that B ∼ = ( U 1 ×···×U m ) B ′ := [ B ′ 1 ∣ ∣ . . . ∣ ∣ B ′ n ] , and furthermore, for all i = 1 , . . . , m , B ′ i x i ≥ 0 for all x i ∈ U i .

Proof of Lemma B.5. Since 0 / ∈ aff U i for all i = 1 , . . . , m by hypothesis, then there exist vectors τ i such that τ glyph[latticetop] i x i = 1 for all x i ∈ U i . For any k ∈ { 1 , . . . , d } and i ∈ { 1 , . . . , m -1 } , let

Furthermore, let β m := ∑ m -1 i =1 β i and B ′ m := B i + β m τ glyph[latticetop] m . It is immediate to check that the matrix B ′ := [ B ′ 1 | · · · | B ′ m ] is such that B ′ x = B x for all x ∈ U 1 × · · · × U m , that is, B ′ ∼ = ( U 1 ×···×U m ) B . We now show that B ′ i x i ≥ 0 for all i = 1 , . . . , m and x i ∈ U i . Expanding the definition of B ′ i and β i , for all i ∈ { 1 , . . . , m -1 } , x i ∈ U i and k ∈ { 1 , . . . , d } ,

Hence, it only remains to prove that the same holds for i = m . To that end, fix any k ∈ { 1 , . . . , d } and x m ∈ U m , and let x ∗ i ∈ arg min x i ∈U i ( B i x i )[ k ] for all i ∈ { 1 , . . . , m -1 } . Using the fact that all vectors in P ( b ) are nonnegative, x ∗ := ( x ∗ 1 , . . . , x ∗ m -1 , x m ) ∈ U 1 ×··· × U m must satisfy B x ∗ ≥ 0 . Hence,

thus concluding the proof of the lemma.

Since B ′ ∼ = ( U 1 ×···×U m ) B , and B maps to P ( b ) , for all x = ( x 1 , . . . , x m ) ∈ U 1 ×···×U m we must have

Since we can pick the x i for different indices i independently, it follows that PB i x i must be a constant function of x i ∈ U i , that is, there must exist vectors b 1 , . . . , b m ∈ R k such that

Since in addition B ′ i x i ≥ 0 (by construction of the B ′ i ), this means that B ′ i ∈ L U i →P ( b i ) . Finally, using the hypothesis that L U i →P ( b i ) ∼ = U i M U i →P ( b i ) , there must exist A i ∈ M U i →P ( b i ) , with A i ∼ = U i B ′ i , for all i = 1 , . . . , m . This concludes the proof.

B.3 Characterization of Linear Functions of Subtrees

The following result can be understood as a version of Theorem 3.1 stated for each subtree, rooted at some decision node, of the decision space.

Theorem B.6. For any decision node j ∈ J and vector b j ∈ R k , let

Before continuing with the proof, we remark a subtle point: unlike (7), which constraints each column to have entries in [0 , γ ] , (18) only specifies the lower bound at zero, but no upper bound. Hence the tilde above the symbol of this Theorem. Consequently, the matrices in the set ˜ M Q glyph[followsequal] j →P ( b j ) need not have bounded entries. In that sense, Theorem B.6 is slightly different from Theorem 3.1. We will strengthen (18) to enforce a bound on each column when completing the proof of Theorem 3.1 in the next subsection.

Proof. To aid us with the proof, we first express the definition of ˜ M Q glyph[followsequal] j →P ( b j ) in a way that better captures the inductive structure we need. By direct inspection of the constraints, the set ˜ M Q glyph[followsequal] j →P ( b j ) satisfies the inductive definition

We prove the result by structural induction on the tree-form decision process.

Base case. We start by establishing the result for any terminal decision node j ∈ J , that is, one for which all sequences { ja : a ∈ A j } are terminal. In this case, the set Q glyph[followsequal] j is the probability simplex ∆( { ja : a ∈ A j } ) . Thus, for a matrix A to map all x ∈ Q glyph[followsequal] j to elements in the convex polytope P ( b j ) it is both necessary and sufficient that all columns of A be elements of P ( b j ) . It is necessary because if A x ∈ P ( b j ) for all x ∈ Q glyph[followsequal] j , then for the indicator vector x with x [ ja ] = 1 we get A x = A ( ja ) ∈ P ( b j ) . And, it is sufficient because any x ∈ Q glyph[followsequal] j represents a convex combination of the columns A ( ja ) .

The set defined by these constraints matches exactly the set ˜ M Q glyph[followsequal] j →P ( b j ) defined in the statement: since all sequences ja are terminal, in this case it reduces to

that is, the set of matrices whose columns are elements of P ( b j ) . So, we have ˜ M Q glyph[followsequal] j →P ( b j ) = L Q glyph[followsequal] j →P ( b j ) with equality, which immediately implies the claim ˜ M Q glyph[followsequal] j →P ( b j ) ∼ = Σ glyph[followsequal] j L Q glyph[followsequal] j →P ( b j ) .

Inductive step. We now look at a general decision node j ∈ J , assuming as inductive hypothesis that the claim holds for any j ′ glyph[follows] j . Below we prove that ˜ M Q glyph[followsequal] j →P ( b j ) ∼ = Q glyph[followsequal] j L Q glyph[followsequal] j →P ( b j ) as well.

( ⊆ ) We start by showing that for any b j ∈ R k , x ∈ Π glyph[followsequal] j and A ∈ ˜ M Q glyph[followsequal] j →P ( b j ) , we have A x ∈ P ( b j ) . From (18) it is immediate that A has nonnegative entries, and since any vector x ∈ Π glyph[followsequal] j also has nonnegative entries, it follows that A x ≥ 0 . Hence, it only remains to show that P ( A x ) = b j . Using Lemma A.2, for any j ′ ∈ glyph[unionsq] a ∈A j C ja there exists x glyph[followsequal] j ′ ∈ Q glyph[followsequal] j ′ such that x [Σ glyph[followsequal] j ′ ] = x [ ja ] · x glyph[followsequal] j ′ . Hence, we have

( ⊇ ) Conversely, consider any B ∈ L Q glyph[followsequal] j →P ( b j ) . We will show that there exists a matrix A ∈ ˜ M Q glyph[followsequal] j →P ( b j ) such that B ∼ = Q glyph[followsequal] j A . First, we argue that there exists a matrix B ′ ∼ = Q glyph[followsequal] j B with the property that the column B ′ ( ja ) corresponding to any nonterminal sequence is identically zero.

Lemma B.7. There exists B ′ ∼ = Q glyph[followsequal] j B such that B ′ ( ja ) = 0 for all a ∈ A j such that ja ∈ Σ glyph[followsequal] j \ Σ ⊥ .

Proof. Fix any a ∈ A j such that ja is nonterminal. Then, by definition there exists at least one decision node j ′ whose parent sequence is ja . Consider now the matrix B ′′ obtained from 'spreading' column B ( ja ) onto B ( j ′ a ′ ) ( a ′ ∈ A j ′ ), that is, the matrix whose columns are defined according to the following rules: (i) B ′′ ( ja ) = 0 , (ii) B ′′ ( j ′ a ′ ) = B ( j ′ a ′ ) + B ( ja ) for all a ′ ∈ A j ′ , (iii) B ′′ ( σ ) = B ( σ ) everywhere else. The column B ′′ ( ja ) is identically zero by construction, and all other columns B ′′ ( ja ′ ) , a ′ ∈ A j \ { a } , are the same as B . Most importantly, since from the sequence-form constraints Definition 2.2 any sequence-form strategy x ∈ Q glyph[followsequal] j satisfies the equality x [ ja ] = ∑ a ′ ∈A j ′ x [ j ′ a ′ ] , the matrix B ′′ satisfies B ′′ x = B x for all x ∈ Q glyph[followsequal] j , i.e. , B ′′ ∼ = Q glyph[followsequal] j B . Iterating the argument for all actions a ∈ A j yields the statement.

Consider now any a ∈ A j that leads to a terminal sequence ja ∈ Σ glyph[followsequal] j ∩ Σ ⊥ . The vector 1 ja defined as having a 1 in the position corresponding to ja and 0 everywhere else is a valid sequence-form strategy vector, that is, 1 ja ∈ Q glyph[followsequal] j . Hence, since B ′ maps Q glyph[followsequal] j to P ( b j ) , it is necessary that B ′ ( ja ) ∈ P ( b j ) , that is, B ′ ( ja ) ≥ 0 and PB ′ ( ja ) = b j . In other words, we have just proved the following.

Lemma B.8. For any a ∈ A j such that ja ∈ Σ glyph[followsequal] j ∩ Σ ⊥ , B ′ ( ja ) ≥ 0 and PB ′ ( ja ) = b j .

Combined, Lemmas B.7 and B.8 show that B ′ satisfies constraints (21), (22), and (25). Consider now any action a ∈ A j that defines a nonterminal sequence ja ∈ Σ glyph[followsequal] j \ Σ ⊥ . For each child decision point j ′ ∈ C ja , let x glyph[followsequal] j ′ ∈ Q glyph[followsequal] j ′ be a choice of strategy for that decision point, and denote B ′ glyph[followsequal] j ′ the submatrix of B ′ obtained by only considering the columns B ′ ( σ ) corresponding to sequences σ glyph[followsequal] j ′ . The vector x defined according to x [ ja ] = 1 , x [Σ glyph[followsequal] j ′ ] = x glyph[followsequal] j ′ for all j ′ ∈ C ja , and 0 everywhere else is a valid sequence-form strategy x ∈ Q glyph[followsequal] j , and therefore B ′ x ∈ P ( b j ) since B ′ ∼ = Q glyph[followsequal] j B and B ∈ L Q glyph[followsequal] j →P ( b j ) by hypothesis. Therefore, using the fact that B ′ ( ja ) = 0 by Lemma B.7, we conclude that

Because the above holds for any choice of x glyph[followsequal] j ′ ∈ Q glyph[followsequal] j ′ , it follows that the matrix [ · · · | B ′ glyph[followsequal] j ′ | · · · ] ∈ L × j ′ ∈C ja Q glyph[followsequal] j ′ →P ( b j ) . Hence, applying Proposition B.4 (note that 0 / ∈ aff Q glyph[followsequal] j ′ since ∑ a ∈A j ′ x [ j ′ a ] = 1 for all x ∈ Q glyph[followsequal] j ′ by Definition A.1) together with the inductive hypothesis, we conclude that for each j ′ ∈ C ja there exist a vector b j ′ ∈ R k and a matrix A glyph[followsequal] j ′ ∈ M Q glyph[followsequal] j ′ →P ( b j ′ ) , such that ∑ j ′ ∈C ja b j ′ = b j and [ · · · | B ′ glyph[followsequal] j ′ | · · · ] ∼ = × j ′ ∈C ja Q glyph[followsequal] j ′ [ · · · | A glyph[followsequal] j ′ | · · · ] . We can therefore replace all columns corresponding to B ′ glyph[followsequal] j ′ with those of A glyph[followsequal] j ′ , obtaining a new matrix ∼ = Q glyph[followsequal] j B ′ . Repeating the argument for each ja ∈ Σ glyph[followsequal] j \ Σ ⊥ finally yields a new matrix that is ∼ = Σ glyph[followsequal] j B and satisfies all constraints given in (27), as we wanted to show.

B.4 Putting all the Pieces Together

Finally, we are ready to prove the main result of the paper.

Theorem 3.1. Let Q be a sequence-form strategy space and let P be any bounded polytope of the form P := { x ∈ R d : P x = p , x ≥ 0 } ⊆ [0 , γ ] d , where P ∈ R k × d . Then, for any linear function f : Q → P , there exists a matrix A in the polytope

such that f ( x ) = A x for all x ∈ Q . Conversely, any A ∈ M Q→P defines a linear function x ↦→ A x from Q to P , that is, such that A x ∈ P for all x ∈ Q .

Proof of Theorem 3.1. We prove the result in two steps. First, we show that

where the difference between ˜ M Q→P and M Q→P lies in constraint (34), which only sets a lower bound (at zero) for each entry of the matrix, as opposed to a bound [0 , γ ] as in (7). Using the definition of ˜ M Q glyph[followsequal] j →P ( b j ) given in (20) (Theorem B.6), the set ˜ M Q→P can be equivalently written as

To show that ˜ M Q→P ∼ = Q L Q→P , we proceed exactly like in the inductive step of the proof of Theorem B.6. Specifically, let A ∈ ˜ M Q→P and x ∈ Q be arbitrary. From Definition 2.2 it follows that x [Σ glyph[followsequal] j ] ∈ Q glyph[followsequal] j for any j ∈ C ∅ and therefore, denoting A glyph[followsequal] j := [ A ( σ ) ] σ glyph[followsequal] j ,

which shows that A x ∈ P . Since A and x were arbitrary, it follows that ˜ M Q→P ⊆ L Q→P . Conversely, let B ∈ L Q→P be arbitrary, and fix a root decision node j ∈ C ∅ . Then, we can 'spread out' the column B ( ∅ ) by adding it to each B ( ja ) : a ∈ A j by constructing the matrix L Q→P glyph[owner] B ′ ∼ = Q B defined by (i) B ′ ( ∅ ) = 0 , (ii) B ′ ( ja ) = B ( ja ) + B ( ∅ ) for any a ∈ A j , and (iii) B ′ ( σ ) = B ( σ ) everywhere else. Pick now any vectors { x glyph[followsequal] j ∈ Q glyph[followsequal] j : j ∈ C ∅ } , and consider the vector x defined as x [ ∅ ] = 1 , and x [Σ glyph[followsequal] j ] = x glyph[followsequal] j for all j ∈ C ∅ . The vector x is a valid sequence-form strategy, that is, x ∈ Q . Let now B ′ glyph[followsequal] j := [ B ′ ( σ ) ] σ glyph[followsequal] j . From the fact that B ′ x ∈ P , together with the fact that by construction B ′ ( ∅ ) = 0 , we conclude that

Since the inclusion above holds for any choice of { x glyph[followsequal] j ∈ Q glyph[followsequal] j : j ∈ C ∅ } , and since for all j ∈ C ∅ the vector 0 / ∈ aff Q glyph[followsequal] j (indeed, ∑ a ∈A j x [ ja ] = 1 for all x ∈ Σ glyph[followsequal] j by Definition A.1), from Proposition B.4 together with Theorem B.6 we conclude that for each j ∈ C ∅ there exists a vector b j ∈ R k and a matrix A glyph[followsequal] j ∈ ˜ M Q glyph[followsequal] j →P ( b j ) , such that ∑ j ∈C ∅ b j = p and A glyph[followsequal] j ∼ = Q glyph[followsequal] j B ′ glyph[followsequal] j . By replacing the submatrices B ′ glyph[followsequal] j with A glyph[followsequal] j in B ′ we then obtain an equivalent matrix that satisfies all constraints that define ˜ M Q→P . In summary, we have ˜ M Q→P ∼ = L Q→P .

To conclude the proof, we now show that ˜ M Q→P = M Q→P . First, we make the straightforward observation that any A ∈ M Q→P also belongs to ˜ M Q→P , as the constraint that define the latter set are only looser. Hence, we only need to show that any B ∈ ˜ M Q→P also satisfies constraint (7). Since B ∈ ˜ M Q→P , all columns of B are nonnegative (constraint (34)). Furthermore, since ˜ M Q→P ∼ = Q L Q→P , clearly B x ∈ P for all x ∈ Q . Fix now any sequence σ ∈ Σ , and consider any strategy x ∈ Q that puts proability mass 1 on all the actions on the path from the root to σ included, that is, any x ∈ Q with x [ σ ] = 1 . Then, from the nonnegativity of the columns of B , it follows that

Since by definition of γ any point in P belongs to [0 , γ ] d , we then conclude that B ( σ ) ∈ [0 , γ ] d , implying that B ∈ M Q→P as we wanted to show.

C Corollaries of the Characterization Theorem

Wemention two direct corollaries of Theorem 3.1 that slightly extend the scope of the characterization. The first corollary asserts that the polytope M Q→P characterizes not only all linear functions from Q to P , but also all affine functions.

Corollary C.1 (From linear to affine functions) . Let Q be a sequence-form strategy space and let P be any polytope. Then, for any affine function g : Q → P , there exists a matrix A in the polytope M Q→P defined in Theorem 3.1 such that g ( x ) = A x for all x ∈ Q . Conversely, any A ∈ M Q→P induces an affine function from Q to P .

Proof. The second part of the statement is trivial since any linear function is also affine, and any A ∈ M Q→P induces a linear function from Q to P . Let g ( x ) = f ( x ) + b be any affine function, where f is an appropriate linear function from Q to P and b ∈ R n . Since q [ ∅ ] = 1 for all q ∈ Q , the function g coincides on Q with the function ˜ g : Q glyph[owner] x ↦→ f ( x ) + b · x [ ∅ ] , which is a linear function of x . Hence, from the first part of Theorem 3.1 there exists A ∈ M Q→P such that A x = ˜ g ( x ) = g ( x ) for all x ∈ Q .

The second corollary of Theorem 3.1 extends the characterization to the alternative definition of polytope as a bounded set of the form C := { y ∈ R n : C y ≤ c } , by first introducing slack variables and rewriting the polytope in the form handled by Theorem 3.1.

Corollary C.2 (Alternative polytope representations) . Let Q be a sequence-form strategy polytope, and C := { y ∈ R n : C y ≤ c } ⊆ [ -γ, γ ] n be a bounded polytope, where C ∈ R m × n . Let k := max {‖ C ‖ ∞ , ‖ c ‖ ∞ } and introduce the polytope

which is of the form handled by Theorem 3.1. For any affine function g : Q → C , there exists a matrix A in the polytope

such that g ( x ) = A x for all x ∈ Q . Conversely, any A ∈ ˜ M Q→C induces an affine function from Q to C .

Proof. We begin by proving that C ⊆ [ -γ, γ ] n implies ˜ P ⊆ [0 , 2 γ ] n + m . Consider any (˜ y , s ) ∈ ˜ P and set y = ˜ y -γ 1 . Then, this is a valid y ∈ C ⊆ [ -γ, γ ] n and, consequently, ˜ y ∈ [0 , 2 γ ] n . For the slack variables s it holds that kn s = c -C y , where y = ˜ y -γ 1 from before. By definition of k and by y ≥ -γ 1 , we conclude that c -C y ≤ ( k + nγk ) 1 = ⇒ s ∈ [0 , 2 γ ] m .

Now let g : Q → C be any affine function from Q to C . Then we can define an affine function f : Q → ˜ P such that

for all x ∈ Q . By Corollary C.1 we know that M Q→ ˜ P characterizes all affine functions from Q to ˜ P , including the previous function f . Thus, there exists an ˜ M ∈ R n × Σ such that g ( x ) + γ 1 = ˜ M x for all x ∈ Q . Since x [ ∅ ] = 1 for all x ∈ Q , we conclude that there exists A ∈ ˜ M Q→C such that g ( x ) = A x = ˜ M x -γ 1 for all x ∈ Q .

Conversely, consider any A ∈ ˜ M Q→C and define g ( x ) = A x . That is, there exist suitable ˜ M ∈ R n × Σ , ˜ Z ∈ R m × Σ that satisfy the constraints of polytope ˜ M Q→C . Then for all x ∈ Q it holds g ( x ) = ˜ M x -γ 1 , and ( ˜ y , s ) ∈ ˜ P where ˜ y = ˜ M x and s = ˜ Z x . Thus, by construction of ˜ P , as we also argued in the beginning of the proof, we conclude that g ( x ) = ˜ y -γ 1 = y ∈ C .

D Additional proofs

Theorem D.1. Let Σ denote the set of sequences of the learning player in the extensive-form game, and let η ( t ) = 1 / √ t for all t . Then, for any sequence of loss vectors glyph[lscript] ( t ) ∈ [0 , 1] Σ , Algorithm 1 guarantees linear-swap regret O ( | Σ | 2 √ T ) after any number T of iterations, and runs in O ( | Σ | 10 log( | Σ | ) log 2 t ) time for each iteration t .

Proof of Theorem 3.2. First we focus on the linear-swap regret bound. Based on Gordon et al. [2008] the Φ -regret equals external regret over the set Φ of transformations. In our case Φ is the set M Q→Q of all valid linear transformations and the losses for the external regret minimizer are functions A ↦→〈 glyph[lscript] (1) , A x (1) 〉 , A ↦→〈 glyph[lscript] (2) , A x (2) 〉 , . . . Equivalently, we can write these as A ↦→〈 L ( t ) , A 〉 F , where 〈· , ·〉 F is the component-wise inner product for matrices and L ( t ) = glyph[lscript] ( t ) ( x ( t ) ) glyph[latticetop] , which is a rank-one matrix. Let D be an upper bound on the diameter of M Q→Q , and L be such that ‖ L ( t ) ‖ F ≤ L for all t . Then, based on Orabona [2022] we can bound the external regret for this instance of Online Linear Optimization by picking η ( t ) = D L √ t which gives a regret of O ( DL √ T ) . Since A ∈ [0 , 1] Σ × Σ we get D = | Σ | , and since glyph[lscript] ( t ) , x ( t ) ∈ [0 , 1] Σ we get L = | Σ | . This results in the desired linear-swap regret of O ( | Σ | 2 √ T ) .

However, in our previous analysis we assumed that it is possible to compute an exact solution A ( t +1) = Π M Q→Q ( A ( t ) -η ( t ) L ( t ) ) of the projection step, which is an instance of convex quadratic programming, meaning that we can only get an approximate solution [Vishnoi, 2021]. In the following Lemma we prove that an glyph[epsilon1] -approximate projection does not affect the regret for a single iteration of

the algorithm, if glyph[epsilon1] is sufficiently small. The inequality we prove is similar to the one in Lemma 2.12 from Orabona [2022].

Lemma D.2. Let Y ∗ = Π M Q→Q ( A ( t ) -η ( t ) L ( t ) ) and suppose that A ( t +1) ∈ M Q→Q is such that ‖ A ( t +1) -Y ∗ ‖ 2 F ≤ glyph[epsilon1] ( t ) , then for any X ∈ M Q→Q it holds

Proof. From Lemma 2.12 of Orabona [2022] we know that

Additionally, it holds that

The Lemma follows by combining the previous two inequalities.

If we set glyph[epsilon1] ( t ) = 1 /t 5 / 2 then for our choice of η ( t ) = D L √ t , the error term becomes L/t 2 . Summing over T timesteps we thus get an additive error of O ( L ) = O ( | Σ | ) in the regret, and our total linear-swap regret bound remains O ( | Σ | 2 √ T ) .

We now move to analyzing the per-iteration time complexity of Algorithm 1. The most computationally heavy steps are (5) and (6). To compute the fixed point at step (6) we can use any polynomial-time LP algorithm. We can also perform the projection step (5) in polynomial time using the ellipsoid method [Vishnoi, 2021]. For this we reduce it to a suitable Semidefinite Program with O ( | Σ | 2 ) variables and use the Cholesky factorization [G¨ artner and Matousek, 2014] as the separation oracle, thus responding to separation queries in O ( | Σ | 6 ) time. Note that it is possible to guarantee ‖ A ( t +1) -Y ∗ ‖ 2 F ≤ glyph[epsilon1] ( t ) as the ellipsoid method can output a point A ( t +1) ∈ M Q→Q such that ‖ A ( t ) -η ( t ) L ( t ) -A ( t +1) ‖ F ≤ ‖ A ( t ) -η ( t ) L ( t ) -Y ∗ ‖ F + glyph[epsilon1] and furthermore, the Frobenius norm is a 2-strongly convex function.

To apply the ellipsoid method we further need bounds on the Frobenius norm of Y ∈ M Q→Q and the maximum projection distance. For the norm of Y , we pick an upper bound of R = D = | Σ | . For the lower bound r we note that Y x ∈ Q for all x ∈ Q , which implies Y [ ∅ ] x = 1 = ⇒ ‖ Y [ ∅ ] ‖ 1 ≥ 1 . Thus we get r ≥ 1 | Σ | . Similarly, we bound the projection distance between 0 and D . Based on these bounds and on Theorem 13.1 from Vishnoi [2021] we conclude that the total per-iteration time complexity of Algorithm 1 is O ( | Σ | 10 log( | Σ | ) log 2 ( t ) ) .

Theorem 4.1. For two-player, perfect-recall extensive-form games with chance moves, the problem MAXPAY-LCE is not solvable in polynomial time, unless P=NP.

Proof. We use the exact same argument employed in the paper by von Stengel and Forges [2008] by reducing SAT to the MAXPAY-LCE problem. Specifically, for each instance of SAT having n clauses and m variables we construct a two-player extensive-form game of size polynomial in n and m . In the beginning, there is a chance move that picks one of n possible actions uniformly at random - one for each SAT clause. Then is the turn of Player 2 who has n distinct singleton information sets corresponding to the chance node actions, and respectively to the n clauses of the SAT instance. Let L i be the set of literals (negated or non-negated variables) included in the i -th clause of the SAT

instance. In information set i of Player 2 there exist | L i | actions, one for each literal in L i . Finally, each literal leads to a different decision node for Player 1 who has as many decision nodes as the number of literals in the SAT instance, and in each node there exist exactly 2 possible actions: TRUE and FALSE. However, Player 1 only has m information sets corresponding to the m SAT variables with each information set x grouping together all nodes corresponding to literals of the variable x . This way, Player 1 only chooses the truth value of a variable without knowing from which literal Player 2 has picked this variable. The utilities for both players are equal to 1 if the truth value picked by Player 1 satisfies the literal picked by Player 2, and both utilities are 0 otherwise.

In this game, there exists a pure strategy attaining payoff 1 for each player if and only if the SAT instance is satisfiable - namely Player 1 always acts based on the satisfying assignment and Player 2 picks for every clause a literal that is known to be TRUE. Otherwise, the maximum payoff for each pure strategy is at most 1 -1 /n . Given that a LCE describes a convex combination of pure strategies, it follows that the maximum total expected payoff of the the sum of the two players' utilities will either be 2 when the SAT instance is satisfiable, or it will be at most 2(1 -1 /n ) when the instance is not satisfiable. Additionally, this holds not just for the sum but for any linear combination of player utilities. Thus, the problem of deciding whether MAXPAY-LCE can attain a value of at least k is NP-hard for any linear combination of utilities.

E Examples

glyph[negationslash]

G

B

X

Y

4

10

0

6

6

0

0

6

4

10

0

6

6

0

0

6

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

l

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

r

X

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

l

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

r

Y

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

X

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

Y

G

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

X

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Y

B

Player 1

Nature

Player 2

Terminal node

Y

Information set

Legend

1

2

l

X

l

Y

l

X

r

Y

r

X

l

Y

r

X

r

Y

X

G

X

B

5, 5

5, 5

0, 6

0, 6

X

G

Y

B

5, 5

2, 8

3, 3

0, 6

Y

G

X

B

5, 5

3, 3

2, 8

0, 6

Y

G

Y

B

5, 5

0, 6

5, 5

0, 6

However, we can observe that this EFCE is not a linear-deviation correlated equilibrium because, for example, Player 1 can increase their payoff by a value of 3 / 2 using the following transformation:

(Above, we have implicitly assumed that the strategy vectors encode probability of actions in the arbitrary order X G , Y G , X B , Y B ). The above transformation is linear, since it can be represented via the matrix

This would swap the pure strategy X G Y B with X G X B and strategy Y G X B with Y G Y B . Crucially, the strategy at the subtree of information set B is determined by the strategy at the subtree of information set G . Thus, the transformed value for each sequence does not purely depend on the ancestors of that sequence, but can also depend on strategies belonging to 'sibling' subtrees. Hence, this example proves that LCE = EFCE.

glyph[negationslash]

In fact, we can even verify that in this toy example the set of LCE coincides with the set of all normal-form CE. We refer the interested reader to von Stengel and Forges [2008] for a more detailed discussion of the different behaviors that players can exhibit at an EFCE vs a normal-form CE, which in this specific example also corresponds to a comparison between the EFCE and the LCE.

Remark E.2. The linear transformation given in Example E.1 also serves to show that the Behavioral Deviations defined in Morrill et al. [2021] are not a superset of all linear transformations. Additionally, behavioral deviations are not a subset of linear transformations, as the latter act only on reduced strategies. Thus the two sets of deviations are incomparable.

glyph[negationslash]

Example E.3 (LCE = CE) . This example was found through computational search. Consider the 2-player game with the following game tree:

A

B

Q

W

0

0

-3

0

-2

0

0

0

0

0

-310

-315

0

0

0

0

0

2

0

0

0

0

2

4

0

0

0

0

202

0

0

0

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

1/4

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

l

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

Q

r

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

l

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

W

r

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

2

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

A

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

2

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

B

1

Player 1

Nature

Player 2

Terminal node

Y

Information set

Legend

We can now verify that the following is a linear-deviation correlated equilibrium for this game. The verifcation can be done computationally by expressing all constraints of Theorem 3.1 as a Linear Program.

However, the swap { A 1 B 2 ↦→ A 1 B 1 , A 2 B 1 ↦→ A 1 B 1 } can increase player 1's payoff by 50 . 5 .

Furthermore, we can verify that this swap is not linear as follows. First assume that it was linear and could be written as a matrix A ∈ [0 , 1] Σ × Σ . Then the matrix has to be consistent with the following transformations:

For convenience of referring to matrix rows and columns we number the four sequences as:

If we focus on the second transformation, A 1 B 2 ↦→ A 1 B 1 , we conclude that A [: , 0] + A [: , 3] = (1 , 0 , 1 , 0) glyph[latticetop] which implies that A [1 , 0] = A [3 , 0] = A [1 , 3] = A [3 , 3] = 0 . Now, if we subtract the respective equations of the last two swaps we get A [: , 2] -A [: , 3] = (1 , -1 , 1 , -1) glyph[latticetop] . Since A ∈ [0 , 1] Σ × Σ , the last equation implies A [1 , 3] = 1 which contradicts the previous constraint of A [1 , 3] = 0 . Thus, we have found a valid normal-form swap that cannot be expressed as a linear transformation of sequence-form strategies. Hence, this example proves that LCE = CE.

F Details on Empirical Evaluation

In this section we provide details about the implementation of our algorithm, as well as the compute resources and game instances used.

glyph[negationslash]

Implementation of our no-linear-swap-regret dynamics We implemented our no-linear-swapregret algorithm (Algorithm 1) in the C++ programming language using the Gurobi commercial optimization solver [Gurobi Optimization, LLC, 2023], version 10. We use Gurobi for the following purposes.

To compute the projection needed on Line 5 of Algorithm 1. We remark that while Gurobi is typically recognized as a linear and integer linear programming solver, modern versions include tuned code for convex quadratic programming. In particular, we used the barrier algorithm to compute the Euclidean projections onto the polytope M Q→Q required at every iteration of our algorithm.

To compute the fixed points of the matrices A ∈ M Q→Q , that is, finding Q glyph[owner] x = A x . As discussed in Section 3 this is a polynomially-sized linear program.

To measure the linear-swap regret incurred after any T iterations, which is plotted on the y-axes of Figure 2. This corresponds to solving the linear optimization problem

We did very minimal tuning of the constant learning rate η used for online projected gradient descent, trying values η ∈ { 0 . 05 , 0 . 1 , 0 . 5 } (we remark that a constant value of η ≈ 1 / √ T is theoretically sound). We found that η = 0 . 1 , which is used in the plots of Figure 2, performed best.

Implementation of no-trigger-regret dynamics We implemented the no-trigger-regret algorithm of Farina et al. [2022] in the C++ programming language. In this case, there is no need to use Gurobi, since, as the original authors show, the polytope of trigger deviation functions admits a convenient combinatorial characterization that enables us to sidestep linear programming. Rather, we implemented the algorithm and the computation of the trigger regret directly leveraging the combinatorial structure.

Computational resources used Minimal computational resources were used. All code ran on a personal laptop for roughly 12 hours.

Game instance used We ran our code on the standard benchmark game of Kuhn poker Kuhn [1950]. We used a three-player variant of the game. Compared to the original game, which only considers a simplified deck make of cards out of only three possible ranks (Jack, Queen, or Kind), we use a full deck of 13 possible card ranks. The game has 156 information sets, 315 sequences, and 22308 terminal states.

G Further Remarks on the Reduction from NoΦ -Regret to External Regret

A noΦ -regret algorithm is typically defined as outputting deterministic behavior from a finite set X (for example, deterministic reduced-normal-form plans in extensive-form games, or actions in normal-form games), which can be potentially sampled at random. This is the setting used by, for example, Hart and Mas-Colell [2000] and Farina et al. [2022]. However, we remark that when the transformations φ ∈ Φ are linear, any such device can be constructed starting from an algorithm that outputs points x ′ ∈ X ′ := conv ( X ) , and then sampling x unbiasedly in accordance with x ′ , that is, so that E [ x ] = x ′ . The reason why this is useful is that constructing the latter object is usually simpler, as X ′ is a closed and convex set, and is therefore amenable to the wide array of online optimization techniques that have been developed over the years.

More formally, let Φ be a set of linear transformations that map X to itself. A noΦ -regret algorithm for X ′ = conv ( X ) guarantees that, no matter the sequence of loss vectors glyph[lscript] ( t ) ,

grows sublinearly. Consider now an algorithm that, after receiving x ′ ( t ) ∈ conv ( X ) , samples x ( t ) ∈ X unbiasedly, that is, so that E [ x ( t ) ] = x ′ ( t ) . Then, by linearity of the transformations and

using the Azuma-Hoeffind concentration inequality, we obtain that for any glyph[epsilon1] > 0 ,

where the big-theta notation hides constants that depend on the payoff range of the game and the diameter of X (a polynomial quantity in the game tree size). This shows that as long as the regret of the noΦ -algorithm that operates over X ′ is sublinear, then so is that of an algorithm that outputs points on X by sampling unbiasedly from X . We refer the interested reader to Section 4.2 ('From Deterministic to Mixed Strategies') of Farina et al. [2022].

Peer reviews:
comments: SUMMARY:
Authors introduce a new class of correlated equilibria called linear-deviation correlated equilibria, which can be approached efficiently if all players attain sublinear linear-swap regret.
They show LCEs are distinct from correlated equilibria and extensive-form correlated equilibria and the hardness of maximizing social welfare in LCEs. Finally, they show the difference in no-linear-swap-regret dynamics and no-trigger-regret dynamics to support their theoretical results on a small game.

STRENGTHS:
- Introduction of new correlated equilibrium class.
- Technical contribution of a polynomial characterization of the set of all linear transformations from a sequence-form strategy polytope to itself.
- General algorithm for minimizing linear-swap regret.
- Shows relation to existing equilibrium classes.
- Very clearly written paper that I expect will be built upon in the future.

WEAKNESSES:
- This is a theoretical paper and it is not clear what would be the motivation to approach LCEs in practice, compared to another correlation classes.

QUESTIONS:
- I think it would be of great pedagogical value to make a visual depiction of the different classes of equilibria similar to https://www.cs.cmu.edu/~ggordon/CE/ -- but I understand this might not be possible because the high dimensionality of the joint strategy space. 
- Do you think there exists a linear program to find LCEs? If not, why?

LIMITATIONS:
Limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence of uncoupled strategies to a weakened notion of equilibria called \"linear-deviation correlated equilibrium\". This equilibrium is reached when all players minimise the no-linear-swap regret which is a specialisation of Phi-equilibria when the set of deviations Phi is the set of all linear transformation from the set (mixed) strategy to itself.\nThe authors provide efficient implementation of such no-linear-swap regret algorithms and prove that the set of linear-deviation correlated equilibria is a subset of extensive form correlated equilibria. The technical tool used is the of of Gordon 2008 which converts a regret minimiser for the space of deviations to a regret minimiser for the strategy set.\n\nSTRENGTHS:\nThe paper aims to solve an important problem, which is the complexity of CE in EFG.\nThe paper is easy to read, clear and well written and the results seems non-trivial\n\nWEAKNESSES:\nThe notion of LCE is not well motivated. The authors prove that LCE are EFCE but, while EFCE are well motivated in EFG, LCE do not seems to have any practical meaning. Also the algorithm presented in the paper is slower at converging to EFCEs then the one specifically designed for them (Figure 2 left) and thus do not contribuite in this regard.\nThe main weakness is that LCE are not an interesting equilibrium to converge to, except from their connection to EFCEs. The author should motivate the introduction of this new equilibria for example by finding examples in which LCE are a reasonable solution concept while other EFCEs are not. Without this the point the paper seems a technical exercise.\n\nQUESTIONS:\n1) Th 31 seems to be the main technical statement of the paper. What are the main challenges of proving this statement? A better intuition of its implications/design would be appriciated\n2) Motivate LCE (see Weaknesses)\n3) Figure 2 only proves that the inclusions of CE, LCE and EFCE are strict, but is somewhat uninteresting once we have a theorem stating it. What happens is in Figure 2 left you put there time instead of iterations? If the no-linear-swap regret algorithm runs faster then the one that minimises only the trigger-regret this would be a better algorithm for finding EFCEs.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies regret minimization in extensive-form games (EFG). Specifically, they study a notion of regret called linear-swap regret, which measures the regret against linear transformations of the player's sequence-form strategies. This notion is stronger than trigger regret (as trigger deviations can be described as a linear transformation of sequence-form strategies) but weaker than swap regret.  \n\nThe main technical contribution lies in Theorem 3.1, which states that the set of linear transformations (from the set of sequence-form strategies to itself) is a compact polytope defined by a polynomial (in the game tree size) number of linear constraints. \n\nWith that in hand, they leverage the template of Gordon et al. [2008] that produces a $\\Phi$-regret minimization algorithm given a no-external-regret minimizer $\\Phi$ and a fixed point oracle which returns fixed point of transformations in $\\Phi$. In this context, the $\\Phi$-regret minimization can be chosen, for example, to be Online Gradient Decent; and the fixed point oracle can be implemented using a Linear Programming Solver (e.g., Ellipsoid algorithm). This results in an efficient algorithm that achieves $\\sqrt{T}$ linear-swap regret; which in turn also implies an efficient method for finding the correlated equilibrium that corresponds to linear transformations, that the authors call linear-deviation correlated equilibrium (LCE).\n\nFinally, the authors provide examples to show that the set of LCEs strictly contain**s** the set of CEs, and is strictly contain**ed** in the set of Extensive-form correlated equilibrium (EFCE). They also show that the set of Behavioral correlated equilibrium (BCE) and the set of LCEs are incomparable.\n\nSTRENGTHS:\nThe paper is written well; the authors give great background on EFGs, on different notions of correlated equilibrium, and their relation to different notions of regret. \nThe construction of the algorithm and results are fairly easy to understand, and the authors express well the difference between their results and previous work by providing examples that compare their notion of equilibrium with existing notions.\nFinally, I agree that determining *\"what is the strongest notion of rationality that can be attained efficiently\"* is an interesting open problem, and indeed the paper provide some progress in this direction.\n\nWEAKNESSES:\n- There is no obvious algorithmic contribution here: As noted, Theorem 3.1 is the main technical contribution which is purely geometric. Given that, constructing the algorithm is fairly straightforward with the scheme of Gordon et al. [2008].\n- While the background the paper gives is quite solid, it is not very proportional to the relatively small amount of attention the authors give to their technical contribution. In particular, while the statement of Theorem 3.1 is clear, it is not very intuitive, and it is hard to understand why it is correct. This is also the main reason that my confidence score is relatively low. I think that some of the preliminaries, as well as the empirical evaluations, could be deferred to the appendix - instead, I think that giving the main proof ideas is more important.\n\n**Missing related work:** I think that the work of Anagnostides et al. (2022) is very relevant here and motivates important questions for future work. Specifically, this work achieves a much better rate of $\\log T$ for trigger regret. Thus, it is fairly natural to ask whether this is also attainable for linear-swap regret, especially since they also built upon the template of Gordon et al. [2008].\n\nAnagnostides, I., Farina, G., & Sandholm, T. (2022). Near-Optimal $\\Phi $-Regret Learning in Extensive-Form Games. arXiv preprint arXiv:2208.09747.‏\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on addressing the challenge of minimizing linear swap regret in extensive form games, which is considered a stronger notion compared to trigger regret in Extensive Form Correlated Equilibrium (EFCE). To achieve efficient implementation of the Phi-regret minimization problem (where Phi set contains all linear swap transformation), the authors provide a polynomial-sized description of all linear swap operators. This result shows that there exists a no linear swap regret algorithm with polynomial-time iteration complexity based on the size of the game tree. Additionally, the paper presents experimental results that illustrate how the equilibria achieved through the no linear swap regret algorithm exhibit greater strength compared to EFCE.\n\nSTRENGTHS:\nThis paper introduces a novel and essential contribution by providing a polynomial characterization of the set of all linear transformations from a sequence-form strategy polytope to itself. This characterization holds significant value for future research on linear-swap-regret.\n\nWEAKNESSES:\nThe contribution of this paper may be perceived as limited. The primary and significant contribution lies in the polynomial description of the linear swap transformation set. Given the sequential structure of the sequence form strategy, this finding is not entirely unexpected.\n\nIt appears that the paper lacks some intuitive explanations regarding the concept of linear swap (further details can be found in the questions raised). There is a sense that LCE (Linear Swap Correlated Equilibrium) is merely one equilibrium between CE (Correlated Equilibrium) and EFCE (Extensive Form Correlated Equilibrium), which is relatively easy to compute. The experimental results may not comprehensively demonstrate the potential and strength of LCE\n\nQUESTIONS:\n1. Is there any intuition behind the linear swap transformation, apart from its linearity? For instance, can LCE be interpreted with the presence of a mediator? Why do you believe LCE is an equilibrium worth considering?\n\n2. Maximizing social welfare is known to be challenging. However, in your experiments, did you observe any significant differences in social welfare between No-linear-swap-regret dynamics and No-trigger-regret dynamics?\n\n3. Since No-linear-swap-regret dynamics are stronger than No-trigger-regret dynamics. The result in Figure 2 is expected. What about the running times? Is the running time of no-linear-swap-regret dynamics substantially larger?\n\n[1] Zhang, Brian, and Tuomas Sandholm. \"Polynomial-time optimal equilibria with a mediator in extensive-form games.\" Advances in Neural Information Processing Systems 35 (2022): 24851-24863.\n\nLIMITATIONS:\nThe authors have partially addressed the limitations of their work, though there is space for improvement (see the section Weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}