Title: Guarantees for Self-Play in Multiplayer Games via Polymatrix Decomposability
Venue: NeurIPS 2023
Accepted: yes
Keywords: Algorithmic Game Theory

Self-Play

Regret-Minimization

Multi-agent RL

Multiplayer Games

General-Sum Games
Abstract: Self-play is a technique for machine learning in multi-agent systems where a learning algorithm learns by interacting with copies of itself. Self-play is useful for generating large quantities of data for learning, but has the drawback that the agents the learner will face post-training may have dramatically different behavior than the learner came to expect by interacting with itself. For the special case of two-player constant-sum games, self-play that reaches Nash equilibrium is guaranteed to produce strategies that perform well against any post-training opponent; however, no such guarantee exists for multiplayer games. We show that in games that approximately decompose into a set of two-player constant-sum games (called constant-sum polymatrix games) where global $\epsilon$-Nash equilibria are boundedly far from Nash equilibria in each subgame (called subgame stability), any no-external-regret algorithm that learns by self-play will produce a strategy with bounded vulnerability. For the first time, our results identify a structural property of multiplayer games that enable performance guarantees for the strategies produced by a broad class of self-play algorithms. We demonstrate our findings through experiments on Leduc poker.

Guarantees for Self-Play in Multiplayer Games via Polymatrix Decomposability

Abstract

Self-play is a technique for machine learning in multi-agent systems where a learning algorithm learns by interacting with copies of itself. Self-play is useful for generating large quantities of data for learning, but has the drawback that the agents the learner will face post-training may have dramatically different behavior than the learner came to expect by interacting with itself. For the special case of two-player constant-sum games, self-play that reaches Nash equilibrium is guaranteed to produce strategies that perform well against any post-training opponent; however, no such guarantee exists for multiplayer games. We show that in games that approximately decompose into a set of two-player constant-sum games (called constant-sum polymatrix games) where global ϵ -Nash equilibria are boundedly far from Nash equilibria in each subgame (called subgame stability), any no-external-regret algorithm that learns by self-play will produce a strategy with bounded vulnerability. For the first time, our results identify a structural property of multiplayer games that enable performance guarantees for the strategies produced by a broad class of self-play algorithms. We demonstrate our findings through experiments on Leduc poker.

1 Introduction

Self-play is one of the most commonly used approaches for machine learning in multi-agent systems. In self-play, a learner interacts with copies of itself to produce data that will be used for training. Some of the most noteworthy successes of AI in the past decade have been based on self-play; by employing the procedure, algorithms have been able to achieve super-human abilities in various games, including Poker (Moravcik et al., 2017; Brown & Sandholm, 2018, 2019), Go and Chess (Silver et al., 2016, 2018), Starcraft (Vinyals et al., 2019), Diplomacy (Paquette et al., 2019), and Stratego (Perolat et al., 2022).

Self-play has the desirable property that unbounded quantities of training data can be generated (assuming access to a simulator). But using self-play necessarily involves a choice of agents for the learner to train with: copies of itself. Strategies that perform well during training may perform poorly against new agents, whose behavior may differ dramatically from that of the agents that the learner trained against.

The problem of learning strategies during training that perform well against new agents is a central challenge in algorithmic game theory and multi-agent reinforcement learning (MARL) (Matignon et al., 2012; Lanctot et al., 2017). In particular, a self-play trained agent interacting with agents from an independent self-play instance-differing only by random seed-can lead to dramatically worse performance (Lanctot et al., 2017). The self-play-based DORA (Bakhtin et al., 2021) performs well aginst copies of itself in the game of no-press Diplomacy, but poorly against human-like agents

(Bakhtin et al., 2022). Self-play has also been known to perform poorly against new agents in games with highly specialized conventions (Hu et al., 2020), such as Hanabi (Bard et al., 2020). In training, one instance of a self-play algorithm may learn conventions that are incompatible with the conventions of another instance.

There are special classes of games where the strategies learned through self-play generalize well to new agents. In two-player, constant-sum games there exist strong theoretical results guaranteeing the performance of a strategy learned through self-play: Nash equilibrium strategies are maxmin strategies, which will perform equally well against any optimal opponent and can guarantee the value of the game against any opponent.

We lose these guarantees outside of two-player constantsum games. For example, consider the simple two-player coordination game of Figure 1. If both players choose the same action, both receive a utility of 1, otherwise they receive 0. Suppose the row player learned in self-play to choose a (which performs well against another a -player). Similarly, column learned to play b . If these two players

a

b

a

1

,

1

0

,

0

b

0

,

0

1

,

1

played against each other, both agents would regret their actions. Upon the introduction of a new agent who did not train with a learner, despite a and b being optimal strategies during training, they fail to generalize to new agents. As this example demonstrates, equilibrium strategies in general are vulnerable : agents are not guaranteed the equilibrium's value against new agents-even if the new agent's also play an equilibrium strategy.

In multiplayer, general-sum games, no-regret self-play is no longer guaranteed to produce Nash equilibra-instead, algorithms converge to a mediated equilibrium , where a mediator recommends actions to each player (Von Stengel & Forges, 2008; Farina et al., 2019, 2020; Morrill et al., 2021b). The mediator can represent an external entity that makes explicit recommendations, such as traffic lights mediating traffic flows. More commonly in machine learning, correlation can arise through the shared history of actions of learning agents interacting with each other (Hart & Mas-Colell, 2000). If a strategy learned in self-play were played against new agents, these new agents may not have access to the actions taken by other agents during training, so agents would no longer be able to correlate their actions. In fact, even if all agents play a decorrelated strategy from the same mediated equilibrium, the result may not be an equilibrium. 1

Despite the problems of vulnerability and loss of correlation, self-play has shown promising results outside of two-player constant-sum games. For example, algorithms based on self-play have outperformed professional poker players in multiplayer Texas hold 'em, despite the lack of theoretical guarantees (Brown & Sandholm, 2019).

We seek to understand what structure in multiplayer games will allow self-play to compute a good strategy. We show that any multiplayer game can be projected into the set of two-player constant-sum games between each pair of players, called constant-sum polymatrix games . The closer a game is to this space, the less the problem of correlation affects the removal of a mediator. We identify a second important property, called subgame stability , where global ϵ -Nash equilibria are boundedly far from Nash equilibria in each two-player subgame. We show that if a multiplayer, general-sum game is close to a subgame stable constant-sum polymatrix game, this is sufficient for strategies learned via self-play to generalize to new agents, as they do in two-player constant-sum games.

Throughout this work, we take an algorithm-agnostic approach by assuming only that self-play is performed by a regret minimizing algorithm . This is accomplished by analyzing directly the equilibria that no-regret algorithms converge to-namely coarse correlated equilibria. As a result, our analysis applies to a broad class of game-theoretically-inspired learning algorithms but also to MARL algorithms that converge to coarse correlated equilibria (Marris et al., 2021; Liu et al., 2021; Jin et al., 2021), since any policy can be transformed into a mixed strategy with Kuhn's Theorem (Kuhn, 1953). For the remainder of this work, when we say 'self-play" we are referring to self-play using a no-regret algorithm.

Decomposition-based approaches have been used in prior work to show convergence of fictitious play to Nash equilibria in two-player games (Chen et al., 2022) and evolutionary dynamics (Tuyls et al., 2018). Cheung & Tao (2020) decompose games into zero-sum and cooperative parts to analyse

the chaotic nature of Multiplicative Weights and Follow-the-Regularized-Leader. We focus less on the convergence of algorithms per se, and focus instead on the generalization of learned strategies to new agents. We are, to the best of our knowledge, the first to do so.

After defining our structural properties and proving our main results, we conclude with experiments on Leduc poker to elucidate why self-play performs well in multiplayer poker. Our results suggest that regret-minimization techniques converge to a subset of the game's strategy space that is wellapproximated by a subgame stable constant-sum polymatrix game.

2 Background

Normal Form Games A normal form game is a 3 tuple G = ( N, P , u ) where N is a set of players, P = × i ∈ N P i is a joint pure strategy space and P i is a set of pure strategies for player i . Let n = | N | . Pure strategies are deterministic choices of actions in the game. We call ρ ∈ P a pure strategy profile . u = ( u i ) i ∈ N is a set of utility functions where u i : P → R . A player i can randomize by playing a mixed strategy , a probability distribution s i over i 's pure strategies. Let S i = ∆(P i ) be the set of player i 's mixed strategies (where ∆( X ) denotes the set of probability distributions over a domain X ), and let S = × i ∈ N S i be the set of mixed strategy profiles. We overload the definition of utility function to accept mixed strategies as follows: u i ( s ) = ∑ ρ ∈ P (∏ i ∈ N s i ( ρ i ) ) u i ( ρ ) . We use ρ -i and s -i to denote a joint assignment of pure (resp. mixed) strategies to all players except for i , thus s = ( s i , s -i ) . We use P -i and S -i to denote the sets of all such assignments.

Hindsight Rationality The hindsight rationality framework (Morrill et al., 2021b) conceptualizes the goal of an agent as finding a strategy that minimizes regret with respect to a set of deviations Φ . A deviation ϕ ∈ Φ is a mapping ϕ : S i → S i that transforms a learner's strategy into some other strategy. Regret measures the amount the learner would prefer to deviate to ϕ ( s i ) : u i ( ϕ ( s i ) , s -i ) -u i ( s i , s -i ) . An agent is hindsight rational with respect to a set of deviations Φ if the agent does not have positive regret with respect to any deviation in Φ , i.e. ∀ ϕ ∈ Φ , u i ( ϕ ( s i ) , s -i ) -u i ( s i , s -i ) ≤ 0 . Let µ ∈ ∆(P) be a distribution over pure strategy profiles and (Φ i ) i ∈ N be a choice of deviation sets for each player.

Definition 2.1 ( ϵ -Mediated Equilibrium (Morrill et al., 2021b)) . We say m = ( µ, (Φ i ) i ∈ N ) is an ϵ -mediated equilibrium if ∀ i ∈ N,ϕ ∈ Φ i we have E ρ ∼ µ [ u i ( ϕ ( ρ i ) , ρ -i ) -u i ( ρ )] ≤ ϵ . A mediated equilibrium is a 0-mediated equilibrium.

Learning takes place in an online learning environment. At each iteration t , a learning agent i chooses a strategy s t i while all other agents choose a strategy profile s t -i . NoΦ -regret learning algorithms ensure that the maximum average positive regret tends to 0:

If all agents use a no-regret learning algorithms w.r.t. a set of deviations Φ i , the empirical distribution of play converges to a mediated equilibrium. Formally, let ˆ µ ∈ ∆(P) be the empirical distribution of play, where the weight on ρ ∈ P is ˆ µ ( ρ ) . = ∑ T t =1 (∏ i ∈ N s t i ( ρ i ) ) . As T →∞ , ˆ µ converges to µ of a mediated equilibrium ( µ, (Φ i ) i ∈ N ) .

̸

Different sets of deviations determine the strength of a mediated equilibrium. For normal-form games, the set of swap deviations, Φ SW , are all possible mappings ϕ : P i → P i . We may apply a swap deviation ϕ to a mixed strategy s i by taking its pushforward measure: [ ϕ ( s i )]( ρ i ) = ∑ ρ ′ i ∈ ϕ -1 ( ρ i ) s i ( ρ ′ i ) , where ϕ -1 ( ρ i ) = { ρ ′ i ∈ P i | ϕ ( ρ ′ i ) = ρ i } . The set of internal deviations Φ I , which replace a single pure strategy with another, offer the same strategic power as swap deviations (Foster & Vohra, 1999). Formally, Φ I = { ϕ ∈ Φ SW | ∃ ρ i , ρ ′ i : [ ϕ ( ρ i ) = ρ ′ i ] ∧ [ ∀ ρ ′′ i = ρ i , ϕ ( ρ ′′ i ) = ρ ′′ i ] } . The set of external deviations, Φ EX , is even more restricted: any ϕ ∈ Φ EX maps all (mixed) strategies to some particular pure strategy; i.e. Φ EX = { ϕ ∈ Φ SW | ∃ ρ i : ∀ ρ ′ i , ϕ ( ρ ′ i ) = ρ i } . The choice of (Φ i ) i ∈ N determines the nature of the mediated equilibrium-provided the learning algorithm for player i is noΦ i -regret (Greenwald et al., 2011). For example, if all players are hindsight rational w.r.t. Φ EX , then ˆ µ converges to the set of coarse correlated equilibria (CCE) (Moulin & Vial, 1978) and if all players are hindsight rational w.r.t. Φ I then ˆ µ converges to the set of correlated equilibria (Aumann, 1974).

A special case of mediated equilibria are Nash equilibria . If some mediated equilibrium m = ( µ, (Φ i ) i ∈ N ) is a product distribution (i.e. µ ( ρ ) = ∏ i ∈ N s i ( ρ ) ∀ ρ ∈ P ) and Φ i ⊇ Φ EX ∀ i ∈ N then µ is a Nash equilibrium. Similarly, an ϵ -mediated equilibrium is an ϵ -Nash equilibrium if µ is a product distribution and Φ i ⊇ Φ EX ∀ i ∈ N .

In sequential decision making scenarios (often modelled as extensive form games), the set of deviations is even more rich (Morrill et al., 2021b). All of these deviation classes-with the exception of action deviations (Selten, 1988) (which are so weak they do not even imply Nash equilibria in two-player constant-sum games, see appendix)-are stronger than external deviations. This means that the equilibria of any algorithm that minimizes regret w.r.t. a stronger class of deviations than external deviations still inherit all the properties of CCE (for example Hart & Mas-Colell (2000); Zinkevich et al. (2008); Celli et al. (2020); Steinberger et al. (2020); Morrill et al. (2021a)). Thus, we focus on CCE since the analysis generalizes broadly. Moreover, CCE can be computed efficiently, either analytically (Jiang & Leyton-Brown, 2011) or by a learning algorithm. When we refer to CCE, we use the distribution µ to refer to the CCE, since Φ is implicit.

3 Vulnerability

The choice of other agents during learning affects the strategy that is learned. Choosing which agents make good 'opponents" during training is an open research question (Lanctot et al., 2017; Marris et al., 2021). One common approach, self-play , is to have a learning algorithm train with copies of itself as the other agents. If the algorithm is a noΦ -regret algorithm, the learned behavior will converge to a mediated equilibrium; this gives a nice characterization of the convergence behavior of the algorithm.

However, in general the strategies in a mediated equilibrium are correlated with each other. This means that in order to deploy a strategy learned in self-play, an agent must first extract it by marginalizing out other agent's strategies. This new marginal strategy can then be played against new agents with whom the agent did not train (and thus correlate).

Definition 3.1 (Marginal strategy) . Given some mediated equilibrium ( µ, (Φ i ) N i =1 ) , let s µ i be the marginal strategy for i , where s µ i ( ρ i ) . = ∑ ρ -i ∈ P -i µ ( ρ i , ρ -i ) . Let s µ be a marginal strategy profile , where each ∀ i ∈ N plays s µ i .

Once a strategy has been extracted via marginalization, learning can either continue with the new agents (and potentially re-correlate), or the strategy can remain fixed. 2 We focus on the case where the strategy remains fixed. In doing so we can guarantee the performance of this strategy if learning stops, but also the show guarantees about the initial performance of a strategy that continues to learn; this is especially important in safety-critical domains.

Given a marginal strategy s µ i , we can bound its underperformance against new agents that behave differently from the (decorrelated) training opponents by a quantity which we call vulnerability .

Definition 3.2 (Vulnerability) . The vulnerability of a strategy profile s for player i with respect to S ′ -i ⊆ S -i is

Vulnerability gives a measure of how much worse s will perform with new agents compared to its training performance under pessimistic assumptions-that -i play the strategy profile in S ′ -i that is worst for i . We assume that -i are not able to correlate their strategies.

Thus, if a marginal strategy profile s µ is learned through self-play and Vul i ( s µ , S ′ -i ) is small, then s µ i performs roughly as well against new agents -i playing some strategy profile in S ′ -i . S ′ -i is used to encode assumptions about the strategies of opponents. S ′ -i = S -i means opponents could play any strategy, but we could also set S ′ -i to be the set of strategies learnable through self-play if we believe that opponents would also be using self-play as a training procedure.

Some games have properties that make the vulnerability low. For example, in two-player constantsum games the marginal strategies learned in self-play generalize well to new opponents since any Nash equilibrium strategy is also a maxmin strategy (von Neumann, 1928).

4 Guarantees via Polymatrix Decomposability

Multiplayer games are fundamentally more complex than two-player constant-sum games (Daskalakis &Papadimitriou, 2005; Daskalakis et al., 2009). However, certain multiplayer games can be decomposed into a graph of two-player games, where a player's payoffs depend only on their actions and the actions of players who are neighbors in the graph (Bergman & Fokin, 1998). In these polymatrix games (a subset of graphical games (Kearns et al., 2013)) Nash equilibria can be computed efficiently if player's utilities sum to a constant (Cai & Daskalakis, 2011; Cai et al., 2016).

Definition 4.1 (Polymatrix game) . A polymatrix game G = ( N,E, P , u ) consists of a set N of players, a set of edges E between players, a set of pure strategy profiles P , and a set of utility functions u = { u ij , u ji | ∀ ( i, j ) ∈ E } where u ij , u ji : P i × P j → R are utility functions associated with the edge ( i, j ) for players i and j , respectively.

We refer to the normal-form subgame between ( i, j ) as G ij = ( { i, j } , P i × P j , ( u ij , u ji )) . We use u i to denote the global utility function u i : P → R where u i ( ρ ) = ∑ ( i,j ) ∈ E u ij ( ρ i , ρ j ) for each player. We use E i ⊆ E to denote the set of edges where i is a player and | E i | to denote the number of such edges.

Definition 4.2 (Constant-sum polymatrix) . We say a polymatrix game G is constant-sum if for some constant c we have that ∀ ρ ∈ P , ∑ i ∈ N u i ( ρ ) = c .

Constant-sum polymatrix (CSP) games have the desirable property that all CCE factor into a product distribution; i.e., are Nash equilibria (Cai et al., 2016). We give a relaxed version:

Proposition 4.3. If µ is an ϵ -CCE of a CSP game G , s µ is an nϵ -Nash of G .

This means no-external-regret learning algorithms will converge to Nash equilibria, and thus do not require a mediator to enable the equilibrium. However, they do not necessarily have the property of two-player constant-sum games that all (marginal) equilibrium strategies are maxmin strategies (Cai et al., 2016). Thus Nash equilibrium strategies in CSP games have no vulnerability guarantees. Cai et al. (2016) show that CSP games that are constant sum in each subgame are no more or less general than CSP games that are constant sum globally, since there exists a payoff preserving transformation between the two sets. For this reason we focus on CSP games that are constant sum in each subgame without loss of generality. Note that the constant need not be the same in each subgame.

4.1 Vulnerability on a Simple Constant-Sum Polymatrix Game

We next demonstrate why CSP games do not have bounded vulnerability on their own without additional properties. Consider the simple 3-player CSP game called Offense-Defense (Figure 2a). There are 3 players: 0 , 1 and 2 . Players 1 and 2 have the option to either attack 0 ( a 0 ) or attack the other (e.g. a 1 ); player 0 , on the other hand, may either relax ( r ) or defend ( d ). If either 1 or 2 attacks the other while the other is attacking 0 , the attacker gets β and the other gets -β in that subgame. If both 1 and 2 attack 0 , 1 and 2 get 0 in their subgame and if they attack each other, their attacks cancel out and they both get 0 . If 0 plays d , they defend and will always get 0 . If they relax, they get -β if they are attacked and 0 otherwise. Offense-Defense is a CSP game, so any CCE is a Nash equilibrium.

Note that ρ = ( r, a 2 , a 1 ) is a Nash equilibrium. Each i ∈ { 1 , 2 } are attacking the other j ∈ { 1 , 2 } \ { i } , so has expected utility of 0 . Deviating to attacking 0 would leave them open against the other, so a 0 is not a profitable deviation, as it would also give utility 0 . Additionally, 0 has no incentive to deviate to d , since this would also give them a utility of 0 .

However, ρ is not a Nash equilibrium of the subgames-all i ∈ { 1 , 2 } have a profitable deviation in their subgame against 0 , which leaves 0 vulnerable in that subgame. If 1 and 2 were to both deviate to a 0 , and 0 continues to play their Nash equilibrium strategy of r , 0 would lose 2 β utility from their equilibrium value; in other words, the vulnerability of player 0 is 2 β .

4.2 Subgame Stability

However, some constant-sum polymatrix games do have have bounded vulnerability; we call these subgame stable games . In subgame stable games, global equilibria imply equilibria at each pairwise subgame.

Definition 4.4 (Subgame stable profile) . Let G be a polymatrix game with global utility functions ( u i ) i ∈ N . We say a strategy profile s is γ -subgame stable if ∀ ( i, j ) ∈ E , we have ( s i , s j ) is a γ -Nash of G ij ; that is u ij ( ρ i , s j ) -u ij ( s i , s j ) ≤ γ ∀ ρ i ∈ P i and u ji ( ρ j , s i ) -u ji ( s j , s i ) ≤ γ ∀ ρ j ∈ P j

For example, in Offense-Defense, ( r, a 2 , a 1 ) is β -subgame stable; it is a Nash equilibrium but is a β -Nash of the subgame between 0 and 1 and the subgame between 0 and 2 .

Definition 4.5 (Subgame stable game) . Let G be a polymatrix game. We say G is ( ϵ, γ ) -subgame stable if for any ϵ -Nash equilibrium s of G , s is γ -subgame stable.

Subgame stability connects the global behavior of play ( ϵ -Nash equilibrium in G ) to local behavior in a subgame ( γ -Nash in G ij ). If a polymatrix game is both constant-sum and is (0 , γ ) -subgame stable then we can bound the vulnerability of any marginal strategy.

Theorem 4.6. Let G be a CSP game. If G is (0 , γ ) -subgame stable, then for any player i ∈ N and CCE µ of G , we have Vul i ( s µ , S -i ) ≤ | E i | γ .

Theorem 4.6 tells us that using self-play to compute a marginal strategy s µ on constant-sum polymatrix games will have low vulnerability against worst-case opponents if γ is low. Thus, these are a set of multiplayer games where self-play is an effective training procedure.

Proof idea. Since G is a CSP game, s µ is a Nash equilbrium. Since G is (0 , γ ) -subgame stable, ( s µ i , s µ j ) is a γ -Nash equilibrium in each subgame, which bounds the vulnerability in each subgame. This is because

since players j = i can minimize i 's utility without coordinating, as G is a polymatrix game.

We give an algorithm for finding the minimum value of γ such that a CSP game is (0 , γ ) -subgame stable in Appendix D.1.

̸

0

1

2

1

a

0

a

2

2

a

0

0

-

β

a

1

β

0

2

a

0

a

1

0

d

0

0

r

-

β

0

1

a

0

a

2

0

d

0

0

r

-

β

0

(a)

2

d

0

...

1

...

2

0







0

0

0

0





f

c

1

f

c







0

β

-

β/

2

-

β/

2





f







0

-

β/

2

-

β/

2

β





c







0

-

β

β

0





f







0

-

β

β/

2

β/

2





c

(b)

4.3 Approximate Constant-Sum Polymatrix Games

Most games are not factorizable into CSP games. However, we can take any game G and project it into the space of CSP games.

Definition 4.7 ( δ -constant sum polymatrix) . Agame G is δ -constant sum polymatrix ( δ -CSP) if there exists a CSP game ˇ G with global utility function ˇ u such that ∀ i ∈ N,ρ ∈ P , | u i ( ρ ) -ˇ u i ( ρ ) | ≤ δ . Given G , we denote the set of such CSP games as CSP δ ( G ) .

Proposition 4.8. In a δ -CSP game G the following hold.

Any CCE of G is a 2 δ -CCE of any ˇ G ∈ CSP δ ( G ) .

The marginal strategy profile of any CCE of G is a 2 nδ -Nash equilibrium of any ˇ G ∈ CSP δ ( G ) .

The marginal strategy profile of any CCE of G is a 2( n +1) δ -Nash equilibrium of G .

From (3) we have that the removal of the mediator impacts players utilities by a bounded amount in δ -CSP games. We give a linear program in Appendix D.2 that will find the minimum δ such that G is δ -CSP and returns a CSP game ˇ G ∈ CSP δ ( G ) .

Combining δ -CSP with ( ϵ, γ ) -subgame stability lets us bound vulnerability in any game.

Theorem 4.9. If G is δ -CSP and ∃ ˇ G ∈ CSP δ ( G ) that is (2 nδ, γ ) -subgame stable and µ is a CCE of G , then

Theorem 4.9 shows that games which are close to the space of subgame stable CSP (SS-CSP) games are cases where the marginal strategies learned through self-play have bounded worst-case performance. This makes them suitable for any no-external-regret learning algorithm.

5 Vulnerability Against Other Self-Taught Agents

Theorem 4.9 bounds vulnerability in worst-case scenarios, where -i play any strategy profile to minimize i 's utility. In reality, however, each player j ∈ -i has their own interests and would only play a strategy that is reasonable under these own interests. In particular, what if each agent were also determining their own strategy via self-play in a separate training instance. How much utility can i guarantee themselves in this setup?

While no-external-regret learning algorithms converge to the set of CCE, other assumptions can be made with additional information about the type of regret being minimized. For example, no-externalregret learning algorithms will play strictly dominated strategies with vanishing probability and CFR will play dominated actions with vanishing probability (Gibson, 2014). These refinements can tighten our bounds, since the part of the game that no-regret learning algorithms converge to might be closer to a CSP game than the game overall.

Consider the game shown in Figure 2b, called 'Bad Card'. The game starts with each player except the dealer putting β/ 2 into the pot. A dealer player d -who always receives utility 0 regardless of the strategies of the other players-then selects a player from { 0 , 1 , 2 } to receive a 'bad card', while the other two players receive a 'good card'. The player who receives the bad card has an option to fold, after which the game ends and all players receive their ante back. Otherwise if this player calls, the other two players can either fold or call. The pot of β is divided among the players with good cards who call. If one player with a good card calls, they win the pot of β . If both good card players call then they split the pot. If both players with good cards fold, then the player with the bad card wins the pot.

As we shall soon show, Bad Card does not have a CSP decomposition-in fact it does not have any polymatrix decomposition. Since Bad Card is an extensive-form game without chance, each pure strategy profile leads to a single terminal history. Let P( z ) be the set of pure strategy profiles that play to a terminal z . In order for Bad Card to be polymatrix, we would need to find subgame utility functions such that ∀ ρ ∈ P , u 0 ( ρ ) = u 0 ,d ( ρ 0 , ρ d ) + u 0 , 1 ( ρ 0 , ρ 1 ) + u 0 , 2 ( ρ 0 , ρ 2 ) . Equivalently, we

could write ∀ z ∈ Z, ρ ∈ P( z ) , u 0 ( z ) = u 0 ,d ( ρ 0 , ρ d ) + u 0 , 1 ( ρ 0 , ρ 1 ) + u 0 , 2 ( ρ 0 , ρ 2 ) where Z is the set of terminals. A subset of these constraints results in an infeasible system of equations.

Consider the terminals in the subtree shown in Figure 2b: z 1 = (0 , c, c, c ) , z 2 = (0 , c, c, f ) , z 3 = (0 , c, f, c ) and z 4 = (0 , c, f, f ) . Let ρ c i be any pure strategy that plays c in this subtree and ρ f i be any strategy that plays f in this subtree for player i . In order for Bad Card to decompose into a polymatrix game we would need to solve the following infeasible system of linear equations:

Thus, Bad Card is not a CSP game, although it is a β -CSP game. However, if we prune out dominated actions (namely, those in which a player folds after receiving a good card), the resulting game is indeed a 0 -CSP game.

Let M ( A ) be the set of mediated equilibria than an algorithm A converges to in self-play. For example, if A is a no-external-regret algorithm, M ( A ) is the set of CCE without strictly dominated strategies in their support. Let S ( A ) . = { s µ | ( µ, (Φ i ) i ∈ N ) ∈ M ( A ) } be the set of marginal strategy profiles of M ( A ) , and let S i ( A ) . = { s i | s ∈ S ( A ) } be the set of i 's marginal strategies from S ( A ) .

Now, consider if each player i learns with their own self-play algorithm A i . Let A N . = ( A 1 , ... A n be the profile of learning algorithms, then let S × ( N ) . = S i ( i ) and S × ( N ) . =

j ∈-i S j ( A j ) . Summarizing, if each player learns with a noΦ i -regret learning algorithm A i , they will converge to the set of M ( A i ) equilibria. The set of marginal strategies from this set of equilibria is S i ( A i ) and the set of marginal strategy profiles is S ( A i ) . If each player plays a (potentially) different learning algorithm, S × ( A N ) is the set of possible joint match-ups if each player plays a marginal strategy from their own algorithm's set of equilibria and S × -i ( A N ) is the set of profiles for -i .

) A × i ∈ N A -i A ×

Definition 5.1. We say a game G is δ -CSP in the neighborhood of S ′ ⊆ S if there exists a CSP game ˇ G such that ∀ s ∈ S ′ we have | u i ( s ) -ˇ u i ( s ) | ≤ δ . We denote the set of such CSP games as CSP δ ( G,S ′ ) .

Definition 5.2. We say a polymatrix game G is γ -subgame stable in the neighborhood of S ′ if ∀ s ∈ S ′ , ∀ ( i, j ) ∈ E we have that ( s i , s j ) is a γ -Nash of G ij .

These definitions allow us to prove the following generalization of Theorem 4.9.

Theorem 5.3. For any i ∈ N , if G is δ -CSP in the neighborhood of S × ( A N ) and ∃ ˇ G ∈ CSP δ ( G,S × ( A N )) that is γ -subgame stable in the the neighborhood of S ( A i ) , then for any s ∈ S ( A i )

An implication of Theorem 5.3 is that if agents use self-play to compute a marginal strategy from some mediated equilibrium and there is an SS-CSP game that is close to the original game for these strategies, then this is sufficient to bound vulnerability against strategies learned in self-play.

6 Computing an SS-CSP Decomposition in a Neighborhood

How might one determine if a game is well-approximated by an SS-CSP game? In addition to the algorithms presented in Appendix D, we give an algorithm, SGDecompose , that finds an SS-CSP decomposition for a game in a given neighborhood of strategy profiles. Pseudocode is given in Algorithm 1. SGDecompose could be used to test whether a game is well-approximated by an SS-CSP game before potentially analytically showing this property holds. We will use this algorithm in the following section to decompose Leduc poker.

As input, SGDecompose receives a neighborhood of strategies S ′ and the set of match-ups between strategies in S ′ , given by S × . = × i ∈ N S ′ i . The idea is to compute a CSP game ˇ G that minimizes

a loss function with two components: how close ˇ G is to G in the neighborhood of S × and how subgame stable ˇ G is for the neighborhood of S ′ . First, L δ is the error between the utility functions of G and ˇ G ( u and ˇ u , respectively); it is a proxy for δ in δ -CSP. The loss for a single strategy profile s is

The other component of the overall loss function, L γ , measures the subgame stability. First, we define L γ ij , which only applies to a single subgame. Let s ij = ( s i , s j ) be a profile for a subgame and s ∗ ij = ( s ∗ i , s ∗ j ) is a profile of deviations for that subgame. The L γ ij loss for this subgame is

Then, given a strategy profile s and deviation profile s ∗ for all players N , we have

SGDecompose repeats over a number of epoches T . At the start of an epoch, we compute a bestresponse (for example, via sequence-form linear programming in extensive-form games) to each strategy s ′ i in S ′ in each subgame; the full process is shown in Algorithm 4 in the appendix. After computing these best-responses for the current utility function of ˇ G , SGDecompose fits ˇ u to be nearly CSP in the neighborhood of S × and subgame stable in the neighborhood of S ′ . Since S × is exponentially larger than S ′ , we partition it into batches, then use batch gradient descent. 3

We use the following batch loss function, which computes the average values of L δ and L γ over the batch then weights the losses with λ . Let S b denote a batch of strategy profiles from S × with size B ,

We use this loss function to update ˇ u , which is guaranteed to a be a valid utility function for a CSP game via its representation, see Appendix F for details. In Appendix F, we give the procedure in terms of a more efficient representation of polymatrix decompositions for extensive-form games, which we call poly-EFGs ; which we describe in Appendix E.

Algorithm 1 SGDecompose

̸

7 Experiments

Approaches using regret-minimization in self-play have been shown to outperform expert human players in some multiplayer games, the most notable example being multiplayer no-limit Texas hold 'em (Brown & Sandholm, 2019), despite no formal guarantees.

Conjecture 7.1. Self-play with regret minimization performs well in multiplayer Texas hold 'em because 'good" players (whether professional players or strategies learned by self-play) play in a part of the games' strategy space that is close to an SS-CSP game (i.e. low values of γ, δ ).

While multiplayer no-limit Texas hold 'em is too large to directly check the properties developed in this work, we use a smaller poker game, called Leduc poker (Southey et al., 2012), to suggest why regret-minimization 'works" in multiplayer Texas hold 'em. Leduc poker was originally developed for two players but was extended to a 3-player variant by Abou Risk & Szafron (2010); we use the 3-player variant here. The game has 8 cards, two rounds of betting, one private and one public card.

We first use a self-play algorithm to learn strategies, then use SGDecompose to see if this part of Leduc Poker is close to an SS-CSP game. We give a summary of results here, please refer to Appendix G for full details. We use CFR+ (Tammelin, 2014; Tammelin et al., 2015) as a self-play algorithm to compute a set of approximate marginal strategies. 4 CFR+ was chosen because of its improved efficiency over CFR. CFR+ is a deterministic algorithm, so we use different random initializations of CFR+'s initial strategy in order to generate a set of CCE. We will use 30 runs of CFR+ as input to SGDecompose; and have 30 runs of SGDecompose (i.e. we trained CFR+ 900 times in total in self-play). This will give us a value of δ and γ for each run.

We found that Leduc poker was well-approximated by an SS-CSP game in the neighborhood of strategies learned by CFR+. In particular, across runs, Leduc poker was on average (with standard errors) δ = 0 . 009 ± 0 . 00046 -CSP and γ = 0 . 004 ± 0 . 00016 -subgame stable in the neighborhood of CFR+-learned strategies. How well do these values bound vulnerability with respect to other CFR -learned strategies? For each of the runs, we computed the vulnerability with respect to the strategies of that run, by evaluating each strategy against each other and taking the maximum vulnerability. We compare these values to the upper bounds implies by the values of δ and γ for each run and Theorem 5.3. We found the computed values of δ and γ do a good job of upper bounding the vulnerability. Across the runs, the bounds are at minimum 1 . 89 times the vulnerability, at maximum 3 . 05 times the vulnerability and on average 2 . 51 times as large, with a standard error of 0 . 049 .

We repeated these experiments with a toy hanabi game-where strategies learned in self-play are highly vulnerable-which we found to have much higher values of δ and γ ; details are in Appendix H.

It was previously believed that CFR does not compute an ϵ -Nash equilibrium on 3-player Leduc for any reasonable value of ϵ . Abou Risk & Szafron (2010) found CFR computed a 0 . 130 -Nash equilibrium. We found that CFR+ always computed an approximate Nash equilbrium with ϵ ≤ 0 . 013 . Appendix G.2 shows that CFR also computes an approximate Nash equilbrium with ϵ ≤ 0 . 017 .

8 Conclusion

Self-play has been incredibly successful in producing strategies that perform well against new opponents in two-player constant-sum games. Despite a lack of theoretical guarantees, self-play seems to also produce good strategies in some multiplayer games (Brown & Sandholm, 2019). We identify a structural property of multiplayer, general-sum game that allow us to establish guarantees on the performance of strategies learned via self-play against new opponents. We show that any game can be projected into the space of constant-sum polymatrix games, and if there exists a game with this set with high subgame stability (low γ ), strategies learned through self-play have bounded loss of performance against new opponents.

We conjecture that Texas hold 'em is one such game. We investigate this claim on Leduc poker, and find that CFR+ plays strategies from a part of the strategy space in Leduc poker that is wellapproximated by a subgame stable constant-sum polymatrix game. This work lays the groundwork for guarantees for self-play in multiplayer games. However, there is room for algorithmic improvement and efficiency gains for checking these properties in very large extensive-form games.

References

Abou Risk, N. and Szafron, D. Using counterfactual regret minimization to create competitive multiplayer poker agents. International Conference on Autonomous Agents and Multiagent Systems , 2010.

Aumann, R. J. Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics , 1(1):67-96, 1974.

Bakhtin, A., Wu, D., Lerer, A., and Brown, N. No-press diplomacy from scratch. Advances in Neural Information Processing Systems , 34:18063-18074, 2021.

Bakhtin, A., Wu, D. J., Lerer, A., Gray, J., Jacob, A. P., Farina, G., Miller, A. H., and Brown, N. Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning. arXiv preprint arXiv:2210.05492 , 2022.

Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., Parisotto, E., Dumoulin, V ., Moitra, S., Hughes, E., Dunning, I., Mourad, S., Larochelle, H., Bellemare, M. G., and Bowling, M. The hanabi challenge: A new frontier for ai research. Artificial Intelligence , 2020.

Bergman, L. and Fokin, I. On separable non-cooperative zero-sum games. Optimization , 44(1): 69-84, 1998.

Brown, N. and Sandholm, T. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Brown, N. and Sandholm, T. Superhuman AI for multiplayer poker. Science , 365(6456):885-890, 2019.

Cai, Y. and Daskalakis, C. On minmax theorems for multiplayer games. ACM-SIAM Symposium on Discrete algorithms , 2011.

Cai, Y., Candogan, O., Daskalakis, C., and Papadimitriou, C. Zero-sum polymatrix games: A generalization of minmax. Mathematics of Operations Research , 41(2):648-655, 2016.

Celli, A., Marchesi, A., Farina, G., and Gatti, N. No-regret learning dynamics for extensive-form correlated equilibrium. Neural Information Processing Systems , 2020.

Chen, Y., Deng, X., Li, C., Mguni, D., Wang, J., Yan, X., and Yang, Y. On the convergence of fictitious play: A decomposition approach. International Joint Conferences on Artificial Intelligence , 2022.

Cheung, Y. K. and Tao, Y. Chaos of learning beyond zero-sum and coordination via game decompositions. arXiv preprint arXiv:2008.00540 , 2020.

Daskalakis, C. and Papadimitriou, C. H. Three-player games are hard. Electron. Colloquium Comput. Complex. , 2005.

Daskalakis, C., Goldberg, P. W., and Papadimitriou, C. H. The complexity of computing a nash equilibrium. SIAM Journal on Computing , 39(1):195-259, 2009.

Farina, G., Ling, C. K., Fang, F., and Sandholm, T. Correlation in extensive-form games: Saddle-point formulation and benchmarks. Neural Information Processing Systems , 2019.

Farina, G., Bianchi, T., and Sandholm, T. Coarse correlation in extensive-form games. AAAI conference on Artificial Intelligence , 2020.

Foster, D. P. and Vohra, R. Regret in the on-line decision problem. Games and Economic Behavior , 29(1):7-35, 1999.

Gibson, R. G. Regret minimization in games and the development of champion multiplayer computer poker-playing agents. Ph.D. Thesis , 2014.

Greenwald, A., Jafari, A., and Marks, C. Noϕ -regret: A connection between computational learning theory and game theory. Games, Norms and Reasons: Logic at the Crossroads , 2011.

Hart, S. and Mas-Colell, A. A simple adaptive procedure leading to correlated equilibrium. Econometrica , 68(5):1127-1150, 2000.

Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. 'other-play' for zero-shot coordination. International Conference on Machine Learning , 2020.

Jiang, A. X. and Leyton-Brown, K. A general framework for computing optimal correlated equilibria in compact games. Internet and Network Economics: 7th International Workshop, WINE 2011 , 2011.

Jin, C., Liu, Q., Wang, Y., and Yu, T. V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 , 2021.

Kearns, M., Littman, M. L., and Singh, S. Graphical models for game theory, 2013.

Kuhn, H. W. Extensive games and the problem of information. Contributions to the Theory of Games , 2(28):193-216, 1953.

Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Perolat, J., Silver, D., and Graepel, T. A unified game-theoretic approach to multiagent reinforcement learning. Neural Information Processing Systems , 2017.

Lanctot, M., Lockhart, E., Lespiau, J.-B., Zambaldi, V., Upadhyay, S., Pérolat, J., Srinivasan, S., Timbers, F., Tuyls, K., Omidshafiei, S., et al. Openspiel: A framework for reinforcement learning in games. arXiv preprint arXiv:1908.09453 , 2019.

Liu, Q., Yu, T., Bai, Y., and Jin, C. A sharp analysis of model-based reinforcement learning with self-play. International Conference on Machine Learning , 2021.

Marris, L., Muller, P., Lanctot, M., Tuyls, K., and Graepel, T. Multi-agent training beyond zero-sum with correlated equilibrium meta-solvers. International Conference on Machine Learning , 2021.

Matignon, L., Laurent, G. J., and Le Fort-Piat, N. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review , 27(1):1-31, 2012.

Moravcik, M., Schmid, M., Burch, N., Lisy, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , 356(6337):508-513, 2017.

Morrill, D., D'Orazio, R., Lanctot, M., Wright, J. R., Bowling, M., and Greenwald, A. R. Efficient deviation types and learning for hindsight rationality in extensive-form games. International Conference on Machine Learning , 2021a.

Morrill, D., D'Orazio, R., Sarfati, R., Lanctot, M., Wright, J. R., Greenwald, A. R., and Bowling, M. Hindsight and sequential rationality of correlated play. AAAI Conference on Artificial Intelligence , 2021b.

Moulin, H. and Vial, J. Strategically zero-sum games: The class of games whose completely mixed equilibria cannot be improved upon. International Journal of Game Theory , 7(3):201-221, 1978.

Paquette, P., Lu, Y., Bocco, S. S., Smith, M., O-G, S., Kummerfeld, J. K., Pineau, J., Singh, S., and Courville, A. C. No-press diplomacy: Modeling multi-agent gameplay. Neural Information Processing Systems , 2019.

Perolat, J., Vylder, B. D., Hennes, D., Tarassov, E., Strub, F., de Boer, V., Muller, P., Connor, J. T., Burch, N., Anthony, T., McAleer, S., Elie, R., Cen, S. H., Wang, Z., Gruslys, A., Malysheva, A., Khan, M., Ozair, S., Timbers, F., Pohlen, T., Eccles, T., Rowland, M., Lanctot, M., Lespiau, J.-B., Piot, B., Omidshafiei, S., Lockhart, E., Sifre, L., Beauguerlange, N., Munos, R., Silver, D., Singh, S., Hassabis, D., and Tuyls, K. Mastering the game of stratego with model-free multiagent reinforcement learning. Science , 378(6623):990-996, 2022.

Selten, R. Reexamination of the perfectness concept for equilibrium points in extensive games. In Models of Strategic Rationality , pp. 1-31. Springer, 1988.

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science , 362(6419): 1140-1144, 2018.

Southey, F., Bowling, M. P., Larson, B., Piccione, C., Burch, N., Billings, D., and Rayner, C. Bayes' bluff: Opponent modelling in poker. arXiv preprint arXiv:1207.1411 , 2012.

Steinberger, E., Lerer, A., and Brown, N. Dream: Deep regret minimization with advantage baselines and model-free learning. arXiv preprint arXiv:2006.10410 , 2020.

Tammelin, O. Solving large imperfect information games using cfr+. arXiv preprint arXiv:1407.5042 , 2014.

Tammelin, O., Burch, N., Johanson, M., and Bowling, M. Solving heads-up limit texas hold'em. In Twenty-fourth international joint conference on artificial intelligence , 2015.

Tuyls, K., Pérolat, J., Lanctot, M., Ostrovski, G., Savani, R., Leibo, J. Z., Ord, T., Graepel, T., and Legg, S. Symmetric decomposition of asymmetric games. Scientific reports , 8(1):1-20, 2018.

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V ., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gülçehre, Ç., Wang, Z., Pfaff, T., Wu, Y ., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

von Neumann, J. Zur theorie der gesellschaftsspiele. Mathematische annalen , 100(1):295-320, 1928.

Von Stengel, B. and Forges, F. Extensive-form correlated equilibrium: Definition and computational complexity. Mathematics of Operations Research , 33(4):1002-1022, 2008.

Zinkevich, M., Johanson, M., Bowling, M., and Piccione, C. Regret minimization in games with incomplete information. Neural Information Processing Systems , 2008.

A Marginals of a CCE May Not Be a CCE

a

b

a

1

,

0

-

1

,

-

1

a

1

,

1

0

,

1

-

-

Here we give an example showing the marginal strategies of a CCE may not form a CCE. Consider µ s.t. µ ( a, a ) = 0 . 5 and µ ( b, b ) = 0 . 5 . µ is a CCE. E ρ ∼ µ [ u i ( ρ )] = 0 . 5 for each player. If row ( r ) were to player a and column continues to play according to µ , row's utility is 0 ; if r plays b instead, their utility is now -0 . 5 . Thus r has no profitable deviations from the CCE recommendations. Column does not either, this can be shown with a symmetric argument.

Row's marginal strategy s µ r plays a with probability 0 . 5 and b with probability 0 . 5 , s µ c does likewise. u r ( s µ r , s µ c ) = u c ( s µ r , s µ c ) = -0 . 25 . However, a is a profitable deviation for r now since 0 > -0 . 25 , thus the decorrelated strategies from the same CCE are also not a CCE.

B Hindsight Rationality With Respect to Action Deviations Does Not Imply Nash

Here we show that hindsight rationality with respect to action deviations does not imply Nash equilibrium in 2 player constant-sum games. We show this with a 1 player game. Consider the agents strategy, shown in blue, which receives utility of 1. Deviating to [ I 1 : b, I 2 : a ] will increase the player's utility to 2, so the blue strategy is not a Nash equilibrium. However, this would require two simultaneous action deviations, one at I 1 to b and one at I 2 to a . Neither of these deviations increases the player's utility on their own, so the player is hindsight rational w.r.t. action deviations.

I

1

1

a

b

I

2

2

a

0

b

C Omitted Proofs

Proposition C.1 (Cai et al. (2016)) . In CSP games, for any CCE µ , if i deviates to s i , then their expected utility if other players continue to play µ is equal to their utility if other player where to play the marginal strategy profile s µ -i :

C.1 Proof of Proposition 4.3

Proposition 4.3. If µ is an ϵ -CCE of a CSP game G , s µ is an nϵ -Nash of G .

Proof. Since µ is an ϵ -CCE, ∀ i ∈ N , we have

which implies (by Proposition C.1) that ∀ i ∈ N ,

Summing over N , we get

Where (4) and (5) use the fact that ∀ ρ ∈ P , ∑ i ∈ N u i ( ρ ) = c for some constant. The above inequalities give us

All terms in the sum are are non-negative because ρ ′ i is a best-response to s µ -i . Then any particular term in the summation is upper bounded by nϵ .

C.2 Proof of Theorem 4.6

Proposition C.3. In two-player constant-sum games, for any ϵ -Nash equilibrium s and player i , we have

The proof of Proposition C.3 is immediate from the fact that no player can gain more than ϵ utility by deviating and the game is constant-sum.

Theorem 4.6. Let G be a CSP game. If G is (0 , γ ) -subgame stable, then for any player i ∈ N and CCE µ of G , we have Vul i ( s µ , S -i ) ≤ | E i | γ .

Proof. First we show 1. Any marginal strategy s µ of a CCE µ is a Nash equilibrium of G Cai et al. (2016). Then,

Rearranging, we get

Where the last line uses the fact that -i minimize i ′ s utility, so can do so without coordinating since G is polymatrix. Continuing,

where by (0 , γ ) -subgame stability of each G ij , ( s µ i , s µ i ) is a γ -Nash of G ij . By Proposition C.3, we have u ij ( s µ i , s µ j ) -min s j ∈ S j u i ( s µ i , s j ) ≤ γ .

C.3 Proof of Proposition 4.8

Proposition 4.8. In a δ -CSP game G the following hold

Any CCE of G is a 2 δ -CCE of any ˇ G ∈ CSP δ ( G ) .

The marginalized strategy profile of any CCE of G is a 2 nδ -Nash equilibrium of any ˇ G ∈ CSP δ ( G ) .

The marginalized strategy profile of any CCE is a 2( n +1) δ -Nash equilibrium of G

Proof. First we prove claim 1. Let ˇ u i denote the utility function of i in ˇ G . Note that ∀ ρ ∈ P we have | ˇ u i ( ρ ) -u i ( ρ ) | ≤ δ ∀ i ∈ N . Let µ be any CCE of G . The definition of CCE states

It is sufficient to consider only player i . We can preserve the inequality by substituting ˇ u i ( ρ ′ i , ρ -i ) -δ in place of u i ( ρ ′ i , ρ -i ) and ˇ u i ( ρ ) + δ in place of u i ( ρ ) . This gives us

Thus claim 1 is shown. Claim 2 is an immediate corollary of claim 1 and Proposition 4.3. Lastly, we show claim 3. By claim 2, we have the marginalized strategy of µ , s µ , is a 2 nδ -Nash equilibrium of ˇ G ∈ CSP δ ( G ) . That is for any i ∈ N ,

However, since G is δ -CSP, we may substitute u i ( ρ ′ i , s µ -i ) -δ in place of ˇ u i ( ρ ′ i , s µ -i ) and u i ( s µ i , s µ -i )+ δ in place of ˇ u i ( s µ i , s µ -i ) as preserve the inequality.

Rearranging, this gives us

C.4 Proof of Theorem 4.9

Theorem 4.9. If G is δ -CSP and ∃ ˇ G ∈ CSP δ ( G ) that is (2 nδ, γ ) -subgame stable and µ is a CCE of G , then

The proof is largely the same as Theorem 4.6, with added approximation since G is no longer CSP.

Proof. Let ˇ G be a polymatrix game that is (2 nδ, γ ) -subgame stable such that ˇ G ∈ CSP δ ( G ) . Let ˇ u i denote the utility function of i in ˇ G . By Proposition 4.8, µ is a 2 nδ -Nash equilibrium of ˇ G . Then,

since G is δ -CSP. Then expanding ˇ u i across i 's subgames we have

Where, as in Theorem 4.6, the last line uses the fact that ˇ G is polymatrix, G ij is constant-sum and -i minimize i ′ s utility and can do so by without coordinating. Continuing, we have

Where by (2 nδ, γ ) -subgame stability of each G ij , ( s µ i , s µ i ) is a γ -Nash of G ij . By Proposition C.3, s µ i can lose at most γ to a worst case opponent s ′ j in each subgame, since ˇ G ij is two-player constantsum.

C.5 Proof of Theorem 5.3

Theorem 5.3. If G is δ -CSP in the neighborhood of S × ( A N ) and ∃ ˇ G ∈ CSP δ ( G,S × ( A N )) that is γ -subgame stable in the the neighborhood of S ( A i ) , then for any s ∈ S ( A i )

Proof. The proof is very similar to Theorem 4.9. Writing the definition of vulnerability we have

since G is δ -CSP in the neighborhood of S × ( A N ) . Swapping out the utility of u i for ˇ u , we have

Since ˇ G is a polymatrix game,

Where, as in Theorem 4.6 and Theorem 4.9, the last line uses the fact that ˇ G is polymatrix, G ij is constant-sum and -i minimize i ′ s utility and can do so by without coordinating.

Since ˇ G is γ -subgame stable in the neighborhood of S ( A i ) and s ∈ S ( A i ) , then means ( s i , s j ) is a γ -Nash for each subgame ˇ G ij , so has bounded vulnerability within that subgame.

D Normal-Form Algorithms

D.1 Computing Subgame Stability

Let γ be the minimum γ such that a CSP game G is (0 , γ ) -subgame stable. How do we compute γ ? Does it involve computing all equilibria of G and checking their subgame stability? The answer is no, it can be done in polynomial time in the number of pure strategies. We next provide an algorithm for computing γ . The algorithm involves solving a linear program for each edge in the graph and each pure strategy of those players. This linear program takes a pure strategy ρ ′ i , and finds a Nash equilibrium of G that maximizes i 's incentive to deviate to ρ ′ i when only considering their utility in G ij ; call this quantity γ ρ ′ i ij . If there are no such Nash equilibria the solver returns 'infeasible'. If the solver does not return 'infeasible', we update γ = max ( i,j ) ∈ E i max ρ ′ i ∈ P i γ ρ ′ i ij .

Algorithm 2 Compute

Let a i ( ρ ′ i , µ ) be the advantage of deviating to ρ ′ i from a joint distribution over pure strategies:

Note that ( a ) is a linear function of µ , since s µ j is a marginal strategy. ( b ) is also a linear function of µ , and so a i ( ρ ′ i , µ ) is a linear function of µ . Likewise, let

be the advantage of ρ ′ i in the subgame between i and j . LP1 ( i, j, ρ ′ i ) is given below. The decision variables are the weights of µ for each ρ ∈ P and γ ρ ′ i ij .

We can get away with computing a CCE rather than targeting Nash equilibria because the marginals of any CCE are Nash equilibria in CSP games Cai et al. (2016).

The whole procedure runs in polynomial time in the size of the game. We need to solve an LP, which takes polynomial time, at most n 2 max i ∈ N | P i | times.

D.2 Finding Constant-Sum Polymatrix Decomposition

Projecting a game into the space of CSP games with minimum δ can be done with a linear program. Let δ be the minimum δ such that G is δ -CSP. We give a linear program that finds δ and returns a CSP game ˇ G ∈ CSP δ ( G ) . The decision variables are the values of ˇ u ij ( ρ ) for all i = j ∈ N,ρ ∈ P , δ and constants for each subgame c ij , for all i = j .

LP 2

E Extensive-Form Games

Section 4 developed notions of approximately CSP and subgame stable in the context of normal-form games. Here, we apply these concepts to extensive-form games. While any extensive-form game has an equivalent induced normal-form game, analysing properties of an EFG through its induced normal is intractable for moderately-sized EFGs, since the size the normal-form representation is exponentially larger.

After background on extensive-form games, we introduce a novel 'extensive-form version' of normalform polymatrix games, which we call poly-EFGs . The major benefit of poly-EFGs over normal-form polymatrix games is their efficiently: poly-EFGs are exponentially more compact than an equivalent normal-form polymatrix game. The results of this section extend the theory of Sections 4 and 5 using this more efficient representation. We also give a proof-of-concept showing that poly-EFGs can be used to efficiently decompose extensive-form games by giving an algorithm for decomposing a perfect information EFG into a poly-EFG.

̸

̸

̸

E.1 Background on Extensive-Form Games

We use the imperfect information extensive-form game (EFG) as a model for sequential multi-agent strategic situations. An imperfect information extensive-form game is a 10-tuple ( N, A , H, Z, A, P, u, I , c, π c ) where N is a set of players; A is a set of actions; H is a set of sequences of actions, called histories ; Z ⊆ H is a set of terminal histories; A : H →A is a function that maps a history to available actions; P : H → N is the player function, which assigns a player to choose an action at each non-terminal history; u = { u i } i ∈ N is a set of utility functions where u i : Z → R is the utility function for player i ; I = {I i } i ∈ N where I i is a partition of the set { h ∈ H : P ( h ) = i } such that if h, h ′ ∈ I ∈ I i then A ( h ) = A ( h ′ ) . We call an element I ∈ I i an information set. The chance player c has a function π c ( a, h ) ∀ h : P ( h ) = c which returns the probability of random nature events a ∈ A . Let N c = N ∪{ c } be the set of players including chance.

For some history h , the j th action in h is written h j . A sub-history of h from the j th to k th actions is denoted h j : k and we use h : k as a short-hand for h 0: k . If a history h ′ is a prefix of history h , we write h ′ ⊑ h and if h ′ is a proper prefix of h , we write h ′ ⊏ h .

A pure strategy in an EFG is a deterministic choice of actions for the player at every decision point. We use ρ i : I i → A to denote a pure strategy of player i , and the set of all pure strategies as P i . Likewise, s i ∈ ∆(P i ) = S i is a mixed strategy , where ∆( X ) denotes the set of probability distributions over a domain X .

There are an exponential number of pure strategies in the number of information sets. A behavior strategy is a compact representation of the behavior of an agent that assigns a probability distribution over actions to each information set. We use π i ∈ Π i = (∆( A ( I ))) I ∈I i to denote a behavior strategy of player i and π i ( a, I ) as the probability of playing action a at I . Let I ( h ) be the unique information set such that h ∈ I . We overload π i ( a, h ) = π i ( a, I ( h )) . We use ρ ∈ P , s ∈ S and π ∈ Π to denote pure, mixed and behavior strategy profiles, respectively. Note that P is a subset of both Π and S .

Given a behavior strategy profile, let

be the probability of transitioning from history h 1 to h 2 according to π i , π and π -i , respectively. Let p i ( z, π i ) , p -i ( z, π i ) and p ( z, π i ) be short-hands for p i ( ∅ , z, π i ) , p -i ( ∅ , z, π -i ) and p ( ∅ , z, π ) , respectively, where ∅ is the empty history.

We define the utility of a behavior strategy as:

Perfect recall is a common assumption made on the structure of information sets in EFGs that prevents players from forgetting information they once possessed. Formally, for any h ∈ I let X i ( h ) denote the set of ( I, a ) s.t. I ∈ I i and ∃ h ′ ∈ I and h ′ a ⊑ h . Let X -i ( h ) be defined analogously for -i and X ( h ) for all players.

Definition E.1 (Perfect recall) . If ∀ I ∈ I i , ∀ h, h ′ ∈ I, X i ( h ) = X i ( h ′ ) then i has perfect recall. If all players possess perfect recall in some EFG G , we call G a game of perfect recall.

In games of perfect recall, the set of behavior strategies and mixed strategies are equivalent: any behavior strategy can be converted into a mixed strategy which is outcome equivalent over the set of terminal histories (i.e. has the same distribution over Z ) and vice-versa.

Theorem E.2 (Kuhn (1953)) . In games of perfect recall, any behavior strategy π i has an equivalent mixed strategy s i (and vice versa), such that

Theorem E.2 establishes a connection between equilibria in behavior and mixed strategies: a Nash equilibrium behavior strategy profile implies the equivalent mixed strategy profile is also a Nash equilibrium in mixed strategies and vice-versa.

We may also reduce any extensive-form game into an equivalent normal-form game.

Definition E.3 (Induced normal-form) . The induced normal-form of an extensive-form game G (with utility functions u i ) is a normal-form game G ′ = ( N, P , u ′ ) such that u ′ i ( ρ ) = u i ( ρ )

The induced normal-form of an EFG has players making all decisions up-front. It is not always practical to construct an induced normal-form, but the concept is useful for proving things about EFGs.

E.2 Poly-EFGs

What is the appropriate extension of polymatrix games to EFGs? Given some n -player EFG G , what should the subgame between i and j be in the graph? Unlike in normal form games, players act sequentially, and i and j may either observe actions or have their utility impacted by other players. There is additional structure present in EFGs beyond the players' pure strategies.

The approach we take is to have an EFG for each pair of players i and j . For simplicity, we assume that each subgame G ′ ij shares the same structure as some n -player game G , but information sets where P ( I ) / ∈ { i, j } now belong to the chance player c .

Definition E.4 (Subgame) . Let G = ( N, A , H, Z, A, P, u, I , c, π c ) be some EFG. We define a subgame G ′ ij = ( { i, j } , A , H, Z, A, P ′ , ( u ′ ij , u ′ ji ) , I , c, π ′ c ) as a structurally identical game to G between i and j with player function P ′ and utility functions ( u ′ ij , u ′ ji ) , then

and let π ′ c be the strategy of the chance player in G ′ ij . We put no restrictions on π ′ c .

Note that u ′ ij , u ′ ji are not necessarily defined in the above definition. They may take any values. We merely want the subgame G ′ ij to share the structure of G . In G ′ ij , i and j 's utility only depends on their strategies π i , π j and chance's actions π ′ c :

What should π ′ c be defined as? This turns out to not matter very much. Given any subgame G ′ ij with chance strategy π ′ c and utility functions u ′ ij , u ′ ji , for any G ′′ ij with chance strategy π ′′ c , we can find u ′′ ij , u ′′ ji so that the utility of players between the two games will always be equal for any strategy profile ( π i , π j ) .

Definition E.5. We say π c is fully mixed if π ′ c ( a, h ) > 0 ∀ h ∈ { h ∈ H | P ( h ) = c } , a ∈ A ( h )

Proposition E.6. Let G be an EFG and G ′ ij a subgame between i and j with utility functions u ′ ij , u ′ ji and fully mixed chance strategy π ′ c . Given G ′′ ij , a subgame between i and j with fully mixed chance strategy π ′′ c , we may find u ′′ ij , u ′′ ji such that ∀ π i , π j we have u ′ ij ( π i , π j ) = u ′′ ij ( π i , π j ) and u ′ ji ( π i , π j ) = u ′′ ji ( π i , π j ) .

̸

Proof. Note that p c ( z, π ′ c ) , p c ( z, π ′′ c ) = 0 . Then ∀ z ∈ Z , define u ′′ ij ( z ) = p c ( z,π ′ c ) p c ( z,π ′′ c ) u ′ ij ( z ) and u ′′ ji ( z ) = p c ( z,π ′ c ) p c ( z,π ′′ c ) u ′ ji ( z ) . Then

For the remainder of this work, when defining a subgame between i and j given an EFG G , we define the subgame chance player's strategy to be equal to π c in G at information sets I where P ( I ) = c in G and uniform randomly otherwise.

Having defined subgames, we may now define our representation of extensive-form polymatrix games.

Definition E.7 (Poly-EFG) . A poly-EFG ( N,E, G ) is defined by a graph with nodes N , one for each player, a set of edges E and a set of games G = { G ij | ∀ ( i, j ) ∈ E } where G ij ∈ G is a two player EFG between i and j and all G ij ∈ G are a subgame between i and j and all subgames are defined with respect to some EFG G .

Let G ij ∈ G denote the subgame between i and j . We use subscript ij (e.g. Z ij ) to refer to parts of G ij .

Since each subgame of the poly-EFG is the subgame of the same G , the space of pure, mixed and behavior strategies is the same for each subgame for any player. A player chooses a strategy (whether pure, mixed or behavior) and plays this strategy in each subgame. A player's global utility is the sum of their utility in each of their subgames. We have

where E i ⊆ E is the set of edges connected to i in the graph and u ij is the utility function of i in subgame G ij .

E.3 Constant-Sum and Subgame Stable Poly-EFGs

Here, we give definitions of constant-sum and subgame stability for poly-EFGs. These definitions are largely identical to their normal-form counterparts, we merely provide them here for completeness.

Poly-EFGs are constant-sum if, for each subgame, the utilities at the terminals add up to a constant.

Definition E.8 (Constant-sum) . We say a poly-EFG G is constant-sum if ∀ G ij ∈ G , z ∈ Z ij , u ij ( z ) + u ji ( z ) = c ij where Z ij is the set of terminal histories of ∀ G ij and c ij is a constant.

We may also define approximate poly-EFGs in the same way as in normal-form games.

Definition E.9 ( δ -constant sum poly-EFG) . An EFG G is δ -constant sum poly-EFG ( δ -CSP) if there exists a constant-sum poly-EFG ˇ G with global utility function ˇ u such that ∀ i ∈ N,π ∈ Π , | u i ( π ) -ˇ u i ( π ) | ≤ δ . We denote the set of such CSP games as CSP δ ( G ) .

Finally, we define subgame stability for poly-EFGs. Our definitions are near-identical from the normal-form definitions.

Definition E.10 (Subgame stable profile) . Let G be a poly-EFG. We say a strategy profile π is γ -subgame stable if ∀ ( i, j ) ∈ E , we have ( π i , π j ) is a γ -Nash of G ij .

Definition E.11 (Subgame stable game) . Let G be a poly-EFG. We say G is ( ϵ, γ ) -subgame stable if for any ϵ -Nash equilibrium π of G , π is γ -subgame stable.

E.4 Theoretical Results For Poly-EFGs

Our theoretical results from Sections 4 and 5 continue to hold in poly-EFGs. The idea is to use the induced normal-form of a poly-EFG. We assume perfect recall.

First, we will characterize what self-play will produce in EFGs. These are called marginal behavior strategies .

Definition E.12 (Marginal behavior strategy) . Given some mediated equilibrium ( µ, (Φ i ) N i =1 ) , let π µ i be the marginal behavior strategy for i where π µ i ( a, I ) is defined arbitrarily if ∑ ρ ′ i ∈ P i ( I ) s µ i ( ρ ′ i ) = 0 and otherwise

where s µ i ( ρ i ) . = ∑ ρ -i ∈ P -i µ ( ρ i , ρ -i ) .

Definition E.13 (Marginal behavior strategy profile) . Given some mediated equilibrium ( µ, (Φ i ) N i =1 ) , let π µ be a marginal behavior strategy profile , where π µ i is a marginal behavior strategy ∀ i ∈ N .

Definition E.14 (Induced normal-form polymatrix game) . Given a poly-EFG G = ( N,E, G ) , the induced normal-form polymatrix game is a polymatrix game G ′ = ( N,E, P , u ′ ) such that P i is equal to i 's set of pure strategies in each G ij and u ′ ij ( ρ i , ρ j ) = u ij ( ρ i , ρ j ) = ∑ z ∈ Z p i ( z, ρ i ) p j ( z, ρ j ) p c ( z, π ′ c ) u ij ( z ) where u ij is the utility function of i in G ij .

In games of perfect recall, every behavior strategy π i has an equivalent mixed strategy s π i i (by Kuhn's Theorem), which means for any perfect recall EFG G , we can use the poly-EFG representation instead of turning G into a normal-form game then using a normal-form polymatrix game to get the same vulnerability bounds on G . Given π , let s π be a profile of equivalent mixed strategies.

From Kuhn's Theorem and an assumption that each G ij ∈ G has perfect recall, we derive two immediate corollaries.

Corollary E.15. If π is γ subgame stable for a poly-EFG ˇ G where each subgame has perfect recall , then s π is γ subgame stable in the induced normal-form polymatrix game of ˇ G .

Corollary E.16. For EFG of perfect recall G , if G is δ -CSP-EFG then the induced normal form of G is δ -CSP.

An extension of Theorem 4.9 holds for poly-EFGs. Let Π µ be the set of marginal behavior strategy profiles for any CCE of G and Π µ i be the set of marginal behavior strategies for i .

Proposition E.17. If an EFG G is δ -CSP and ∃ ˇ G ∈ CSP δ ( G ) that is (2 nδ, γ ) -subgame stable and µ is a CCE 5 of G , then

Proof. Transform ˇ G into its induced normal-form polymatrix game ˇ G ′ . By Corollaries E.15 and E.16 the induced normal form of G is δ -CSP and (2 nδ, γ ) -subgame stable. By perfect recall, we can convert π µ to an equivalent mixed strategy profile s µ and do likewise with any π -i ∈ Π -i . Then apply Theorem 4.9 using s µ , S -i and the induced normal-form polymatrix game of ˇ G to bound vulnerability on G 's induced normal form, and hence G .

E.5 Vulnerability Against Self-Taught Agents in EFGs

Next we show an analogue of Theorem 5.3 for extensive-form games. In Section 5 we defined S ( A ) as the set of marginal strategy profiles for a no-regret learning algorithm A . Many algorithms for EFGs will compute behavior strategies, so we use Π( A ) . = { π µ | ( µ, (Φ i ) i ∈ N ) ∈ M ( A ) } as the set of marginal behavior strategy profiles of M ( A ) (recall that M ( A ) is the set of mediated equilibria reachable by learning algorithm A . Then let Π i ( A ) . = { π i | π ∈ Π( A ) } be the set of i 's marginal strategies from Π( A ) . For example, if A is CFR, M ( A ) is the set of observably sequentially rational CFCCE Morrill et al. (2021b) and Π( A ) is the set of behavior strategy profiles computable by CFR.

Next, suppose each player i learns with their own self-play algorithm A i . Let A N . = ( A 1 , ... A n ) be the profile of learning algorithms; Π × ( A N ) . = × i ∈ N Π i ( A i ) be the set of all possible match-ups between strategies learned in self-play by those learning algorithms and Π × -i ( A N ) . = × j ∈-i Π j ( A j ) be the profiles of -i amongst these match-ups.

Definition E.18. We say a game G is δ -CSP in the neighborhood of Π ′ ⊆ Π if there exists a constant sum poly-EFG ˇ G such that ∀ π ∈ Π ′ we have | u i ( π ) -ˇ u i ( π ) | ≤ δ . We denote the set of such CSP games as CSP δ ( G, Π ′ ) .

Definition E.19. We say a poly-EFG game G is γ -subgame stable in the neighborhood of Π ′ if ∀ π ∈ Π ′ , ∀ ( i, j ) ∈ E we have that ( π i , π j ) is a γ -Nash of G ij .

Proposition E.20. If G is δ -CSP in the neighborhood of Π × ( A N ) and ∃ ˇ G ∈ CSP δ ( G, Π × ( A N )) that is γ -subgame stable in the the neighborhood of Π( A i ) , then for any Π ∈ Π( A i )

The proof goes the same way as in Corollary E.17. Use the induced normal-form polymatrix game of ˇ G and Theorem 5.3 to derive bounds for the induced normal form of G , which then apply to G .

E.6 Leveraging the Poly-EFG Representation for Computing CSP Decompositions

The poly-EFG representation gives rise to more efficient algorithms for computing a poly-EFG decomposition. As a proof of concept, we show that in perfect information EFGs, we can write a linear program to compute the optimal polymatrix decomposition for an EFG that is exponentially smaller than LP 2 from Section D.2. Recall that δ is the minimum value of δ such that a game is δ -CSP. In an perfect information EFG, we can compute δ with the following LP. The decision variables are δ , the values of ˇ u ij ( z ) ∀ z ∈ Z, ( i, j ) ∈ E and c ij ∀ ( i, j ) ∈ E .

LP 3

The trick is that in perfect information EFGs, each pure strategy profile leads to a single terminal. Hence, rather than having a constraint for each pure strategy profile, a constraint for each terminal will suffice. This leads to an exponential reduction in the number of constraints over LP 2.

F Details of SGDecompose

Next, we give the full details of SGDecompose. We give details here using our poly-EFG representation, since this is the representation we use in our experiments.

For each subgame ˇ G ij we store a single vector ˇ u ij where the entry ˇ u ij ( z ) gives the value of the utility for corresponding terminal z . We additionally store a constant ˇ c ij for each subgame. Player i will use ˇ u ij when computing ˇ u ij ( π i , π j ) :

Whereas we compute ˇ u ji ( π i , π j ) as follows.

For simplicity, let ˇ u denote the stacked vector of all ˇ u ij and ˇ c ij . We additionally initialize ˇ G as a fully connected graph.

̸

The overall loss function which we minimize has two components: first, L δ is the error between the utility functions of G and ˇ G ; it is a proxy for δ in δ -CSP.

The other component of the overall loss function, L γ , measures the subgame stability. First, we define L γ ij , which only applies to a single subgame.

Where π ij = ( π i , π j ) is a profile and π ∗ ij = ( π ∗ i , π ∗ j ) is a profile of deviations. Then

Algorithm 3 shows how to compute a subgame stable constant-sum polymatrix decomposition via SGD. As input, the algorithm receives a game G , a finite set of strategy profiles Π ′ , learning rate η , number of training epochs T , hyperparameter λ ∈ [0 , 1] and batch size B . First, we initialize Π × as the set of all match-ups amongst strategies in Π ′ .

We then repeat the following steps for each epoch. First, we compute a best-response (for example, via sequence-form linear programming) to each strategy π ′ i in Π ′ in each subgame; the full process is shown in Algorithm 4. After computing these best-responses for the current utility function of ˇ G , we try to fit ˇ u to be nearly CSP in the neighborhood of Π × and subgame stable in the neighborhood of Π ′ . Since Π × is exponentially larger than Π ′ , we partition it into batches, then use batch gradient descent. We use the following batch loss function, which computes the average values of L δ and L γ over the batches, then weights the losses with λ . Let Π b denote a batch of strategy profiles from Π × with size B , the overall loss function is

We take this loss and find its gradient with respect to ˇ u , then update ˇ u :

We found that in practise the gradient can be quite large relative to ˇ u , which has the potential to destabilize optimization. This is alleviated by normalizing the gradient by its l 2 norm.

G Experiment Details

The codebase for our experiments is available at https://github.com/RevanMacQueen/ Self-Play-Polymatrix .

G.1 Leduc Poker

We ran SGDecompose 30 times, each time with its own set of 30 strategy profiles. These 900 strategy profiles are generated with CFR+ in self-play for 1000 iterations with random initializations. We

Algorithm 3

Algorithm 4

̸

randomly initialize CFR+ with regrets between 0 and 0.001 chosen uniformly at random, which are the default values in OpenSpiel.

Interestingly, we found that CFR+ converges to approximate Nash equilibria in Leduc poker, with a maximum value of ϵ equal to 0 . 013 after 1000 iterations. As we will show in Appendix G.2, CFR also empirically produces approximate Nash equilibria in Leduc poker.

Let Π(CFR+) j denote the set of CFR+ -learned strategy profiles for run j ; and Π × (CFR+) j denote the set of all match-ups between these 30 strategy profiles. Figure 5 shows diversity of Π(CFR+) j for each of the 30 runs. We measure the diversity of each Π(CFR) j by taking each pair of strategy profiles π, π ′ ∈ Π(CFR) j and computing the total variation between these two probability distributions induced over the terminal histories of Leduc poker. We denote the maximum total variation for run j as TV j , where

TV j is constrained to be between 0 and 1 , where 0 means the two distributions are the same and 1 means they are maximally different. Figure 5 shows the maximum total variation between any two of the strategy profiles used in each run.

We used the same parameters for each run of SGDecompose: λ = 0 . 5 , B = 30 , T = 200 . We used a learning rate schedule where the learning rate η begins at 2 -6 , then halves every 5 epochs until reaching 2 -17 to encourage convergence. Our results are shown in Figure 6. We see that across the 30 runs of SGDecompose, Leduc poker is at most 0 . 021 -CSP in the neighborhood of Π × (CFR+) j and 0 . 009 -subgame stable in the neighborhood of Π(CFR+) j . Figure 7 shows the maximum vulnerability with respect to the strategies in each of the runs compared to the bounds on vulnerability given by Proposition E.20. We compute the maximum vulnerability as

̸

j

0.18

0.20

0.22

0.24

0.26

0.28

TVj

We see that the bounds are between 1 . 89 and 3 . 05 times the actual vulnerability, and are on average 2 . 51 times larger with a standard error of 0 . 049 .

0.0075

0.0100

0.0125

0.0150

0.0175

0.0200

j

(a)

0.003

0.004

0.005

0.006

0.007

0.008

j

(b)

G.2 CFR Finds Approximate Nash in Leduc Poker

It was previously believed that CFR does not compute an ϵ -Nash equilibrium on 3-player Leduc for any reasonable value of ϵ . Previous work found that CFR computed a 0 . 130 -Nash equilibrium after 10 8 iterations Abou Risk & Szafron (2010). We saw in the previous section that CFR+ computes approximate Nash equilibria in Leduc poker-does this hold for CFR as well?

We ran 30 runs of CFR in self-play for 10,000 iterations and found that all of our strategies converged to an approximate Nash equilibrium with the maximum ϵ = 0 . 017 after only 10 4 iterations. Figure 8b

j

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Expected Utility

Vulj

2(

j

+

j

)

shows the shows the maximum deviation incentive

for each of the CFR strategies π computed by CFR in Leduc Poker. Each column is for one of the players and each point is one of the random seeds. We see the maximum value of ϵ after 10,000 iterations is 0 . 017 . Figure 8a shows the maximum deviation incentive ϵ over 10,000 iterations. We average learning curves over 30 random seeds.

H Toy Hanabi

In games with low values of δ and γ , self-play will perform well against new opponents; however is the converse also true? Do games where self-play performs poorly against new opponents have high values of δ and γ ? As mentioned earlier, self-play struggles to generalize to new agents in some games with specialized conventions (Hu et al., 2020). Hanabi is one such game (Bard et al., 2020). Hanabi is a cooperative game where players cannot see their own hands, but can see all other players hands; therefore players must give each other hints on how to act.

We show that a small version of the game of Hanabi is not close to the space of CSP games and self-play is quite vulnerable. We use Tiny Hanabi in the Openspiel framework (Lanctot et al., 2019) with our own payoffs, shown in Figure 9. Chance deals one of two hands, A or B with equal probability. Only player 1 may observe this hand and must signal to other players through their actions, σ 1 and σ 2 , which hand chance has dealt. If both players 2 and 3 then correctly choose their actions to match chance's deal ( ( a, a ) for A or ( b, b ) for B ) then all players get payoff equal to 1 , otherwise all players get 0 .

σ 1 and σ 2 can come to mean different things, σ 1 could signal to 2 and 3 to play a , or b . Self-play may learn either of these conventions. However, if a player trained in self-play encounters a set of players trained in an independent instance of self-play, they may not have compatible conventions.

This is indeed what happens when we train CFR on Tiny Hanabi in Figure 9. We trained 30 runs in self-play with different random initializations for 10,000 iterations. Some of these runs converged to

0

5000

10000

Iteration

10

2

10

1

10 0

Player 1

0

5000

10000

Iteration

10

2

10

1

10 0

Player 2

0

5000

10000

Iteration

10

2

10

1

10 0

Player 3

(a)

1

2

3

Player

0.008

0.010

0.012

0.014

0.016

(b)

each convention and when played against each other miscoordinated. We found

as expected.

We decomposed Tiny Hanabi and found δ = 0 . 5 and γ ≈ 0 , meaning the true vulnerability matched what our bounds predicted since ( n -1) γ +2 δ ≈ 1 . Why is γ ≈ 0 ? We found that this was because our algorithm was setting the payoffs to equal to 0 . 50 for all terminal histories, which is trivially polymatrix.

c

A

B

1

σ

1

σ

2

1

σ

1

σ

2

2

a

b

2

a

b

2

a

b

2

a

b

3

1

a

b

3

a

1

b

3

1

a

b

3

a

1

b

3

1

a

b

3

a

1

b

3

1

a

b

3

a

1

b

Peer reviews:
comments: SUMMARY:
The paper studies theoretical performance guarantees for agents learned using self-play in multiplayer games. Self-play is a common approach of machine learning in multi-agent systems for generating unbounded quantities of training data, but agents trained using self-play may perform poorly against new agents whose behavior differ dramatically from those seen during training. Despite guarantees previously established for self-play agents in 2 player constant-sum games, these guarantees do not extend outside of two-player constant-sum games. To solve this problem, the authors identify a structural property of multiplayer, general-sum game and use it to establish guarantees on the performance of strategies learned via self-play against new opponents. They show that any game can be projected into the space of constant-sum polymatrix games, which enables performance guarantees for the strategies produced by a broad class of self-play algorithms. The findings are empirically demonstrated on Leduc poker.

STRENGTHS:
- I find the studied topic which extends the theoretical guarantees of self-play beyond two-player (constant-sum) games to be important to the multi-agent learning community.
- The proposed method, by projecting a general game into the space of constant-sum polymatrix games, is novel and interesting.
- The theoretical results are verified on 3-player Leduc poker

WEAKNESSES:
The technical sections of the paper is slightly hard to follow for a non-expert in this field.

QUESTIONS:
- Can the authors pleaes explicitly list the assumptions/conditions needed for the theoretical results?
- Can the structural property identified in this paper extends to other application domains (outside of multi-agent self-play)?

LIMITATIONS:
I don't see negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper asks and answers the question: \"In what games does self-play (with regret-minimizing algorithms) in multiplayer imperfect-information games perform well?\"\n\nThe motivation behind the question is that such algorithms, such as multiplayer versions of CFR (which has theoretic guarantees in 2-player zero-sum games), have empirically performed well in some multiplayer games, despite no previous theory guaranteeing that they will do so. (The salient example is no-limit Texas Hold'em, in the work \"Superhuman AI for multiplayer poker\", Brown & Sandholm 2019.)\n\nThis paper answers the question primarily by looking at the concept of the coarse correlated equilibrium (CCE), since that is what the policies of the regret-minimizing players will converge to. The authors show that if games can be decomposed into a bunch of 2-player constant-sum games, called a constant-sum polymatrix (CSP) game, and the CSP game fulfills some additional properties, then we can bound the worst-case performance of a player's CCE strategy. They also show that even in games which can't be factored into polymatrix games (which is most games), we can project them into the space of CSP games, and still bound the worst-case performance of each player's CCE strategy.\n\nThey also refine the analysis further by examining worst-case performance only against other self-play strategies. Finally, they perform experiments on multiplayer Leduc poker, and show that the empirical worst-case performances are within the bounds predicted by their theory.\n\nSTRENGTHS:\n- I think this is an excellent paper. The question it asks and answers is fascinating, and one that I think was begging to be examined for several years now. Therefore, I judge this paper to be a significant contribution to science.\n- The results of the paper are novel and nontrivial. \n- The paper is well-written. The writing is clear and understandable. The introduction explains the background and motivates the problem well.\n\nWEAKNESSES:\nThe paper mentions no-limit Texas hold'em as a motivating real-world positive example of self-play working in multiplayer games. I also suggest that the authors mention some real-world negative examples -- for example, FAIR's line of work on Diplomacy, particularly the self-play agents performing poorly against humans in \"No-Press Diplomacy from Scratch\", Bakhtin et al 2021 https://arxiv.org/pdf/2110.02924.pdf (the problem is the motivation of \"Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning\" Bakhtin et al 2022: \"the resulting agent DORA does very well when playing with other copies of itself. However, DORA performs poorly in games with 6 human human-like agents.\")\n\ntypo on line 336: \"with ... be the set\"\n\nQUESTIONS:\nIt's not clear to me how important the caveat in Footnote 3 on page 8 (section 6) is. So CFR does not necessarily converge to a CCE. The experiments in Section 6 use CFR. Is my interpretation correct that you take the marginal strategies of the CFR iterates, and hand-wavingly say that we can assume they are marginal strategies of (a distribution converging to) \na CCE, even though they're not?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn multiplayer games, the authors derive bounds for the vulnerability of marginal strategies trained via no-regret self-play against other, uncorrelated agents independently trained via no-regret self-play. This is done by projecting games onto the space of constant-sum polymatrix (CSP) games, which can be decomposed into a set of 2-player constant-sum games between individual players. The closer this projection is, the tighter the bound on self-play's vulnerability to other self-play-trained agents is. This claim is validated by demonstrating that 3-payer Leduc can indeed be closely approximated as a CSP game and that the calculated vulnerability bounds are relatively close to the empirically measured vulnerability among marginal CFR strategies from many seeds.\n\nSTRENGTHS:\n- A bound on the vulnerability of marginal no-regret strategies to other uncorrelated no-regret strategies is novel and highly useful to the game theory/AI community.\n- The paper is well written and concepts are clearly explained.\n\nWEAKNESSES:\n- I would have preferred to see a range of common tractable games examined to get an intuition on when and how often a game can be closely approximated as a CSP game (and thus provide a useful bound on vulnerability). Currently, experiments only include Leduc Poker.\n- The analysis method is not immediately transferable to large games, and this is clearly stated as a limitation.\n\n\n- Minor note: It's not explicitly stated until the very end of section 6 that the 3-player variant of Leduc Poker is used for experiments.\n\nQUESTIONS:\nAre there any heuristics one may be able to look for in larger games that might indicate when a game has a high chance of being approximately decomposable as a CSP game? If there is intuition on this to be gained from explicitly analyzing smaller games, it would be a great thing to discuss.\n\nLIMITATIONS:\nAll limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the intriguing problem of why no-regret algorithms seem to approximate well in multiplayer games, a phenomenon that has been empirically demonstrated in multiagent poker. The authors identify a structural property in multi-player games that allows performance guarantees for strategies derived through self-play algorithms. They propose that multi-player games can be projected into a series of two-player constant-sum games, known as polymatrix games. The proximity of a game to this structure is hypothesized to diminish the effect of correlation issues on the removal of a mediator. The researchers take an algorithm-agnostic approach, which broadens the applicability of their analysis to a variety of game-theoretically inspired learning algorithms and MARL algorithms that converge to coarse correlated equilibria.\n\nSTRENGTHS:\nThe paper's strength lies in its novel approach to an important problem. The authors introduce theory that all multiplayer games can be projected onto the space of polymatrix games, offering a promising direction for further research. If a high subgame stability game exists within this space, no-regret learning algorithms are predicted to converge to low-exploitability strategies. Additionally, the authors' algorithm-agnostic approach ensures that the analysis remains broadly applicable to a variety of game-theoretically inspired learning and MARL algorithms.\n\nWEAKNESSES:\nThe paper's main weaknesses lie in its experimental section, which comes across as somewhat unclear. The section seems intended to demonstrate that CFR will always converge to a low-exploitable strategy due to the subgame stability of the game, but this is not clearly stated. If the goal is to show that no-regret algorithms will always converge, the experiment design should include many seeds with different hyperparameters and many different no-regret algorithms. Secondly, it's unclear how this theory can be practically applied. For instance, if a new game is introduced, can it be predicted ahead of time whether CFR or another no-regret algorithm will converge to a low-exploitability strategy? This potential application isn't clear and should be further highlighted if it is indeed feasible.\n\nQUESTIONS:\nHow can the theory be applied practically? Can it be used to predict whether a no-regret algorithm will converge to a low-exploitability strategy in a new game?\nCould the use of the term 'vulnerability' over 'exploitability' be explained or justified in this context?\nWhat are the detailed parameters and specifics of the experiments conducted, such as the number of seeds and the range of no-regret algorithms tested?\n\nLIMITATIONS:\nThere are certain limitations in the terminology used in the paper. The authors use 'vulnerability' instead of 'exploitability', although 'exploitability' is the term most widely used and recognized in this field. In addition, the term 'self-play' can lead to confusion, as it can generally refer to methods where RL agents play against themselves, which are not no-regret. The term 'no-regret' would be clearer. Lastly, at the end of page 9: “and if there exists a game with this set with high subgame stability” do you mean “and if there exists a game in this set with high subgame stability”?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}