SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Revan MacQueen  
Department of Computing Science  
University of Alberta / Amii
revan@ualberta.ca
James R. Wright  
Department of Computing Science  
University of Alberta / Amii
james.wright@ualberta.ca
Self-play is a technique for machine learning in multi-agent systems where a learning algorithm learns by interacting with copies of itself. Self-play is useful for generating large quantities of data for learning, but has the drawback that the agents the learner will face post-training may have dramatically different behavior than the learner came to expect by interacting with itself. For the special case of two-player constant-sum games, self-play that reaches Nash equilibrium is guaranteed to produce strategies that perform well against any post-training opponent; however, no such guarantee exists for multiplayer games. We show that in games that approximately decompose into a set of two-player constant-sum games (called constant-sum polymatrix games) where global  $\epsilon$ -Nash equilibria are boundedly far from Nash equilibria in each subgame (called subgame stability), any no-external-regret algorithm that learns by self-play will produce a strategy with bounded vulnerability. For the first time, our results identify a structural property of multiplayer games that enable performance guarantees for the strategies produced by a broad class of self-play algorithms. We demonstrate our findings through experiments on Leduc poker.
Self-play is one of the most commonly used approaches for machine learning in multi-agent systems. In self-play, a learner interacts with copies of itself to produce data that will be used for training. Some of the most noteworthy successes of AI in the past decade have been based on self-play; by employing the procedure, algorithms have been able to achieve super-human abilities in various games, including Poker (Moravcik et al., 2017; Brown & Sandholm, 2018, 2019), Go and Chess (Silver et al., 2016, 2018), Starcraft (Vinyals et al., 2019), Diplomacy (Paquette et al., 2019), and Stratego (Perolat et al., 2022).
Self-play has the desirable property that unbounded quantities of training data can be generated (assuming access to a simulator). But using self-play necessarily involves a choice of agents for the learner to train with: copies of itself. Strategies that perform well during training may perform poorly against new agents, whose behavior may differ dramatically from that of the agents that the learner trained against.
The problem of learning strategies during training that perform well against new agents is a central challenge in algorithmic game theory and multi-agent reinforcement learning (MARL) (Matignon et al., 2012; Lanctot et al., 2017). In particular, a self-play trained agent interacting with agents from an independent self-play instance—differing only by random seed—can lead to dramatically worse performance (Lanctot et al., 2017). The self-play-based DORA (Bakhtin et al., 2021) performs well against copies of itself in the game of no-press Diplomacy, but poorly against human-like agents
(Bakhtin et al., 2022). Self-play has also been known to perform poorly against new agents in games with highly specialized conventions (Hu et al., 2020), such as Hanabi (Bard et al., 2020). In training, one instance of a self-play algorithm may learn conventions that are incompatible with the conventions of another instance.
There are special classes of games where the strategies learned through self-play generalize well to new agents. In two-player, constant-sum games there exist strong theoretical results guaranteeing the performance of a strategy learned through self-play: Nash equilibrium strategies are maxmin strategies, which will perform equally well against any optimal opponent and can guarantee the value of the game against any opponent.
We lose these guarantees outside of two-player constant-sum games. For example, consider the simple two-player coordination game of Figure 1. If both players choose the same action, both receive a utility of 1, otherwise they receive 0. Suppose the row player learned in self-play to choose  $a$  (which performs well against another  $a$ -player). Similarly, column learned to play  $b$ . If these two players
Figure 1: A simple coordination game
played against each other, both agents would regret their actions. Upon the introduction of a new agent who did not train with a learner, despite  $a$  and  $b$  being optimal strategies during training, they fail to generalize to new agents. As this example demonstrates, equilibrium strategies in general are vulnerable: agents are not guaranteed the equilibrium's value against new agents—even if the new agent's also play an equilibrium strategy.
In multiplayer, general-sum games, no-regret self-play is no longer guaranteed to produce Nash equilibra--instead, algorithms converge to a mediated equilibrium, where a mediator recommends actions to each player (Von Stengel & Forges, 2008; Farina et al., 2019, 2020; Morrill et al., 2021b). The mediator can represent an external entity that makes explicit recommendations, such as traffic lights mediating traffic flows. More commonly in machine learning, correlation can arise through the shared history of actions of learning agents interacting with each other (Hart & Mas-Colell, 2000). If a strategy learned in self-play were played against new agents, these new agents may not have access to the actions taken by other agents during training, so agents would no longer be able to correlate their actions. In fact, even if all agents play a decorrelated strategy from the same mediated equilibrium, the result may not be an equilibrium.<sup>1</sup>
Despite the problems of vulnerability and loss of correlation, self-play has shown promising results outside of two-player constant-sum games. For example, algorithms based on self-play have out-performed professional poker players in multiplayer Texas hold 'em, despite the lack of theoretical guarantees (Brown & Sandholm, 2019).
We seek to understand what structure in multiplayer games will allow self-play to compute a good strategy. We show that any multiplayer game can be projected into the set of two-player constant-sum games between each pair of players, called constant-sum polymatrix games. The closer a game is to this space, the less the problem of correlation affects the removal of a mediator. We identify a second important property, called subgame stability, where global  $\epsilon$ -Nash equilibria are boundedly far from Nash equilibria in each two-player subgame. We show that if a multiplayer, general-sum game is close to a subgame stable constant-sum polymatrix game, this is sufficient for strategies learned via self-play to generalize to new agents, as they do in two-player constant-sum games.
Throughout this work, we take an algorithm-agnostic approach by assuming only that self-play is performed by a regret minimizing algorithm. This is accomplished by analyzing directly the equilibria that no-regret algorithms converge to—namely coarse correlated equilibria. As a result, our analysis applies to a broad class of game-theoretically-inspired learning algorithms but also to MARL algorithms that converge to coarse correlated equilibria (Marris et al., 2021; Liu et al., 2021; Jin et al., 2021), since any policy can be transformed into a mixed strategy with Kuhn's Theorem (Kuhn, 1953). For the remainder of this work, when we say "self-play" we are referring to self-play using a no-regret algorithm.
Decomposition-based approaches have been used in prior work to show convergence of fictitious play to Nash equilibria in two-player games (Chen et al., 2022) and evolutionary dynamics (Tuyls et al., 2018). Cheung & Tao (2020) decompose games into zero-sum and cooperative parts to analyse
1Please refer to the Appendix A for an example
the chaotic nature of Multiplicative Weights and Follow-the-Regularized-Leader. We focus less on the convergence of algorithms per se, and focus instead on the generalization of learned strategies to new agents. We are, to the best of our knowledge, the first to do so.
After defining our structural properties and proving our main results, we conclude with experiments on Leduc poker to elucidate why self-play performs well in multiplayer poker. Our results suggest that regret-minimization techniques converge to a subset of the game's strategy space that is well-approximated by a subgame stable constant-sum polymatrix game.
Normal Form Games A normal form game is a 3 tuple  $G = (N, \mathrm{P}, u)$  where  $N$  is a set of players,  $\mathrm{P} = \times_{i \in N} \mathrm{P}_i$  is a joint pure strategy space and  $\mathrm{P}_i$  is a set of pure strategies for player  $i$ . Let  $n = |N|$ . Pure strategies are deterministic choices of actions in the game. We call  $\rho \in \mathrm{P}$  a pure strategy profile.  $u = (u_i)_{i \in N}$  is a set of utility functions where  $u_i : \mathrm{P} \to \mathbb{R}$ . A player  $i$  can randomize by playing a mixed strategy, a probability distribution  $s_i$  over  $i$ 's pure strategies. Let  $S_i = \Delta(\mathrm{P}_i)$  be the set of player  $i$ 's mixed strategies (where  $\Delta(X)$  denotes the set of probability distributions over a domain  $X$ ), and let  $S = \times_{i \in N} S_i$  be the set of mixed strategy profiles. We overload the definition of utility function to accept mixed strategies as follows:  $u_i(s) = \sum_{\rho \in \mathrm{P}} \left( \prod_{i \in N} s_i(\rho_i) \right) u_i(\rho)$ . We use  $\rho_{-i}$  and  $s_{-i}$  to denote a joint assignment of pure (resp. mixed) strategies to all players except for  $i$ , thus  $s = (s_i, s_{-i})$ . We use  $\mathrm{P}_{-i}$  and  $S_{-i}$  to denote the sets of all such assignments.
Hindsight Rationality The hindsight rationality framework (Morrill et al., 2021b) conceptualizes the goal of an agent as finding a strategy that minimizes regret with respect to a set of deviations  $\Phi$ . A deviation  $\phi \in \Phi$  is a mapping  $\phi : S_i \to S_i$  that transforms a learner's strategy into some other strategy. Regret measures the amount the learner would prefer to deviate to  $\phi(s_i) \colon u_i(\phi(s_i), s_{-i}) - u_i(s_i, s_{-i})$ . An agent is hindsight rational with respect to a set of deviations  $\Phi$  if the agent does not have positive regret with respect to any deviation in  $\Phi$ , i.e.  $\forall \phi \in \Phi, u_i(\phi(s_i), s_{-i}) - u_i(s_i, s_{-i}) \leq 0$ . Let  $\mu \in \Delta(P)$  be a distribution over pure strategy profiles and  $(\Phi_i)_{i \in N}$  be a choice of deviation sets for each player.
Definition 2.1 ( $\epsilon$ -Mediated Equilibrium (Morrill et al., 2021b)). We say  $m = (\mu, (\Phi_i)_{i \in N})$  is an  $\epsilon$ -mediated equilibrium if  $\forall i \in N, \phi \in \Phi_i$  we have  $\mathbb{E}_{\rho \sim \mu}[u_i(\phi(\rho_i), \rho_{-i}) - u_i(\rho)] \leq \epsilon$ . A mediated equilibrium is a 0-mediated equilibrium.
Learning takes place in an online learning environment. At each iteration  $t$ , a learning agent  $i$  chooses a strategy  $s_i^t$  while all other agents choose a strategy profile  $s_{-i}^t$ . No-  $\Phi$ -regret learning algorithms ensure that the maximum average positive regret tends to 0:
$$
\lim _ {T \to \infty} \frac {1}{T} \left(\max _ {\phi \in \Phi} \sum_ {t = 1} ^ {T} u _ {i} \big (\phi (s _ {i} ^ {t}), s _ {- i} ^ {t} \big) - u _ {i} (s _ {i} ^ {t}, s _ {- i} ^ {t})\right) \to 0.
$$
If all agents use a no-regret learning algorithms w.r.t. a set of deviations  $\Phi_{i}$ , the empirical distribution of play converges to a mediated equilibrium. Formally, let  $\hat{\mu} \in \Delta(\mathrm{P})$  be the empirical distribution of play, where the weight on  $\rho \in \mathrm{P}$  is  $\hat{\mu}(\rho) \doteq \sum_{t=1}^{T} \left( \prod_{i \in N} s_{i}^{t}(\rho_{i}) \right)$ . As  $T \to \infty$ ,  $\hat{\mu}$  converges to  $\mu$  of a mediated equilibrium  $(\mu, (\Phi_{i})_{i \in N})$ .
Different sets of deviations determine the strength of a mediated equilibrium. For normal-form games, the set of swap deviations,  $\Phi_{SW}$ , are all possible mappings  $\phi : \mathrm{P}_i \to \mathrm{P}_i$ . We may apply a swap deviation  $\phi$  to a mixed strategy  $s_i$  by taking its pushforward measure:  $[\phi(s_i)](\rho_i) = \sum_{\rho_i' \in \phi^{-1}(\rho_i)} s_i(\rho_i')$ , where  $\phi^{-1}(\rho_i) = \{\rho_i' \in \mathrm{P}_i \mid \phi(\rho_i') = \rho_i\}$ . The set of internal deviations  $\Phi_I$ , which replace a single pure strategy with another, offer the same strategic power as swap deviations (Foster & Vohra, 1999). Formally,  $\Phi_I = \{\phi \in \Phi_{SW} \mid \exists \rho_i, \rho_i' : [\phi(\rho_i) = \rho_i'] \land [\forall \rho_i'' \neq \rho_i, \phi(\rho_i'') = \rho_i'']\}$ . The set of external deviations,  $\Phi_{EX}$ , is even more restricted: any  $\phi \in \Phi_{EX}$  maps all (mixed) strategies to some particular pure strategy; i.e.,  $\Phi_{EX} = \{\phi \in \Phi_{SW} \mid \exists \rho_i : \forall \rho_i', \phi(\rho_i') = \rho_i\}$ . The choice of  $(\Phi_i)_{i \in N}$  determines the nature of the mediated equilibrium—provided the learning algorithm for player  $i$  is no- $\Phi_i$ -regret (Greenwald et al., 2011). For example, if all players are hindsight rational w.r.t.  $\Phi_{EX}$ , then  $\hat{\mu}$  converges to the set of coarse correlated equilibria (CCE) (Moulin & Vial, 1978) and if all players are hindsight rational w.r.t.  $\Phi_I$  then  $\hat{\mu}$  converges to the set of correlated equilibria (Aumann, 1974).
A special case of mediated equilibria are Nash equilibria. If some mediated equilibrium  $m = (\mu, (\Phi_i)_{i \in N})$  is a product distribution (i.e.,  $\mu(\rho) = \prod_{i \in N} s_i(\rho) \forall \rho \in \mathrm{P}$ ) and  $\Phi_i \supseteq \Phi_{EX} \forall i \in N$ , then  $\mu$  is a Nash equilibrium. Similarly, an  $\epsilon$ -mediated equilibrium is an  $\epsilon$ -Nash equilibrium if  $\mu$  is a product distribution and  $\Phi_i \supseteq \Phi_{EX} \forall i \in N$ .
In sequential decision making scenarios (often modelled as extensive form games), the set of deviations is even more rich (Morrill et al., 2021b). All of these deviation classes—with the exception of action deviations (Selten, 1988) (which are so weak they do not even imply Nash equilibria in two-player constant-sum games, see appendix)—are stronger than external deviations. This means that the equilibria of any algorithm that minimizes regret w.r.t. a stronger class of deviations than external deviations still inherit all the properties of CCE (for example Hart & Mas-Colell (2000); Zinkevich et al. (2008); Celli et al. (2020); Steinberger et al. (2020); Morrill et al. (2021a)). Thus, we focus on CCE since the analysis generalizes broadly. Moreover, CCE can be computed efficiently, either analytically (Jiang & Leyton-Brown, 2011) or by a learning algorithm. When we refer to CCE, we use the distribution  $\mu$  to refer to the CCE, since  $\Phi$  is implicit.
3 Vulnerability
The choice of other agents during learning affects the strategy that is learned. Choosing which agents make good "opponents" during training is an open research question (Lanctot et al., 2017; Marris et al., 2021). One common approach, self-play, is to have a learning algorithm train with copies of itself as the other agents. If the algorithm is a no- $\Phi$ -regret algorithm, the learned behavior will converge to a mediated equilibrium; this gives a nice characterization of the convergence behavior of the algorithm.
However, in general the strategies in a mediated equilibrium are correlated with each other. This means that in order to deploy a strategy learned in self-play, an agent must first extract it by marginalizing out other agent's strategies. This new marginal strategy can then be played against new agents with whom the agent did not train (and thus correlate).
Definition 3.1 (Marginal strategy). Given some mediated equilibrium  $(\mu, (\Phi_i)_{i=1}^N)$ , let  $s_i^\mu$  be the marginal strategy for  $i$ , where  $s_i^\mu(\rho_i) \doteq \sum_{\rho_{-i} \in \mathrm{P}_{-i}} \mu(\rho_i, \rho_{-i})$ . Let  $s^\mu$  be a marginal strategy profile, where each  $\forall i \in N$  plays  $s_i^\mu$ .
Once a strategy has been extracted via marginalization, learning can either continue with the new agents (and potentially re-correlate), or the strategy can remain fixed.2 We focus on the case where the strategy remains fixed. In doing so we can guarantee the performance of this strategy if learning stops, but also the show guarantees about the initial performance of a strategy that continues to learn; this is especially important in safety-critical domains.
Given a marginal strategy  $s_i^\mu$ , we can bound its underperformance against new agents that behave differently from the (decorrelated) training opponents by a quantity which we call vulnerability.
Definition 3.2 (Vulnerability). The vulnerability of a strategy profile  $s$  for player  $i$  with respect to  $S_{-i}^{\prime} \subseteq S_{-i}$  is
$$
\operatorname {Vul} _ {i} \left(s, S _ {- i} ^ {\prime}\right) \doteq u _ {i} (s) - \min  _ {s _ {- i} ^ {\prime} \in S _ {- i} ^ {\prime}} u _ {i} (s _ {i}, s _ {- i} ^ {\prime}).
$$
Vulnerability gives a measure of how much worse  $s$  will perform with new agents compared to its training performance under pessimistic assumptions—that  $-i$  play the strategy profile in  $S_{-i}'$  that is worst for  $i$ . We assume that  $-i$  are not able to correlate their strategies.
Thus, if a marginal strategy profile  $s^{\mu}$  is learned through self-play and  $\mathrm{Vul}_i\left(s^\mu ,S_{-i}'\right)$  is small, then  $s_i^\mu$  performs roughly as well against new agents  $-i$  playing some strategy profile in  $S_{-i}'$ .  $S_{-i}'$  is used to encode assumptions about the strategies of opponents.  $S_{-i}' = S_{-i}$  means opponents could play any strategy, but we could also set  $S_{-i}'$  to be the set of strategies learnable through self-play if we believe that opponents would also be using self-play as a training procedure.
2The strategy learned in self-play prior to online learning has been called a blueprint strategy (Brown & Sandholm, 2019).
Some games have properties that make the vulnerability low. For example, in two-player constant-sum games the marginal strategies learned in self-play generalize well to new opponents since any Nash equilibrium strategy is also a maxmin strategy (von Neumann, 1928).
4 Guarantees via Polymatrix Decomposability
Multiplayer games are fundamentally more complex than two-player constant-sum games (Daskalakis & Papadimitriou, 2005; Daskalakis et al., 2009). However, certain multiplayer games can be decomposed into a graph of two-player games, where a player's payoffs depend only on their actions and the actions of players who are neighbors in the graph (Bergman & Fokin, 1998). In these polymatrix games (a subset of graphical games (Kearns et al., 2013)) Nash equilibria can be computed efficiently if player's utilities sum to a constant (Cai & Daskalakis, 2011; Cai et al., 2016).
Definition 4.1 (Polymatrix game). A polymatrix game  $G = (N, E, \mathrm{P}, u)$  consists of a set  $N$  of players, a set of edges  $E$  between players, a set of pure strategy profiles  $\mathrm{P}$ , and a set of utility functions  $u = \{u_{ij}, u_{ji} \mid \forall (i,j) \in E\}$  where  $u_{ij}, u_{ji}: \mathrm{P}_i \times \mathrm{P}_j \to \mathbb{R}$  are utility functions associated with the edge  $(i,j)$  for players  $i$  and  $j$ , respectively.
We refer to the normal-form subgame between  $(i,j)$  as  $G_{ij} = (\{i,j\}, \mathrm{P}_i \times \mathrm{P}_j, (u_{ij}, u_{ji}))$ . We use  $u_i$  to denote the global utility function  $u_i: \mathrm{P} \to \mathbb{R}$  where  $u_i(\rho) = \sum_{(i,j) \in E} u_{ij}(\rho_i, \rho_j)$  for each player. We use  $E_i \subseteq E$  to denote the set of edges where  $i$  is a player and  $|E_i|$  to denote the number of such edges.
Definition 4.2 (Constant-sum polymatrix). We say a polymatrix game  $G$  is constant-sum if for some constant  $c$  we have that  $\forall \rho \in \mathrm{P}$ ,  $\sum_{i\in N}u_i(\rho) = c$ .
Constant-sum polymatrix (CSP) games have the desirable property that all CCE factor into a product distribution; i.e., are Nash equilibria (Cai et al., 2016). We give a relaxed version:
Proposition 4.3. If  $\mu$  is an  $\epsilon$ -CCE of a CSP game  $G$ ,  $s^{\mu}$  is an  $n\epsilon$ -Nash of  $G$ .
This means no-external-regret learning algorithms will converge to Nash equilibria, and thus do not require a mediator to enable the equilibrium. However, they do not necessarily have the property of two-player constant-sum games that all (marginal) equilibrium strategies are maxmin strategies (Cai et al., 2016). Thus Nash equilibrium strategies in CSP games have no vulnerability guarantees. Cai et al. (2016) show that CSP games that are constant sum in each subgame are no more or less general than CSP games that are constant sum globally, since there exists a payoff preserving transformation between the two sets. For this reason we focus on CSP games that are constant sum in each subgame without loss of generality. Note that the constant need not be the same in each subgame.
4.1 Vulnerability on a Simple Constant-Sum Polymatrix Game
We next demonstrate why CSP games do not have bounded vulnerability on their own without additional properties. Consider the simple 3-player CSP game called Offense-Defense (Figure 2a). There are 3 players: 0, 1 and 2. Players 1 and 2 have the option to either attack  $0(a_0)$  or attack the other (e.g.  $a_1$ ); player 0, on the other hand, may either relax  $(r)$  or defend  $(d)$ . If either 1 or 2 attacks the other while the other is attacking 0, the attacker gets  $\beta$  and the other gets  $-\beta$  in that subgame. If both 1 and 2 attack 0, 1 and 2 get 0 in their subgame and if they attack each other, their attacks cancel out and they both get 0. If 0 plays  $d$ , they defend and will always get 0. If they relax, they get  $-\beta$  if they are attacked and 0 otherwise. Offense-Defense is a CSP game, so any CCE is a Nash equilibrium.
Note that  $\rho = (r, a_2, a_1)$  is a Nash equilibrium. Each  $i \in \{1, 2\}$  are attacking the other  $j \in \{1, 2\} \setminus \{i\}$ , so has expected utility of 0. Deviating to attacking 0 would leave them open against the other, so  $a_0$  is not a profitable deviation, as it would also give utility 0. Additionally, 0 has no incentive to deviate to  $d$ , since this would also give them a utility of 0.
However,  $\rho$  is not a Nash equilibrium of the subgames—all  $i \in \{1, 2\}$  have a profitable deviation in their subgame against 0, which leaves 0 vulnerable in that subgame. If 1 and 2 were to both deviate to  $a_0$ , and 0 continues to play their Nash equilibrium strategy of  $r$ , 0 would lose  $2\beta$  utility from their equilibrium value; in other words, the vulnerability of player 0 is  $2\beta$ .
4.2 Subgame Stability
However, some constant-sum polymatrix games do have bounded vulnerability; we call these subgame stable games. In subgame stable games, global equilibria imply equilibria at each pairwise subgame.
Definition 4.4 (Subgame stable profile). Let  $G$  be a polymatrix game with global utility functions  $(u_{i})_{i\in N}$ . We say a strategy profile  $s$  is  $\gamma$ -subgame stable if  $\forall (i,j)\in E$ , we have  $(s_i,s_j)$  is a  $\gamma$ -Nash of  $G_{ij}$ ; that is  $u_{ij}(\rho_i,s_j) - u_{ij}(s_i,s_j)\leq \gamma \quad \forall \rho_i\in \mathrm{P}_i$  and  $u_{ji}(\rho_j,s_i) - u_{ji}(s_j,s_i)\leq \gamma \quad \forall \rho_j\in \mathrm{P}_j$
For example, in Offense-Defense,  $(r,a_2,a_1)$  is  $\beta$ -subgame stable; it is a Nash equilibrium but is a  $\beta$ -Nash of the subgame between 0 and 1 and the subgame between 0 and 2.
Definition 4.5 (Subgame stable game). Let  $G$  be a polymatrix game. We say  $G$  is  $(\epsilon, \gamma)$ -subgame stable if for any  $\epsilon$ -Nash equilibrium  $s$  of  $G$ ,  $s$  is  $\gamma$ -subgame stable.
Subgame stability connects the global behavior of play  $(\epsilon$ -Nash equilibrium in  $G$ ) to local behavior in a subgame  $(\gamma$ -Nash in  $G_{ij}$ ). If a polymatrix game is both constant-sum and is  $(0, \gamma)$ -subgame stable then we can bound the vulnerability of any marginal strategy.