SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stephen McAleer  
Carnegie Mellon University
Gabriele Farina  
MIT
Gaoyue Zhou  
Carnegie Mellon University
Mingzhi Wang  
Peking University
Yaodong Yang  
Peking University
Tuomas Sandholm* Carnegie Mellon University
Recent algorithms have achieved superhuman performance at a number of two-player zero-sum games such as poker and go. However, many real-world situations are multi-player games. Zero-sum two-team games, such as bridge and football, involve two teams where each member of the team shares the same reward with every other member of that team, and each team has the negative of the reward of the other team. A popular solution concept in this setting, called TMECor, assumes that teams can jointly correlate their strategies before play, but are not able to communicate during play. This setting is harder than two-player zero-sum games because each player on a team has different information and must use their public actions to signal to other members of the team. Prior works either have game-theoretic guarantees but only work in very small games, or are able to scale to large games but do not have game-theoretic guarantees. In this paper we introduce two algorithms: Team-PSRO, an extension of PSRO from two-player games to team games, and Team-PSRO Mix-and-Match which improves upon Team PSRO by better using population policies. In Team-PSRO, in every iteration both teams learn a joint best response to the opponent's meta-strategy via reinforcement learning. As the reinforcement learning joint best response approaches the optimal best response, Team-PSRO is guaranteed to converge to a TMECor. In experiments on Kuhn poker and Liar's Dice, we show that a tabular version of Team-PSRO converges to TMECor, and a version of Team PSRO using deep cooperative reinforcement learning beats self-play reinforcement learning in the large game of Google Research Football.
Two-player zero-sum games have served as testbeds for artificial intelligence research. Algorithms that have achieved superhuman performance in games such as go (Silver et al., 2017) and poker (Bowling et al., 2015; Brown & Sandholm, 2017) are widely seen as milestones. These algorithms use game-theoretic techniques to find an approximate Nash equilibrium, in other words a strategy that cannot be exploited by humans. Despite these successes in two-player games, real-world scenarios often involve more than two players.
In this paper we study a class of games with multiple players where each player shares reward with the other players in one of two teams, which we call two-team games. A canonical example of a
*Additional affiliations: Strategy Robot, Inc., Strategic Machine, Inc., Optimized Markets, Inc.
two-team game is bridge, a game in which current algorithms fail to compete with expert humans. There are multiple solution concepts for team games, but the one we consider is known as TMECor. In this setting, we allow each team to jointly sample a value from a distribution to correlate their strategies before play. This setting has many nice properties and is the natural solution concept for many team games. Two joint team strategies are in a TMECor if the value that a joint best response for an opponent team gets against each team is the same as the value when both teams are playing their current strategy.
There are two strands of research surrounding two-team games. The first has developed tabular methods that have game-theoretic guarantees. These algorithms work well on very small games but do not scale to large team games. The second has developed deep reinforcement learning algorithms that can scale to large two-team games, but end up being exploitable because they are not based on game-theoretic algorithms.
In this paper, we introduce the first scalable game-theoretic techniques for two-team games. We show that a straightforward extension of PSRO to team games, called Team PSRO, and is guaranteed to converge to an approximate TMECorr. We also introduce a novel algorithm called Team PSRO Mix-and-Match (Team PSRO-MM) that mixes and matches best responses to create a larger population and show that it outperforms Team PSRO. In our experiments, we show that Team PSRO-MM outperforms Team PSRO and that both methods outperform self-play on the large benchmark game of Google Research Football.
Team PSRO is based on policy space response oracles (PSRO) (Lanctot et al., 2017). PSRO is already one of the most promising methods for finding approximate Nash equilibrium in large two-player zero-sum games because it is simple to use with existing RL methods, it naturally provides a measure of approximate exploitability, and doesn't require full game-tree traversals. Methods based on PSRO such as AlphaStar (Vinyals et al., 2019b) and Pipeline PSRO (McAleer et al., 2020) have achieved state-of-the-art performance on Starcraft and Barrage Stratego, respectively. Our method, called Team PSRO is the first PSRO-based algorithm for team games and we show that it converges to TMECor when the RL best response is strong enough.
Despite being the first scalable method for computing TMECor, Team PSRO does not efficiently use the population of best responses because each policy must be deployed with the corresponding policy in its joint best response. Building on Team PSRO we fix this issue by allowing policies to be mixed and matched with other policies in the population that are not the corresponding policy in its joint best response. We show that the resulting algorithm, called Team PSRO Mix-and-Match (Team PSRO-MM) is able to empirically improve upon Team PSRO. We hypothesize that this is because computing best responses is more costly than evaluating the expected value of policies when the number of policies is low enough.
Our contributions include the following:
1. We show that a straightforward extension of PSRO to team games converges to TMECor.
2. We introduce a novel method based on Team PSRO, called Team PSRO Mix-and-Match, that empirically converges faster by better using the best responses in the population.
Extensive-form games (EFGs) model games that are played on a game tree, and can capture both sequential and simultaneous moves, as well as private information. In this section, for simplicity we focus on four-player zero-sum games where two players—T1 and T2—play as a team against two opponent players, denoted by O1 and O2. However, these methods generalize to games with more than two players, as shown in our experiments on Google Research Football.
Each node  $v$  in the game tree belongs to exactly one player  $i \in \{\mathrm{T}1, \mathrm{T}2, \mathrm{O}1, \mathrm{O}2\} \cup \{\mathrm{c}\}$  whose turn is to move. Player  $\mathbb{C}$  is a special player, called the chance player. It models exogenous stochasticity in the environment, such as drawing a card from a deck or tossing a coin. The edges leaving  $v$  represent the actions available at that node. Any node without outgoing edges is called a leaf and represents an end state of the game. We denote the set of such nodes by  $Z$ . Each  $z \in Z$  is associated with a tuple of payoffs specifying the payoff  $u_{i}(z)$  of each player  $i \in \{\mathrm{T}1, \mathrm{T}2, \mathrm{O}1, \mathrm{O}2\}$  at  $z$ . The product of the probabilities of all actions of  $\mathbb{C}$  on the path from the root of the game to leaf  $z$  is denoted by  $p_{\mathrm{c}}(z)$ .
Private information is represented via information set (infoset). In particular, the set of nodes belonging to  $i \in \{\mathsf{T}1, \mathsf{T}2, \mathsf{O}1, \mathsf{O}2\}$  is partitioned into a collection  $\mathcal{I}_i$  of non-empty sets: each  $I \in \mathcal{I}_i$  groups together nodes that Player  $i$  cannot distinguish among, given what they have observed. Necessarily, for any  $I \in \mathcal{I}_i$  and  $v, w \in I$ , nodes  $v$  and  $w$  must have the same set of available actions. Consequently, we denote the set of actions available at all nodes of  $I$  by  $A_I$ . As it is customary in the related literature, we assume perfect recall, that is, no player forgets what he/she knew earlier in the game. Finally, given players  $i$  and  $j$ , two_infosets  $I_i \in \mathcal{I}_i$ ,  $I_j \in \mathcal{I}_j$  are connected, denoted by  $I_i \rightleftharpoons I_j$ , if there exist  $v \in I_i$  and  $w \in I_j$  such that the path from the root to  $v$  passes through  $w$  or vice versa.
Sequences. The set of sequences of Player  $i$ , denoted by  $\Sigma_{i}$ , is defined as  $\Sigma_{i} := \{(I, a) : I \in \mathcal{I}_{i}, a \in A_{I}\} \cup \{\varnothing\}$ , where the special element  $\varnothing$  is called the empty sequence of Player  $i$ . The parent sequence of a node  $v$  of Player  $i$ , denoted  $\sigma(v)$ , is the last sequence (information set-action pair) for Player  $i$  encountered on the path from the root of the game to that node. Since the game has perfect recall, for each  $I \in \mathcal{I}_{i}$ , nodes belonging to  $I$  share the same parent sequence. So, given  $I \in \mathcal{I}_{i}$ , we denote by  $\sigma(I) \in \Sigma_{i}$  the unique parent sequence of nodes in  $I$ . Additionally, we let  $\sigma(I) = \emptyset$  if Player  $i$  never acts before信息发布  $I$ .
Reduced-normal-form plans. A reduced-normal-form plan  $\pi_i$  for Player  $i$  defines a choice of action for every information set  $I \in \mathcal{I}_i$  that is still reachable as a result of the other choices in  $\pi$  itself. The set of reduced-normal-form plans of Player  $i$  is denoted  $\Pi_i$ . We denote by  $\Pi_i(I)$  the subset of reduced-normal-form plans that prescribe all actions for Player  $i$  on the path from the root to information set  $I \in \mathcal{I}_i$ . Similarly, given  $\sigma = (I, a) \in \Sigma_i$ , let  $\Pi_i(\sigma) \subseteq \Pi_i(I)$  be the set of reduced-normal-form plans belonging to  $\Pi_i(I)$  where Player  $i$  plays action  $a$  at  $I$ , and let  $\Pi_i(\emptyset) := \Pi_i$ . Finally, given a leaf  $z \in Z$ , we denote with  $\Pi_i(z) \subseteq \Pi_i$  the set of reduced-normal-form plans where Player  $i$  plays so as to reach  $z$ .
Sequence-form strategies. A sequence-form strategy is a compact strategy representation for perfect-recall players in EFGs (Romanovskii, 1962; Koller et al., 1996). Given a player  $i \in \{\mathsf{T}1, \mathsf{T}2, \mathsf{O}1, \mathsf{O}2\}$  and a normal-form strategy  $\mu \in \Delta(\Pi_i)^2$ , the sequence-form strategy induced by  $\mu$  is the real vector  $\pmb{y}$ , indexed over  $\sigma \in \Sigma_i$ , defined as  $y[\sigma] := \sum_{\pi \in \Pi_i(\sigma)} \mu(\pi)$ . The set of sequence-form strategies that can be induced as  $\mu$  varies over  $\Delta(\Pi_i)$  is denoted by  $\mathcal{Y}_i$  and is known to be a convex polytope (called the sequence-form polytope) defined by a number of constraints equal to  $|\mathcal{I}_i|$  (Koller et al., 1996).
TMECor as a Bilinear Saddle-Point Problem. A TMECor strategy is a probability distribution  $\mu_{\mathsf{T}}$  over the set of randomized strategy profiles  $\mathcal{V}_{\mathsf{T1}} \times \mathcal{V}_{\mathsf{T2}}$  that guarantees maximum expected utility for the team against the best-responding opponent team  $\{\mathsf{O1}, \mathsf{O2}\}$ . Since each player has perfect recall, any randomized strategy for a player is equivalent to a distribution over reduced-normal-form pure strategies (Kuhn, 1953). Hence, any distribution over profiles of randomized strategies of the team members can be expressed in an equivalent way as a distribution over deterministic strategy profiles  $\Pi_{\mathsf{T1}} \times \Pi_{\mathsf{T2}}$ . Denote by  $\mathcal{P}(\Pi_{\mathsf{T}})$  all possible combinations of pure strategies of a population  $\Pi_{\mathsf{T}}$ :  $\{(\pi_{\mathsf{T1}}, \pi_{\mathsf{T2}}) | \pi_{\mathsf{T1}}, \pi_{\mathsf{T2}} \in \Pi_{\mathsf{T}}\}$ . The benefit of this transformation is that  $\Pi_{\mathsf{T1}} \times \Pi_{\mathsf{T2}}$  is a finite set, unlike  $\mathcal{V}_{\mathsf{T1}} \times \mathcal{V}_{\mathsf{T2}}$ . For this reason, TMECor is usually defined in the literature as a distribution over  $\Pi_{\mathsf{T1}} \times \Pi_{\mathsf{T2}}$  without loss of generality. We will follow the same approach in our characterization.
For each leaf  $z$ , let  $\hat{u}_{\mathsf{T}}(z) \coloneqq (u_{\mathsf{T}1}(z) + u_{\mathsf{T}2}(z))p_{\mathsf{C}}(z)$ . The expected utility of the team can be written as the following function of the distributions of play  $\mu_{\mathsf{T}} \in \Delta (\Pi_{\mathsf{T}1} \times \Pi_{\mathsf{T}2})$ ,  $\mu_{\mathsf{O}} \in \Delta (\Pi_{\mathsf{O}1} \times \Pi_{\mathsf{O}2})$ :
$$
u_{\mathsf{T}}(\mu_{\mathsf{T}},\mu_{\mathsf{O}}):= \sum_{z\in Z}\hat{u}_{\mathsf{T}}(z)\left(\sum_{\substack{\pi_{\mathsf{T}1}\in \Pi_{\mathsf{T}1}(z)\\ \pi_{\mathsf{T}2}\in \Pi_{\mathsf{T}2}(z)}}\mu_{\mathsf{T}}(\pi_{\mathsf{T}1},\pi_{\mathsf{T}2})\right)\left(\sum_{\substack{\pi_{\mathsf{O}1}\in \Pi_{\mathsf{O}1}(z)\\ \pi_{\mathsf{O}2}\in \Pi_{\mathsf{O}2}(z)}}\mu_{\mathsf{O}}(\pi_{\mathsf{O}1},\pi_{\mathsf{O}2})\right).
$$
For a given opponent coordinated strategy  $\mu_{\mathrm{O}} \in \Delta(\Pi_{\mathrm{O1}} \times \Pi_{\mathrm{O2}})$ , a best response  $\mathbb{BR}_{\top}(\mu_{\mathrm{O}})$  is defined as a coordinated strategy that gives the highest expected utility against the opponent coordinated strategy:
$$
\mathbb {B} \mathbb {R} _ {\top} (\mu_ {\circ}) = \underset {\mu_ {\top} \in \Delta \left(\Pi_ {\top 1} \times \Pi_ {\top 2}\right)} {\arg \max } u _ {\top} (\mu_ {\top}, \mu_ {\circ}). \tag {1}
$$
By definition, a team-maxmin equilibrium with coordination device (TMECor) (Celli & Gatti, 2018; Farina et al., 2018b) is a Nash equilibrium of the game where the team plays according to the
$^2\Delta(X)$  denotes the probability simplex over the finite set  $X$ .
coordinated strategy  $\mu_{\mathsf{T}} \in \Delta(\Pi_{\mathsf{T}1} \times \Pi_{\mathsf{T}2})$  and the opponent team plays according to the coordinated strategy  $\mu_{\mathsf{O}} \in \Delta(\Pi_{\mathsf{O}1} \times \Pi_{\mathsf{O}2})$ . In the zero-sum setting, this amounts to finding a solution of the optimization problem
$$
\max  _ {\mu_ {T} \in \Delta \left(\Pi_ {T 1} \times \Pi_ {T 2}\right)} \min  _ {\mu_ {O} \in \Delta \left(\Pi_ {O 1} \times \Pi_ {O 2}\right)} u _ {T} \left(\mu_ {T}, \mu_ {O}\right). \tag {2}
$$
Note that if a pair of joint strategies  $(\mu_{\mathrm{T}},\mu_{\mathrm{O}})$  are in a TMECor, then
$$
u _ {\top} \left(\mu_ {\top}, \mu_ {\circ}\right) = u _ {\top} \left(\mathbb {B R} _ {\top} \left(\mu_ {\circ}\right), \mu_ {\circ}\right) = u _ {\top} \left(\mu_ {\top}, \mathbb {B R} _ {\circ} \left(\mu_ {\top}\right)\right) \tag {3}
$$
2.1 Approximate TMECor
Define an  $\epsilon$ -best response ( $\epsilon$ -BR)  $\mathbb{BR}_{\top}^{\epsilon}(\mu_{\mathsf{O}})$  as any coordinated strategy that achieves expected utility against the opponent coordinated strategy within  $\epsilon$  of optimal:  $u_{\top}(\mathbb{BR}_{\top}(\mu_{\mathsf{O}}), \mu_{\mathsf{O}}) - \epsilon \leq u_{\top}(\mathbb{BR}_{\top}^{\epsilon}(\mu_{\mathsf{O}}), \mu_{\mathsf{O}})$ . The exploitability  $e(\mu_{\top}, \mu_{\mathsf{O}})$  of a pair of correlated strategies ( $\mu_{\top}, \mu_{\mathsf{O}}$ ) is defined as  $e(\mu_{\top}, \mu_{\mathsf{O}}) = u_{\top}(\mathbb{BR}_{\top}(\mu_{\mathsf{O}}), \mu_{\mathsf{O}}) + u_{\mathsf{O}}(\mu_{\top}, \mathbb{BR}_{\mathsf{O}}(\mu_{\top}))$ . A pair of joint strategies ( $\mu_{\top}, \mu_{\mathsf{O}}$ ) is in an  $\epsilon$ -approximate TMECor if  $e(\mu_{\top}, \mu_{\mathsf{O}}) \leq \epsilon$
3.1 Double Oracle (DO) and Policy Space Response Oracles (PSRO)
Double Oracle (McMahan et al., 2003) is an algorithm for finding a NE in two-player zero-sum normal-form games. The algorithm works by maintaining a population of strategies for each player. Each iteration a NE is computed for the game restricted to strategies in each player's population. Then, a best response to this NE for each player is computed and added to the population. Although in the worst case DO must expand all pure strategies, in many games DO empirically terminates early and outperforms existing methods. Policy-Space Response Oracles (PSRO) Lanctot et al. (2017) scales DO to large games by using reinforcement learning to approximate a best response. The restricted-game NE is computed on the empirical game matrix generated by having each policy in the population play each opponent policy and tracking average utility in a payoff matrix (Wellman, 2006).
There are a number of methods related to PSRO. NXDO (McAleer et al., 2021; Tang et al., 2023) iteratively adds reinforcement learning policies to a population but solve an extensive-form restricted game and has shown to be more efficient than PSRO in certain games. AlphaStar (Vinyals et al., 2019b) trains a population of policies through a procedure that is somewhat similar to PSRO. AlphaStar also uses some elements of self-play when constructing its population, and outputs a meta-Nash equilibrium of the population at test time. P2SRO (McAleer et al., 2020) parallelizes PSRO with convergence guarantees. Other work has generalized PSRO to more players (Muller et al., 2020; Harris et al., 2021), incorporated diversity objectives to cover more of the strategy space (Liu et al., 2021b; Perez-Nieves et al., 2021; McAleer et al., 2022a; Slumbers et al., 2023; Yao et al., 2023), and meta-learned the meta-distribution (Feng et al., 2021). PSRO has also been applied to core problems in game theory (Zhang et al., 2023) and reinforcement learning (Liang et al., 2023).
3.2 Team Games
In this paper we study the TMECor Celli & Gatti (2018); Farina et al. (2018a); Zhang & Sandholm (2021); Zhang et al. (2022) setting where players on the same team are allowed to coordinate before playing. A related solution concept is an equilibrium where team members are not able to coordinate, which is known as a team-maxmin equilibrium (TME) strategy (von Stengel & Koller, 1997; Basilico et al., 2017; Zhang & An, 2020a,b; Kalogiannis et al., 2021; Anagnostides et al., 2023). A solution of this type yields the maximum expected utility for the team players against a best-responding opponent. The TMECor solution concept has several advantages over TME. First, the team is guaranteed at least as much expected utility under TMECor than under TME (Celli & Gatti, 2018). Second, the TMECor objective is convex while the TME objective is not convex.
3.3 Cooperative Reinforcement Learning
In this section we focus on the regime of centralized training and decentralized execution (CTDE) in cooperative reinforcement learning. In the CTDE regime, we assume to have access to all agent's
observations during training time, which is a reasonable assumption for our setting of computational game solving. CTDE methods such as MADDPG Lowe et al. (2017) and COMA Foerster et al. (2018) make use of this centralized training learning a centralized critic that can condition on private information. Although this critic is taken away at test time and only the actor is used in decentralized execution, by conditioning on all relevant information, the centralized critic can reduce much more variance than an individual agent's critic.
Multi-Agent PPO (MAPPO) (Yu et al., 2021; Kuba et al., 2022) trains one PPO agent for the whole team with a centralized critic, among other tricks. Although off-policy CTDE methods such as MADDPG and QMix(Rashid et al., 2018; Schäfer et al., 2023; Mguni et al., 2023) tend to outperform on-policy methods such as COMA (Papoudakis et al., 2020; Wang et al., 2021), in practice MAPPO can be a surprisingly effective and stable baseline, and we use it as our joint best response in Team PSRO.
3.4 Deep RL for Team Games
A number of previous works have studied deep reinforcement learning methods in team games. Jaderberg et al. (2019) and Liu et al. (2021a, 2019) train deep RL agents via self-play and population based training to achieve high performance on a capture the flag video game and a simulated soccer environment, respectively. Berner et al. (2019a) use self-play reinforcement learning to achieve expert level performance on Dota. None of these methods are grounded in game theoretic techniques, however, and may not be expected to converge to a TMECor. Concurrent to our work, Xu et al. (2023) propose an algorithm that is similar to Team PSRO, but they do not make a connection to TMECor and do not propose the idea of mixing and matching.
4 Team PSRO
As described in the previous section, existing methods for learning approximate TMECorrs are tabular, and as a result will not scale to large games. In this paper we propose Team-PSRO. Team-PSRO makes the simple observation that approximate joint best responses can be learned via cooperative reinforcement learning. The resulting algorithm is very similar to PSRO in that every iteration, a best response is computed against the opponent's restricted distribution and the resulting policy is added to a population. However, instead of computing single-agent best responses for a single player, the best responses for Team-PSRO are joint best responses for a single team. These joint best responses are a set of policies for each member of the team and are learned through cooperative reinforcement learning.
4.1 Tabular Team Double Oracle
In this section we introduce team double oracle, a double oracle method for computing TMECor in zero-sum games with two teams. Team DO is conceptually the same as existing single oracle algorithms (Celli & Gatti, 2018) for team games but extends these algorithms to two team games. Team DO is very similar to the original double oracle algorithm for two-player zero-sum normal form games. Team DO maintains a population of pure joint strategies for each team  $\Pi_{\mathsf{T}}$ ,  $\Pi_{\mathsf{O}}$ . Each pure strategy in a team's population is a pure joint strategy for each player on that team  $\mu_{\mathsf{T}} = (\pi_{\mathsf{T}1}, \pi_{\mathsf{T}2})$ . Similar to DO, we construct a restricted game in the space of pure strategies of the populations for each team. Our restricted game  $G(\Pi_{\mathsf{T}}, \Pi_{\mathsf{O}})$  is a two-player zero-sum normal-form game that takes as actions the pure strategies  $\mu_{\mathsf{T}} \in \Pi_{\mathsf{T}}$  for the team and  $\mu_{\mathsf{O}} \in \Pi_{\mathsf{O}}$  for the opponent team. Every iteration, the NE  $(\mu_{\mathsf{T}}^r, \mu_{\mathsf{O}}^r)$  of the restricted game  $G(\Pi_{\mathsf{T}}^t, \Pi_{\mathsf{O}}^t)$  is computed. Next, a best response to this NE in the original game  $\mathbb{B}\mathbb{R}_{\mathsf{O}}(\mu_{\mathsf{T}}^r)$  is computed and added to the population. Team DO terminates when the best response for both players does not improve over the Nash value of the restricted game. Like DO, since there are a finite number of pure strategies, team DO must terminate (potentially in a number of iterations exponential in the size of the game). When it does terminate, the NE of the restricted game is a TMECor of the original game because no best response can improve on the Nash value. Team DO is described in Algorithm 1.
Compared to other methods for finding TMECor, team DO has a number of downsides. First, there are no known convergence rates for double oracle algorithms in general and team DO in particular. In fact, there exist games where Team DO must expand all joint pure strategies of the game before