SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The sequence-form representation of a policy remembers the realization probability of reaching a state-action pair. We follow the definition in [11, 31], where the sequence form representation  $\pmb{x}_i \in \mathcal{X}_i \subseteq [0,1]^{|\mathcal{S}_i \times \mathcal{A}_i|}$  of  $\pi_i$  is a vector:
$$
\boldsymbol {x} _ {i} (s, a) = \prod_ {\widetilde {s} _ {i}, \widetilde {a} \in \tau (s, a)} \pi_ {i} (\widetilde {a} | \widetilde {s} _ {i}), \tag {11}
$$
where  $\tau(s, a)$  is a trajectory from the beginning to  $(s, a)$ . By the perfect-recall assumption, there is a unique  $\tau$  that leads to  $(s, a)$ . The policy  $\pi_i$  can be written as  $\pi_i(a|s) = \pmb{x}_i(s, a) / \| \pmb{x}_i(s) \|_1$ , where  $\pmb{x}_i(s)$  is  $(\pmb{x}_i(s, a_1), \ldots, \pmb{x}_i(s, a_n))$  with  $a_1, \ldots, a_n \in \mathcal{A}(s)$ . Unlike the payoff vector representation or the occupancy measure representation,  $\pmb{x}_i$  is independent of the opponent's policy as well as the environmental dynamics. Therefore, it should be more appropriate in representing a policy for the diversity optimization. Without loss of generality and following [21, 11, 31], we define the distance  $\mathrm{dist}(\pi_i, \pi_i')$  between two policies as the Bregman divergence on the sequence form representation  $\mathcal{B}_d(\pmb{x}_i \| \pmb{x}_i')$ , which can be further written in terms of state-action pairs (the derivation is presented in Appendix B.1) in the following:
$$
\operatorname {d i s t} \left(\pi_ {i}, \pi_ {i} ^ {\prime}\right) := \mathcal {B} _ {d} \left(\boldsymbol {x} _ {i} \| \boldsymbol {x} _ {i} ^ {\prime}\right) = \sum_ {s, a \in \mathcal {S} _ {i} \times \mathcal {A} _ {i}} \left(\prod_ {\widetilde {s} _ {i}, \widetilde {a} \in \tau (s, a)} \pi_ {i} (\widetilde {a} | \widetilde {s} _ {i})\right) \beta_ {s} \mathcal {B} _ {d _ {s}} \left(\pi_ {i} (s) \| \pi_ {i} ^ {\prime} (s)\right), \tag {12}
$$
where  $\mathcal{B}_{d_s}(\pi_i(s)\| \pi_i'(s))$  is the Bregman divergence between  $\pi_i(s)$  and  $\pi_i'(s)$ . In our experiment, we let  $\mathcal{B}_{d_s}(\pi_i(s)\| \pi_i'(s)) = \sum_a\pi_i(a|s)\log \pi_i(a|s) / \pi_i'(a|s) = \mathrm{KL}(\pi_i(s)\| \pi_i'(s))$ , i.e., the KL divergence. In previous work [21, 11, 31], the coefficient  $\beta_{s}$  usually declines monotonically as the length of the sequence increases. In our case, we make  $\beta_{s}$  depend on an opponent's policy  $b_{-i}$ :  $\beta_{s} = \prod_{\widetilde{s}_{-i},\widetilde{a}\in \tau (s,a)}b_{-i}(\widetilde{a} |\widetilde{s}_{-i})^2$ . This weighting method allows us to estimate the distance using the sampled average KL divergence and avoids importance sampling:
$$
\begin{array}{l} \operatorname {dist} \left(\pi_ {i}, \pi_ {i} ^ {\prime}\right) = \sum_ {s, a \in \mathcal {S} _ {i} \times \mathcal {A} _ {i}} \left(\prod_ {\widetilde {s} _ {i}, \widetilde {a} \in \tau (s, a)} \pi_ {i} (\widetilde {a} | \widetilde {s} _ {i})\right) \left(\prod_ {\widetilde {s} _ {- i}, \widetilde {a} \in \tau (s, a)} b _ {- i} (\widetilde {a} | \widetilde {s} _ {- i})\right) \mathcal {B} _ {d _ {s}} \left(\pi_ {i} (s) \| \pi_ {i} ^ {\prime} (s)\right) \\ = \mathbb {E} _ {s _ {i} \sim \pi_ {i}, b _ {- i}} \left[ \mathrm {K L} \left(\pi_ {i} \left(s _ {i}\right) \| \pi_ {i} ^ {\prime} \left(s _ {i}\right)\right) \right], \tag {13} \\ \end{array}
$$
where  $s_i \sim \pi_i, b_{-i}$  means sampling player  $i$ 's information states from the trajectories that are collected by playing  $\pi_i$  against  $b_{-i}$ . As a result, we can rewrite the diversity regularized BR solving objective in PSD-PSRO as follows:
$$
\pi_ {i} ^ {t + 1} = \arg \max  _ {\pi_ {i}} \left\{u \left(\pi_ {i}, \sigma_ {- i} ^ {t}\right) + \lambda \min  _ {\pi_ {i} ^ {k} \in \mathcal {H} \left(\Pi_ {i} ^ {t}\right)} \mathbb {E} _ {s _ {i} \sim \pi_ {i}, b _ {- i}} \left[ \mathrm {K L} \left(\pi_ {i} \left(s _ {i}\right) \| \pi_ {i} ^ {k} \left(s _ {i}\right)\right) \right] \right\}. \tag {14}
$$
Now we provide a practical way to optimize Equation 14. By regarding the opponent and chance as the environment, we can use the policy gradient method [47, 43, 44] in RL to train  $\pi_i^{t+1}$ . Denote the probability of generating  $\tau$  by  $\pi_i(\tau)$ , and the payoff of player  $i$  for the trajectory  $\tau$  by  $R(\tau)$ . Let  $\pi_i^{min} = \arg\min_{\pi_i^k \in \mathcal{H}(\Pi_i^t)} \mathbb{E}_{s_i \sim \pi_i, b_{-i}}[\mathrm{KL}(\pi_i(s_i) \| \pi_i^k(s_i))]$  be the policy in  $\mathcal{H}(\Pi_i^t)$  that minimizes the distance and let  $R^{KL}(\tau) = \mathbb{E}_{s_i \sim \tau}[\mathrm{KL}(\pi_i(s_i) \| \pi_i^{min}(s_i))]$ .
2Assume  $\beta_{s} > 0$  , i.e.,  $b_{-i}(a|s_{-i}) > 0,\forall (s_{-i},a)\in S_{-i}\times \mathcal{A}_{-i}$
The gradient of the first term  $u(\pi_i, \sigma_{-i}^t)$  in Equation 14 with respect to  $\pi_i$  can be written as:
$$
\begin{array}{l} \nabla u \left(\pi_ {i}, \sigma_ {- i} ^ {t}\right) = \nabla \int \pi_ {i} (\tau) R (\tau) d \tau = \int \nabla \pi_ {\mathrm {i}} (\tau) \mathrm {R} (\tau) d \tau \\ = \int \pi_ {i} (\tau) \nabla \log \pi_ {i} (\tau) R (\tau) d \tau \\ = \mathbb {E} _ {\tau \sim \pi_ {i}, \sigma_ {- i} ^ {t}} [ \nabla \log \pi_ {i} (\tau) R (\tau) ]. \tag {15} \\ \end{array}
$$
The gradient of the diversity term in Equation 14 with respect to  $\pi_{i}$  can be written as:
$$
\begin{array}{l} \nabla_ {\pi_ {i}} \min  _ {\pi_ {i} ^ {k} \in \mathcal {H} \left(\Pi_ {i} ^ {t}\right)} \mathbb {E} _ {s _ {i} \in \tau , \tau \sim \pi_ {i}, b _ {- i}} \left[ \mathrm {K L} \left(\pi_ {i} \left(s _ {i}\right) \| \pi_ {i} ^ {k} \left(s _ {i}\right)\right) \right] \\ = \nabla \mathbb {E} _ {s _ {i} \in \tau , \tau \sim \pi_ {i}, b _ {- i}} [ \mathrm {K L} (\pi_ {i} (s _ {i}) \| \pi_ {i} ^ {\min } (s _ {i})) ] \\ = \nabla \int \pi_ {i} (\tau) R ^ {K L} (\tau) \mathrm {d} \tau \\ = \int \nabla \pi_ {i} (\tau) R ^ {K L} (\tau) \mathrm {d} \tau + \int \pi_ {i} (\tau) \mathbb {E} _ {s _ {i} \sim \tau} [ \nabla \operatorname {K L} (\pi_ {i} (s _ {i}) \| \pi_ {i} ^ {\min } (s _ {i})) ] \mathrm {d} \tau \\ = \mathbb {E} _ {\tau \sim \pi_ {i}, b _ {- i}} [ \nabla \log \pi_ {i} (\tau) R ^ {K L} (\tau) ] + \mathbb {E} _ {s _ {i} \sim \pi_ {i}, b _ {- i}} [ \nabla \mathrm {K L} (\pi_ {i} (s _ {i}) \| \pi_ {i} ^ {\min } (s _ {i})) ], \tag {16} \\ \end{array}
$$
where we use the property that  $\nabla \mathbb{E}[KL(\pi_i(s_i)|\pi_i^{min}(s_i)]\in \partial \min_{\pi_i^k}\mathbb{E}[KL(\pi_i(s_i)|\pi_i^k (s_i)]$  , whose correctness can be shown from the following proposition:
Proposition 4.2. For any local Lipschitz continuous function  $f(x,y)$ , assume  $\forall x, \min_y f(x,y)$  exists, then  $\partial_x f(x,y)|_{y \in \arg \min_f f(x,y)} \in \partial_x \min_y f(x,y)$ , where  $\partial f$  is the generalized gradient [7].
Proof. According to Theorem 2.1 (property (4) in [7], the result is immediate, as  $\partial_x\min_yf(x,y)$  is the convex hull of  $\{\partial_x f(x,y)|y\in \arg \max g(x,y)\}$ .
Combining the above two equations, we have,
$$
\begin{array}{l} \nabla_ {\pi_ {i}} \left(u (\pi_ {i}, \sigma_ {- i} ^ {t}) + \lambda \min  _ {\pi_ {i} ^ {k} \in \mathcal {H} \left(\Pi_ {i} ^ {t}\right)} \mathbb {E} _ {s _ {i} \sim \pi_ {i}, b _ {- i}} \left[ \mathrm {K L} \left(\pi_ {i} (s _ {i}) \| \pi_ {i} ^ {k} (s _ {i})\right) \right]\right) \\ = \mathbb {E} _ {\tau \sim \pi_ {i}, \sigma_ {- i} ^ {t}} [ \nabla \log \pi_ {i} (\tau) R (\tau) ] + \lambda \mathbb {E} _ {\tau \sim \pi_ {i}, b _ {- i}} [ \nabla \log \pi_ {i} (\tau) R ^ {K L} (\tau) ] \tag {17} \\ + \lambda \mathbb {E} _ {s _ {i} \sim \pi_ {i}, b _ {- i}} [ \nabla \operatorname {K L} (\pi_ {i} (s _ {i}) \| \pi_ {i} ^ {\min } (s _ {i})) ]. \\ \end{array}
$$
According to Equation 17, we can see that optimizing  $\pi_i^{t + 1}$  requires maximizing two types of rewards:  $R(\tau)$  and  $R^{KL}(\tau)$ . This can be done by averaging the gradients using samples that are generated by playing  $\pi_{i}$  against  $\sigma_{-i}^{t}$  and  $b_{-i}$  separately. The last term in Equation 17 is easily estimated by sampling states in the sampled trajectories via playing  $\pi_{i}$  against  $b_{-i}$ . For training efficiency, we simply set  $b_{-i} = \sigma_{-i}^{t}$  in our experiments, although other settings of  $b_{-i}$  are possible, e.g., the uniform random policy. We also want to emphasize that we optimize  $\pi_i^{t + 1}$  for each iteration  $t$ , and  $\sigma_{-i}^{t}$  is fixed during an iteration and thus can be viewed as a part of the environment. Finally, to estimate the distance between  $\pi_{i}$  and  $\pi_i^{min}$ , we should ideally compute  $\pi_i^{min}$  exactly first by solving a convex optimization problem. In practice, we find sampling the policies in  $\mathcal{H}(\Pi_i^t)$  and using the minimal sampled distance already gives us satisfactory performance. The pseudo-code of PSD-PSRO is provided in Appendix C.
4.3 The Convergence Property of PSD-PSRO
We first show how the PH evolves in the original PSRO:
Proposition 4.3. Before the PE of the joint population reaches 0, adding a  $BR$  to the meta NE policy from last iteration in PSRO will strictly enlarge the  $PH$  of the current population.
The proof is in Appendix B.3. From Proposition 4.3, we can see that adding a BR serves one way (an implicit way) of enlarging the PH and hence reducing the PE. In contrast, the optimization of our diversity term in Equation 14 aims to explicitly enlarge the PH. In other words, adding a diversity regularized BR in PSD-PSRO serves as a mixed way of enlarging the PH. More formally, we have the following theorem:
Theorem 4.4. (1) In PSD-PSRO, before the PE of the joint population reaches 0, adding an optimal solution in Equation 14 will strictly enlarge the PH and hence reduce the PE. (2) Once the PH can not be enlarged (i.e., PSD-PSRO converges) by adding an optimal solution in Equation 14, the PE reaches 0, and PSD-PSRO finds a full game NE.
The proof is in Appendix B.6. In terms of convergence property, one significant benefit of PSD-PSRO over other state-of-the-art diversity-enhancing PSRO variants [2, 41, 32, 33] is the convergence of PH guarantees a full game NE in PSD-PSRO. Yet, it is not clear in those papers whether a full game NE is found once the PH of their populations converge. Notably,  $\mathrm{PSRO}_{rN}$  [2] is not guaranteed to find a NE once converged [36]. In practice, we expect a significant performance improvement of PSD-PSRO over PSRO in approximating a NE. As for different games, there might exist different optimal trade-offs between 'exploitation' (adding a BR) and 'exploration' (optimizing our diversity metric) in enlarging the PH and reducing the PE. In other words, PSD-PSRO generalizes PSRO (a PSD-PSRO instance when  $\lambda = 0$ ) in ways of enlarging the PH and reducing the PE.
Diversity has been widely studied in evolutionary computation [13], with a central focus that mimics the natural evolution process. One of the ideas is novelty search [28], which searches for policies that lead to novel outcomes. By hybridizing novelty search with a fitness objective, quality-diversity [42] aims for diverse behaviors of good qualities. Despite these methods achieving good empirical results [8, 23], the diversity metric is often hand-crafted for different tasks.
Promoting diversity is also intensively studied in RL. By adding a distance regularization between the current policy and a previous policy, a diversity-driven approach has been proposed for good exploration [22]. Unsupervised learning of diverse policies [10] has been studied to serve as an effective pretraining mechanism for downstream RL tasks. A diversity metric based on DPP [39] has been proposed to improve exploration in population-based training. Diverse behaviors were learned in order to improve generalization ability for test environments that are different from training [25]. A diversity-regularized collaborative exploration strategy has been proposed in [40]. Reward randomization [49] has been employed to discover diverse strategies in multi-agent games. Trajectory diversity has been studied for better zero-shot coordination in a multi-agent environment [34]. Quality-similar diversity has been investigated in [51].
Diversity also plays a role in game-theoretic methods. Smooth FP [17] adds a policy entropy term when finding a BR.  $\mathrm{PSRO}_{rN}$  [2] encourages effective diversity, which considers amplifying the strength over the weakness in a policy. DPP-PSRO [41] introduces a diversity metric based on DPP and provides a geometric interpretation of behavioral diversity. BD&D-RD-PSRO [32] combines the occupancy measure mismatch and the diversity on payoff vectors as a unified diversity metric. UDM [33] summarizes existing diversity metrics, by providing a unified diversity framework. In both opponent modeling [15] and opponent-limited subgame solving [30], diversity has been shown to have a large impact on the performance.
Figure 1: (a): Exploitability of the meta NE. (b): PE of the joint population.
Figure 2: Non-Transitive Mixture Game. Exploration trajectories during training. For each method, the final exploitability  $\times 100$  (Exp) is reported at the bottom.
The main purpose of the experiments is to compare PSD-PSRO with existing state-of-the-art PSRO variants in terms of approximating a full game NE. The baseline methods include PSRO [26], Pipeline-PSRO (P-PSRO) [36],  $\mathrm{PSRO}_{rN}$  [2], DPP-PSRO [41], and BD&RD-PSRO [32]. The benchmarks consist of single-state games (AlphaStar888 and non-transitive mixture game) and complex extensive games (Leduc poker and Goofspiel). For AlphaStar888 and Leduc poker, we report the exploitability of the meta NE and the PE of the joint population through the training process. For the non-transitive mixture game, we illustrate the 'diversity' of the population and report the final exploitability. For Goofspiel where the exact exploitability is intractable, we report the win rate between the final agents. In addition, we illustrate how the PH evolves for each method using the Disc game [2] in Appendix D.3, where PSD-PSRO is more effective at enlarging the PH and approximating a NE. An ablation study on  $\lambda$  of PSD-PSRO in Appendix D.2 reveals that, for different benchmarks, an optimal trade-off between 'exploitation' and 'exploration' in enlarging the PH to approximate a NE usually happens when  $\lambda$  is greater than zero. Error bars or stds in the results are obtained via 5 independent runs. In Appendix D.3, we also investigate the time cost of calculating our policy space diversity. More details for the environments and hyper-parameters are given in Appendix E.
AlphaStar888 is an empirical game generated from the process of solving Starcraft II [50], which contains a payoff table for 888 RL policies. be viewed as a zero-sum symmetric two-player game where there is only one state  $s_0$ . In  $s_0$ , there are 888 legal actions. Any mixed strategy is a discrete probability distribution over the 888 actions. Hence, the distance function in Equation 14 for AlphaStar888 reduces to the KL divergence between two 888-dim discrete probability distributions.
In Figure 1, we can see that PSD-PSRO is more effective at reducing both the exploitability and PE than other methods.
Non-Transitive Mixture Game consists of 7 equally-distanced Gaussian humps on the 2D plane. Each strategy can be represented by a point on the 2D plane, which is equivalent to the weights (the likelihood of that point in each Gaussian distribution) that each player puts on the humps. The optimal strategy is to stay close to the center of the Gaussian humps and explore all the distributions. In Figure 2, we show the exploration trajectories for different methods during training, where PSRO and  $\mathrm{PSRO}_{rN}$  get trapped and fail in this non-transitive game. In contrast, PSD-PSRO tends to find diverse strategies and explore all Gaussians. Also, the exploitability of the meta NE of the final population is significantly lower in PSD-PSRO than others.
Figure 3: (a): Exploitability of the meta NE. (b): PE of the joint population.
Leduc Poker is a simplified poker [46], where the deck consists of two suits with three cards in each suit. Each player bets one chip as an ante, and a single private card is dealt to each player. Since DPP-PSRO cannot scale to the RL setting [32] and the code of BD&D-RD-PSRO for complex games is not available, we compare PSD-PSRO only to P-PSRO,  $\mathrm{PSRO}_{rN}$  and PSRO. As demonstrated in Figure 3, PSD-PSRO is more effective at reducing both the exploitability and PE.
PSRO	PSROrN	P-PSRO	PSD-PSRO(OURS)
PSRO	-	0.613±0.019	0.469±0.034	0.422±0.025
PSROrN	0.387±0.019	-	0.412±0.030	0.358±0.019
P-PSRO	0.531±0.034	0.588±0.030	-	0.370±0.031
PSD-PSRO(OURS)	0.578±0.025	0.642±0.019	0.630±0.031	-
Table 1: The win rate of the row agents against the column agents on Goofspiel.
Goofspiel is commonly used as a large-scale multi-stage simultaneous move game. Goofspiel features strong non-transitivity, as every pure strategy can be exploited by a simple counter-strategy. We compare PSD-PSRO with PSRO, P-PSRO, and  $\mathrm{PSRO}_{rN}$  on Goofspiel with 5 point cards and 8 point cards settings. In the game with 5 point cards setting, due to the relatively small game size, we can calculate the exact exploitability. We report the results in Appendix D.1, in which we see that PSD-PSRO reduces the exploitability more effectively than other methods. In the game with 8 point cards setting, the game size is too large to show exact exploitability for each iteration. In this setting, we provide a comparison among final solutions produced by different methods. We report the win rate between each two methods in Table 1, where we can see that PSD-PSRO consistently beats existing methods with a  $62\%$  win rate on average.
In this paper, we point out a major and common weakness of existing diversity metrics in previous diversity-enhancing PSRO variants, which is their goal of enlarging the gamescape does not necessarily result in a better approximation to a full game NE. Based on the insight that a larger PH means a lower PE (a better approximation to a NE), we develop a new diversity metric (policy space diversity) that explicitly encourages the enlargement of a population's PH. We then develop a practical method to optimize our diversity metric using only state-action samples, which is derived based on the Bregman divergence on the sequence form of policies. We incorporate our diversity metric into the BR solving in PSRO to obtain PSD-PSRO. We present the convergence property of PSD-PSRO, and extensive experiments demonstrate that PSD-PSRO is significantly more effective in approximating a NE than state-of-the-art PSRO variants.
The diversity regularization term  $\lambda$  in PSD-PSRO plays an important role in balancing the 'exploitation' and 'exploration' in terms of enlarging the PH to approximate a NE. In this paper, other than showing that different problems have different optimal settings of  $\lambda$ , we have not discussed about the guidance of choosing  $\lambda$  optimally. Although there has been related work on how to adapt a diversity regularization term using online bandits [39], future work is still needed on how our policy space diversity could benefit the approximation of a full game NE most. Another interesting direction of future work is to extend PSD-PSRO to larger scale games, such as poker [5], Mahjong [14], and dark chess [52].
[1] Yu Bai, Chi Jin, Song Mei, and Tiancheng Yu. Near-optimal learning of extensive-form games with imperfect information. In International Conference on Machine Learning, pages 1337-1382. PMLR, 2022.
[2] David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pages 434-443. PMLR, 2019.
[3] David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games. In International Conference on Machine Learning, pages 354-363. PMLR, 2018.
[4] George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1):374, 1951.
[47] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[48] Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In Proceedings of the 25th international conference on Machine learning, pages 1032-1039, 2008.
[49] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In International Conference on Learning Representations, 2020.
[50] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
[51] Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian, Yaodong Yang, Qiang Fu, and Yang Wei. Quality-similar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[52] Brian Zhang and Tuomas Sandholm. Subgame solving without common knowledge. Advances in Neural Information Processing Systems, 34:23993-24004, 2021.
[53] Ming Zhou, Jingxiao Chen, Ying Wen, Weinan Zhang, Yaodong Yang, Yong Yu, and Jun Wang. Efficient policy space response oracles. arXiv preprint arXiv:2202.00633, 2022.
[54] Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in neural information processing systems, volume 20, 2007.
We provide the notation in Table 2.
Table 2: Notations
Notation	Meaning
Extensive-Form Games
N	N={1,2}; set of players
S	set of information states
s	s∈S; state node
A(s)	set of actions allowed to be performed at state s
P	player function
i/-i	i∈N; player i / players except i
si	player i's information state
Si	Si={s∈S|P(s)=i}; set of player's information state
Ai	Ai=∪s∈SiA(s); set of player i's actions
πi	player i's behavioral strategy
π	π=(πi,π-i) strategy profile
ui(π)	ui(π)=ui(πi,π-i); the payoff for player i
BR(π-i)	BR(π-i): arg maxπi'∈ΔSi u(πi',π-i); best response for player i against players -i
E(π)	E(π)=1/N ∑i∈N [maxπi' u(πi',π-i)-u(πi,π-i)]; exploitability for policy π
Meta Games
i/-i	i∈{1,2}; player i / players except i
πi^j/πi	j-th policy / any policy for player i
π	π=(π1,π2); joint policy
Πi	Πi:={πi^1,πi^2,...,}; policy set for player i
Π	Π=Π1×Π2; joint policy set
σi	σi∈ΔΠi; meta-policy over Πi
σ	σ=(σ1,σ2); joint meta-policy
MΠi,Π-i	(MΠi,Π-i)jk:=ui(πi^j,π^k_i); player i's payoff table in the meta-games (Πi,Π-i)
Ωi	the full set of all possible mixed strategies of player i
H(Πi)	H(Πi)={∑jβjπi^j|βj≥0, ∑j=1tβj=1}; Policy Hull of Πi={πi^1,πi^2,...,πi^t}
Pi(·,·)	Pi(Πi,Π-i)=πi*T MΠi,Π-i,π*-i; Relative Population Performance for player i
PE(·)	PE(Π)=1/2 ∑i=1,2 maxΠi'⊆Ωi P_i(Π_i',Π-i); Population Exploitability
GS(Πi|Π-i)	GS(Πi|Π-i)={∑jαj m_j: α≥0, α^T 1 = 1, m_j=MΠi,Π-i[j,:]}; gamescape of Πi
PSRO / PSD-PSRO Training
Πi^t	Πi^t={πi^1,...,πi^t}; the policy set of player i at the t-th iteration in PSRO
σi^t	σi^t∈ΔΠi^t; the meta-policy of player i at t-th iteration in PSRO
ρ(·)	policy representation
x_i	x_i∈X_i⊆[0,1] |S_i × A_i| sequence form representation for a policy
τ(s,a)	trajectory from the initial state to (s,a)
Bds(πi(s)||πi'(s))	Bregman divergence between πi(s) and πi'(s)
Div(·)	diversity on the policy set specified in different methods
dist(·,·)	distance function
λ	diversity weight
KL	Kullback-Leibler divergence
B.1 Derivation of Bregman Divergence
We define the Bregman divergence on the sequence-form representation of the policies. Given a strictly convex function  $d$  defined on  $\mathbb{R}^{|S_i\times \mathcal{A}_i|}$ , a Bregman divergence is defined as