SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {B} _ {d} \left(\boldsymbol {x} _ {i} \| \boldsymbol {x} _ {j} ^ {\prime}\right) = d \left(\boldsymbol {x} _ {i}\right) - d \left(\boldsymbol {x} _ {j} ^ {\prime}\right) - \left\langle \nabla d \left(\boldsymbol {x} _ {i} ^ {\prime}\right), \boldsymbol {x} _ {i} - \boldsymbol {x} _ {i} ^ {\prime} \right\rangle , \tag {18}
$$
for any  $\boldsymbol{x}_i, \boldsymbol{x}_i' \in \mathcal{X}_i$ . Intuitively, Bregman divergence is the gap between  $d(\boldsymbol{x}_i)$  and its first-order approximation at  $\boldsymbol{x}_i'$ . It is non-negative (zero is achieved if and only if  $\boldsymbol{x}_i = \boldsymbol{x}_i'$ ) and strictly convex in its first argument. So, it can be used to measure the difference between  $\boldsymbol{x}_i$  and  $\boldsymbol{x}_i'$ . Examples of Bregman divergence include the Squared Euclidean distance, which is generated by the  $l_2$  function, and the Kullback-Leibler (KL) divergence, which is generated by the negative entropy function. For sequence-form policies, we use the dilated Distance-Generating Function (dilated DGF) [21, 11, 31]:
$$
d \left(\boldsymbol {x} _ {i}\right) = \sum_ {s, a \in \mathcal {S} _ {i} \times \mathcal {A} _ {i}} \boldsymbol {x} _ {i} (s, a) \beta_ {s} d _ {s} \left(\frac {\boldsymbol {x} _ {i} (s)}{\| \boldsymbol {x} _ {i} (s) \| _ {1}}\right), \tag {19}
$$
where  $\beta_{s} > 0$  is a hyper-parameter and  $d_{s}$  is a strictly convex function defined on  $\mathbb{R}^{|A(s)|}$ , for example, the negative entropy function. It is proven that for the dilated DGF (Lemma D.2 in [31]), we have the Bregman divergence
$$
\mathcal {B} _ {d} \left(\boldsymbol {x} _ {i} \| \boldsymbol {x} _ {i} ^ {\prime}\right) = \sum_ {s, a \in \mathcal {S} _ {i} \times \mathcal {A} _ {i}} \boldsymbol {x} _ {i} (s, a) \beta_ {s} \mathcal {B} _ {d _ {s}} \left(\frac {\boldsymbol {x} _ {i} (s)}{\| \boldsymbol {x} _ {i} (s) \| _ {1}} \left\| \frac {\boldsymbol {x} _ {i} ^ {\prime} (s)}{\| \boldsymbol {x} _ {i} ^ {\prime} (s) \| _ {1}} \right.\right). \tag {20}
$$
The result allows us to compute the Bergman divergence using the behavior policy instead of explicitly using the sequence-form representation, which is prohibited in large-scale games. Formally, we define the distance of two policies  $\pi_i, \pi_i'$  as
$$
\operatorname {d i s t} \left(\pi_ {i}, \pi_ {i} ^ {\prime}\right) := \mathcal {B} _ {d} \left(\boldsymbol {x} _ {i} \| \boldsymbol {x} _ {i} ^ {\prime}\right) = \sum_ {s, a \in \mathcal {S} _ {i} \times \mathcal {A} _ {i}} \left(\prod_ {\widetilde {s} _ {i}, \widetilde {a} \in \tau (s, a)} \pi_ {i} (\widetilde {a} | \widetilde {s} _ {i})\right) \beta_ {s} \mathcal {B} _ {d _ {s}} \left(\pi_ {i} (s) \| \pi_ {i} ^ {\prime} (s)\right). \tag {21}
$$
Proof. Recall that the PE of joint policy set  $\Pi$  in two-player zero-sum games can be written into
$$
\begin{array}{l} \mathcal{PE}(\Pi) = \frac{1}{2}\sum_{i = 1,2}\max_{\Pi^{\prime}_{i}\subseteq \Omega_{i}}\mathcal{P}_{i}(\Pi^{\prime}_{i},\Pi_{-i}) \\ = \frac {1}{2} \left(\max  _ {\Pi_ {i} ^ {\prime} \subseteq \Omega_ {i}} \mathcal {P} _ {i} \left(\Pi_ {i} ^ {\prime}, \Pi_ {2}\right) - \min  _ {\Pi_ {- i} ^ {\prime} \subseteq \Omega_ {2}} \mathcal {P} _ {i} \left(\Pi_ {i}, \Pi_ {2} ^ {\prime}\right)\right), \\ \end{array}
$$
since  $\mathcal{P}_{-i}(\Pi_{-i}^{\prime},\Pi_{1}) = -\mathcal{P}_{i}(\Pi_{i},\Pi_{2}^{\prime})$
1. We can check this property directly, since
$$
\begin{array}{l} \mathcal {P E} (\Pi) = \frac {1}{2} \left(\max  _ {\Pi_ {i} ^ {\prime} \subseteq \Omega_ {i}} \mathcal {P} _ {i} \left(\Pi_ {i} ^ {\prime}, \Pi_ {2}\right) - \min  _ {\Pi_ {2} ^ {\prime} \subseteq \Omega_ {2}} \mathcal {P} _ {i} \left(\Pi_ {i}, \Pi_ {2} ^ {\prime}\right)\right) \\ \geq \frac {1}{2} (\mathcal {P} _ {i} (\Pi_ {i}, \Pi_ {2}) - \mathcal {P} _ {i} (\Pi_ {i}, \Pi_ {2})) = 0 \\ \end{array}
$$
2. We first note that  $\mathcal{H}(\Pi_1) \times \mathcal{H}(\Pi_2) \subseteq \mathcal{H}(\Pi_i') \times \mathcal{H}(\Pi_{-i}')$  means  $\mathcal{H}(\Pi_1) \subseteq \mathcal{H}(\Pi_i')$  and  $\mathcal{H}(\Pi_2) \subseteq \mathcal{H}(\Pi_{-i}')$ .
$\mathcal{H}(\Pi_2) \subseteq \mathcal{H}(\Pi_2')$  suggests that  $\forall \Phi_i \in \Omega_i$ , we have  $\mathcal{P}_i(\Phi_i, \Pi_2) \geq \mathcal{P}_i(\Phi_i, \Pi_2')$ , which is a property of relative population performance [2]. Take the maximum for both sides, we have,
$$
\max  _ {\Phi_ {i} \subseteq \Omega_ {i}} \mathcal {P} _ {i} \left(\Phi_ {i}, \Pi_ {- i}\right) \geq \max  _ {\Phi_ {i} \subseteq \Omega_ {i}} \mathcal {P} _ {i} \left(\Phi_ {i}, \Pi_ {- i} ^ {\prime}\right) \tag {22}
$$
Similarly, using the condition  $\mathcal{H}(\Pi_1) \subseteq \mathcal{H}(\Pi_i')$ , we have,
$$
\min  _ {\Phi_ {- i} \subseteq \Omega_ {- i}} \mathcal {P} _ {i} \left(\Pi_ {i}, \Phi_ {- i}\right) \leq \min  _ {\Phi_ {- i} \subseteq \Omega_ {- i}} \mathcal {P} _ {i} \left(\Pi_ {i} ^ {\prime}, \Phi_ {- i}\right) \tag {23}
$$
Combining Equation 22 and 23, we obtain  $\mathcal{P}\mathcal{E}(\Pi) \geq \mathcal{P}\mathcal{E}(\Pi')$
3. By the monotonicity of PE, we have
$$
\mathcal {P E} (\Pi) = \frac {1}{2} \left(\mathcal {P} _ {i} \left(\Omega_ {i}, \Pi_ {- i}\right) - \mathcal {P} _ {i} \left(\Pi_ {i}, \Omega_ {- i}\right)\right) \tag {24}
$$
Since  $\Pi_{-i} = \{\pi_{-i}\}$  contains only one policy, by the definition of relative population performance, we have,
$$
\mathcal {P} _ {i} \left(\Omega_ {i}, \left\{\pi_ {- i} \right\}\right) = u \left(B R \left(\pi_ {- i}\right), \pi_ {- i}\right); \mathcal {P} _ {i} \left(\left\{\pi_ {i} \right\}, \Omega_ {- i}\right)) = u \left(\pi_ {i}, B R \left(\pi_ {i}\right)\right) \tag {25}
$$
Substituting Equation 25 into Equation 24, we obtain
$$
\mathcal {P} \mathcal {E} (\Pi) = \mathcal {E} (\pi) \tag {26}
$$
4. By Equation 24 and the definition of the relative population performance, there exists NE  $(\sigma_{i},\pi_{-i})$  in game  $\mathbf{M}_{\Omega_i,\Pi_{-i}}$  and NE  $(\pi_{i},\sigma_{-i})$  in game  $\mathbf{M}_{\Pi_i,\Omega_{-i}}$  s.t.
$$
\mathcal {P E} (\Pi) = \frac {1}{2} (u (\sigma_ {i}, \pi_ {- i}) - u (\pi_ {i}, \sigma_ {- i}))
$$
By the definition of NE, we have,
$$
\begin{array}{l} \mathcal {P E} (\Pi) = \frac {1}{2} \left(\max _ {\pi_ {i} ^ {\prime} \in \Omega_ {i}} u \left(\pi_ {i} ^ {\prime}, \pi_ {- i}\right) - \min _ {\pi_ {- i} ^ {\prime} \in \Omega_ {- i}} u \left(\pi_ {i}, \pi_ {- i} ^ {\prime}\right)\right) \\ = \mathcal {E} (\pi) \tag {27} \\ \end{array}
$$
We now prove  $(\pi_i, \pi_{-i})$  is also the least exploitable policy in the joint policy set.  $\forall \pi' = (\pi_i', \pi_{-i}') \in \Pi_i \times \Pi_{-i}$ , we have,
$$
\mathcal {E} (\pi) = \mathcal {P E} (\Pi) \leq \mathcal {P E} \left(\left\{\pi_ {i} ^ {\prime} \right\} \times \left\{\pi_ {- i} ^ {\prime} \right\}\right) = \mathcal {E} \left(\pi^ {\prime}\right) \tag {28}
$$
where the inequality is from the monotonicity of PE.
Hence,  $(\pi_i,\pi_{-i})$  is the least exploitable policy in the Policy Hull of the population.
5. Necessity If  $(\sigma_{i}^{*},\sigma_{-i}^{*})$  is the NE of the game, we can regard the  $\{\sigma_i^*\}$  and  $\{\sigma_{-i}^{*}\}$  as the populations which contain a single policy. Then by property 2, we have,
$$
\begin{array}{l} \mathcal {P E} (\Pi) \leq \mathcal {P E} \left(\left\{\sigma_ {i} ^ {*} \right\} \times \left\{\sigma_ {- i} ^ {*} \right\}\right) \\ = \frac {1}{2} \left(\mathcal {P} _ {i} \left(\Omega_ {i}, \left\{\sigma_ {- i} ^ {*} \right\}\right) - \mathcal {P} _ {i} \left(\left\{\sigma_ {i} ^ {*} \right\}, \Omega_ {- i}\right)\right) \\ = \frac {1}{2} \left(u \left(\sigma_ {i} ^ {*}, \sigma_ {- i} ^ {*}\right) - u \left(\sigma_ {i} ^ {*}, \sigma_ {- i} ^ {*}\right)\right) = 0 \\ \end{array}
$$
Note that by property 1, we have  $\mathcal{P}\mathcal{E}(\Pi)\geq 0$  , so we have  $\mathcal{P}\mathcal{E}(\Pi) = 0$
Sufficiency Using the conclusion from 3, It is easy to check the sufficiency. If  $\mathcal{PE}(\Pi) = 0$ , there exists  $\sigma = (\sigma_{i},\sigma_{-i})$  s.t.  $\mathcal{E}(\sigma) = 0$ , which means  $\sigma$  is the NE of the game.
Proof. Recall that at each iteration  $t$ , the PSRO calculates the NE  $(\sigma_i^t, \sigma_{-i}^t)$  of the meta-game  $\mathbf{M}_{\Pi_i^t, \Pi_{-i}^t}$ , then adds the new policy  $\pi^t = (BR(\sigma_{-i}^t), BR(\sigma_i^t))$  to the population.
If  $PE(\Pi^t) > 0$ , then by the second property in Proposition 3.2, we know the population does not contain the NE of the game. Therefore, since  $(\sigma_i^t,\sigma_{-i}^t)\in \Pi^t$ , it is not the NE of the full game, which means at least one of the conditions holds: (1)  $BR(\sigma_{-i}^{t})\notin \mathcal{H}(\Pi_{i}^{t})$ ; (2)  $BR(\sigma_i^t)\notin \mathcal{H}(\Pi_{-i}^{t})$ , which suggests the enlargement of the Policy Hull.
In finite games, since the PSRO adds pure BR at each iteration, this Policy Hull extension stops at some iteration, saying  $T$ , in which case we have  $BR(\sigma_{-i}^{T}) \in \mathcal{H}(\Pi_{i}^{T})$  and  $BR(\sigma_{i}^{T}) \in \mathcal{H}(\Pi_{-i}^{T})$  (by above analysis). Let  $(\pi_{i}^{T+1}, \pi_{-i}^{T+1}) := (BR(\sigma_{-i}^{T}), BR(\sigma_{i}^{T}))$ , we claim that  $(\pi_{i}^{T+1}, \pi_{-i}^{T+1})$  is
the NE of the whole game. To see this, note that  $(\sigma_i^T,\sigma_{-i}^T)$  is the NE of the game  $\mathbf{M}_{\Pi_i^T,\Pi_{-i}^T}$  and  $\pi_i^{T + 1}\in \mathcal{H}(\Pi_i^T)$ , hence we have,
$$
u \left(\sigma_ {i} ^ {T}, \sigma_ {- i} ^ {T}\right) \geq u \left(\pi_ {i} ^ {T + 1}, \sigma_ {- i} ^ {T}\right). \tag {29}
$$
On the other hand, since  $\pi_i^{T + 1}$  is the BR to  $\sigma_{-i}^{T}$ , we have,
$$
u \left(\pi_ {i} ^ {T + 1}, \sigma_ {- i} ^ {T}\right) \geq u \left(\sigma_ {i} ^ {T}, \sigma_ {- i} ^ {T}\right). \tag {30}
$$
Combining Equation 29 and 30, we obtain  $u(\pi_i^{T + 1},\sigma_{-i}^T) = u(\sigma_i^T,\sigma_{-i}^T)$ , which suggests  $\sigma_i^T$  is also the BR of  $\sigma_{-i}^T$ . Similarly, we can prove that  $\sigma_{-i}^{T}$  is the BR of  $\sigma_i^T$ , and hence  $(\sigma_i^T,\sigma_{-i}^T)$  is the NE of the game.
Finally, by the fourth property in Proposition 3.2, we have  $\mathcal{P}\mathcal{E}(\Pi^T) = 0$ .
B.4 Proof of Proposition 4.3
This is an intermediate result in the proof of 3.3 (Appendix B.3).
B.5 Proof of Theorem 3.4
Proof. We prove this theorem by providing the counterexamples. Let's first focus on the following two-player zero-sum game:
$$
\begin{array}{c c c} R & P & S \\ R & 0 & - 1 \\ P & 1 & 0 \\ S & - 1 & 1 \\ R ^ {\prime} & 1 & - 1 \\ P ^ {\prime} & 1 & 1 \\ S ^ {\prime} & - 1 & 1 \end{array} \tag {31}
$$
The game is inspired by Rock-Paper-Scissors. We add high-level strategies  $\{R', P', S'\}$  to player  $i$  in the original R-P-S game. The high-level strategy can beat the low-level strategy with the same type.
Let  $\Pi_{i} = \{S,R^{\prime},P^{\prime}\}$  and  $\Pi_{-i} = \{R,P,\frac{1}{2} R + \frac{1}{2} P\}$ , where  $\frac{1}{2} R + \frac{1}{2} P$  means playing  $[R,P]$  with probability  $[\frac{1}{2},\frac{1}{2}]$ . The payoff matrix  $\mathbf{M}_{\Pi_i,\Pi_{-i}}$  is
$$
\begin{array}{c c c} R & P & \frac {1}{2} R + \frac {1}{2} P \\ S & - 1 & 1 \\ R ^ {\prime} & 1 & - 1 \\ P ^ {\prime} & 1 & 1 \end{array} \tag {32}
$$
It is in player  $i$ 's turn to choose his new strategy. Considering two candidates  $\pi_i^1 = S'$  and  $\pi_{-i}^2 = \frac{1}{2} R + \frac{1}{2} R'$  (There are many choices for  $\pi_i^2$  to construct the counterexample, we just take one for example).
The payoff of  $\pi_i^1$  against  $\Pi_{-i}$  is  $(-1, 1, 0)$ , the same as the first row in  $\mathbf{M}_{\Pi_i, \Pi_{-i}}$ . The payoff of  $\pi_i^2$  is  $(\frac{1}{2}, -1, -\frac{1}{4})$ , which is out of the gamescape of  $\Pi_i$ .
Denote  $\Pi_i^1 = \Pi_i\cup \{\pi_i^1\}$ $\Pi_i^2 = \Pi_i\cup \{\pi_i^2\}$  . By the analysis above, we have
$$
\mathcal {G S} \left(\Pi_ {i} ^ {1} \mid \Pi_ {- i}\right) \subsetneq \mathcal {G S} \left(\Pi_ {i} ^ {2} \mid \Pi_ {- i}\right) \tag {33}
$$
Hence, to promote the diversity defined on the payoff matrix and aim to enlarge the gamescape [2, 41], we should choose  $\pi_i^2$  to add. However, in this case, we show adding  $\pi_i^1$  is more helpful to decrease PE.
Denote  $\Pi^1 = \Pi_i^1 \times \Pi_{-i}$  and  $\Pi^2 = \Pi_i^2 \times \Pi_{-i}$ , by Equation 24 and a few calculation, we have,
$$
\mathcal {P E} \left(\Pi^ {1}\right) - \mathcal {P E} \left(\Pi^ {2}\right) = - \frac {1}{2} \mathcal {P} \left(\Pi_ {i} ^ {1}, \Omega_ {- i}\right) + \frac {1}{2} \mathcal {P} \left(\Pi_ {i} ^ {2}, \Omega_ {- i}\right) \approx - 0. 1 7 + 0. 1 0 <   0, \tag {34}
$$
i.e.,  $\mathcal{PE}(\Pi^1) < \mathcal{PE}(\Pi^2)$ . We can see that adding a new policy following the guidance of enlarging gamescape may not be the best choice to decrease PE.
Back to the Proposition 3.4, since we have found an example that,
$$
\mathcal {G S} \left(\Pi_ {i} ^ {1} \mid \Pi_ {- i}\right) \subsetneq \mathcal {G S} \left(\Pi_ {i} ^ {2} \mid \Pi_ {- i}\right) \quad \text {b u t} \quad \mathcal {P E} \left(\Pi^ {1}\right) <   \mathcal {P E} \left(\Pi^ {2}\right) \tag {35}
$$
hence,
$$
\mathcal {G S} \left(\Pi_ {i} ^ {1} \mid \Pi_ {- i}\right) \subsetneq \mathcal {G S} \left(\Pi_ {i} ^ {2} \mid \Pi_ {- i}\right) \Leftrightarrow \mathcal {P E} \left(\Pi^ {1}\right) \geq \mathcal {P E} \left(\Pi^ {2}\right) \tag {36}
$$
Rewrite the example in Equation 35, we have,
$$
\mathcal {P E} \left(\Pi^ {2}\right) \geq \mathcal {P E} \left(\Pi^ {1}\right) \quad \text {b u t} \quad \mathcal {G S} \left(\Pi_ {i} ^ {2} \mid \Pi_ {- i}\right) \supsetneq \mathcal {G S} \left(\Pi_ {i} ^ {1} \mid \Pi_ {- i}\right) \tag {37}
$$
Rename the superscripts (exchange the identity of 1, 2 in the superscript), we obtain,
$$
\mathcal {P E} \left(\Pi^ {1}\right) \geq \mathcal {P E} \left(\Pi^ {2}\right) \nRightarrow \mathcal {G S} \left(\Pi_ {i} ^ {1} \mid \Pi_ {- i}\right) \subsetneq \mathcal {G S} \left(\Pi_ {i} ^ {2} \mid \Pi_ {- i}\right) \tag {38}
$$
Combine Equation 36 and 38, we complete the proof.
Remark We offer another example in a symmetric game, proving that this proposition is also valid in this type of game.
$$
\begin{array}{c c c c c} & A & B & C & D \\ A & 0 & - 1 & - 0. 5 & - 1 \\ B & 1 & 0 & 0. 5 & - 1 \\ C & 0. 5 & - 0. 5 & 0 & 0 \\ D & 1 & 1 & 0 & 0 \\ E & 4 & 4 & - 4 & 4 \end{array} \tag {39}
$$
Current policy set  $\Pi_{i} = \Pi_{-i} = \{A,B\}$ , the payoff is
$$
\begin{array}{c c} A & B \\ A & \left( \begin{array}{l l} 0 & - 1 \\ 1 & 0 \end{array} \right) \end{array} \tag {40}
$$
Considering  $\pi_i^1 = C, \pi_i^2 = D$ , the payoff of  $\pi_i^1 = C$  against  $\Pi_{-i}$  is  $(0.5, -0.5)$ , contained in  $\mathcal{GS}(\Pi_i|\Pi_{-i})$ , while the payoff of  $\pi_i^2 = D$  against  $\Pi_{-i}$  is  $(1, 1)$ , which is out of  $\mathcal{GS}(\Pi_i|\Pi_{-i})$ .
Denote  $\Pi_i^1 = \Pi_i \cup \{\pi_i^1\}$ ,  $\Pi_i^2 = \Pi_i \cup \{\pi_i^2\}$ ,  $\Pi^1 = \Pi_i^1 \times \Pi_{-i}$  and  $\Pi^2 = \Pi_i^2 \times \Pi_{-i}$ , we have  $\mathcal{GS}(\Pi_i^1|\Pi_{-i}) \subsetneq \mathcal{GS}(\Pi_i^2|\Pi_{-i})$ . However,
$$
\mathcal {P E} \left(\Pi^ {1}\right) - \mathcal {P E} \left(\Pi^ {2}\right) = - \frac {1}{2} \mathcal {P} _ {i} \left(\Pi_ {i} ^ {1}, \Omega_ {- i}\right) + \frac {1}{2} \mathcal {P} _ {i} \left(\Pi_ {i} ^ {2}, \Omega_ {- i}\right) \approx 0. 1 7 - 2 <   0 \tag {41}
$$
Hence, similar to the above analysis, we can prove the Proposition 3.4 in symmetric games.
B.6 Proof of Theorem 4.4
Proof. Similar to Appendix B.3, we denote the meta-strategy at iteration  $t$  as  $(\sigma_i^t, \sigma_{-i}^t)$ . At iteration  $t$  of PSRO, if  $\mathcal{PE}(\Pi^t) > 0$ , then from Appendix B.3, we know that at least one of the conditions holds: (1)  $BR(\sigma_{-i}^t) \notin \mathcal{H}(\Pi_i^t)$ ; (2)  $BR(\sigma_i^t) \notin \mathcal{H}(\Pi_{-i}^t)$ . Without loss of generality, we assume
$$
B R \left(\sigma_ {- i} ^ {t}\right) \notin \mathcal {H} \left(\Pi_ {i} ^ {t}\right). \tag {42}
$$
Then  $\forall \hat{\sigma}_i\in \mathcal{H}(\Pi_i^t)$  and  $\lambda >0$
$$
\begin{array}{l} u (\hat {\sigma} _ {i}, \sigma_ {- i} ^ {t}) + \lambda \min  _ {\pi_ {i} ^ {k} \in \mathcal {H} (\Pi_ {i} ^ {t})} \mathbb {E} _ {s _ {i} \sim \hat {\sigma} _ {i}, b _ {2}} [ \mathrm {K L} (\hat {\sigma} _ {i} (s _ {i}) | | \pi_ {i} ^ {k} (s _ {i})) ] \\ = u \left(\hat {\sigma} _ {i}, \sigma_ {- i} ^ {t}\right) + 0 \\ <   u \left(B R \left(\sigma_ {- i} ^ {t}\right), \sigma_ {- i} ^ {t}\right) \\ <   u \left(B R \left(\sigma_ {- i} ^ {t}\right), \sigma_ {- i} ^ {t}\right) + \lambda \min  _ {\pi_ {i} ^ {k} \in \mathcal {H} \left(\Pi_ {i} ^ {t}\right)} \mathbb {E} _ {s _ {i} \sim \hat {\sigma} _ {i}, b _ {2}} \left[ \mathrm {K L} \left(\hat {\sigma} _ {i} \left(s _ {i}\right) \| \pi_ {i} ^ {k} \left(s _ {i}\right)\right) \right] \tag {43} \\ \end{array}
$$
where the first equation is because  $\hat{\sigma}_i\in \mathcal{H}(\Pi_i^t)$  and the last two inequalities is due to the definition of BR and Equation 42 respectively.
Algorithm 1 PSD-PSRO
Input: Initial policy sets  $\Pi_i^1,\Pi_{-i}^1$    
Compute payoff matrix  $\mathbf{M}_{\Pi_i^1,\Pi_{-i}^1}$    
Initialize meta policies  $\sigma_i^1\sim \mathrm{UNIFORM}(\Pi_i^1)$    
for  $t$  in  $\{1,2,\ldots \}$  do for player  $i\in \{1,2\}$  do Initialize  $\pi_i = \pi_i^t$  and sample  $K$  policies  $\{\pi_i^k\}_{k = 1}^K$  from Policy Hull  $\Pi_i^t$  for many episodes do Sample  $\pi_{-i}\sim \sigma_{-i}^{t}$  and collect the trajectory  $\tau$  by playing  $\pi_{i}$  against  $\pi_{-i}$  Discount the terminal reward  $u(\pi_i,\pi_{-i})$  to each state as the extrinsic reward  $r_1$  Discount  $R^{KL}(\tau)$  to each state as the intrinsic reward  $r_2$  Store  $(s,a,s',r)$  to the buffer, where  $s^\prime$  is the next state and  $r = r_1 + r_2$  Estimate the gradient in Equation 17 with the samples in the buffer and update  $\pi_{i}$  end for  $\pi_i^{t + 1} = \pi_i$  and  $\Pi_i^{t + 1} = \Pi_i^t\cup \{\pi_i^{t + 1}\}$  end for Compute missing entries in  $\mathbf{M}_{\Pi_i^t,\Pi_{-i}^t}$  Compute meta-strategies  $(\sigma_i^{t + 1},\sigma_{-i}^{t + 1})$  from  $\mathbf{M}_{\Pi_i^t,\Pi_{-i}^t}$    
end for   
Output: current meta-strategy for each player.
Table 3: The exploitability  $\times 100$  for populations generated by PSD with different diversity weights in Non-transitive Mixture Games. The standard error is calculated by running 5 experiments.
λ	0	1	2	3	5	10	50
Exp	40.88 ± 2.18	2.81 ± 0.66	1.72 ± 0.54	2.87 ± 0.70	3.73 ± 0.77	4.31 ± 0.67	8.22 ± 1.73
From Equation 43, we know that any policy in  $\mathcal{H}(\Pi_i^t)$  will not be considered as the optimal solution in Equation 14, since they are dominated by at least one of the policy that out of  $\mathcal{H}(\Pi_i^t)$ , i.e., the BR. Hence, before PE reaches 0, solving Equation 14 will find a policy not in  $\mathcal{H}(\Pi_i^t)$ , which guarantees the enlargement of the Policy Hull. Once we can not enlarge the Policy Hull by adding the optimal solution in Equation 14, the PE of the Policy Hull must be 0, otherwise the optimal solution will be outside of the Policy Hull.
C Algorithm for PSD-PSRO
We provide the algorithm for PSD-PSRO in Algorithm 1, which generates the algorithm of PSRO to incorporate the estimation of the diversity in Equation 14.
D Additional Study
D.1 PE on Leduc
In Figure 4, we report the exploitability of the meta NE on Goofspiel with 5 point cards setting.
D.2 Ablation Study on Diversity Weight
We conduct the ablation study on the sensitivity of the diversity weight  $\lambda$  on the Non-Transitive Mixture Game and Leduc Poker. In Table 3 and Table 4, we show the exploitability of the final population against different diversity weights in Non-Transitive Mixture Game and Leduc Poker, respectively. We can see that the  $\lambda$  can significantly affect the training convergence and the final exploitability. A large or small  $\lambda$  can lead to high exploitability. This suggests that properly combing the BR (implicitly enlarges the Policy Hull) and PSD (explicitly enlarges the Policy Hull) is critical to improving the efficiency at reducing the exploitability.
Figure 4: Exploitability on Goofspiel (5 point cards).
Table 4: The exploitability for populations generated by PSD with different diversity weights in Leduc Poker. The standard error is calculated by running 3 experiments.
λ	0	0.1	0.5	1	2	5
EXP	0.590 ± 0.017	0.403 ± 0.027	0.356 ± 0.012	0.398 ± 0.030	0.412 ± 0.042	0.664 ± 0.062
D.3 The Visualization of PH
we demonstrate the expansion of PH on the Disc game [2]. In Disc game, each pure strategy is represented by a point in the circle. Due to the linearity in the payoff function, a mixed strategy is equivalent to a pure strategy and thus can be equivalently visualized. Figure 5 shows the expansion of PH on the Disc game [2]. It demonstrates that PSD-PSRO is more effective than other PSRO variants at enlarging the PH.
D.4 Time Consumption in PSD-PSRO
We run an experiment on Leduc poker to estimate the time portion of different parts in our algorithm PSD-PSRO. We can see from Table 5 that the majority of time in PSD-PSRO is in BR solving and computing the payoff matrix, which is shared by PSRO and all its variants. The time spent on calculating our diversity metric is not significant, which accounts for  $7.9\%$  computational time.
E Benchmark and Implementation Details
E.1 AlphaStar888
AlphaStar888 is an empirical game generated from the process of solving Starcraft II [50], which contains a payoff table for 888 RL policies. The occupancy measure is reduced to the action distribution as the game can be regarded as a single-state Markov Game. We choose diversity weight  $\lambda$  to be 0.85 in this game.
E.2 Non-Transitive Mixture Game
This game consists 7 equally-distanced Gaussian humps on the 2D plane. We represent each strategy by a point on the 2D plane, which is equivalent to the weights (the likelihood of that point in each Gaussian distribution) that each player puts on the humps. The payoff of the game which contains
Figure 5: Equivalent visualization of PH of different methods across iterations in the Disc game
both non-transitive and transitive components is  $\pi_i^T S\pi_{-i} + \frac{1}{2}\sum_{k = 1}^{7}(\pi_i^k -\pi_{-i}^k)$ , where
$$
\mathcal {S} = \left[ \begin{array}{c c c c c c c} 0 & 1 & 1 & 1 & - 1 & - 1 & - 1 \\ - 1 & 0 & 1 & 1 & 1 & - 1 & - 1 \\ - 1 & - 1 & 0 & 1 & 1 & 1 & - 1 \\ - 1 & - 1 & - 1 & 0 & 1 & 1 & 1 \\ 1 & - 1 & - 1 & - 1 & 0 & 1 & 1 \\ 1 & 1 & - 1 & - 1 & - 1 & 0 & 1 \\ 1 & 1 & 1 & - 1 & - 1 & - 1 & 0 \end{array} \right].
$$
We choose diversity weight  $\lambda$  to be 1.9 in this game.
Table 5: Time Consumption in PSD-PSRO on Leduc poker
MAIN PART OF PSD-PSRO	PERCENTAGE
COMPUTE NE OF THE META-GAME	0.9%
COMPUTE THE APPROXIMATE BR	59.9%
COMPUTE THE EXTRA DIVERSITY METRIC	7.9%
COMPUTE THE PAYOFF MATRIX	31.3%
E.3 Leduc Poker
Since the DPP-PSRO needs evolutionary updates and cannot scale to the RL setting, and the code for BD&D-RD-PSRO on complex games, where f-divergence on occupancy measure is approximated using prediction errors of neural networks, is not available, we compare PSD-PSRO with P-PSRO,  $\mathrm{PSRO}_{r,N}$  and PSRO in this benchmark. We implement the PSRO framework with Nash solver, using PPO as the oracle agent. Hyper-parameters are shown in Table 6.
Table 6: Hyperparameters for Leduc poker
Hyperparameters	Value
Oracle
Oracle agent	PPO
Replay buffer size	104
Mini-batch size	512
Optimizer	Adam
Learning rate	3 × 10-4
Discount factor (γ)	0.99
Clip	0.2
Max Gradient Norm	0.05
Policy network	MLP (state_dim-256-256-256-action_dim)
Activation function in MLP	ReLU
PSRO
Episodes for each BR training	2 × 104
meta-policy solver	Nash
PSROrN
Episodes for each BR training	2 × 104
meta-policy solver	rectified Nash
P-PSRO
Episodes for each BR training	2 × 104
meta-policy solver	Nash
number of threads	3
PSD-PSRO
Episodes for each BR training	2 × 104
meta-policy solver	Nash
diversity weight λ	0.1
E.4 Goofspiel