SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We ran SGDecompose 30 times, each time with its own set of 30 strategy profiles. These 900 strategy profiles are generated with  $\mathrm{CFR + }$  in self-play for 1000 iterations with random initializations. We
Algorithm 3 SGDecompose with behavior strategies
Input:  $G,\Pi^{\prime},\eta ,T,\lambda ,B$    
Initialize  $\tilde{u}$  to all O   
 $\Pi^{\times}\gets \times_{i\in N}\hat{\Pi}_{i}$    
for  $t\in 1\dots T$  do  $\Pi^{*}\gets \mathrm{getBRs}(\check{G},\Pi^{\prime})$ $\mathcal{B}\gets$  partition of  $\Pi^{\times}$  into batches of size B   
for  $\Pi^b\in \mathcal{B}$  do  $g\leftarrow \nabla_{\tilde{u}}\mathcal{L}(\Pi^{b},\Pi^{\prime},\Pi^{*};\breve{u},u)$ $\breve{u}\gets \breve{u} -\eta \cdot \frac{g}{\|g\|_2}$    
end for   
end for   
{Lastly, output  $\delta$  and  $\gamma \}$ $\delta \gets \max_{\pi \in \Pi^{\times}}|u_i(\pi) - \breve{u}_i(\pi)|$ $\gamma \gets \max_{\pi \in \Pi^{\prime}}\max_{i\neq j\in N\times N}(\breve{u}_{ij}(BR_{ij}(\pi_j),\pi_j) - \breve{u}_{ij}(\pi_i,\pi_j))$    
return  $\tilde{u},\gamma ,\delta$
Algorithm 4 getBRs
Input:  $\check{G},\Pi^{\prime}$ $\Pi_i^*\gets \varnothing \forall i\in N$    
for  $i\neq j\in N\times N$  do   
for  $\pi_j\in \Pi_j^\prime$  do compute  $\pi_{ij}^{*}\in \arg \max_{\pi_{i}^{\prime}\in \Pi_{i}}\check{u}_{ij}(\pi_{i}^{\prime},\pi_{j})$ $\Pi_i^*\gets \Pi_i^*\cup \{\pi_{ij}^*\}$    
end for   
end for   
return  $\times_{i\in N}\Pi_i^*$
randomly initialize  $\mathrm{CFR + }$  with regrets between 0 and 0.001 chosen uniformly at random, which are the default values in OpenSpiel.
Interestingly, we found that  $\mathrm{CFR + }$  converges to approximate Nash equilibria in Leduc poker, with a maximum value of  $\epsilon$  equal to 0.013 after 1000 iterations. As we will show in Appendix G.2, CFR also empirically produces approximate Nash equilibria in Leduc poker.
Let  $\Pi (\mathrm{CFR} + )_j$  denote the set of  $\mathrm{CFR}+$ -learned strategy profiles for run  $j$ ; and  $\Pi^{\times}(\mathrm{CFR} + )_j$  denote the set of all match-ups between these 30 strategy profiles. Figure 5 shows diversity of  $\Pi (\mathrm{CFR} + )_j$  for each of the 30 runs. We measure the diversity of each  $\Pi (\mathrm{CFR})_j$  by taking each pair of strategy profiles  $\pi, \pi' \in \Pi (\mathrm{CFR})_j$  and computing the total variation between these two probability distributions induced over the terminal histories of Leduc poker. We denote the maximum total variation for run  $j$  as  $TV_j$ , where
$$
T V _ {j} \doteq \max  _ {\pi , \pi^ {\prime} \in \Pi (\mathrm {C F R}) _ {j}} \frac {1}{2} \sum_ {z \in Z} | p (z, \pi) - p (z, \pi^ {\prime}) |.
$$
$TV_{j}$  is constrained to be between 0 and 1, where 0 means the two distributions are the same and 1 means they are maximally different. Figure 5 shows the maximum total variation between any two of the strategy profiles used in each run.
We used the same parameters for each run of SGDecompose:  $\lambda = 0.5$ ,  $B = 30$ ,  $T = 200$ . We used a learning rate schedule where the learning rate  $\eta$  begins at  $2^{-6}$ , then halves every 5 epochs until reaching  $2^{-17}$  to encourage convergence. Our results are shown in Figure 6. We see that across the 30 runs of SGDecompose, Leduc poker is at most 0.021-CSP in the neighborhood of  $\Pi^{\times}(\mathrm{CFR} + )_{j}$  and 0.009-subgame stable in the neighborhood of  $\Pi (\mathrm{CFR} + )_{j}$ . Figure 7 shows the maximum vulnerability with respect to the strategies in each of the runs compared to the bounds on vulnerability given by Proposition E.20. We compute the maximum vulnerability as
$$
\operatorname {V u l} _ {j} \doteq \max  _ {i \in N} \max  _ {\pi \in \Pi (\mathrm {C F R}) _ {j}} \operatorname {V u l} _ {i} \left(\pi , \Pi_ {- i} ^ {\times} (\mathrm {C F R}) _ {j}\right). \tag {10}
$$
Figure 5: The maximum total variation for each  $\Pi (\mathrm{CFR})_j$  used in different runs of SGDecompose in Leduc Poker. Different runs are shown on the x-axis, and the corresponding  $TV_{j}$  for run  $j$  is shown with the bars. A value of 0 indicates minimal diversity and 1 means maximal diversity. The minimum, mean, maximum and standard error across runs are 0.21, 0.22, 0.26 and 0.0016, respectively.
We see that the bounds are between 1.89 and 3.05 times the actual vulnerability, and are on average 2.51 times larger with a standard error of 0.049.
Figure 6: Boxplots showing the values of  $\delta_{j}$  and  $\gamma_{j}$  for each of the 30 runs of SGDecompose in Leduc Poker. Figure 6a shows the values of  $\delta_{j}$ , with the minimum, mean, maximum and standard error being 0.006, 0.009, 0.021 and 0.00046, respectively. Figure 6b shows the values of  $\gamma_{j}$ , with the minimum, mean, maximum and standard error being 0.003, 0.004, 0.009 and 0.00016, respectively.
G.2 CFR Finds Approximate Nash in Leduc Poker
It was previously believed that CFR does not compute an  $\epsilon$ -Nash equilibrium on 3-player Leduc for any reasonable value of  $\epsilon$ . Previous work found that CFR computed a 0.130-Nash equilibrium after  $10^{8}$  iterations Abou Risk & Szafron (2010). We saw in the previous section that  $\mathrm{CFR + }$  computes approximate Nash equilibria in Leduc poker—does this hold for CFR as well?
We ran 30 runs of CFR in self-play for 10,000 iterations and found that all of our strategies converged to an approximate Nash equilibrium with the maximum  $\epsilon = 0.017$  after only  $10^{4}$  iterations. Figure 8b
Figure 7: Bounds on vulnerability compared to true vulnerability in Leduc Poker for each run. Each of the 30 runs are shown on the x-axis. For each run  $j$ , we compute the bounds determined by Proposition E.20, which are  $(n - 1)\gamma_j + 2\delta_j = 2(\gamma_j + \delta_j)$ . These values are shown in orange. The blue bars are the maximum vulnerability in each run, computed using (10). The ordering of bars in this plot matches the ordering of bars in Figure 5. The rightmost run had both the highest vulnerability and highest diversity.
shows the shows the maximum deviation incentive
$$
\epsilon = \max  _ {\pi_ {i} ^ {\prime}} u _ {i} \left(\pi_ {i} ^ {\prime}, \pi_ {- i}\right) - u _ {i} (\pi)
$$
for each of the CFR strategies  $\pi$  computed by CFR in Leduc Poker. Each column is for one of the players and each point is one of the random seeds. We see the maximum value of  $\epsilon$  after 10,000 iterations is 0.017. Figure 8a shows the maximum deviation incentive  $\epsilon$  over 10,000 iterations. We average learning curves over 30 random seeds.
H Toy Hanabi
In games with low values of  $\delta$  and  $\gamma$ , self-play will perform well against new opponents; however, it is the converse also true? Do games where self-play performs poorly against new opponents have high values of  $\delta$  and  $\gamma$ ? As mentioned earlier, self-play struggles to generalize to new agents in some games with specialized conventions (Hu et al., 2020). Hanabi is one such game (Bard et al., 2020). Hanabi is a cooperative game where players cannot see their own hands, but can see all other players hands; therefore players must give each other hints on how to act.
We show that a small version of the game of Hanabi is not close to the space of CSP games and self-play is quite vulnerable. We use Tiny Hanabi in the Openspiel framework (Lanctot et al., 2019) with our own payoffs, shown in Figure 9. Chance deals one of two hands,  $A$  or  $B$  with equal probability. Only player 1 may observe this hand and must signal to other players through their actions,  $\sigma_{1}$  and  $\sigma_{2}$ , which hand chance has dealt. If both players 2 and 3 then correctly choose their actions to match chance's deal (( $a, a$ ) for  $A$  or  $(b, b)$  for  $B$ ) then all players get payoff equal to 1, otherwise all players get 0.
$\sigma_{1}$  and  $\sigma_{2}$  can come to mean different things,  $\sigma_{1}$  could signal to 2 and 3 to play  $a$ , or  $b$ . Self-play may learn either of these conventions. However, if a player trained in self-play encounters a set of players trained in an independent instance of self-play, they may not have compatible conventions.
This is indeed what happens when we train CFR on Tiny Hanabi in Figure 9. We trained 30 runs in self-play with different random initializations for 10,000 iterations. Some of these runs converged to
(b)
Figure 8: CFR empirically computes Nash Equilibria in Leduc Poker. (a) shows learning curves over iterations for each of the players. We measure  $\epsilon$  by finding a best-response with sequence-form linear programming every 1000 iterations. We show each of the individual instances of CFR with different random initializations in light-coloured lines and the average across seeds in bold. (b) shows the distribution of  $\epsilon$  at iteration 10,000.
each convention and when played against each other miscoordinated. We found
$$
\max  _ {i \in N} \max  _ {\pi \in \Pi (\mathrm {C F R})} \operatorname {V u l} _ {i} \left(\pi , \Pi_ {- i} ^ {\times} (\mathrm {C F R})\right) \approx 1,
$$
We decomposed Tiny Hanabi and found  $\delta = 0.5$  and  $\gamma \approx 0$ , meaning the true vulnerability matched what our bounds predicted since  $(n - 1)\gamma + 2\delta \approx 1$ . Why is  $\gamma \approx 0$ ? We found that this was because our algorithm was setting the payoffs to equal to 0.50 for all terminal histories, which is trivially polymatrix.
Figure 9: Tiny Hanabi. We omit payoffs of 0 at terminals.