SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can represent the set of plans  $(\pi_{\mathsf{T}1}^{*},\pi_{\mathsf{T}2}^{*})\in \Pi_{\mathsf{T}1}\times \Pi_{\mathsf{T}2}$  using binary variables and linear constraints, by using the well-known fact that  $\Pi_{\mathsf{T}1} = \mathcal{V}_{\mathsf{T}1}\cap \{0,1\}^{\Sigma_{\mathsf{T}1}}$ $\Pi_{\mathsf{T2}} = \mathcal{V}_{\mathsf{T2}}\cap \{0,1\}^{\Sigma_{\mathsf{T2}}}$ . Finally, for each  $z\in Z$  we model the product  $\pi_{\mathsf{T}1}^{*}[\sigma_{\mathsf{T}1}(z)]\pi_{\mathsf{T}2}^{*}[\sigma_{\mathsf{T}2}(z)]$  in the objective function by introducing additional binary variables  $\xi_z$  subject to the constraints (i)  $\xi_z\leq \pi_{\mathsf{T}1}^{*}[\sigma_{\mathsf{T}1}(z)]$  (ii)  $\xi_z\leq \pi_{\mathsf{T}2}^{*}[\sigma_{\mathsf{T}2}(z)]$  and (iii)  $\xi_z\geq \pi_{\mathsf{T}1}^{*}[\sigma_{\mathsf{T}1}(z)] + \pi_{\mathsf{T}2}^{*}[\sigma_{\mathsf{T}2}(z)] - 1$ . Putting everything together, we arrive to the best-response integer program
$$
\left\{ \begin{array}{l} \arg \max  \sum_ {z \in Z} \hat {u} _ {T} (z) \xi_ {z} \left(\sum_ {\substack {\pi_ {O 1} \in \Pi_ {O 1} (z) \\ \pi_ {O 2} \in \Pi_ {O 2} (z)}} \mu_ {O} (\pi_ {O 1}, \pi_ {O 2})\right), \quad \text {s u b j e c t t o :} \\ \begin{array}{l l} ①   \xi_ {z} \leq \pi_ {T 1} ^ {*} [ \sigma_ {T 1} (z) ] & \forall z \in Z \\ ②   \xi_ {z} \leq \pi_ {T 2} ^ {*} [ \sigma_ {T 2} (z) ] & \forall z \in Z \\ ③   \xi_ {z} \geq \pi_ {T 1} ^ {*} [ \sigma_ {T 1} (z) ] + \pi_ {T 2} ^ {*} [ \sigma_ {T 2} (z) ] - 1 & \forall z \in Z \\ ④   \pi_ {T 1} \in \mathcal {Y} _ {T 1} \\ ⑤   \pi_ {T 2} \in \mathcal {Y} _ {T 2} \\ ⑥   \xi_ {z} \in \{0, 1 \} & \forall z \in Z \\ ⑦   \pi_ {T 1} \in \{0, 1 \} ^ {\Sigma_ {T 1}} \\ ⑧   \pi_ {T 2} \in \{0, 1 \} ^ {\Sigma_ {T 2}}. \end{array} \end{array} \right.
$$
C Tabular Experiment Details
For 4-player Kuhn poker with 13 ranks and 4-player Liars dice (coin) with 2 outcomes we used the original multiplayer game but had player 0 and player 2 be on one team and player 1 and player 3 be on the other. Then each player's utility is the average of their original utility and their partner's. This version of Kuhn poker has 566280 leaves and this version of Liar's coin has 4080 leaves.
D Details of Google Research Football Training
The Google Research Football (GRF) Kurach et al. (2020) environment is a simulation environment for real-world football games, where each game consists of 3000 steps. Players need to control one or more players to compete with the opposing team, and the rules are the same as in general football games. The team with the highest number of goals at the end of the match wins. To validate the effectiveness of our algorithm, we conducted training and evaluation of our algorithm on the full 5 vs. 5 game in GRF, where the goalkeeper is controlled by built-in AI, while the rest of the four players are controlled by our model.
Feature Engineering and Action Space. Given the default setting where every player has access to complete game information, we have meticulously crafted a 133-dimensional feature vector to
enhance the training process. The input layer of the network is structured as  $[133 \times 128]$  to ensure effective feature representation and processing. Furthermore, the player's action space is defined using a default set of 19 discrete actions, encompassing fundamental moves such as movement, passing, and shooting. Consequently, the output layer of the network is configured as  $[128 \times 19]$ , allowing for the mapping of features to appropriate action selections.
Reward Shaping. Due to the sparsity of rewards in the GRF environment, it is crucial to design appropriate reward mechanisms. In addition to the rewards provided by the environment, we have designed specific reward components, including win reward, goal reward, yellow card reward, checkpoint reward and lost ball penalty, to expedite our training process and enhance its effectiveness. We adopt a consistent reward configuration across all our experiments.
Training. For each algorithm, our models were trained for 1500M steps on a cluster with 256 CPUs and 8 RTX3090 GPUs, using three different random seeds. The batch size for each GPU was set to 512 and The discount factor  $\gamma$  was set to 0.9999. We employed the Adam optimizer with a learning rate of 0.00005. Specifically, for team PSRO-MM, to mitigate the increased training cost associated with incorporating excessive mix strategies, we implemented a strategy of introducing two mix strategies to the population in each round (We set two here, however, you can add more if you wish). Each mix strategy was constructed by randomly selecting pure strategies from past checkpoints and assigning them to individual players within the team. This approach allows for a controlled inclusion of mix strategies while managing the overall training overhead, enabling an efficient training process while benefiting from the incorporation of diverse strategies within the population. More hyper-parameters are summarized as below:
Hyper-Parameters	Value
game length	3000
batch size	512
max clipped value loss	0.2
gradient clip norm	10
value loss	huber loss
discount factor γ	0.9999
learning rate	5e-4
ppo update number	5
gain	0.01
GAE λ	0.95
entropy coefficient	0.001
Table 1: Hyper-parameters
Evaluation Metrics. Due to the complexity of the GRF, calculating the exploitability of a specific policy or Predictive Error (PE) for a certain population becomes challenging. Both metrics involve max or min operators, and approximating the best response can be significantly inaccurate in such a complex game. We use the relative population performance (RPP)Vinyals et al. (2019a) as the metric. RPP represents the game interaction results of two populations A and B under their meta-nash equilibrium. Besides, considering our objective of identifying robust policies with strong performance in real-world games, we assess the effectiveness by comparing the average goal differences, score and Elo between the policies obtained from different methods and the built-in bots with varying difficulty levels within the General Reinforcement Learning Framework (GRF). This evaluation allows us to gauge the capabilities of the policies in challenging game scenarios and their ability to outperform predefined bot opponents.
Elo Calculation Elo refers to the Elo rating system used to measure the relative skill levels of the algorithms. For the computation of Elo ratings, they was computed by making agents play against each other. Here is a detailed explanation of how they were computed:
- Assign an initial Elo rating (1000 in our setting) to each player.
- Determine the expected score of each player in a game. This is calculated as follows:
- For player A: expected score =  $\frac{1}{1 + 10^{\frac{(B - A)}{400}}}$
- For player B: expected score =  $\frac{1}{1 + 10^{\frac{(A - B)}{400}}}$  where A and B are the current Elo ratings of the two players.
- Play the game and determine the actual score of each player.
- Update the Elo ratings of the two players based on the outcome of the game and the expected scores:
- For player A: new rating = old rating +  $K \times$  (actual score - expected score)
- For player B: new rating = old rating +  $K \times$  (expected score - actual score) where  $K$  is a constant that determines the "weight" of the update (set as 10 in this case).
- Repeat the process for each game, using the updated Elo ratings from the previous game as the starting point for the next game.
Additional Experiments. Since each player can observe the entire game information in GRF, we conducted experiments involving feature engineering to create an imperfect information scenario in the main body. In this scenario, each player is only able to observe information related to teammates, the ball, and the three nearest opposing players, which means there is always a lack of information regarding two opposing players. However, we also conducted experiments with perfect information, as depicted in the figures. Remarkably, we observed that the same pattern held true in both imperfect and perfect information scenarios, with Team PSRO-MM outperforming Team PSRO, which in turn outperformed self-play.
(b) Goal Diff. vs. AI
Figure 3: Team PSRO Results on Google Research Football perfect information scenario. Both Team PSRO and Team PSRO-MM outperform self play, with Team PSRO-MM performing the best.
Additionally, we aimed to compare the performance of both Team PSRO and Team PSRO-MM as the training time steps increased, using the policy trained by self-play as the benchmark. Since draws can occur in football, win rate may not accurately reflect the policies' performance. Therefore, we used average scores as the evaluation metric, where the score for a single game is 1 for a win, 0.5 for a draw, and 0 for a loss. To better reflect the relative strength of each policy against the other, we multiplied the average scores by 100.
The results of the experiments are presented in Figure 4, with the dashed line representing the performance of the policy trained through self-play. We can see that both Team PSRO and Team PSRO-MM ultimately outperform self-play. Additionally, Team PSRO and Team PSRO-MM exhibit comparable performance in the initial stages of training. However, as the training timesteps progress, Team PSRO-MM exhibits superior performance, surpassing Team PSRO. This signifies that the integration of mixed strategies enhance the training effectiveness and robustness of the models. With an extended duration of training, Team PSRO-MM demonstrates a heightened ability to adapt to complex game environments, yielding higher-quality strategies.
Finally, we include additional comparisons with fictitious play (FP) and prioritized fictitious self play (PFSP). These methods learn a team best response to either the opponent average strategy (in FP) or a mixture of the average and the most recent (PFSP). We show in figure 5 that Team PSRO and Team PSRO-MM outperform these additional baselines as well.
(a) Perfect Information
(b) Imperfect Information
Figure 4: Team PSRO results on both perfect and imperfect information scenarios of Google Research Football. Both Team PSRO and Team PSRO-MM outperform self-play, with Team PSRO-MM performing the best.
(a) Additional Baselines Elo
Figure 5: We include additional baselines of fictitious play and prioritized fictitious self play and show that Team PSRO and Team PSRO-MM are able to outperform all existing baselines.
(b) Additional Baselines Head to Head