SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[33] Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 6(2):215-219, 1994.
[34] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
[35] Haoning Xi, Didier Aussel, Wei Liu, S Travis Waller, and David Rey. Single-leader multi-follower games for the regulation of two-sided mobility-as-a-service markets. European Journal of Operational Research, 2022.
[36] Haoning Xi, Wei Liu, S Travis Waller, David A Hensher, Philip Kilby, and David Rey. Incentive-compatible mechanisms for online resource allocation in mobility-as-a-service systems. Transportation Research Part B: Methodological, 170:119-147, 2023.
[37] Haoning Xi, Yili Tang, S Travis Waller, and Amer Shalaby. Modeling, equilibrium, and demand management for mobility and delivery services in mobility-as-a-service ecosystems. Computer-Aided Civil and Infrastructure Engineering, 38(11):1403-1423, 2023.
[38] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3974-3983, 2018.
[39] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems (NeurIPS), 33:15737-15749, 2020.
[40] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems (NeurIPS), 35:24611-24624, 2022.
[41] Yi Zhang, Ping Sun, Yuhan Yin, Lin Lin, and Xuesong Wang. Human-like autonomous vehicle speed control by deep reinforcement learning with double q-learning. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1251-1256. IEEE, 2018.
[42] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145–6153, 2023.
A Proposition Proofs
Proof. The mixture partner policy is defined as  $\pi_{\mathrm{p}} = (1 - \epsilon)\pi_{\mathrm{e}} + \epsilon \pi_{\mathrm{r}}$  with the balance parameter  $\epsilon$ . Detailed derivations of Equation (5) in Proposition 1 are as follows:
$$
\begin{array}{l} \mathcal {H} (\pi_ {p} (. | s _ {t})) \\ = \sum_ {a} - \pi_ {p} (a | s _ {t}) \log \pi_ {p} (a | s _ {t}) \\ = \sum_ {a} - [ \epsilon \pi_ {\mathrm {r}} (a) + (1 - \epsilon) \pi_ {\mathrm {e}} (a | s _ {t}) ] \log \pi_ {p} (a | s _ {t}) \\ = \sum_ {a} - \epsilon \pi_ {\mathrm {r}} (a) \log \pi_ {p} (a | s _ {t}) - (1 - \epsilon) \pi_ {\mathrm {e}} (a | s _ {t}) \log \pi_ {p} (a | s _ {t}) \\ \geq \sum_ {a} ^ {(1)} - \epsilon \pi_ {\mathrm {r}} (a) \log \pi_ {p} (a | s _ {t}) \\ = \sum_ {a} - \epsilon \pi_ {\mathrm {r}} (a) \log \left[ \epsilon \pi_ {\mathrm {r}} (a) \left(1 + \frac {(1 - \epsilon) \pi_ {\mathrm {e}} (a | s _ {t})}{\epsilon \pi_ {\mathrm {r}} (a)}\right) \right] \tag {8} \\ \stackrel {(2)} {\geq} \sum_ {a} - \epsilon \pi_ {\mathrm {r}} (a) \left[ \log \epsilon + \log \pi_ {\mathrm {r}} (a) + \frac {(1 - \epsilon) \pi_ {\mathrm {e}} (a | s _ {t})}{\epsilon \pi_ {\mathrm {r}} (a)} \right] \\ = \sum_ {a} - \epsilon \pi_ {\mathrm {r}} (a) \log \epsilon - \epsilon \pi_ {\mathrm {r}} (a) \log \pi_ {\mathrm {r}} (a) - (1 - \epsilon) \pi_ {\mathrm {e}} (a | s _ {t}) \\ = \epsilon \mathcal {H} \left(\pi_ {\mathrm {r}} (.)\right) - \epsilon \log \epsilon - | A | (1 - \epsilon) \pi_ {\mathrm {e}} (a | s _ {t}) \\ \stackrel {(3)} {\geq} \epsilon \mathcal {H} (\pi_ {\mathrm {r}} (.) - \epsilon \log \epsilon - | A | (1 - \epsilon) \\ = \epsilon \mathcal {H} (\pi_ {\mathrm {r}} (.) - C _ {1}, \\ \end{array}
$$
where  $C_1 = \epsilon \log \epsilon + |\mathcal{A}| (1 - \epsilon)$ , and  $|\mathcal{A}|$  is the cardinality of the action apace  $\mathcal{A}$ . Inequality (1) and (3) hold because  $\pi_{\mathrm{p}}(a|s_t) \leq 1$  and  $\pi_{\mathrm{e}}(a|s_t) \leq 1$ . Inequality (2) holds because  $\log (1 + x) \leq x$ . Since  $C_1$  decreases in the range  $\epsilon \in (0, e^{|A| - 1}]$  and  $\epsilon$  belongs to  $[0, 1]$ , the lower bound of  $\mathcal{H}(\pi_p(|s_t|))$  increases as  $\epsilon$  increases.
A.2 Necessary Notations and Lemmas for Proofs of Proposition 2 and 3
Notations As the previous statement in section 4.2, that when the partner policy  $\pi_p$  is given with  $\mathbf{a}_t^p = \pi_p(.|s_t)$ , the two-player MDP with transition function  $\mathcal{T}$  can be converted in to a single-agent MDP with transition function denoted as  $\mathcal{T}_{\pi_p}(s'|s_t,a) = \sum_{a^{\mathrm{p}}}\pi_p(a^{\mathrm{p}}|s_t)\mathcal{T}(s'|s_t,a^{\mathrm{p}},a)$ . Note that the value estimation  $V(\pi_1,\pi_2)$  under the two-player transition function  $\mathcal{T}$  is equal to the value estimation  $V^{\mathcal{T}_{\pi_2}}(\pi_1) = \frac{1}{1 - \gamma}\mathbb{E}_{(s,a^1)\sim \rho_{\pi_1}(\mathcal{T}_{\pi_2})}[\sum_{a^2}\pi_2(a^2 |s)R(s,a^1,a^2)]$  under the single-agent transition function  $\mathcal{T}_{\pi_2}$ .  $\rho_{\pi_1}(\mathcal{T}_{\pi_2})$  denotes the stationary state-action distribution under the single-agent transition function  $\mathcal{T}_{\pi_2}$ , formally written as:
$$
\rho_ {\pi_ {1}} \left(s, a ^ {1}; \mathcal {T} _ {\pi_ {2}}\right) = (1 - \gamma) \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbb {P} \left(s _ {t} = s, a _ {t} ^ {1} = a ^ {1}; \pi_ {1}, \mathcal {T} _ {\pi_ {2}}\right). \tag {9}
$$
Then, the equivalence between the value estimations  $V(\pi_1,\pi_2),V^{\mathcal{T}_{\pi_2}}(\pi_1)$  is proven as:
$$
\begin{array}{l} V ^ {\mathcal {T} _ {\pi_ {2}}} (\pi_ {1}) = \frac {1}{1 - \gamma} \mathbb {E} _ {(s, a ^ {1}) \sim \rho_ {\pi_ {1}} (\mathcal {T} _ {\pi_ {2}})} [ \sum_ {a ^ {2}} \pi_ {2} (a ^ {2} | s) R (s, a ^ {1}, a ^ {2}) ] \\ = \sum_ {s, a ^ {1}, a ^ {2}} \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbb {P} \left(s _ {t} = s, a _ {t} ^ {1} = a ^ {1}; \pi_ {1}, \mathcal {T} _ {\pi_ {2}}\right) \pi_ {2} \left(a ^ {2} | s\right) R \left(s, a ^ {1}, a ^ {2}\right) \tag {10} \\ = \sum_ {s, a ^ {1}, a ^ {2}} \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbb {P} (s _ {t} = s, a _ {t} ^ {1} = a ^ {1}, a _ {t} ^ {2} = a ^ {2}; \pi_ {1}, \pi_ {2}, \mathcal {T}) R (s, a ^ {1}, a ^ {2}) \\ = V (\pi_ {1}, \pi_ {2}) \\ \end{array}
$$
We next give the necessary lemmas for proofs of Proposition 2 and 3. We first introduce the following Lemma, which describes the relationship between value estimation and policy distribution discrepancy on single-agent MDP.
Lemma 1. Given a single agent MDP with transition  $\mathcal{T}$ , assume there are two policies  $\pi_{a}$  and  $\pi_{b}$  with  $D_{TV}(\pi_a(.|s),\pi_b(.|s))\leq \epsilon ,\forall s,a$  holding. Then the difference between value functions satisfies that  $|V^{\mathcal{T}}(\pi_a) - V^{\mathcal{T}}(\pi_b)|\leq \frac{2\alpha\epsilon}{(1 - \gamma)^2}$ , where rewards are bounded by  $\alpha$  and  $\gamma$  is the discounted factor.
Proof. The Lemma 1 can be proved directly following the proof process of Theorem 1 in [39].
We provide the error bound on value estimations under different transitions used for the proof of Proposition 3.
Lemma 2. Given an MDP with true transition function  $\mathcal{T}^*$  and assume the estimated transition function  $\mathcal{T}'$  satisfies that, for a policy  $\pi$ ,  $\mathbb{E}_{\rho_{\pi(\mathcal{T}^*)}}[D_{TV}(\mathcal{T}^*(|s,a),\mathcal{T}'(|s,a))]\leq \epsilon_m$ , then we have:
$$
\left| V ^ {\mathcal {T} ^ {\prime}} (\pi) - V ^ {\mathcal {T} ^ {*}} (\pi) \right| \leq \frac {\alpha \gamma}{(1 - \gamma) ^ {2}} \epsilon_ {m}, \tag {11}
$$
where rewards are bounded by  $\alpha$  and  $\gamma$  is the discounted factor
Proof. The Lemma 2 can be proved directly following the prove process of Lemma 7 in [39].
A.3 Proof of Proposition 2
Proof. The total variance distance between the ego policy  $\pi_{\mathrm{e}}$  and the mixture partner policy policy  $\pi_{\mathrm{p}}$  satisfies  $D_{TV}(\pi_{\mathrm{e}}(.|s), \pi_{\mathrm{p}}(.|s)) \leq \epsilon$ , derived by:
$$
\begin{array}{l} D _ {T V} \left(\pi_ {\mathrm {e}} (.. | s), \pi_ {\mathrm {p}} (.. | s)\right) = \frac {1}{2} \sum_ {a} \left| \pi_ {\mathrm {e}} (a | s) - \pi_ {\mathrm {p}} (a | s) \right| \\ = \frac {1}{2} \sum_ {a} | \pi_ {\mathrm {e}} (a | s) - (1 - \epsilon) \pi_ {\mathrm {e}} (a | s) - \epsilon \pi_ {\mathrm {r}} (a | s) | \\ = \frac {1}{2} \sum_ {a} | \epsilon \pi_ {\mathrm {e}} (a | s) - \epsilon \pi_ {\mathrm {r}} (a | s) | \\ = \frac {\epsilon}{2} \sum_ {a} \left| \pi_ {\mathrm {e}} (a | s) - \pi_ {\mathrm {r}} (a | s) \right| \tag {12} \\ \leq \frac {\epsilon}{2} \sum_ {a} \left(\left| \pi_ {\mathrm {e}} (a | s) \right| + \left| \pi_ {\mathrm {r}} (a | s) \right|\right) \\ = \frac {\epsilon}{2} \sum_ {a} \left(\pi_ {\mathrm {e}} (a | s) + \pi_ {\mathrm {r}} (a | s)\right) \\ = \epsilon \\ \end{array}
$$
Next, following the Lemma 1 and the equivalence of the value estimations as shown in Equation (10), we have that  $|V(\pi, \pi_{\mathrm{p}}) - V(\pi, \pi_{\mathrm{e}})| < = \frac{2\alpha\epsilon}{(1 - \gamma)^2}, \forall \pi$ , where  $\delta = \frac{2\alpha\epsilon}{(1 - \gamma)^2}$ . Combining the definition of  $\pi_{\mathrm{e}}' = \arg \max_{\pi} V(\pi, \pi_{\mathrm{p}})$ , we have
$$
V \left(\pi_ {\mathrm {e}} ^ {\prime}, \pi_ {\mathrm {e}}\right) \geq V \left(\pi_ {\mathrm {e}} ^ {\prime}, \pi_ {\mathrm {p}}\right) - \delta \geq V \left(\pi_ {\mathrm {e}}, \pi_ {\mathrm {p}}\right) - \delta \geq V \left(\pi_ {\mathrm {e}}, \pi_ {\mathrm {e}}\right) - 2 \delta , \tag {13}
$$
let  $\zeta = 2\delta$  , then the proof is completed.
A.4 Proof of Proposition 3
Proof. Recall that, when the partner policy  $\pi_p$  is given with  $\mathbf{a}_t^p = \pi_p(.|s_t)$ , the two-player MDP with transition function  $\mathcal{T}$  can be converted in to a single-agent MDP with transition function denoted as  $\mathcal{T}_{\pi_p}(s'|s_t,a) = \sum_{a^{\mathrm{p}}}\pi_p(a^{\mathrm{p}}|s_t)\mathcal{T}(s'|s_t,a^{\mathrm{p}},a)$ . The total variance distance between single-agent transition functions under the partner prediction module or the ground-truth partner policy is given by:
$$
\begin{array}{l} D _ {T V} \left(\mathcal {T} _ {\pi_ {\tau}} (. | s, a), \mathcal {T} _ {\pi_ {\mathrm {p}}} (. | s, a)\right) = \frac {1}{2} \sum_ {s ^ {\prime}} \sum_ {a ^ {1}} \left| \pi_ {\tau} (a ^ {1} | s _ {t}) - \pi_ {\mathrm {p}} (a ^ {1} | s) \right| \mathcal {T} (s ^ {\prime} | s, a ^ {1}, a) \\ \stackrel {(1)} {=} \frac {1}{2} \sum_ {a ^ {1}} | \pi \left(a ^ {1} | s\right) - \pi_ {\mathrm {p}} \left(a ^ {1} | s\right) \mid \mathcal {T} \left(s ^ {\prime \prime} \mid s, a ^ {1}, a\right) \tag {14} \\ = \frac {1}{2} \sum_ {a ^ {1}} | \pi (a ^ {1} | s) - \pi_ {\mathrm {p}} (a ^ {1} | s) | \\ = D _ {T V} \left(\pi_ {\tau} (\cdot | s), \pi_ {\mathfrak {p}} (\cdot | s)\right) \\ \leq \epsilon_ {m} \\ \end{array}
$$
The equation (1) is holds because that the transition function  $\mathcal{T}$  is deterministic and equals to 1 at a unique next state  $s''$ . Define the  $\mu_{\tau}^{*}$  and  $\mu_{\mathfrak{p}}^{*}$  are the best response to the partner prediction module and the ground-truth partner policy respectively. Then the performance gap of these best responses collaborating to the partner policy is bounded by:
$$
\begin{array}{l} \left| V (\mu_ {\tau} ^ {*}, \pi_ {\mathrm {p}}) - V (\mu_ {\mathrm {p}} ^ {*}, \pi_ {\mathrm {p}}) \right| \\ = \left| V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) \right| \\ = \left| V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) + V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) + V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) \right| \\ = \left| V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) + V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) + V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) \right| \\ = \left| V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) + V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) + \delta \right| \tag {15} \\ \stackrel {(1)} {\leq} \left| V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) + V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) \right| \\ \leq \left| V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\tau} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\tau} ^ {*}\right) \right| + \left| V ^ {\mathcal {T} _ {\pi_ {\tau}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) - V ^ {\mathcal {T} _ {\pi_ {\mathrm {p}}}} \left(\mu_ {\mathrm {p}} ^ {*}\right) \right| \\ \stackrel {(2)} {\leq} \frac {2 \gamma \epsilon_ {m} \alpha}{(1 - \gamma) ^ {2}} \\ \end{array}
$$
Due to the optimality of  $\mu_{\mathrm{p}}^{*}$  when collaborating with the partner policy  $\pi_{\mathrm{p}}$ , thus  $V(\mu_{\tau}^{*}, \pi_{\mathrm{p}}) - V(\mu_{\mathrm{p}}^{*}, \pi_{\mathrm{p}}) \leq 0$ . In addition, according to the definitions of  $\mu_{\mathrm{p}}^{*}$ , i.e., the optimal policy under the MDP with transition function  $\mathcal{T}_{\pi_p}$ , and  $\mu_{\tau}^{*}$ , i.e., the optimal policy under the MDP with transition function  $\mathcal{T}_{\pi_{\tau}}$ , thus we have  $V^{\mathcal{T}_{\pi_{\mathrm{p}}}}(\mu_{\tau}^{*}) - V^{\mathcal{T}_{\pi_{\mathrm{p}}}}(\mu_{\mathrm{p}}^{*}) \leq 0$ ,  $V^{\mathcal{T}_{\pi_{\tau}}}(\mu_{\mathrm{p}}^{*}) - V^{\mathcal{T}_{\pi_{\tau}}}(\mu_{\tau}^{*}) \leq 0$ . Let  $\delta =$
$V^{\mathcal{T}_{\pi_{\tau}}}\left(\mu_{\tau}^{*}\right) - V^{\mathcal{T}_{\pi_{\mathrm{p}}}}\left(\mu_{\tau}^{*}\right)\geq 0.$  In summary, the inequality (1) holds because  $V(\mu_{\tau}^{*},\pi_{\mathrm{p}}) - V(\mu_{\mathrm{p}}^{*},\pi_{\mathrm{p}})\leq 0,$ $V^{\pi_{\mathrm{p}}}\left(\mu_{\tau}^{*}\right) - V^{\pi_{\mathrm{p}}}\left(\mu_{\mathrm{p}}^{*}\right)\leq 0,V^{\pi_{\tau}}\left(\mu_{\mathrm{p}}^{*}\right) - V^{\pi_{\tau}}\left(\mu_{\tau}^{*}\right)\leq 0$  and  $\delta \geq 0$
The inequality (2) holds by applying Lemma 2 with the condition  $D_{TV}\left(\mathcal{T}_{\pi_{\tau}}(.|s,a),\mathcal{T}_{\pi_{\mathrm{p}}}(.|s,a)\right)\leq \epsilon_{m},\forall s,a$  satisfied as proven in Equation 14.
(a) Matrix Game
(b) Returns
Figure 7: (a) The illustration of  $10 \times 10$  sub-matrix of in the upper left of the  $100 \times 100$  matrix. (b) The returns of E3T with varying balance parameters  $\epsilon$ . The left subfigure shows the self-play returns and the right subfigure shows the coordination returns across 10 independently trained policies.
Figure 8: Overcooked Layouts. From left to right: the layouts are Cramped Room, Asymmetric Advantages, Coordination Ring, Forced Coordination, and Counter Circuit.
B Experiment Settings
B.1 Environments
A  $100 \times 100$  Matrix game We verify the effectiveness of the mixed partner policy for learning a robust policy on a one-step matrix game. The matrix is partially shown in Figure 7(a), where we zoom in on the upper left  $10 \times 10$  sub-matrix. The matrix denoted as  $M$  is constructed such that  $M[i, i] = 1$ ,  $M[i + 1, i] = M[i, i + 1] = 0.2$  for  $i \geq 3$ , and  $M[1, 1] = M[2, 1] = 1$ . All other elements are zero. In the matrix game, two players select a row and a column independently, and the reward is determined by the intersection of their choices. We use a one-step matrix game to test the robustness of our mixture partner policy. The game has a  $100 \times 100$  matrix, where each element represents the reward for both players if they choose the corresponding row and column.
Overcooked We validate the mixed partner policy and the partner modeling module for improving zero-shot coordination on the Overcooked environment [8]. In this environment, the target of players is to make soup using onions, and dishes, and then serve it. After successfully serving a soup, players gain 20 rewards. Its discrete action space contains 6 actions: up, down, left, right, noop, and interact.
We conduct experiments on 5 layouts illustrated in Figure 8. The calculation of the cardinality of state spaces shown in Table 1 is explained as follows. Let's take the example of calculating the state space cardinality for the Counter Circuit layout. In the Counter Circuit layout, there are 16 positions on the kitchen counter where plates, onions, soup, or nothing can be placed. Each pot can be in one of 24 states, which includes being empty, having one onion, two onions, three onions, or waiting for
20 counts. There are two players, each of whom can stand at one of 14 grids and face one of four directions. Each player can hold either a plate, an onion, soup, or nothing in their hands. Therefore, the total number of possible states for this layout is:  $4^{16}C_{24}^2 24^2 4^2 4^2 \approx 5.8 \times 10^{16}$ . Despite having a smaller state space, Forced Coordination is more challenging than Asymmetric Advantages for zero-shot coordination. This is because the two players are separated and must complete different tasks in a timely manner to finish a soup in Forced Coordination, while players can independently complete a soup in Asymmetric Advantages.
B.2 Baselines
We compare proposed E3T with 6 baselines: Self-Play (SP) [28], Population-Based Training (PBT) [16, 8], Fictious Co-Play (FCP) [32], Trajectory Diversity (TrajeDi) [21], Maximum-Entropy Population-based training (MEP) [42] and Other-Play (OP) [15]. The baselines are described in detail below:
Self-Play (SP) [28] In coordination self-play, two players share the same policy during sampling and iteratively improve the policy by focusing on coordinating well with itself. If it converges, the self-play policy is able to coordinate with itself but is poor in generalization when meeting unseen policies.
Population Based Training (PBT) [16, 8] PBT is an online evolutionary algorithm, which is widely used in multi-agent RL to iteratively improve policies and finally converge to a population with high quality. At each iteration, paired policies randomly sampled from the population are improved by interacting with a number of timesteps, and then the population is updated according to the recorded performance. Specifically, the worst policy in the population is replaced with a mutation of the best policy.
Fictitious Co-Play (FCP) [32] FCP trains a policy that coordinates well with unseen humans by making the best response to a diverse population of partners. To obtain the population, it trains several self-play policies with different random seeds, in addition, it saves past checkpoints with different levels of coordination during training self-play policies.
Trajectory Diversity (TrajeDi) [21] TrajeDi solves the two-player zero-shot coordination problem by jointly training a diverse population and its common best response (BR). To be as diverse as possible, the training of the population is regularized by maximizing the Jensen-Shannon divergence (JSD) on the trajectory distributions of policies. Additionally, it trains a common BR to a diverse population, which indicates that the BR is able to coordinate well with unseen policies. We reuse the results of TrajeDi coordinating with human proxies reported in [42].
Maximum Entropy Population-based training (MEP) [42] MEP first trains a diverse population through PBT, and this population is encouraged to be diverse with the entropy of the mean policies in the population as an auxiliary reward. Secondly, a common best agent is trained to cooperate with policies in this diversified population, and the selection of partner policies is via prioritized sampling. The prioritization is dynamically updated based on coordination rewards during training progress.
Other-Play [15] Both E3T and other-play [15] both belong to the self-play training framework, and we discuss the difference between them as follows. Other-play avoids trapping into specialized conventions by breaking known symmetries in environments, for example the colors in Card games are symmetric, and thus improves the robustness of the trained ego agent. However, this approach relies heavily on the assumption of the state or action space is strongly symmetric, for example, the colors of the cards are symmetric on Hanabi. But it does not adapt to the Overcooked environment, a population two-player cooperative game, since there do not exist obvious and easy to exchange symmetries. Our work shares the benefits of other-play with end-to-end training and breaking the symmetric of self-play training, but E3T is more general since it incents partner exploration by employing a constant random policy and does not rely on any assumptions on environments.
E3T We propose an efficient and robust training approach, combing a mixed partner policy for enhancing the diversity of partner policy and the partner modeling module for better adaptation. An ego policy is trained by pairing with the mixture partner which consists of the parameter-copied ego policy and the random policy. We set the length of the context  $k = 5$ .
B.3 Experiments design of zero-shot coordination with humans
We describe the details of Human-AI coordination experiments. The results of baselines coordinating with real humans are shown in Figure 4(b). We follow the Human-AI coordination testing protocol proposed in [32]. We recruit 20 individuals to evaluate the coordination ability of E3T and state-
(a) Rating on the intelligence level and coordination ability
(b) Preference for MEP and E3T
Figure 9: Subjective assessments of E3T and MEP providing by real humans. (a). Average scores of the intelligence degree and the coordination ability, both scores are rated in the range of [1, 5], and a larger score indicates higher intelligence and coordination ability. (b). The proportion of real humans' preference for E3T and MEP.
of-the-art MEP, humans play a total of 600 rounds with them. For cost and time consideration, we reuse the Human-AI coordination results of other baselines reported in [8, 42]. We select models with the best coordination performance with human proxies for both E3T and MEP. For E3T, we set the  $\epsilon = 0.5$  on layouts except for Forced Coord. with  $\epsilon = 0.0$  on it. We use the code provided in [20] to implement the Human-Ai interaction interface.
Participants first watched a description video explaining the game rules and then they were familiar with the experimental process on a single-agent demo layout with the other player not moving to experience the game. During formal experiments, each participant cooperated with the AI agents to complete the 5 layouts and is suggested to get as larger rewards as possible. Pairing with each agent on each layout will repeat 3 times, and the experiment has a total of 30 rounds, with each round lasting one minute. We incentivized game performance: participants earned 50 yuan to complete 30 rounds, and the participants with the top 5 highest rewards earned an extra 50 yuan.