SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To obtain the predicted partner action  $\hat{\mathbf{a}}_t^{\mathrm{p}}$ , we first devise an encoder network  $E_{\mathrm{c}}$  to extract historical partner context  $z_{t}^{\mathrm{c}}$  from a sequence of past state-action pairs  $\tau_{t} = \{(s_{t - i},a_{t - i}^{\mathrm{p}})_{i = 1}^{k}\}$ . The historical partner context  $z_{t}^{\mathrm{c}}$  contains rich information of the partner's behavior and coordination pattern, enabling our action prediction network  $F$  to predict the partner's action distribution based on  $z_{t}^{\mathrm{c}}$  and current state  $s_t$ . This procedure is formally written as follows:
$$
z _ {t} ^ {\mathrm {c}} = E _ {c} \left(\tau_ {t}\right), \hat {\mathbf {a}} _ {t} ^ {\mathrm {p}} = F \left(s _ {t}, z _ {t} ^ {\mathrm {c}}\right) \tag {6}
$$
The parameters of the context encoder  $E_{\mathrm{c}}$  and the action prediction network  $F$  are jointly optimized by minimizing a cross-entropy loss:
$$
\mathcal {L} = \sum_ {i = 1} ^ {N} - \mathbf {y} _ {i} ^ {T} \log \hat {\mathbf {a}} _ {i} ^ {\mathrm {p}}, \tag {7}
$$
where  $N$  is the batch size, and  $\mathbf{y}_i$  is a one-hot vector indicating the actual partner action.
We provide a theoretical analysis of how the accuracy of partner modeling affects the coordination ability as shown in Proposition 3. We analyze the coordination performance gap between the best responses to the ground-truth partner policy and the partner prediction module. It is established that this performance gap is proved to be proportional to the prediction error.
Proposition 3. Denote  $\pi_{\tau}(.|s_t) = \mathbb{E}_{\tau_t}[F(s_t,E_c(\tau_t))]$  as the expectation of the proposed partner prediction module over historical trajectory  $\tau_{t}$ . The ground-truth partner policy is denoted as  $\pi_p(.|s_t)$ . Assume the learned  $\pi_{\tau}(.|s_t)$  satisfies that  $D_{TV}(\pi_{\tau}(.|s_t),\pi_p(.|s_t))\leq \epsilon_m,\forall s_t$ . Define the  $\mu_{\tau}^{*}$  and  $\mu_p^*$  are the best response to the partner prediction module and the ground-truth partner policy respectively. Then the value functions of these best responses collaborating to the partner policy satisfy that:  $|V(\mu_{\tau}^{*},\pi_{p}) - V(\mu_{p}^{*},\pi_{p})|\leq \frac{2\gamma\epsilon_{m}\alpha}{(1 - \gamma)^{2}}$ . Proof. See Appendix.
4.3 Ego Policy Training and Test
E3T separately trains the partner action prediction module via minimizing the loss function in Equation (7) and trains the ego policy that makes decisions depending on both observations and predicted partner action distribution via PPO [24]. The detailed training process of E3T is described in the Algorithm 1. During testing time, the ego agent zero-shot coordinates with an unseen partner, without parameters updating or temporally-extended exploration as [22].
Implement Details The partner modeling module of E3T takes current observation at step  $t$  and partner's past  $k$  state-action pairs as input as shown in Equation (6), we set  $k = 5$  in this work. The ego policy  $\pi_{\mathrm{e}}(.|s_t, \hat{\mathbf{a}}_t^p)$  conditions on current state  $s_t$  and predicted partner action distribution  $\hat{\mathbf{a}}_t^p$ ,
Table 1: The cardinality of the state space of 5 layouts. The state space's cardinality of Counter Circuit is up to  $5.8 \times 10^{16}$ .
Layout	Cramped Rm.	Asymm. Adv.	Coord. Ring	Forced Coord.	Counter Circ.
State Space	5.3 × 107	1.2 × 1014	6.8 × 1010	2.0 × 109	5.8 × 1016
Table 2: Comparisons of baselines in terms of training time and coordination performance with behaviour-cloned human proxies. The training time represents the hours required for training one model on the layout Cramped Rm (training time on other layouts is similar). We report the average and the standard error of coordination rewards over 5 random seeds.
Baseline	SP	PBT	FCP	MEP	E3T
Training Time(h)	0.5	2.7	7.6	17.9	1.9
Average Reward	65.8 ± 5.5	66.8 ± 5.0	71.7 ± 3.7	89.7 ± 7.1	114.1 ± 6.4
which enables the ego agent to analyze and adapt to its partner, while a random action distribution  $\hat{\mathbf{a}}_t^e$  is fed into the partner policy as the predicted ego agent policy. The random action distribution breaks symmetries and conventions commonly seen in self-play agents and introduces randomness into the partner policy, which in turn improves partner diversity. Moreover, this is consistent with the fact that some humans may not take their AI partners' actions into consideration, but the AI agents should adapt to their human partners.
In this section, we evaluate E3T on zero-shot collaborating with behavior-cloned human proxies, real humans, and AI baselines. More details of the experiment setting and additional results can be found in Appendix.
5.1 Environments
We evaluate the zero-shot coordination performance of E3T on a  $100 \times 100$  cooperative matrix game [21, 42] and the Overcooked environment [8]. Overcooked has a discrete action space with 6 actions, and we conduct experiments on 5 different layouts that pose various coordination challenges, such as avoiding blocking each other and preparing the required ingredients for a soup together. In addition, the state spaces of 5 layouts are different as shown in Table 1. For instance, the state space of Counter Circuit reaches up to  $5.8 \times 10^{16}$ . This statistic indicates the 5 layouts are challenging for explorations. We have
(a) Results on  $100 \times 100$  matrix game
(b) AI-AI coordination
Figure 3: (a) We report the cross-play returns between 10 independently trained models for each baseline. (b) These results show the normalized coordination rewards between baselines averaged over 5 layouts.
conducted additional experiments on the Google Football [17] environment's "3 vs 1 with Keeper" layout, which is a three-player cooperative game with a discrete action space comprising 19 actions. More details about the environment descriptions can be found in Appendix.
5.2Baselines
We compare proposed E3T with 6 baselines: Self-Play (SP) [28], Population-Based Training (PBT) [16, 8], Fictious Co-Play (FCP) [32], Trajectory Diversity (TrajeDi) [21], Maximum-Entropy Population-based training (MEP) [42] and Other-Play (OP) [15]. All methods, except SP, OP, and E3T, fall under the category of population-based methods, which aim to develop a robust ego policy through interaction with a partner population. Detailed descriptions of baselines and the network details of E3T can be found in Appendix.
5.3 Result Analysis
We compare E3T with other baselines on zero-shot collaboration with the human proxy, real humans, and AI baselines.
(a) Results on coordinating with human proxy models
(b) Results on collaborating with real humans
Figure 4: (a) We plot the mean and standard error of coordination rewards over 5 random seeds. The parameter  $\epsilon$  is set to 0.5 for all layouts except Forced Coord., where  $\epsilon$  is settoas 0.0. In this layout, the ego agent does not need too much exploration due to the narrow active range. (b) We plot the mean and standard error of humans collaborating with baselines over all layouts. These results show that E3T outperforms all other baselines when collaborating with real humans.
Zero-shot Coordination on  $100 \times 100$  matrix game Figure 3(a) presents the cross-play[15] results of 10 independently trained policies for each baseline on a one-step matrix game. The cross-play result measures the coordination performance between models that belong to the same baseline but have different random seeds. In this game, two players select a row and a column independently, and the reward is determined by the intersection of their choices. As shown in Figure 3(a), E3T and MEP can converge to the optimal strategies with the largest return as 1, while E3T can converge faster than MEP. These results indicate that the mixture partner policy can enable sufficient exploration and avoid getting stuck in local optima that generalize poorly on the game with large action space. More details of the matrix game are given in Appendix.
Training Efficiency Analysis Table 2 compares the training time and average rewards of different baselines, where the rewards are calculated by averaging over 5 layouts on collaboration with human proxies. These results suggest that E3T outperforms existing baselines on collaboration with human proxies, and E3T is more efficient than population-based methods. For instance, it only spends  $10.6\%$  training time of previous state-of-the-art MEP.
AI-AI Coordination The zero-shot coordination performance between AI baselines are shown in Figure 3(b), where each element represents the normalized cooperation rewards averaged over 5 layouts for a pair of baselines. The performance of baselines cooperating with themselves is measured by the cross-play metric. As shown in Figure 3(b), E3T outperforms other baselines in coordinating with AI models. In addition, SP and PBT, which train policies only based on rewards and ignore diversity, have poor generalization to unseen AI partners. MEP and FCP, which train the ego agent by best responding to a pre-trained diverse partner population, perform better than SP and PBT, but worse than E3T. These results indicate that partner diversity is essential for zero-shot coordination, and E3T learns a roust ego policy by providing sufficient partner diversity and being equipped with adaptation ability. More results of AI-AI coordination on 5 layouts can be found in Appendix.
Human Proxy-AI Coordination Figure 4(a) shows the results of baselines cooperating with behavior-cloned human proxies. These results suggest that E3T consistently outperforms existing baselines on all layouts. In addition, FCP, MEP, and TrajeDi have gained better or similar generalization performance than SP and PBT, owing to training an ego agent based on a diverse partner policy. These results indicate that the diversity of partners is important for the zero-shot coordination task. Note that SP achieves comparable performance to population-based MEP on Forced Coord. and E3T achieves the best coordination performance at parameter  $\epsilon = 0.0$ . One possible reason is that the Forced Coord. layout is a difficult layout with a very narrow activity range and two agents being forced to coordinate to complete a soup. Therefore, it requires more cooperation than exploration. Moreover, E3T with  $\epsilon = 0.0$  is better than SP, indicating that the partner modeling module is beneficial to adapting to an unseen partner.
Human-AI Coordination The average rewards over 5 layouts of AI baselines coordinating with real humans are shown in Figure 4(b). We follow the Human-AI coordination testing protocol
proposed in [32] and recruit 20 individuals to evaluate the Human-AI coordination ability of E3T and state-of-the-art MEP, where individuals play a total of 600 rounds with AI. For cost and time consideration, we reuse the Human-AI coordination results of other baselines reported in [8, 42]. Figure 4(b) shows that E3T outperforms all other baselines when zero-shot coordinating with real humans. We also collect individuals' subjective assessment of E3T and MEP about the degree of intelligence, collaborative ability, and preference between the two. More details of the Human-AI coordination setting and individuals' subjective assessment are given in Appendix.
(a) Ablations of E3T's components
(b) Ablations of the balance parameter  $\epsilon$
Figure 5: (a) Ablation of E3T's components on collaborating with AI baselines.  $\mathrm{E3T(+R)}$  and  $\mathrm{E3T(+P)}$  respectively integrate the mixture partner and the partner modeling module into the self-play framework. The balance parameter  $\epsilon$  is set to 0.3 for  $\mathrm{E3T(+R)}$ . (b) Results on E3T with different balance parameters  $\epsilon$ . We plot the average rewards over 5 random seeds when E3T collaborating with AI (MEP, PBT) baselines and human proxy on layout  $Coord$ . Ring. The blue bar shows performance of E3T cooperating with AI baselines, while the orange and the green bars show its performance when cooperating with the human proxy, with starting positions switched.
Figure 6: Relationship between the accuracy of partner action prediction and coordination performance when collaborating with the human proxy on layout  $Coord$ . Ring. We plot the rewards and prediction accuracy obtained by coordinating with the human proxy at different checkpoints along the training process.
Analysis of Component Ablation Results We present the ablation results of E3T's components in Figure 5(a). We evaluate the effectiveness of each component on collaborating with AI baselines (including MEP, FCP, and PBT). We use  $(+\mathrm{R})$  and  $(+\mathrm{P})$  to denote models incorporated with the mixture partner policy and the partner modeling module, respectively. As shown in Figure 5(a), E3T  $(+\mathrm{P})$  achieves a higher average reward than SP on each layout, which shows that the partner modeling module can improve zero-shot coordination by updating actions accordingly. Likewise, E3T  $(+\mathrm{R})$  surpasses SP on 4 layouts except for the Forced Coord., which shows that encouraging more exploration in the partner policy can discover more coordination patterns during training and thus can enhance zero-shot coordination with unseen partners during testing. The lower performance on Forced Coord. may be due to its narrow action space and the lack of reward incentives for coordination when using a too random partner policy during training. Moreover, E3T  $(\mathrm{R} + \mathrm{P})$  that combines both the mixture partner policy and the partner modeling module performs better
than E3T(R) and E3T(P) on the average reward over all layouts.
Analyzing the effect of the hyper-parameter  $\epsilon$  Figure 5(b) shows how the balance parameter  $\epsilon$  of the mixture partner affects the performance of E3T when collaborating with AI baselines and the human proxy. We observe that the coordination rewards of E3T increase and then decrease as  $\epsilon$  increases for both types of partners. The results indicate that both collaboration ability and partner diversity are essential for zero-shot coordination, but simply increasing the partner action diversity is not enough. The optimal values of  $\epsilon$  are 0.3 and 0.5 for AI baselines and human proxy, respectively. One possible reason for this discrepancy is that the human proxy tends to generate more 'noop' actions [8], which reduces the need for coordination behaviors compared to AI baselines.
Table 3: We present the coordination performance of baselines during training and zero-shot testing on a three-player Google Football game, where the coordination performance is measured by the winning rates of scoring a goal. We independently train 3 policies with different random seeds for each baseline. For zero-shot testing, we consider all 6 possible permutations of the 3 trained policies and report the mean and standard error of the winning rates as Zero-shot (Mean). We also report the highest winning rate among the permutations as Zero-shot (Max).
Training Performance	Zero-shot (Mean)	Zero-shot (Max)
Self-play	0.79 ± 0.08	0.07 ± 0.08	0.24
E3T(mixture policy)	0.76 ± 0.09	0.49 ± 0.21	0.78
Analyzing the effect of partner modeling module To demonstrate the effect of the proposed partner modeling module, we plot the accuracy of predicting partner actions and the rewards when cooperating with a human proxy. As shown in Figure 6, the reward and the prediction accuracy show the same tendency: better partner modeling leads to improvement in coordination. In addition, the accuracy at  $\epsilon = 0.5$  reaches around  $55\%$ , which is higher than the accuracy around  $40\%$  when  $\epsilon = 0.0$ . This indicates that mixing a random policy can provide more exploration and thus generalize well to unseen partners.
Does the mixture partner policy accommodate the multi-player game? We verify the effect of the mixture partner policy on improving the zero-shot coordination performance on a three-player Google Football game, the 3 vs 1 with Keeper scenario. We train self-play policies via the Multi-Agent PPO (MAPPO) [40], where policies have shared parameters. E3T(mixture policy) integrates the mixture partner policy into the self-play training process, in detail, the three agents randomly sample an action with a probability as  $\epsilon$ , otherwise, take actions according to the action policies. We independently train 3 policies with different random seeds for each baselines and report the cross-play results between unseen policies to test the zero-shot coordination ability of these models.
Table 3 shows that self-play policies fail to zero-shot cooperate with policies from different training seeds, because they may fall into some specific coordination conventions during training. Our proposed mixture partner policy can increase the behavioral diversity of the partners, allowing the ego policy to encounter different coordination patterns during training. Consequently, our method with the mixture partner policy can train robust ego policies that can adapt well to policies independently trained from different training seeds. Moreover, increasing partner's behavior diversity during training can significantly improve the generalization ability of the trained ego policy.
In this paper, we propose an Efficient End-to-End Training approach for zero-shot human-AI coordination, called E3T. Different from previous population-based methods, E3T employs a mixture policy of ego policy and random policy to construct the partner policy, enabling the partner policy to be both coordination-skilled and diverse. In this way, the ego-agent is able to be end-to-end trained in such a mixture policy without pre-training a population, thus significantly improving the training efficiency. Additionally, we introduce a partner modeling module that allows the ego agent to predict the partner's next action, enabling adaptive collaboration with partners exhibiting different behavior patterns. Empirical results on Overcooked clearly demonstrate that E3T can significantly improve training efficiency while achieving comparable or superior performance compared to existing methods.
Limitations and Future Work In this paper, we only explore the feasibility of our framework in zero-shot two-player human-AI coordination tasks but neglect general multi-player (more than two) human-AI coordination tasks. Future research will focus on expanding the framework to multi-player (more than two) human-AI coordination tasks, with a goal of enhancing both zero-shot coordination performance and training efficiency. Since AI agents are unable to regulate human behavior, indiscriminately collaborating with humans in accomplishing their goals could result in unlawful circumstances. Therefore, future research should concentrate on developing AI agents capable of detecting dangerous intentions and preventing their misuse for illicit or military purposes.
We would like to thank Yang Li for providing the models for the MEP and FCP baselines. Haifeng Zhang thanks to the support of the NSFC Grant Number 62206289.
[1] Stefano V Albrecht, Jacob W Crandall, and Subramanian Ramamoorthy. Belief and truth in hypothesised behaviours. Artificial Intelligence, 235:63-94, 2016.
[2] Stefano V Albrecht, Somchaya Liemhetcharat, and Peter Stone. Special issue on multiagent interaction without prior coordination: Guest editorial. Autonomous Agents and Multi-Agent Systems (AAMAS), 31(4):765-766, 2017.
[3] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
[4] Dipyaman Banerjee and Sandip Sen. Reaching pareto-optimality in prisoner's dilemma using conditional joint action learning. Autonomous Agents and Multi-Agent Systems (AAMAS), 15:91–108, 2007.
[5] Cynthia Breazeal, Cory D Kidd, Andrea Lockerd Thomaz, Guy Hoffman, and Matt Berlin. Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 708–713. IEEE, 2005.
[6] George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1):374, 1951.
[7] Sandra Carberry. Techniques for plan recognition. User Modeling and User-Adapted Interaction, 11(1):31-48, 2001.
[8] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[9] Philip R Cohen, C Raymond Perrault, and James F Allen. Beyond question answering. Strategies for Natural Language Processing, 245274, 1981.
[10] Hao Ding, Chengxing Jia, Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Coordination scheme probing for generalizable multi-agent reinforcement learning, 2023.
[11] Can Erdogan and Manuela Veloso. Action selection via learning behavior patterns in multi-robot systems. In Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI), 2011.
[12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), pages 1861-1870. PMLR, 2018.
[13] Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect-information games. NeurIPS Deep Reinforcement Learning (DRL) Workshop, 2016.
[14] Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems (NeurIPS), 33:20179–20191, 2020.
[15] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pages 4399-4410. PMLR, 2020.
[16] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017.