SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Besides the objective coordination rewards, we also collected participants' subjective assessment of E3T and MEP about the degree of intelligence, collaboration ability, and preference between the two. After the experiments of each layout, participants were required to answer these three questions. Study sessions lasted around 50 minutes on average, including being familiar with the experiment process, playing with AI agents, recording rewards, and filling in subjective assessments.
Figure 4(b) shows that E3T outperforms all other baselines when zero-shot coordinating with real humans on average reward across all layouts. Subjective assessments of E3T and MEP are shown in Figure 9. These results show that participants rate E3T with higher scores on the intelligence level and the coordination ability than MEP, across all layouts. In addition, more participants prefer E3T than MEP.
B.4 The network details of E3T
The network architecture of E3T including the policy network and the partner modeling module is described in Table 4. We describe E3T for the Overcooked environment, which has a discrete action space with 6 actions. E3T are trained based on Tensorflow-GPU 1.15.5 and CUDA11.4+codnn8.8.
C.1 Results on  $100 \times 100$  Matrix Game
The performance of E3T under different balance parameters  $\epsilon$  is illustrated in Figure 7(b). When  $\epsilon = 1.0$ , the ego policy is trained by pairing with the completely random partner policy and fails to coordinate during self-play and cross-play with other independently trained ego policies. When  $\epsilon < 1.0$ , E3T sacrifices some self-play returns during training due to the random policy mixture but
Table 4: The network details of E3T.
Partner Prediction Module
index	Layers	Channels	Kernel Size
1	Conv2D + LeakyReLU	25	5*5
2	Conv2D + LeakyReLU	25	3*3
3	Conv2D + LeakyReLU	25	3*3
4	Concatenate state embedding and action embedding	/	/
5	(FC + LeakyReLU)×3	64	/
6	Concatenate embedding of 5 state-action pairs	64×5	/
7	(FC + LeakyReLU)×3	64	/
8	FC + Tanh	64	/
9	FC + L2 normalize	6	/
Policy Network
index	Layers	Channels	Kernel Size
1	Conv2D + LeakyReLU	25	5*5
2	Conv2D + LeakyReLU	25	3*3
3	Conv2D + LeakyReLU	25	3*3
4	Concatenate state embedding and action embedding	/	/
5	(FC + LeakyReLU)×3	64	/
6	FC	6	/
Figure 10: Results of MEP with varying entropy coefficient on  $100 \times 100$  matrix game.
Figure 11: Results on coordinating with human proxy models. The difference between this figure with the Figure 4(a) is that the starting positions of two agents are switched. We plot the mean and standard error of coordination rewards over 5 random seeds. The parameter  $\epsilon$  is set as 0.5 for all layouts except for Forced Coord., which sets the  $\epsilon$  as 0.0. In this layout, the ego agent does not need too much exploration due to the range of activity being very narrow.
Figure 12: Results on coordinating with other AI baselines. These results plot the normalized cross-play [15] rewards between baselines. The first 5 subfigures show the results on 5 layouts and the last shows the average results over 5 layouts. The last column of each subfigure represents the average result of collaborating with other baselines. For each baseline, we train 5 different models varying only over random seeds. The hyper-parameter  $\epsilon$  is set as 0.5 on Cramped Rm. and Asymm.Adv., as 0.3 on Coord. Ring, and Counter Circ and a 0.0 on Forced Coord.
(a) Ablations about the main components
(b) Ablations about implement details
Figure 13: Ablation of E3T's components on collaborating with human proxies. We plot the average performance over standing in two different positions. (a)  $(+\mathrm{R})$  and  $(+\mathrm{P})$  respectively integrate the mixed partner policy and the partner modeling module into the self-play framework. For versions with the mixed partner policy  $(+\mathrm{R})$ , the balance parameter  $\epsilon$  is defined as 0.5, and as 0.3 on Forced Coord.. (b) E3T(context) means that the policy depends on the concatenation of the state and the context embedding. E3T(-A) means that both the partner policy and ego policy use outputs of the partner prediction module as additional latent. We plot the results of models coordinating with the human proxy on Coord. Ring with  $\epsilon = 0.5$ .
achieves optimal cross-play returns during testing. These results demonstrate that E3T can explore effectively and avoid being stuck in suboptimal on the diagonal line as shown in Figure 7(a).
Results of MEP with varying population entropy coefficients are shown in Figure 10. The entropy coefficient corresponds the  $\lambda$  in Equation (1). MEP with population  $= 2$  can reach the best self-play returns but fail to coverage to the optimal policy (with the first column, the first and second rows)
when  $\lambda = 15$ , which indicates the hyper-parameter  $\lambda$  for reward reshaping is hard to tune. We select  $\epsilon = 0.99$  and  $\lambda = 5.0$  for results in Figure 3(a).
C.2 Results on Overcooked
Results on collaborating with human proxies with starting position switched are shown in Figure 11. E3T achieves superior or comparable performance compared to other baselines.
AI-AI coordination results on 5 layouts Additional results of AI-AI coordination on all layouts and the average performance over all layouts are shown Figure 12. E3T outperforms other baselines on 4 layouts and has comparable performance with population-based method MEP on Asymm. Adv.. In addition, E3T achieves the highest average performance over all layouts when collaborating with AI baselines. Moreover, MEP and FCP training the ego agent by pairing with pre-trained diverse partner populations are better than SP and PBT, which only improve the coordination rewards during training.
Ablation studies of E3T's components on cooperation with human proxies The ablation studies of E3T's components on cooperation with human proxies are shown in Figure 13(a). The average reward of  $\mathrm{E3T(+P)}$  is larger than that of SP on each layout, thus the partner modeling can assist the zero-shot coordination since the ego policy adapts to the predicted partner actions. In addition,  $\mathrm{E3T(+R)}$  also outperforms SP on 4 layouts except for the Forced Coord., which indicates that encouraging the partner policy to be more diverse during training time is beneficial to zero-shot coordinating with unseen partners. Moreover,  $\mathrm{E3T(R + P)}$  that combing the partner modeling and the mixed policy is better than other models on the averaged reward over all layouts. Figure 13(b) compares the ego policy that conditions on the context embedding with the one that conditions on the predicted partner action. The results show that the latter policy performs better, suggesting that the predicted partner action is easier to comprehend and respond to than the context embedding. Moreover, E3T(-A) has lower coordination performance than E3T, which demonstrates the advantage of using asymmetric latent variables for ego and partner policies during training, as explained in section 4.3. This technique enhances zero-shot coordination by introducing more diversity in the latent space and breaking the asymmetry between ego and partner policies.
Figure 14: Ablation studies of  $\epsilon$  on 4 layouts. In general, the  $\epsilon$  can be set small when the layout space is not large, e.g. smaller  $\epsilon$  for the Forced Coordination layout. For layouts with big space, a larger  $\epsilon$  can provide more explanation about different cooperation patterns.
Further ablation studies of  $\epsilon$  on more layouts Figure 14 Ablation studies of epsilon (in the range of [0.1, 0.3, 0.5, 0.7, 0.9]) on the other 4 layouts are shown in Figure 14 of the rebuttal material. In general, the epsilon can be set small when the layout space is not large, e.g. smaller epsilon
for the Forced Coordination layout can achieve better performance because the activation range is narrow and the partner's coordination ability is more important than behavior diversity in this case. For layouts with big spaces, there exist different coordination patterns, such as multiple paths to deliver soups. Therefore, we normally set the epsilon larger to encourage the policy to explore more coordination patterns. For example, our method achieves higher rewards with epsilon=0.5 when coordinating with human proxies in other layouts, and with epsilon=0.3 or 0.5 when coordinating with other AI baselines.
(a) Effect of partner modeling
(b) Effect of asymmetric action distribution
Figure 15: (a) We plot the effect of the partner modeling to Self-play and MEP when collaborating with AI baselines. (b) We plot the effect of the asymmetric input setting. E3T(-A) indicates both ego and partner policies are equipped with the partner modeling module.
The effect of the partner modeling module We have incorporated the partner modeling module into two baselines (Self-play and the state-of-the-art MEP) as illustrated in Figure 15(a). These results demonstrate that the partner modeling module can further enhance the zero-shot coordination performance of baselines when they collaborate with AI baselines.
The effect of the asymmetric inputs for the ego and partner policies This work makes use of predicted partner action distribution for ego policy and random action distribution for the partner policy during training. The motivation of the asymmetric input setting comes from two considerations. Firstly, the random action distribution introduces extra randomness into the partner policy, which in turn improves partner diversity. Secondly, this is consistent with the real-world command that AI agents should adapt to human partners' behavior, while different humans may have various cooperative patterns and are not necessary to try to fit AI agents.
In addition, we have empirically verified that asymmetric input is helpful to zero-shot coordination as illustrated in Figure 15(b) of the rebuttal material, where the performance of our method with asymmetric input slightly outperforms the model that uses the partner modeling for both ego and partner policies on most layouts.
The partner modeling module with only the historical observations accessible In more complex environments, we do not have the ground truth of partner actions, but only the historical observations of the ego policy are available, thus it is worth developing an advanced model to predict the partner action from changes in the observed environment. In the Overcooked environment, agents can observe the global state (this is the environmental setting). We also empirically verified that partner action can be predicted from the sequence of ego states on Coord. Ring layout. As shown in Table 5, the prediction accuracy from only the state sequence is comparable to that from the state-action sequence. Note that the accuracy of random prediction is around 0.17.
Table 5: The accuracy of partner prediction and the coordination performance under the settings of the state or state-action available.
Prediction accuracy	Coordination reward with human proxy
state-action	0.54 ± 0.04	143.3 ± 8.3
state	0.50 ± 0.01	139.3 ± 6.8