Title: An Efficient End-to-End Training Approach for Zero-Shot Human-AI Coordination
Venue: NeurIPS 2023
Accepted: yes
Keywords: Zero-Shot Coordination

Human-AI coordination

Training Efficiency

Partner Modeling
Abstract: The goal of zero-shot human-AI coordination is to develop an agent that can collaborate with humans without relying on human data. Prevailing two-stage population-based methods require a diverse population of mutually distinct policies to simulate diverse human behaviors. The necessity of such populations severely limits their computational efficiency. To address this issue, we propose E3T, an **E**fficient **E**nd-to-**E**nd **T**raining approach for zero-shot human-AI coordination. E3T employs a mixture of ego policy and random policy to construct the partner policy, making it both coordination-skilled and diverse. In this way, the ego agent is end-to-end trained with this mixture policy without the need of a pre-trained population, thus significantly improving the training efficiency.  In addition, a partner modeling module is proposed to predict the partner's action from historical information. With the predicted partner's action, the ego policy is able to adapt its policy and take actions accordingly when collaborating with humans of different behavior patterns. Empirical results on the Overcooked environment show that our method significantly improves the training efficiency while preserving comparable or superior performance than the population-based baselines. Demo videos are available at https://sites.google.com/view/e3t-overcooked.

An Efficient End-to-End Training Approach for Zero-Shot Human-AI Coordination

Abstract

The goal of zero-shot human-AI coordination is to develop an agent capable of collaborating with humans without relying on human data. Prevailing two-stage population-based methods require a diverse population of mutually distinct policies to simulate diverse human behaviors. The necessity of such populations severely limits their computational efficiency. To address this issue, we propose E3T, an E fficient E nd-toE nd T raining approach for zero-shot human-AI coordination. E3T employs a mixture of ego policy and random policy to construct the partner policy, making it both skilled in coordination and diverse. This way, the ego agent is trained end-to-end with this mixture policy, eliminating the need for a pre-trained population, and thus significantly improving training efficiency. In addition, we introduce a partner modeling module designed to predict the partner's actions based on historical contexts. With the predicted partner's action, the ego policy can adapt its strategy and take actions accordingly when collaborating with humans exhibiting different behavior patterns. Empirical results on the Overcooked environment demonstrate that our method substantially improves the training efficiency while preserving comparable or superior performance than the population-based baselines. Demo videos are available at https://sites.google.com/view/e3t-overcooked .

1 Introduction

Cooperation between AI agents and humans has gained significant attention, spanning various fields such as robotics [5, 26], automatic driving [41], Human-AI dialogue [14] and Human-AI coordination games [30]. Due to the high cost of collecting human data and involving humans during training, recent studies [32, 15] have focused on zero-shot human-AI coordination problems, with the aim to train an ego agent capable of collaborating with humans without relying on human data.

To train such an ego agent, the prevailing approach [15, 21] involves training the ego agent with diverse partner agents that can simulate human behavior without the need of human data. One method for achieving this is through self-play [33, 29], where the ego agent plays against itself as the partner agent. However, empirical evidence has shown that self-play can lead to agents getting stuck in a single cooperative pattern during training [15], hindering their ability to adapt to diverse human behaviors. Alternative approaches like other-play [15] aim to introduce diversity by breaking the symmetry of self-play policies. Nevertheless, this method relies on the assumption of strict symmetry in the action or state space, limiting its practical applicability.

In contrast to relying solely on a single partner policy, advanced population-based methods [21, 42, 32, 18, 19] construct a population with multiple policies to simulate diverse human behaviors, and then train the ego agent to collaborate with this population. Among these methods, FCP [32] obtains a diverse population by training several self-play policies with different random seeds and saving multiple past checkpoints reflecting various coordination levels. TrajeDi [21] promotes diversity by maximizing the divergence of trajectory distributions within the partner population. MEP [42] achieves a cooperative and diverse population by maximizing self-play rewards and population entropy as auxiliary rewards.

However, these population-based methods suffer from two major limitations. Firstly, they are highly computationally inefficient due to their two-stage framework, which involves training a population of policies before training the ego agent. For example, training a MEP agent for one environment in overcooked [8] can take over 15 hours on a single 3090Ti GPU. By contrast, a self-play agent can be trained in less than 1 hour. Secondly, these methods only learn a single response for all partners in the population, neglecting the diverse behavior patterns among them. As a result, the learned ego agent lacks the capability to adjust its policy based on each partner's unique behavior pattern, leading to unsatisfactory performance in zero-shot human-AI coordination.

SP

PBT

FCP

MEP

E3T

Algorithm

90

100

110

120

130

AI Coordination Performance

0

5

10

15

Training Hour Requirements

Performance

Training Time

In this paper, we introduce an E fficient E nd-toE nd T raining approach, called E3T, which aims to improve both efficiency and the ability to recognize behavior patterns in zero-shot human-AI coordination. To achieve efficient training, E3T utilizes a blend of ego and random policies as partners during training, and integrates a partner modeling module into the ego policy. This approach is motivated by the idea that the partner policies should be both skilled in coordination and diverse [42]. By breaking down these two objectives into two policies i.e. ego policy and random policy, and merging them as the partner policy, E3T is able to be end-to-end trained without the need for pre-training a population, thus enjoying higher efficiency and a simpler implementation compared to population-based methods.

In order to improve the collaboration capabilities of of ego agents when working with partners exhibiting different behaviors, we introduce a partner modeling module that predicts the partner's actions based on historical contexts. By incorporating this module into the ego policy, the agent is able to know not only the current observation, but also the predicted actions of the partner. This enables the ego agent to adapt its behavior more effectively, aligning it better with the partner's preferences and resulting in improved collaboration performance. To the best of our knowledge, our paper is the first to utilize a partner action prediction module in the context of a two-player human-AI coordination task.

Our contributions are three-fold. Firstly, we introduce an efficient end-to-end ego training approach, named E3T, for zero-shot Human-AI coordination, which offers higher training efficiency compared to population-based methods. Secondly, we propose a partner modeling module that predicts partner actions from the context, allowing the ego policy to adapt when collaborating with humans exhibiting different behavior patterns. Lastly, empirical results on Overcooked show that our method, E3T, significantly improves training efficiency while achieving comparable or superior performance to existing methods as Figure 1 shows. Specifically, E3T demonstrates a significant improvement in training efficiency when compared to state-of-the-art population-based methods.

2 Related Work

Our work investigates how to train an ego agent for zero-shot Human-AI coordination, a field with broad applications in real-world scenarios such as robotics [5, 26], service allocation [36, 37, 35] and Human-AI coordination games [30]. In zero-shot Human-AI coordination tasks, reinforcement learning (RL) algorithms [15] are tasked with training a robust ego agent without relying on human data, due to the high cost of collecting real human data.

Self-Play Training There are a number of methods to tackle zero-shot coordination, broadly categorized into two classes: self-play and population-based training. Self-play has demonstrated impressive capability in competitive games such as Go [29], Dota [38], StarCraft [34] and imperfect information games [13]. Self-play can also master high-level skills in cooperative games [8]. However, self-play agents often struggle to coordinate effectively with previously unseen partners due to their tendency to adopt rigid conventions formed during training against themselves [32]. Other-play [15] attempts to mitigate these conventions by breaking the symmetries of self-play policies. However, it heavily relies on the prior knowledge of strict symmetries in the environment, limiting its applicability in general environments lacking such symmetries. In this work, E3T follows the self-play training framework, which avoids falling into conventions by encouraging the partner policies to be more diverse. In addition, our method offers greater generality compared to other-play, making it applicable in a wider range of environments.

Population-Based Training Methods Population-based methods train a robust AI by collaborating with a diverse population. Experimental results show that population-based methods achieve superior zero-shot coordination performance compared to self-play and other-play [15]. For example, PBT [8] employs an online evolutionary algorithm, which iteratively updates policy parameters and performs policy substitution within the population. FCP [32] achieves diversity within a population by training several self-play policies with different random seeds and saving multiple past checkpoints to obtain agents with various coordination levels. In a similar vein, TrajeDi [21] regulates the partner population to be diverse by maximizing the divergence of trajectory distributions between policies within the population. Furthermore, MEP [42] introduces an auxiliary reward by designing the entropy of the average of policies in the population, thus promoting population diversity. More recently, COLE [18, 19] reformulate two-player coordination as a game-theoretic problem, where the trainer's objective is to estimate the best responses to the most recent population based on the cooperative incompatibility distribution. Our work improves the partner behavior diversity by maximizing the entropy, but E3T directly increases the randomness of a single-agent partner policy, making it more efficient than MEP and TrajeDi.

Agent Modeling Agent modeling is an important ability for autonomous agents, enabling them to reason about various properties of other agents such as actions and goals [3]. Typical agent modeling methods focus on reasoning about types of other agents [1, 2, 31], plan recognition [9, 7], group modeling [23, 11, 20] and especially policy reconstruction [6, 25, 4, 27], which is also adopted in E3T to improve zero-shot human-AI coordination performance. However, a key difference is that E3T's ego agent's partner modeling module is trained by pairing with agents and aims to predict the actions of partners. In addition, we directly employ the predicted partner actions for decision-making, instead of using the embedding of partner behaviors [10].

3 Preliminary

In this section, we introduce the two-player cooperative Markov Game and a representative pipeline of population-based training for two-player zero-shot coordination.

Two-player Cooperative Markov Game A two-player cooperative Markov Game is described by a tuple ⟨S , A , A , T , R ⟩ , where S is a finite set of states, A is the finite set of actions, and suppose two agents have the same action space. p 0 : S → R is the initial state distribution. T : S × A × A × S → [0 , 1] is the transition function and assumes the transition dynamic is deterministic. R : S × A × A → R is the reward function and assume rewards are bounded by α , i.e. R ( s, a 1 , a 2 ) ≤ α, ∀ s, a 1 , a 2 . We assume the transition and reward functions are unknown. In this work, we solve the two-policy cooperative Markov Game via single-agent reinforcement learning and the self-play framework as previous studies [8, 15]. Specifically, the ego policy π e is trained by pairing with a fixed partner policy π p with the object defined as π e = arg max π V ( π, π p ) .The value function of joint policy ( π, π p ) is defined as V ( π, π p ) = E p 0 , T ,a 1 ∼ π,a 2 ∼ π p ∑ ∞ t =0 γ t R ( s t , a 1 t , a 2 t ) , where γ ∈ (0 , 1) is the discounted factor.

Population-Based Training Population-based methods [21, 42, 32] aim to construct a population of policies to simulate diverse human behaviors. Intuitively, we expect that the policy in the population should be both skilled in coordination and diverse. Taking Maximum Entropy Population (MEP) [42] as an example, it constructs the partner population by maximizing the self-play rewardsand simultaneously increasing the entropy of policies in the population. The objective of Maximum

Ego Agent

Partner Agent

Policy

Copy

Partner Action

Prediction

𝜏

!

𝑠

!

# 𝑎

!

"

𝑎

!

#

Random action

distribution

Partner

Policy

Policy Copy

# 𝑎

!

#

𝑠

!

𝑎

!

"

Ego Agent

Partner Agent

𝑠

!

Random

Policy

Policy Mixture

Sampling

Sampling

(a) Self-play

(b) E3T

Figure 2: (a). The illustration of the self-play training framework, which trains the ego policy by pairing it with the copied partner policy. (b). The illustration of E3T. The green box shows the decision process of the ego agent. The blue box shows that of the partner agent.

Entropy Population training is defined as:

Here P = { π 1 , π 2 , ..., π n } denotes a population of policies, ¯ π ( . | s t ) = 1 n ∑ n i =1 π i ( . | s t ) is the mean policy of the population P , and λ determines the relative importance of the population entropy term against the task reward. The task reward under the joint action a t is obtained by self-play with a uniformly sampled policy from the population, and the population entropy term acts as an auxiliary reward. With this competent and diverse population as the partner population, an ego policy is trained by collaborating with the population via prioritized sampling that assigns a higher priority to the partner policy that is difficult to cooperate with. The objective of ego policy training is defined as:

where ρ ( . ) as the prioritized sampling mechanism. However, the training process of MEP is inefficient due to the complex two-stage training and the necessity of maintaining a large population for behavioral diversity. Our work draws inspiration from MEP, which achieves behaviorally diverse and cooperative partners by maximizing the self-play reward and the entropy of partner policy and then trains a robust ego policy by cooperating with those partners. In the next section, we will discuss how to solve zero-shot coordination using an efficient end-to-end training process.

4 Methodology

In this section, we introduce the proposed method, E3T, an efficient end-to-end training approach, which trains an ego policy π e through collaborating with a partner policy π p. To take into account both training efficiency and partner diversity, we employ a mixture of the copied ego policy and a random policy as the partner policy π p. The objective of E3T is defined as follows:

The proposed method contains two key components, a diverse partner policy and a partner modeling module for adapting to unseen partners, which will be introduced in Section 4.1 and Section 4.2 respectively. An overview of E3T is presented in Figure 2.

4.1 A Mixture Policy as Partner

We begin with the motivation of leveraging the proposed mixture partner policy. Recall that MEP trains a partner population by maximizing self-play rewards of policies within it and maximizing the population entropy as shown in Equation (1). The first term on maximizing self-play rewards tends to learn a population with cooperative ability. The second entropy term can force the partner population

to be diverse. However, optimizing the objective of Equation (1) suffers very high computational complexity due to the necessity of training a number of policies to collaborate effectively with themselves. Moreover, it exhibits unstable convergence properties, requiring meticulous hyperparameter tuning for additional entropy term [12].

To circumvent computational cost and non-stable training, we construct the partner policy via dividing Equation (1) into two parts with different partner policies learned, i.e. π p1 and π p2, which aim to maximize the rewards ∑ t E ( s t , a t ) ∼ π p1 ,π p1 [ R ( s t , a t )] and the entropy of the policy output ∑ t E ( s t , a t ) ∼ π p2 [ H ( π p2 ( . | s t ))] , respectively. We can see that maximizing the former objective function is aligned with the goal of learning coordination skills, akin to the self-play training objective. In this way, we can directly use the ego policy π e as π p1. On the other hand, if we directly learn the latter one objective, i.e. , the entropy of the mean policy, policy π p2 will lead to the random policy π r , i.e. , a uniform distribution over the action space.

To construct a partner who has both skills in coordination and diverse behaviors, we directly mix the above-discussed partner policies π p1 and π p2 as the partner policy π p. As previously discussed, π p1 can directly apply ego policy π e to achieve high reward for coordination. π p2 is a random policy π r , which has the maximum policy entropy. In this way, the partner policy π p is the mixture between the self-play ego policy and the random policy with the balance parameter ϵ , formally defined as:

We can see that the computation cost for obtaining this mixture partner policy is significantly lower than population-based methods since the conversion eliminates the need to train a population of policies. Additionally, the partner policy achieves behavioral diversity by simply leveraging a random policy with maximum entropy, whereas population-based approaches rely on more complex mechanisms, such as gradient descent [21] or reward shaping [42], to enforce the diversity. Both diversity and coordination ability are two keys to constructing a partner policy. Intuitively, the diversity reflected by entropy increases, and the coordination performance decreases with the random coefficient epsilon increasing. We attempt to quantitatively verify these intuitions through Propositions 1 and 2. Firstly, we analyze the behavior diversity of the partner policy of E3T. The entropy of partner policy reflects the diversity of partner behaviors. Proposition 1 shows that the entropy of the mixture partner policy is related to that of the random policy with maximum entropy.

Proposition 1 (Entropy of Mixture Partner Policy) . The entropy of the mixture partner policy, defined as π p = (1 -ϵ ) π e + ϵπ r , satisfies that:

where C 1 = ϵ log ϵ + |A| (1 -ϵ ) , and |A| is the cardinality of the action apace A . In addition, the lower bound of H ( π p ( . | s t )) is increased as the increasing of ϵ . Proof. See Appendix.

Secondly, we demonstrate the benefits of improving coordination skills when pairing with the partner policy of E3T. According to Proposition 2, although π ′ e is the best response to the mixture partner policy rather than the ego policy itself, the coordination performance of π ′ e with the ego policy is lower bounded by that of ego policy self-play with a punishment term ζ . The ζ is related to the balance parameter ϵ and smaller ϵ , which means less weight on the random policy, leads to lower ζ .

Proposition 2 (Coordination Skill Improvement) . Let the π ′ e as the best response to π p = (1 -ϵ ) π e + ϵπ r , i.e π ′ e = arg max π V ( π, π p ) , then we have V ( π ′ e , π e ) ≥ V ( π e , π e ) -ζ, where ζ = 4 αϵ (1 -γ ) 2 . Proof. See Appendix.

4.2 Partner Modeling

In this section, we describe the details of the proposed partner modeling module, designed to extract and analyze the behavior pattern of the partner from historical information. This module equips the ego policy with the ability to consider not only the current state but also the partner's behavior pattern, empowering the ego agent to adapt to previously unseen partners

Partner Action Prediction As in previous works [6, 25, 4], we model the partner agent by policy reconstruction. If the actual partner action distribution a p t is given, the two-player cooperative Markov Game will reduce to a single-agent MDP about the ego agent, in which the ego agent can easily find the best response to the current state. Formally, under the partner policy π p with a p t = π p ( . | s t ) , the two-player MDP with transition function T can be converted in to a single-agent MDP with transition

Algorithm 1 E3T: An efficient end-to-end training approach with mixture policies and partner modeling

Input: Parameters of context encoder E c , action prediction network F , ego policy π e , hyperparameter ϵ , k .

Output: Well-trained context encoder E c, action prediction network F and the ego policy π e .

1: while not done do

2: Set data buffer D = ∅ .

3: for t = 1 , 2 , · · · , step do

4: Encode context of the partner: z c t = E c ( τ t )

6: Randomly select an action distribution: ˆ a e t

5: Predict partner action distribution: ˆ a p t = F ( z c t )

7: Mixture partner policy as: π p ( ·| s t , ˆ a e t ) = (1 -ϵ ) π e ( ·| s t , ˆ a e t ) + ϵπ r ( · )

8: Gather data from π e ( ·| s t , ˆ a p t ) and π p ( ·| s t , ˆ a e t ) , then D = D ∪ { ( τ t +1 , s t , a e t , a p t , r t ) }

9: end for

10: Update parameters of partner modeling module by minimizing Equation (7).

11: Update parameters of the ego policy π e via PPO.

12: end while

function denoted as T π p ( s ′ | s t , a ) = ∑ a p π p ( a p | s t ) T ( s ′ | s t , a p , a ) . However, because the agents act simultaneously, the ego agent can only infer the partner's action from historical information instead of directly observing the actual partner's action.

To obtain the predicted partner action ˆ a p t , we first devise an encoder network E c to extract historical partner context z c t from a sequence of past state-action pairs τ t = { ( s t -i , a p t -i ) k i =1 } . The historical partner context z c t contains rich information of the partner's behavior and coordination pattern, enabling our action prediction network F to predict the partner's action distribution based on z c t and current state s t . This procedure is formally written as follows:

The parameters of the context encoder E c and the action prediction network F are jointly optimized by minimizing a cross-entropy loss:

where N is the batch size, and y i is a one-hot vector indicating the actual partner action.

We provide a theoretical analysis of how the accuracy of partner modeling affects the coordination ability as shown in Proposition 3. We analyze the coordination performance gap between the best responses to the ground-truth partner policy and the partner prediction module. It is established that this performance gap is proved to be proportional to the prediction error.

Proposition 3. Denote π τ ( . | s t ) = E τ t [ F ( s t , E c ( τ t ))] as the expectation of the proposed partner prediction module over historical trajectory τ t . The ground-truth partner policy is denoted as π p ( . | s t ) . Assume the learned π τ ( . | s t ) satisfies that D TV ( π τ ( . | s t ) , π p ( . | s t )) ≤ ϵ m , ∀ s t . Define the µ ∗ τ and µ ∗ p are the best response to the partner prediction module and the ground-truth partner policy respectively. Then the value functions of these best responses collaborating to the partner policy satisfy that: | V ( µ ∗ τ , π p ) -V ( µ ∗ p , π p ) | ≤ 2 γϵ m α (1 -γ ) 2 . Proof. See Appendix.

4.3 Ego Policy Training and Test

E3T separately trains the partner action prediction module via minimizing the loss function in Equation (7) and trains the ego policy that makes decisions depending on both observations and predicted partner action distribution via PPO [24]. The detailed training process of E3T is described in the Algorithm 1. During testing time, the ego agent zero-shot coordinates with an unseen partner, without parameters updating or temporally-extended exploration as [22].

Implement Details The partner modeling module of E3T takes current observation at step t and partner's past k state-action pairs as input as shown in Equation (6), we set k = 5 in this work. The ego policy π e ( . | s t , ˆ a p t ) conditions on current state s t and predicted partner action distribution ˆ a p t ,

which enables the ego agent to analyze and adapt to its partner, while a random action distribution ˆ a e t is fed into the partner policy as the predicted ego agent policy. The random action distribution breaks symmetries and conventions commonly seen in self-play agents and introduces randomness into the partner policy, which in turn improves partner diversity. Moreover, this is consistent with the fact that some humans may not take their AI partners' actions into consideration, but the AI agents should adapt to their human partners.

5 Experiments

In this section, we evaluate E3T on zero-shot collaborating with behavior-cloned human proxies, real humans, and AI baselines. More details of the experiment setting and additional results can be found in Appendix.

5.1 Environments

We evaluate the zero-shot coordination performance of E3T on a 100 × 100 cooperative matrix game [21, 42] and the Overcooked environment [8]. Overcooked has a discrete action space with 6 actions, and we conduct experiments on 5 different layouts that pose various coordination challenges, such as avoiding blocking each other and preparing the required ingredients for a soup together. In addition, the state spaces of 5 layouts are different as shown in Table 1. For instance, the state space of Counter Circuit reaches up to 5 . 8 × 10 16 . This statistic indicates the 5 layouts are challenging for explorations. We have

0

25

50

75

100

Training Steps

0.00

0.25

0.50

0.75

1.00

Avg. return

Cross-Play Return

Other-Play

E3T

TrajeDi

MEP

(a) Results on

100

×

100

matrix

game

MEP PBT

SP

FCP E3TMean

MEP

PBT

SP

FCP

E3T

0.85 0.71 0.64 0.68 0.79 0.73

0.71 0.66 0.58 0.68 0.76 0.68

0.64 0.58 0.59 0.73 0.78 0.67

0.68 0.68 0.73 0.67 0.85 0.72

0.79 0.76 0.78 0.85 0.93 0.82

0.6

0.7

0.8

0.9

(b) AI-AI coordination

conducted additional experiments on the Google Football [17] environment's "3 vs 1 with Keeper" layout, which is a three-player cooperative game with a discrete action space comprising 19 actions. More details about the environment descriptions can be found in Appendix.

5.2 Baselines

We compare proposed E3T with 6 baselines: Self-Play (SP) [28], Population-Based Training (PBT) [16, 8], Fictious Co-Play (FCP) [32], Trajectory Diversity (TrajeDi) [21], Maximum-Entropy Population-based training (MEP) [42] and Other-Play (OP) [15]. All methods, except SP, OP, and E3T, fall under the category of population-based methods, which aim to develop a robust ego policy through interaction with a partner population.Detailed descriptions of baselines and the network details of E3T can be found in Appendix.

5.3 Result Analysis

We compare E3T with other baselines on zero-shot collaboration with the human proxy, real humans, and AI baselines.

Asymm. Adv.

Coord. Ring

0

25

50

75

100

125

150

175

Average reward of per episode

Performance with human proxy model

H

proxy

+ SP

H

proxy

+ PBT

H

proxy

+ FCP

H

proxy

+ TrajeDi

H

proxy

+ MEP

H

proxy

+ E3T

0

20

40

60

80

100

120

140

Average reward per episode across layouts

E3T

MEP

TrajeDi

FCP

SP

PBT

Figure 4: (a) We plot the mean and standard error of coordination rewards over 5 random seeds. The parameter ϵ is set to 0 . 5 for all layouts except Forced Coord. , where ϵ is settoas 0 . 0 . In this layout, the ego agent does not need too much exploration due to the narrow active range. (b) We plot the mean and standard error of humans collaborating with baselines over all layouts. These results show that E3T outperforms all other baselines when collaborating with real humans.

Zero-shot Coordination on 100 × 100 matrix game Figure 3(a) presents the cross-play[15] results of 10 independently trained policies for each baseline on a one-step matrix game. The cross-play result measures the coordination performance between models that belong to the same baseline but have different random seeds. In this game, two players select a row and a column independently, and the reward is determined by the intersection of their choices. As shown in Figure 3(a), E3T and MEP can converge to the optimal strategies with the largest return as 1 , while E3T can converge faster than MEP. These results indicate that the mixture partner policy can enable sufficient exploration and avoid getting stuck in local optima that generalize poorly on the game with large action space. More details of the matrix game are given in Appendix.

Training Efficiency Analysis Table 2 compares the training time and average rewards of different baselines, where the rewards are calculated by averaging over 5 layouts on collaboration with human proxies. These results suggest that E3T outperforms existing baselines on collaboration with human proxies, and E3T is more efficient than population-based methods. For instance, it only spends 10 . 6% training time of previous state-of-the-art MEP.

AI-AI Coordination The zero-shot coordination performance between AI baselines are shown in Figure 3(b), where each element represents the normalized cooperation rewards averaged over 5 layouts for a pair of baselines. The performance of baselines cooperating with themselves is measured by the cross-play metric. As shown in Figure 3(b), E3T outperforms other baselines in coordinating with AI models. In addition, SP and PBT, which train policies only based on rewards and ignore diversity, have poor generalization to unseen AI partners. MEP and FCP, which train the ego agent by best responding to a pre-trained diverse partner population, perform better than SP and PBT, but worse than E3T. These results indicate that partner diversity is essential for zero-shot coordination, and E3T learns a roust ego policy by providing sufficient partner diversity and being equipped with adaptation ability. More results of AI-AI coordination on 5 layouts can be found in Appendix.

Human Proxy-AI Coordination Figure 4(a) shows the results of baselines cooperating with behavior-cloned human proxies. These results suggest that E3T consistently outperforms existing baselines on all layouts. In addition, FCP, MEP, and TrajeDi have gained better or similar generalization performance than SP and PBT, owing to training an ego agent based on a diverse partner policy. These results indicate that the diversity of partners is important for the zero-shot coordination task. Note that SP achieves comparable performance to population-based MEP on Forced Coord. and E3T achieves the best coordination performance at parameter ϵ = 0 . 0 . One possible reason is that the Forced Coord. layout is a difficult layout with a very narrow activity range and two agents being forced to coordinate to complete a soup. Therefore, it requires more cooperation than exploration. Moreover, E3T with ϵ = 0 . 0 is better than SP, indicating that the partner modeling module is beneficial to adapting to an unseen partner.

Human-AI Coordination The average rewards over 5 layouts of AI baselines coordinating with real humans are shown in Figure 4(b). We follow the Human-AI coordination testing protocol

proposed in [32] and recruit 20 individuals to evaluate the Human-AI coordination ability of E3T and state-of-the-art MEP, where individuals play a total of 600 rounds with AI. For cost and time consideration, we reuse the Human-AI coordination results of other baselines reported in [8, 42]. Figure 4(b) shows that E3T outperforms all other baselines when zero-shot coordinating with real humans. We also collect individuals' subjective assessment of E3T and MEP about the degree of intelligence, collaborative ability, and preference between the two. More details of the Human-AI coordination setting and individuals' subjective assessment are given in Appendix.

Coord. Ring

Mean

0

25

50

75

100

125

150

175

Average reward of per episode

Ablation Studies of E3T components on AI-AI coordination

SP

E3T(+P)

E3T(+R)

E3T(+P+R)

(a) Ablations of E3T's components

=0.1

=0.3

=0.5

=0.7

=0.9

0

20

40

60

80

100

120

140

Average reward of per episode

The influence of Hyper-parameter

E3T+AI

E3T+H

proxy

H

proxy

+E3T

(b) Ablations of the balance parameter

ϵ

5.4 Ablation Study

0

1

2

3

4

5

6

Environment Steps

1e6

20

40

60

80

100

120

140

Average Reward per Episode

=0.5 Reward

=0.5 Accuracy

=0.0 Reward

=0.0 Accuracy

0.2

0.3

0.4

0.5

Accuracy of Action Prediction

Analysis of Component Ablation Results We present the ablation results of E3T's components in Figure 5(a). We evaluate the effectiveness of each component on collaborating with AI baselines (including MEP, FCP, and PBT). We use (+R) and (+P) to denote models incorporated with the mixture partner policy and the partner modeling module, respectively. As shown in Figure 5(a), E3T(+P) achieves a higher average reward than SP on each layout, which shows that the partner modeling module can improve zero-shot coordination by apdating actions accordingly. Likewise, E3T(+R) surpasses SP on 4 layouts except for the Forced Coord. , which shows that encouraging more exploration in the partner policy can discover more coordination patterns during training and thus can enhance zero-shot coordination with unseen partners during testing. The lower performance on Forced Coord. may be due to its narrow action space and the lack of reward incentives for coordination when using a too random partner policy during training. Moreover, E3T(R+P) that combines both the mixture partner policy and the partner modeling module performs better than E3T(R) and E3T(P) on the average reward over all layouts.

Analyzing the effect of the hyper-parameter ϵ Figure 5(b) shows how the balance parameter ϵ of the mixture partner affects the performance of E3T when collaborating with AI baselines and the human proxy. We observe that the coordination rewards of E3T increase and then decrease as ϵ increases for both types of partners. The results indicate that both collaboration ability and partner diversity are essential for zero-shot coordination, but simply increasing the partner action diversity is not enough. The optimal values of ϵ are 0.3 and 0.5 for AI baselines and human proxy, respectively. One possible reason for this discrepancy is that the human proxy tends to generate more 'noop' actions [8], which reduces the need for coordination behaviors compared to AI baselines.

Analyzing the effect of partner modeling module To demonstrate the effect of the proposed partner modeling module, we plot the accuracy of predicting partner actions and the rewards when cooperating with a human proxy. As shown in Figure 6, the reward and the prediction accuracy show the same tendency: better partner modeling leads to improvement in coordination. In addition, the accuracy at ϵ = 0 . 5 reaches around 55% , which is higher than the accuracy around 40% when ϵ = 0 . 0 . This indicates that mixing a random policy can provide more exploration and thus generalize well to unseen partners.

Does the mixture partner policy accommodate the multi-player game? We verify the effect of the mixture partner policy on improving the zero-shot coordination performance on a three-player Google Football game, the 3 vs 1 with Keeper scenario. We train self-play policies via the MultiAgent PPO (MAPPO) [40], where policies have shared parameters. E3T(mixture policy) integrates the mixture partner policy into the self-play training process, in detail, the three agents randomly sample an action with a probability as ϵ , otherwise, take actions according to the action policies. We independently train 3 policies with different random seeds for each baselines and report the cross-play results between unseen policies to test the zero-shot coordination ability of these models.

Table 3 shows that self-play policies fail to zero-shot cooperate with policies from different training seeds, because they may fall into some specific coordination conventions during training. Our proposed mixture partner policy can increase the behavioral diversity of the partners, allowing the ego policy to encounter different coordination patterns during training. Consequently, our method with the mixture partner policy can train robust ego policies that can adapt well to policies independently trained from different training seeds. Moreover, increasing partner's behavior diversity during training can significantly improve the generalization ability of the trained ego policy.

6 Conclusion

In this paper, we propose an E fficient E nd-toE nd T raining approach for zero-shot human-AI coordination, called E3T. Different from previous population-based methods, E3T employs a mixture policy of ego policy and random policy to construct the partner policy, enabling the partner policy to be both coordination-skilled and diverse. In this way, the ego-agent is able to be end-to-end trained in such a mixture policy without pre-training a population, thus significantly improving the training efficiency. Additionally, we introduce a partner modeling module that allows the ego agent to predict the partner's next action, enabling adaptive collaboration with partners exhibiting different behavior patterns. Empirical results on Overcooked clearly demonstrate that E3T can significantly improve training efficiency while achieving comparable or superior performance compared to existing methods.

Limitations and Future Work In this paper, we only explore the feasibility of our framework in zero-shot two-player human-AI coordination tasks but neglect general multi-player (more than two) human-AI coordination tasks. Future research will focus on expanding the framework to multi-player (more than two) human-AI coordination tasks, with a goal of enhancing both zero-shot coordination performance and training efficiency. Since AI agents are unable to regulate human behavior, indiscriminately collaborating with humans in accomplishing their goals could result in unlawful circumstances. Therefore, future research should concentrate on developing AI agents capable of detecting dangerous intentions and preventing their misuse for illicit or military purposes.

References

Stefano V Albrecht, Jacob W Crandall, and Subramanian Ramamoorthy. Belief and truth in hypothesised behaviours. Artificial Intelligence , 235:63-94, 2016.

Stefano V Albrecht, Somchaya Liemhetcharat, and Peter Stone. Special issue on multiagent interaction without prior coordination: Guest editorial. Autonomous Agents and Multi-Agent Systems (AAMAS) , 31(4):765-766, 2017.

Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence , 258:66-95, 2018.

Dipyaman Banerjee and Sandip Sen. Reaching pareto-optimality in prisoner's dilemma using conditional joint action learning. Autonomous Agents and Multi-Agent Systems (AAMAS) , 15:91-108, 2007.

Cynthia Breazeal, Cory D Kidd, Andrea Lockerd Thomaz, Guy Hoffman, and Matt Berlin. Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 708-713. IEEE, 2005.

George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation , 13(1):374, 1951.

Sandra Carberry. Techniques for plan recognition. User Modeling and User-Adapted Interaction , 11(1):31-48, 2001.

Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Philip R Cohen, C Raymond Perrault, and James F Allen. Beyond question answering. Strategies for Natural Language Processing , 245274, 1981.

Hao Ding, Chengxing Jia, Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Coordination scheme probing for generalizable multi-agent reinforcement learning, 2023.

Can Erdogan and Manuela Veloso. Action selection via learning behavior patterns in multi-robot systems. In Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI) , 2011.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML) , pages 1861-1870. PMLR, 2018.

Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfectinformation games. NeurIPS Deep Reinforcement Learning (DRL) Workshop , 2016.

Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems (NeurIPS) , 33:20179-20191, 2020.

Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. 'other-play' for zero-shot coordination. In International Conference on Machine Learning , pages 4399-4410. PMLR, 2020.

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846 , 2017.

Karol Kurach, Anton Raichuk, Piotr Sta´ nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4501-4510, 2020.

Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Cooperative open-ended learning framework for zero-shot coordination. In International Conference on Machine Learning (ICML) . PMLR, 2023.

Yang Li, Shao Zhang, Jichen Sun, Wenhao Zhang, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Tackling cooperative incompatibility for zero-shot human-ai coordination. arXiv preprint arXiv:2306.03034 , 2023.

Xingzhou Lou, Jiaxian Guo, Junge Zhang, Jun Wang, Kaiqi Huang, and Yali Du. Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination. In Autonomous Agents and Multi-Agent Systems (AAMAS) , 2023.

Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning (ICML) , pages 7204-7213. PMLR, 2021.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International Conference on Machine Learning (ICML) , pages 5331-5340. PMLR, 2019.

Patrick Riley and Manuela Veloso. Recognizing probabilistic opponent movement models. In RoboCup 2001: Robot Soccer World Cup V 5 , pages 453-458. Springer, 2002.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Sandip Sen and Neeraj Arora. Learning to take risks. In AAAI-97 Workshop on Multiagent Learning , pages 59-64, 1997.

Thomas B Sheridan. Human-robot interaction: status and challenges. Human Factors , 58(4):525-532, 2016.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 , 2017.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature , 550(7676):354-359, 2017.

Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. Evaluation of human-ai teams for learned and rule-based agents in hanabi. Advances in Neural Information Processing Systems (NeurIPS) , 34:16183-16195, 2021.

Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Twenty-Fourth AAAI Conference on Artificial Intelligence , 2010.

DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems (NeurIPS) , 34:14502-14515, 2021.

Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation , 6(2):215-219, 1994.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Haoning Xi, Didier Aussel, Wei Liu, S Travis Waller, and David Rey. Single-leader multifollower games for the regulation of two-sided mobility-as-a-service markets. European Journal of Operational Research , 2022.

Haoning Xi, Wei Liu, S Travis Waller, David A Hensher, Philip Kilby, and David Rey. Incentivecompatible mechanisms for online resource allocation in mobility-as-a-service systems. Transportation Research Part B: Methodological , 170:119-147, 2023.

Haoning Xi, Yili Tang, S Travis Waller, and Amer Shalaby. Modeling, equilibrium, and demand management for mobility and delivery services in mobility-as-a-service ecosystems. Computer-Aided Civil and Infrastructure Engineering , 38(11):1403-1423, 2023.

Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3974-3983, 2018.

Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems (NeurIPS) , 33:15737-15749, 2020.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems (NeurIPS) , 35:24611-24624, 2022.

Yi Zhang, Ping Sun, Yuhan Yin, Lin Lin, and Xuesong Wang. Human-like autonomous vehicle speed control by deep reinforcement learning with double q-learning. In 2018 IEEE Intelligent Vehicles Symposium (IV) , pages 1251-1256. IEEE, 2018.

Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 6145-6153, 2023.

A Proposition Proofs

A.1 Proof of Proposition 1

Proposition 1 (Entropy of Mixture Partner Policy) . The entropy of the mixture partner policy, defined as π p = (1 -ϵ ) π e + ϵπ r , satisfies that:

where C 1 = ϵ log ϵ + |A| (1 -ϵ ) , and |A| is the cardinality of the action apace A . In addition, the lower bound of H ( π p ( . | s t )) is increased as the increasing of ϵ . Proof. See Appendix.

Proof. The mixture partner policy is defined as π p = (1 -ϵ ) π e + ϵπ r with the balance parameter ϵ . Detailed derivations of Equation (5) in Proposition 1 are as follows:

where C 1 = ϵ log ϵ + |A| (1 -ϵ ) , and |A| is the cardinality of the action apace A . Inequality (1) and (3) hold because π p ( a | s t ) ≤ 1 and π e ( a | s t ) ≤ 1 . Inequality (2) holds because log (1 + x ) ≤ x . Since C 1 decreases in the range ϵ ∈ (0 , e | A |-1 ] and ϵ belongs to [0 , 1] , the lower bound of H ( π p ( . | s t )) increases as ϵ increases.

A.2 Necessary Notations and Lemmas for Proofs of Proposition 2 and 3

Notations As the previous statement in section 4.2, that when the partner policy π p is given with a p t = π p ( . | s t ) , the two-player MDP with transition function T can be converted in to a single-agent MDP with transition function denoted as T π p ( s ′ | s t , a ) = ∑ a p π p ( a p | s t ) T ( s ′ | s t , a p , a ) . Note that the value estimation V ( π 1 , π 2 ) under the two-player transition function T is equal to the value estimation V T π 2 ( π 1 ) = 1 1 -γ E ( s,a 1 ) ∼ ρ π 1 ( T π 2 ) [ ∑ a 2 π 2 ( a 2 | s ) R ( s, a 1 , a 2 )] under the singleagent transition function T π 2 . ρ π 1 ( T π 2 ) denotes the stationary state-action distribution under the single-agent transition function T π 2 , formally written as:

Then, the equivalence between the the value estimations V ( π 1 , π 2 ) , V T π 2 ( π 1 ) is proven as:

We next give the necessary lemmas for proofs of Proposition 2 and 3. We first introduce the following Lemma, which describes the relationship between value estimation and policy distribution discrepancy on single-agent MDP.

Lemma 1. Given a single agent MDP with transition T , assume there are two policies π a and π b with D TV ( π a ( . | s ) , π b ( . | s )) ≤ ϵ, ∀ s, a holding. Then the difference between value functions satisfies that | V T ( π a ) -V T ( π b ) | ≤ 2 αϵ (1 -γ ) 2 , where rewards are bounded by α and γ is the discounted factor.

Proof. The Lemma 1 can be proved directly following the proof process of Theorem 1 in [39].

We provide the error bound on value estimations under different transitions used for the proof of Proposition 3.

Lemma 2. Given an MDP with true transition function T ∗ and assume the estimated transition function T ′ satisfies that, for a policy π , E ρ π ( T ∗ ) [ D TV ( T ∗ ( . | s, a ) , T ′ ( . | s, a ))] ≤ ϵ m , then we have:

where rewards are bounded by α and γ is the discounted factor

Proof. The Lemma 2 can be proved directly following the prove process of Lemma 7 in [39].

A.3 Proof of Proposition 2

Proposition 2 (Coordination Skill Improvement) . Let the π ′ e as the best response to π p = (1 -ϵ ) π e + ϵπ r , i.e π ′ e = arg max π V ( π, π p ) , then we have V ( π ′ e , π e ) ≥ V ( π e , π e ) -ζ, where ζ = 4 αϵ (1 -γ ) 2 . Proof. See Appendix.

Proof. The total variance distance between the ego policy π e and the mixture partner policy policy π p satisfies D TV ( π e ( . | s ) , π p ( . | s )) ≤ ϵ , derived by:

Next, following the Lemma 1 and the equivalence of the value estimations as shown in Equation (10), we have that | V ( π, π p ) -V ( π, π e ) | < = 2 αϵ (1 -γ ) 2 , ∀ π , where δ = 2 αϵ (1 -γ ) 2 . Combing the definition of π ′ e = arg max π V ( π, π p ) , we have

let ζ = 2 δ , then the proof is completed.

A.4 Proof of Proposition 3

Proposition 3. Denote π τ ( . | s t ) = E τ t [ F ( s t , E c ( τ t ))] as the expectation of the proposed partner prediction module over historical trajectory τ t . The ground-truth partner policy is denoted as π p ( . | s t ) . Assume the learned π τ ( . | s t ) satisfies that D TV ( π τ ( . | s t ) , π p ( . | s t )) ≤ ϵ m , ∀ s t . Define the µ ∗ τ and µ ∗ p are the best response to the partner prediction module and the ground-truth partner policy respectively. Then the value functions of these best responses collaborating to the partner policy satisfy that: | V ( µ ∗ τ , π p ) -V ( µ ∗ p , π p ) | ≤ 2 γϵ m α (1 -γ ) 2 . Proof. See Appendix.

Proof. Recall that, when the partner policy π p is given with a p t = π p ( . | s t ) , the two-player MDP with transition function T can be converted in to a single-agent MDP with transition function denoted as T π p ( s ′ | s t , a ) = ∑ a p π p ( a p | s t ) T ( s ′ | s t , a p , a ) . The total variance distance between single-agent transition functions under the partner prediction module or the ground-truth partner policy is given by:

The equation (1) is holds because that the transition function T is deterministic and equals to 1 at a unique next state s ′′ . Define the µ ∗ τ and µ ∗ p are the best response to the partner prediction module and the ground-truth partner policy respectively. Then the performance gap of these best responses collaborating to the partner policy is bounded by:

Due to the optimality of µ ∗ p when collaborating with the partner policy π p, thus V ( µ ∗ τ , π p ) -V ( µ ∗ p , π p ) ≤ 0 . In addition, according to the definitions of µ ∗ p , i.e, the optimal policy under the MDP with transition function T π p , and µ ∗ τ , i.e, the optimal policy under the MDP with transition function T π τ , thus we have V T π p ( µ ∗ τ ) -V T π p ( µ ∗ p ) ≤ 0 , V T πτ ( µ ∗ p ) -V T πτ ( µ ∗ τ ) ≤ 0 . Let δ =

The inequality (2) holds by applying Lemma 2 with the condition D TV ( T π τ ( . | s, a ) , T π p ( . | s, a ) ) ≤ ϵ m , ∀ s, a satisfied as proven in Equation 14 .

1

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

1

0.02

0

0

0

0

0

0

0

0

0.02

1

0.02

0

0

0

0

0

0

0

0

0.02

1

0.02

0

0

0

0

0

0

0

0

0.02

1

0.02

0

0

0

0

0

0

0

0

0.02

1

0.02

0

0

0

0

0

0

0

0

0.02

1

0.02

0

0

0

0

0

0

0

0

0.02

1

0.02

0

0

0

0

0

0

0

0

0.02

1

(a) Matrix Game

0

50

100

150

200

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Avg. return

Self-Play Return

E3T

=0.1

E3T

=0.3

E3T

=0.5

E3T

=0.7

E3T

=0.9

E3T

=0.99

E3T

=1.0

0

50

100

150

200

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Avg. return

Cross-Play Return

E3T

=0.1

E3T

=0.3

E3T

=0.5

E3T

=0.7

E3T

=0.9

E3T

=0.99

E3T

=1.0

(b) Returns

0

中

B Experiment Settings

B.1 Environments

A 100 × 100 Matrix game We verify the effectiveness of the mixed partner policy for learning a robust policy on a one-step matrix game. The matrix is partially shown in Figure 7(a), where we zoom in on the upper left 10 × 10 sub-matrix. The matrix denoted as M is constructed such that M [ i, i ] = 1 , M [ i + 1 , i ] = M [ i, i + 1] = 0 . 2 for i ≥ 3 , and M [1 , 1] = M [2 , 1] = 1 . All other elements are zero.In the matrix game, two players select a row and a column independently, and the reward is determined by the intersection of their choices. We use a one-step matrix game to test the robustness of our mixture partner policy. The game has a 100 × 100 matrix, where each element represents the reward for both players if they choose the corresponding row and column.

Overcooked We validate the mixed partner policy and the partner modeling module for improving zero-shot coordination on the Overcooked environment [8]. In this environment, the target of players is to make soup using onions, and dishes, and then serve it. After successfully serving a soup, players gain 20 rewards. Its discrete action space contains 6 actions: up, down, left, right, noop, and interact.

We conduct experiments on 5 layouts illustrated in Figure 8. The calculation of the cardinality of state spaces shown in Table 1 is explained as follows. Let's take the example of calculating the state space cardinality for the Counter Circuit layout. In the Counter Circuit layout, there are 16 positions on the kitchen counter where plates, onions, soup, or nothing can be placed. Each pot can be in one of 24 states, which includes being empty, having one onion, two onions, three onions, or waiting for

20 counts. There are two players, each of whom can stand at one of 14 grids and face one of four directions. Each player can hold either a plate, an onion, soup, or nothing in their hands. Therefore, the total number of possible states for this layout is: 4 16 C 2 24 24 2 4 2 4 2 ≈ 5 . 8 × 10 16 . Despite having a smaller state space, Forced Coordination is more challenging than Asymmetric Advantages for zero-shot coordination. This is because the two players are separated and must complete different tasks in a timely manner to finish a soup in Forced Coordination, while players can independently complete a soup in Asymmetric Advantages.

B.2 Baselines

We compare proposed E3T with 6 baselines: Self-Play (SP) [28], Population-Based Training (PBT) [16, 8], Fictious Co-Play (FCP) [32], Trajectory Diversity (TrajeDi) [21], Maximum-Entropy Population-based training (MEP) [42] and Other-Play (OP) [15]. The baselines are described in detail below:

Self-Play (SP) [28] In coordination self-play, two players share the same policy during sampling and iteratively improve the policy by focusing on coordinating well with itself. If it converges, the self-play policy is able to coordinate with itself but is poor in generalization when meeting unseen policies.

Population Based Training (PBT) [16, 8] PBT is an online evolutionary algorithm, which is widely used in multi-agent RL to iteratively improve policies and finally converge to a population with high quality. At each iteration, paired policies randomly sampled from the population are improved by interacting with a number of timesteps, and then the population is updated according to the recorded performance. Specifically, the worst policy in the population is replaced with a mutation of the best policy.

Fictitious Co-Play (FCP) [32] FCP trains a policy that coordinates well with unseen humans by making the best response to a diverse population of partners. To obtain the population, it trains several self-play policies with different random seeds, in addition, it saves past checkpoints with different levels of coordination during training self-play policies.

Trajectory Diversity (TrajeDi) [21] TrajeDi solves the two-player zero-shot coordination problem by jointly training a diverse population and its common best response (BR). To be as diverse as possible, the training of the population is regularized by maximizing the Jensen-Shannon divergence (JSD) on the trajectory distributions of policies. Additionally, it trains a common BR to a diverse population, which indicates that the BR is able to coordinate well with unseen policies. We reuse the results of TrajeDi coordinating with human proxies reported in [42].

Maximum Entropy Population-based training (MEP) [42] MEP first trains a diverse population through PBT, and this population is encouraged to be diverse with the entropy of the mean policies in the population as an auxiliary reward. Secondly, a common best agent is trained to cooperate with policies in this diversified population, and the selection of partner policies is via prioritized sampling. The prioritization is dynamically updated based on coordination rewards during training progress.

Other-Play [15] Both E3T and other-play [15] both belong to the self-play training framework, and we discuss the difference between them as follows. Other-play avoids trapping into specialized conventions by breaking known symmetries in environments, for example the colors in Card games are symmetric, and thus improves the robustness of the trained ego agent. However, this approach relies heavily on the assumption of the state or action space is strongly symmetric, for example, the colors of the cards are symmetric on Hanabi. But it does not adapt to the Overcooked environment, a population two-player cooperative game, since there do not exist obvious and easy to exchange symmetries. Our work shares the benefits of other-play with end-to-end training and breaking the symmetric of self-play training, but E3T is more general since it incents partner exploration by employing a constant random policy and does not rely on any assumptions on environments.

E3T We propose an efficient and robust training approach, combing a mixed partner policy for enhancing the diversity of partner policy and the partner modeling module for better adaptation. An ego policy is trained by pairing with the mixture partner which consists of the parameter-copied ego policy and the random policy. We set the length of the context k = 5 .

B.3 Experiments design of zero-shot coordination with humans

We describe the details of Human-AI coordination experiments. The results of baselines coordinating with real humans are shown in Figure 4(b). We follow the Human-AI coordination testing protocol proposed in [32]. We recruit 20 individuals to evaluate the coordination ability of E3T and state-

Asymm. Adv.

Coord. Ring

Layouts

0

1

2

3

4

5

Scores

E3T Intelligence

MEP Cooperative

E3T Cooperative

Asymm. Adv.

Coord. Ring

Layouts

0.0

0.2

0.4

0.6

0.8

E3T

MEP

(b) Preference for MEP and E3T

of-the-art MEP, humans play a total of 600 rounds with them. For cost and time consideration, we reuse the Human-AI coordination results of other baselines reported in [8, 42]. We select models with the best coordination performance with human proxies for both E3T and MEP. For E3T, we set the ϵ = 0 . 5 on layouts except for Forced Coord. with ϵ = 0 . 0 on it. We use the code provided in [20] to implement the Human-Ai interaction interface.

Participants first watched a description video explaining the game rules and then they were familiar with the experimental process on a single-agent demo layout with the other player not moving to experience the game. During formal experiments, each participant cooperated with the AI agents to complete the 5 layouts and is suggested to get as larger rewards as possible. Pairing with each agent on each layout will repeat 3 times, and the experiment has a total of 30 rounds, with each round lasting one minute. We incentivized game performance: participants earned 50 yuan to complete 30 rounds, and the participants with the top 5 highest rewards earned an extra 50 yuan.

Besides the objective coordination rewards, we also collected participants' subjective assessment of E3T and MEP about the degree of intelligence, collaboration ability, and preference between the two. After the experiments of each layout, participants were required to answer these three questions. Study sessions lasted around 50 minutes on average, including being familiar with the experiment process, playing with AI agents, recording rewards, and filling in subjective assessments.

Figure 4(b) shows that E3T outperforms all other baselines when zero-shot coordinating with real humans on average reward across all layouts. Subjective assessments of E3T and MEP are shown in Figure 9. These results show that participants rate E3T with higher scores on the intelligence level and the coordination ability than MEP, across all layouts. In addition, more participants prefer E3T than MEP.

B.4 The network details of E3T

The network architecture of E3T including the policy network and the partner modeling module is described in Table 4. We describe E3T for the Overcooked environment, which has a discrete action space with 6 actions. E3T are trained based on Tensorflow-GPU 1.15.5 and cuda11.4+cudnn8.8.

C Additional Results

C.1 Results on 100 × 100 Matrix Game

The performance of E3T under different balance parameters ϵ is illustrated in Figure 7(b). When ϵ = 1 . 0 , the ego policy is trained by pairing with the completely random partner policy and fails to coordinate during self-play and cross-play with other independently trained ego policies. When ϵ < 1 . 0 , E3T sacrifices some self-play returns during training due to the random policy mixture but

Proportion

index

1

2

3

4

5

6

Policy Network

Layers

Conv2D + LeakyReLU

Conv2D + LeakyReLU

Conv2D + LeakyReLU

Concatenate state embedding and action embedding

×

(FC + LeakyRelu)

FC

Channels

25

25

25

/

64

6

5*5

3*3

3*3

/

/

/

0

25

50

75

100

125

150

175

200

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Avg. return

Self-Play Return

MEP pop=2  =15.0

MEP pop=2  =10.0

MEP pop=2  =5.0

MEP pop=2  =1.0

0

25

50

75

100

125

150

175

200

Training Steps

0.0

0.2

0.4

0.6

0.8

1.0

Avg. return

Cross-Play Return

MEP pop=2  =15.0

MEP pop=2  =10.0

MEP pop=2  =5.0

MEP pop=2  =1.0

Asymm. Adv.

Coord. Ring

Layouts

0

25

50

75

100

125

150

Average reward of per episode

Performance with human proxy model

H

proxy

+ SP

H

proxy

+ PBT

H

proxy

+ FCP

H

proxy

+ TrajeDi

H

proxy

+ MEP

H

proxy

+ E3T

3

MEP

PBT

SP

FCP

E3T

Mean

MEP

PBT

SP

FCP

E3T

0.80

0.66

0.63

0.82

0.76

0.73

0.66

0.52

0.53

0.71

0.75

0.63

0.63

0.53

0.55

0.75

0.78

0.65

0.82

0.71

0.75

0.79

0.78

0.77

0.76

0.75

0.78

0.78

1.00

0.81

Coord. Ring

0.56

0.64

0.72

0.80

0.88

0.96

MEP

PBT

SP

FCP

E3T

Mean

MEP

PBT

SP

FCP

E3T

0.67

0.77

0.67

0.86

0.87

0.77

0.77

1.00

0.79

0.88

0.87

0.86

0.67

0.79

0.57

0.74

0.83

0.72

0.86

0.88

0.74

0.93

0.91

0.86

0.87

0.87

0.83

0.91

0.88

0.87

0.64

0.72

0.80

0.88

0.96

MEP

PBT

SP

FCP

E3T

Mean

MEP

PBT

SP

FCP

E3T

1.00

0.85

0.82

0.89

0.98

0.91

0.85

0.65

0.69

0.78

0.81

0.76

0.82

0.69

0.66

0.72

0.81

0.74

0.89

0.78

0.72

0.78

0.88

0.81

0.98

0.81

0.81

0.88

0.94

0.89

Asymm. Adv.

0.66

0.72

0.78

0.84

0.90

0.96

MEP

PBT

SP

FCP

E3T

Mean

MEP

PBT

SP

FCP

E3T

1.00

0.60

0.27

0.22

0.58

0.53

0.60

0.53

0.54

0.51

0.50

0.53

0.27

0.54

0.73

0.94

0.83

0.66

0.22

0.51

0.94

0.87

0.95

0.70

0.58

0.50

0.83

0.95

0.83

0.74

0.30

0.45

0.60

0.75

0.90

MEP

PBT

SP

FCP

E3T

Mean

MEP

PBT

SP

FCP

E3T

0.78

0.67

0.82

0.63

0.76

0.73

0.67

0.59

0.36

0.51

0.84

0.59

0.82

0.36

0.45

0.51

0.65

0.56

0.63

0.51

0.51

0.00

0.72

0.48

0.76

0.84

0.65

0.72

1.00

0.79

0.0

0.2

0.4

0.6

0.8

1.0

MEP

PBT

SP

FCP

E3T

Mean

MEP

PBT

SP

FCP

E3T

0.85

0.71

0.64

0.68

0.79

0.73

0.71

0.66

0.58

0.68

0.76

0.68

0.64

0.58

0.59

0.73

0.78

0.67

0.68

0.68

0.73

0.67

0.85

0.72

0.79

0.76

0.78

0.85

0.93

0.82

Mean

0.60

0.66

0.72

0.78

0.84

0.90

Coord. Ring

Mean

0

25

50

75

100

125

150

Average reward of per episode

Ablation Studies of E3T components on AI-Human Proxy coordination

SP

E3T(+P)

E3T(+R)

E3T(+P+R)

(a) Ablations about the main components

0

20

40

60

80

100

120

140

Average reward of per episode

SP

E3T(context)

E3T(-A)

E3T

(b)

Ablations

about

imple-

ment details

achieves optimal cross-play returns during testing. These results demonstrate that E3T can explore effectively and avoid being stuck in suboptimal on the diagonal line as shown in Figure 7(a).

Results of MEP with varying population entropy coefficients are shown in Figure 10. The entropy coefficient corresponds the λ in Equation (1). MEP with population=2 can reach the best self-play returns but fail to coverage to the optimal policy (with the first column, the first and second rows)

when λ = 15 , which indicates the hyper-parameter λ for reward reshaping is hard to tune. We select ϵ = 0 . 99 and λ = 5 . 0 for results in Figure 3(a).

C.2 Results on Overcooked

Results on collaborating with human proxies with starting position switched are shown in Figure 11. E3T achieves superior or comparable performance compared to other baselines.

AI-AI coordination results on 5 layouts Additional results of AI-AI coordination on all layouts and the average performance over all layouts are shown Figure 12. E3T outperforms other baselines on 4 layouts and has comparable performance with population-based method MEP on Asymm. Adv. . In addition, E3T achieves the highest average performance over all layouts when collaborating with AI baselines. Moreover, MEP and FCP training the ego agent by pairing with pre-trained diverse partner populations are better than SP and PBT, which only improve the coordination rewards during training.

Ablation studies of E3T's components on cooperation with human proxies The ablation studies of E3T's components on cooperation with human proxies are shown in Figure 13(a). The average reward of E3T(+P) is larger than that of SP on each layout, thus the partner modeling can assist the zero-shot coordination since the ego policy adapts to the predicted partner actions. In addition, E3T(+R) also outperforms SP on 4 layouts except for the Forced Coord. , which indicates that encouraging the partner policy to be more diverse during training time is beneficial to zero-shot coordinating with unseen partners. Moreover, E3T(R+P) that combing the partner modeling and the mixed policy is better than other models on the averaged reward over all layouts. Figure 13(b) compares the ego policy that conditions on the context embedding with the one that conditions on the predicted partner action. The results show that the latter policy performs better, suggesting that the predicted partner action is easier to comprehend and respond to than the context embedding. Moreover, E3T(-A) has lower coordination performance than E3T, which demonstrates the advantage of using asymmetric latent variables for ego and partner policies during training, as explained in section 4.3. This technique enhances zero-shot coordination by introducing more diversity in the latent space and breaking the asymmetry between ego and partner policies.

The influence of Hyper-parameter   on Cramped Rm.

=0.1

=0.3

=0.5

=0.7

=0.9

0

25

50

75

100

125

150

175

Average reward of per episode

E3T+AI

E3T+H

proxy

H

proxy

+E3T

=0.0

=0.1

=0.3

=0.5

0

10

20

30

40

50

60

70

Average reward of per episode

The influence of Hyper-parameter

E3T+AI

E3T+H

proxy

H

proxy

+E3T

=0.1

=0.3

=0.5

=0.7

=0.9

0

25

50

75

100

125

150

175

Average reward of per episode

The influence of Hyper-parameter   on Asymm. Adv.

E3T+AI

E3T+H

proxy

H

proxy

+E3T

=0.1

=0.3

=0.5

=0.7

=0.9

0

10

20

30

40

50

60

70

80

Average reward of per episode

The influence of Hyper-parameter

E3T+AI

E3T+H

proxy

H

proxy

+E3T

Further ablation studies of ϵ on more layouts Figure 14 Ablation studies of epsilon (in the range of [0.1, 0.3, 0.5, 0.7, 0.9]) on the other 4 layouts are shown in Figure 14 of the rebuttal material. In general, the epsilon can be set small when the layout space is not large, e.g. smaller epsilon

for the Forced Coordination layout can achieve better performance because the activation range is narrow and the partner's coordination ability is more important than behavior diversity in this case. For layouts with big spaces, there exist different coordination patterns, such as multiple paths to deliver soups. Therefore, we normally set the epsilon larger to encourage the policy to explore more coordination patterns. For example, our method achieves higher rewards with epsilon=0.5 when coordinating with human proxies in other layouts, and with epsilon=0.3 or 0.5 when coordinating with other AI baselines.

SP

MEP

0

25

50

75

100

125

150

175

Average reward of per episode

The influence of partner modeling

Baseline

Baseline+Partner Modeling

(a) Effect of partner modeling

Asymm. Adv.

Coord. Ring

Mean

0

25

50

75

100

125

150

175

Average reward of per episode

Ablation Studies of E3T components on AI-Human Proxy coordination

E3T(-A)

E3T

(b) Effect of asymmetric action distribution

The effect of the partner modeling module We have incorporated the partner modeling module into two baselines (Self-play and the state-of-the-art MEP) as illustrated in Figure 15(a). These results demonstrate that the partner modeling module can further enhance the zero-shot coordination performance of baselines when they collaborate with AI baselines.

The effect of the asymmetric inputs for the ego and partner policies This work makes use of predicted partner action distribution for ego policy and random action distribution for the partner policy during training. The motivation of the asymmetric input setting comes from two considerations. Firstly, the random action distribution introduces extra randomness into the partner policy, which in turn improves partner diversity. Secondly, this is consistent with the real-world command that AI agents should adapt to human partners' behavior, while different humans may have various cooperative patterns and are not necessary to try to fit AI agents.

In addition, we have empirically verified that asymmetric input is helpful to zero-shot coordination as illustrated in Figure 15(b) of the rebuttal material, where the performance of our method with asymmetric input slightly outperforms the model that uses the partner modeling for both ego and partner policies on most layouts.

The partner modeling module with only the historical observations accessible In more complex environments, we do not have the ground truth of partner actions, but only the historical observations of the ego policy are available, thus it is worth developing an advanced model to predict the partner action from changes in the observed environment. In the Overcooked environment, agents can observe the global state (this is the environmental setting). We also empirically verified that partner action can be predicted from the sequence of ego states on Coord. Ring layout. As shown in Table 5, the prediction accuracy from only the state sequence is comparable to that from the state-action sequence. Note that the accuracy of random prediction is around 0.17.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an Efficient End-to-End training (E3T) approach for zero-shot human-AI coordination. E3T uses a mixture of an ego policy and a random policy as a simple way of training a diverse policy that is still capable of coordination. Unlike prior population based approaches, E3T does not require training populations of agents and thus is more efficient. The authors also propose a partnering module that models the agent’s partner’s actions, enabling improved zero-shot human collaboration. The experiments show a clear improvement over a range of prior methods, testing with both proxy and real human partners.\n\nSTRENGTHS:\n- The idea is straightforward and easy to employ, and directly addresses a meaningful challenge (efficient zero-shot coordination with humans). The combination of an ego policy and a random policy brings together the strengths of self play for training a coordination policy, whilst also incorporating diversity through the random policy to prevent overfitting to a specific partner. \n-  The inclusion of a partnering module makes sense and shows a clear advantage empirically.\n- The experiments are thorough and show comparisons against a range of prior approaches, including experiments with real people, as well as detailed ablations of the proposed method.\n\nWEAKNESSES:\n- The method seems like it would be sensitive to $\\varepsilon$. While 5b) shows that evaluation for one task, it would be interesting to see of all tasks (i.e. do we need to tune $\\varepsilon $ for each task separately?)\n- nit: missing epsilon in line 7 in the algorithm\n- nit: line 232 has a typo — \"Assume the *learned*...\"\n\nQUESTIONS:\nBased on figure 6, the overall accuracy of action predictions is pretty low (less than 60%) — it’s surprising that the partnering module is still helpful even when accuracy is not very high. Why do the authors think this is?\n\nLIMITATIONS:\nLimitations are clearly discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes E3T, an method to train agent for zero-shot coordination with humans. The main contribution in E3T is that, in contrast to population-based approaches, it can be trained in a single stage, significantly reducing training time. E3T also includes a model to predict the next action of the partner agents, given that trajectory. By conditioning on that action, the coordination agent can adapt to different partner behaviors according to their observed trajectories. The method is tested on Overcooked against both zero-shot coordination agents and real humans, showing improved performance and lower training time than competitive zero-shot coordination methods.\n\nSTRENGTHS:\nOriginality\n- The proposed method is a simple extension to MEP, with the adition of the partner action prediction module, which have been explored before. Despite highly relying on these 2 components the method is simple and provides gains in training time and performance.\n\nQuality:\n- The paper provides thorough experiments showing that the proposed method can coordinate with several zero-shot coordination agents in different tasks and layouts. The proposed method shows better training time than competitive approaches, as well as improved coordination results. More importantly, it shows significant improvements over baselines when coordinating with real humans.\n\n- Clear and simple to implement method, with comparisons with main coordination approaches and analysis of the entropy and reward bounds of the proposed method.\n\n\nClarity:\n- The paper provides a clear overview of the main methods for zero-shot coordination and their limitations. The proposed method is clearly explained, and section 4.1 provides a clear intution for the design choices, building from the limitations of self-play appraoches such as MEP to motivate different design decisions.\n\n- The explanation of the algorithm, figures and pseudocode provide a clear udnerstanding of the proposed method.\n\n- Clear analysis of the results, and ablations to understand the importance of the two components\n\n\nSignificance:\n- Building agents that can coordinate with other agents in a zero-shot manner is an important problem, and as the paper points out, the two stage training of existing approaches make learning highly inefficient. Proposing a zero-shot coordination coordination agent that can be trained in a single stage is an important problem. Overcooked is a very simplified coordination setting, but one that showcases some of the challenges in zero-shot coordination, and thus showing improved performance and training time resuls is a significant result.\n\nWEAKNESSES:\n- Figure 5.b seems to indicate that the value of epsilon has a strong effect on the final performance and changes for different kinds of tasks, making it potentially hard to find a good parameter for novel tasks. How is that epsilon chosen in the other experiments?\n\n- Figure 5.a shows that partner modeling has a significant effect in the coordination performance. Given the importance of that module, which is compatible with the other proposed baselines, it would be worth testing whether adding that into them would improve their performance results, even if training time would still be higher.\n\n- The mixture policy looks a lot like a self-play policy where a temperature to add noise to the policy is added. What are the main differences to that, if the partner modeling module is eliminated?\n\n- Propositions 1 and 2 are sound, and show that both the entropy and performance of the mixture policy are lower bounded by random and ego policy counterparts, safe for some terms depending on the mixture parameter and action space size. Despite the analysis being sound I am not sure about the value they add that was not known before. The entropy is equal to that of the random policy when \\eps is 1, and decreases as we decrease \\eps (increasing C_1). Similar analysis can be done for Proposition 2, meaning that the ego and random policies serve as upper bounds. How do propositions 1 and 2 help?\n\n- Having access to the partners action and state pairs is a big assumption that may not hold in more complex environments.\n\n- It seems strange that an ego policy that is trained with actions coming from the partner action prediction model can generalize when training the partner, where a random action distribution is used at the input. As epsilon becomes smaller, this gap should increase further. Could authors comment on that?\n\n- Related work: Authors should consider citing, and commenting on the differences with https://arxiv.org/pdf/2104.07750.pdf, which also models the partner for multi-agent collaboration.\n\nQUESTIONS:\n- I am surprised at Fig 4.a results, particularly on th efact that E3T coordinates better with self-play than self-play itself. This seems really counterintuitive, could authors comment on that?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple end-to-end training mechanism for zero-shot coordination. Existing works in the ZSC literature often make use of population-based training, where a large pool of policies is trained to play well against itself while maintaining a certain form of diversity to prepare for an unknown online partner. This work proposes that sufficient diversity can be achieved by simply adding epsilon-greedy noise into the self-play policy. Performance is further improved by an additional partner modeling module, which takes the past trajectory of the partner and predicts its next action. Experiments against human players and various baselines in the Overcooked environment show some improvements.\n\nSTRENGTHS:\n- Simplicity of method, where only a single policy is learned. This is much more efficient than population-based training approaches.\n- Experiments are performed against various agents, including human proxy, actual human players, and some baselines.\n- Code is available in the submission.\n\nWEAKNESSES:\n- The limited novelty. The method is a simple combination of existing techniques (epsilon-greedy, partner modeling, etc). Besides ablations, a more detailed analysis would provide more insight as to why this works.\n- The Overcooked environments are out-of-date. It is required to further evaluate the method on more complex Overcooked environments[1], e.g.,  more recipes,  and more layouts.\n\n[1 ]Wu, Sarah A., et al. \"Too Many Cooks: Bayesian Inference for Coordinating Multi‐Agent Collaboration.\" Topics in Cognitive Science 13.2 (2021): 414-432.\n\nQUESTIONS:\n-  Are there any details that differ from those in [2], e.g. the human proxy partner or the reward function?\n- Can the proposed methods be used in more challenging scenarios, e.g., more players, mixed games, or partial observation?\n\n[2] Strouse, D., McKee, K., Botvinick, M., Hughes, E., and Everett, R. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502–14515, 2021.\n\nLIMITATIONS:\nThis paper will be improved by evaluating on more complex environments and showing the scalability of the proposed methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a one-stage training framework for human-AI coordination. The key insight is that partner policies should exhibit both coordination skills and diversity. However, the traditional approach of constructing a competent and diverse partner population in the first stage and training an ego policy given this population in the second stage is complex and inefficient. To address this, the author proposes to decouple the partner policy into two components: one aligned with self-play (ego policy) and another component introducing diversity (random policy). By implementing this approach, the paper significantly improves efficiency compared to the Maximum Entropy Population (MEP) method. The effectiveness of the proposed method is demonstrated through experiments conducted on five different Overcooked layout scenarios.\n\nSTRENGTHS:\n1. The paper is well-motivated and easy to follow.\n2. The key idea behind the proposed method is straightforward and intuitive.\n3. The experiments section of the paper is substantial, covering a range of important aspects such as zero-shot coordination with human proxy, comparisons against human and AI baselines, and a well-conducted ablation study.\n4. This paper offers a sound theoretical analysis of the proposed method.\n\nWEAKNESSES:\nBased on Figure 5, it appears that the proposed method is highly task-sensitive. Therefore, I find it less convincing to solely rely on experiments conducted on five different layouts of Overcooked. It would be beneficial for the authors to consider conducting experiments on various multi-agent reinforcement learning benchmarks, such as matrix games, MPE, Hanabi, SMAC, and others, to provide a more comprehensive evaluation of their method.\n\nQUESTIONS:\n1. The partner policy observes random actions, which implies it is unlikely to exhibit coordinated behavior. And the ego policy achieves the best performance when $\\epsilon = 0.5$, which is a considerably large value. Does the ego policy adopt a more conservative approach and attempt to complete the task independently?\n2. The significant performance improvement of this method compared to MEP is intriguing. It appears that this paper can be viewed as a one-staged MEP combined with partner modeling. Is the performance disparity primarily attributed to the partner modeling?\n3. In the ablation study, when partner modeling is removed, what serves as the input for the ego policy?\n4. Minor comments:\na) Figure 6 is difficult to comprehend. The importance of \"environment steps\" is not clear in the figure. I think it would be helpful to use the \"accuracy of action prediction\" as the x-axis, as it could better illustrate the correlation between \"average reward per episode\" and \"action prediction\".\n\nLIMITATIONS:\nI appreciate that the author acknowledges the limitation of the current method in neglecting general multi-player human-AI coordination tasks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}