Collaborating with Humans without Human Data
Authors: DJ Strouse
Kevin R. McKee
Matthew Botvinick
Edward Hughes
Richard Everett
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: multi-agent
zero-shot coordination
collaboration
cooperation
common-payoff
human-ai interaction
reinforcement learning
deep reinforcement learning

Abstract: Collaborating with humans requires rapidly adapting to their individual strengths, weaknesses, and preferences. Unfortunately, most standard multi-agent reinforcement learning techniques, such as self-play (SP) or population play (PP), produce agents that overfit to their training partners and do not generalize well to humans. Alternatively, researchers can collect human data, train a human model using behavioral cloning, and then use that model to train "human-aware" agents ("behavioral cloning play", or BCP). While such an approach can improve the generalization of agents to new human co-players, it involves the onerous and expensive step of collecting large amounts of human data first. Here, we study the problem of how to train agents that collaborate well with human partners without using human data. We argue that the crux of the problem is to produce a diverse set of training partners. Drawing inspiration from successful multi-agent approaches in competitive domains, we find that a surprisingly simple approach is highly effective. We train our agent partner as the best response to a population of self-play agents and their past checkpoints taken throughout training, a method we call Fictitious Co-Play (FCP). Our experiments focus on a two-player collaborative cooking simulator that has recently been proposed as a challenge problem for coordination with humans. We find that FCP agents score significantly higher than SP, PP, and BCP when paired with novel agent and human partners. Furthermore, humans also report a strong subjective preference to partnering with FCP agents over all baselines.

Collaborating with Humans without Human Data
DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, Richard Everett" DeepMind
{strouse, kevinrmckee, botvinick, edwardhughes, reveratt/@deepmindd com
Abstract
Collaborating with humans requires rapidly adapting to their individual strengths, weaknesses, and preferences. Unfortunately, most standard multi-agent reinforce- ment learning techniques, such as self-play (SP) population play (PP), produce agents that overfit to their training partners and do not generalize well to humans. Alternativll,, researchers can collect human data, train a human model using behav- ioral cloning, and then use that model to train "human-aware" agents ("hehavioral cloning play", BCP). While such approach of agents to new human co-players, it involves the onerous and expensive step of collecting large amounts of human data first. Here, we study the problem of how to train agents that collaborate well with human partners without using human data. We argue that the crux of the problem is produce diverse set of training part- ners. Drawing inspiration from successful multi-agent approaches in competitive domains, find that surprisingly simple approach is highly effective. We train our agent partner as the best response to a population of self-play agents and their past checkpoints taken throughout training, method we call Fictitious Co-Play (FCP). Our experiments focus on two-player collaborative cooking simulator that has recently been proposed as a challenge problem for coordination with humans. We find that FCP agents score significantly higher than SP, PP, and BCP when paired with novel agent and human partners. Furthermore, humans also report a strong subjective preference to partnering with FCP agents over all baselines.
improve the generalization
1
Introduction
Generating agents which collaborate with novel partners is a longstanding challenge for Artificial Intelligence (AT) 44 16,
37 52 Achieving ad-hoc, zero-shot coordination
66 especially
important in situations where an AI must generalize to novel human partners 6 Many successful
approaches have employed human models, either constructed explicitly [4] 35 5
or learnt implicitly
[12 60 By contrast, recent work in competitive domains has shown that it is possible to reach human-
level using model-free reinforcement learning (RL) without human data, via self-play [ 63
This begs the question: Can model-free RL without human data generate agents that can collaborate with novel humans?
We seek an answer to this question in the space of common common-payoff games, where all agents work towards a shared goal and receive the same reward. Self-play (SP), which an agent learns from repeated games played against copies of itself, does not produce agents that generalize well to novel co-players T0
Intuitively, this is because agents trained self-play only ever need to coordinate with themselves, and make for brittle and stubborn collaborators with new partners who act differently. Population play (PP) trains a population of agents, all of whom interact with each other [39 While PP can generate agents capable of cooperation with humans in competitive team games it still fails to produce robust partners for novel humans in pure common-pay payoff settings
EEqual contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Agent training in various populations
Zero-shot coordination agent partners
Zero-shot coordination with novel human partners elicitation preferences agents
Salt-play play
SF
Which agent
did you prater?
Pogulation-play (PP)
Figure In this work, we evaluate variety of agent training methods (Section 2 in zero-shot coordination with agents (Section 4). We then run a human-agent collaborative study designed to elicit human preferences over agents (Section
12 PP in common common-payoff settings naturally encourages agents to play the same way, reducing strategic diversity and producing agents not different from self-play [24]
Our approach starts with the intuition that the key to producing robust agent collaborators is exposure to diverse training partners. We find that surprisingly simple strategy is effective in generating sufficient diversity. We train self-play agents varying only their random seed for neural network initialization. Periodically during training, we save agent "checkpoints" representing their strategy at that point in time. Then, we train an agent partner as the best-response to both the fully-trained agents and their past checkpoints. The different checkpoints simulate different skill levels, and the different random seeds simulate breaking symmetries different ways. We refer to this agent training procedure Fietitious Co-Play (FCP) for its relationship to fictitious self- self-play
27
28 69
We evaluate FCP fully-observable two-player nommon-payote collaborative cooking simulator. Based on the game Overcooked [25] it has recently been proposed as coordination challenge for AI
50 State-of-the-ar art performance in producing agents capable of generalization novel humans was achieved in [12 via behavioral cloning (BC) of human data. More precisely, BC was used to produce models that can stand in as human proxies during training in simulation, a method we call behavioral cloning play (BCP). We demonstrate that FCP outperforms BCP in generalizing to both novel agent human partners, and that humans express a significant preference for partnering with FCP over BCP. Our method avoids the cost and potential privacy concerns of collecting human data for training, while achieving better outcomes for humans at test time.
We summarize the novel contributions of this paper as follows:
We propose Fictitious Co-Play (FCP) to train agents capable of zero-shot coordination with humans (Section 2.1
2. We demonstrate that FCP agents generalize better than SP, PP, and BCP in zero-shot coordination with variety of held-out agents (Section 4.2).
3. We propose a rigorous human-agent aggent interaction study with behavioral analysis and partici- pant feedback (Section 5.1).
4. We demonstrate that FCP significantly outperforms the BCP state-of-the-art, both in task
and in human partner preference (Section 5.2
2 Methods
2.1 Fictitious Co-Play (FCP)
Diverse training conditions have been shown make agents more robust, from environmental variations (i.e. domain randomization [44] 56 67) to heterogeneity in training partners [69]. We seek to train agents that robust partners for humans in common-payonn -ayoff games, and extend this line work to that setting.
One important challenge in collaborating with novel partners is dealing with symmetries 31 For example, two agents A and B facing each other may move past each other by going left and going right, vice versa. Both are valid solutions, but good agent partner will adaptively switch between

---

Reinforcement learning (RL)
Human-human data collection
Behavioral cloning (BC)
te
Ex2
H+
Self-play (SP)
Population-play (PP)
Behavioral cloning play (BCP)
Fictitious co-play (FCP)
Figure 2: The four agent training methods evaluate in this work. Self-play (SP) where agent learns with itself, population-play (PP) where a population of agents are co-trained together, and behavioral cloning play (BCP) where data from human games is used create behaviorally cloned agent with which an RL agent is then trained. In our method, Fictitious Co-Play (FCP), N self-play agents are trained independently and checkpointed throughout training. An agent is then trained to best respond to the entire population of SP agents and their checkpoints.
these conventions if human clearly prefers one over the other. A second important challenge dealing with variations in skill level. Good agent partners should be able to assist both highly-skilled partners, as well as partners who are still learning.
Fictitious co-play (FCP) is a simple two-stage approach for training agents that overcomes both of these challenges (Figure [ right). In the first stage, we train diverse pool of partners. To allow the pool to represent different symmetry breaking conventions, we train N partner agents in self-play. Since these partners are trained independently. they can arrive at different arbitrary conventions for breaking symmetries. To allow the pool to represent different skill levels, we use multiple checkpoints of each self-play partner throughout training. The final checkpoint represents fully-trained "skillful" partner, while earlier checkpoints represent less skilled partners. Notably, by using multiple checkpoints per partner, this additional diversity in skill incurs no extra training cost.
In the second stage, we train an FCP agent as the best response to the pool diverse partners created in the first stage. Importantly, the partner parameters are frozen and thus FCP must learn to adapt to partners, rather than expect partners to adapt to it. In this way, FCP agents are prepared to follow the lead of human partners, and learn general policy across range of strategies and skills. We call our method "fictitious" co-play for its relationship to fictitious self- play in which competitive agents are trained with past checkpoints (in that case, avoid strategy cycling)
39
2.2 Baselines and ablations
We compare FCP agents to the three baseline training methods listed below, each varying only in their set of training partners, with the RL algorithm and architecture consistent across all agents:
1. Self-play (SP), where agents learn solely through interaction with themselves.
2. Population-play (PP), where population of agents are co-trained through random pairings.
3. Behavioral cloning play (BCP), where an agent is trained with BC model of a human
We also evaluate three variations on FCP to better understand the conditions for its success:
1. To test the importance of including past checkpoints in training, evaluate ablation FCP which agents are trained only with the converged checkpoints of their partners (FCP_T for "FCP minus time").
2. To test whether FCP would benefit from additional diversity in its partner population, we evaluate an augmentation of FCP in which the population of SP partners varies not just in random seed, but also in architecture (FCP+A for "FCP plus architectural variation""..
3. To test whether architectural variation can serve as full replacement for playing with past checkpoints, we evaluate the combination of both modifications (FCP-T+4).

---

2.3 Environment
Following prior work on zero-shot coordination in human- aggnn interaction, we study the Overcooked
environment (see Figure 5 12 13 EN, 50) 70] We draw particular inspiration from the environment in Carroll et
For full details, see Appendix A
In this environment, players are placed into gridworld kitchen as chefs and tasked with delivering as many cooked dishes of tomato soup possible within episode. This involves series of sequential high-level actions to which both players can contribute: collecting tomatoes, depositing them into cooking pots, letting the tomatoes cook into soup, collecting dish, getting the soup, and delivering it. Upon a successful delivery, both players are rewarded equally.
To effectively complete the task. players must learn to navigate the kitchen and interact with objects in the correct order, while maintaining awareness their partner's behavior to coordinate with them. This environment therefore presents the challenges of both movement and strategic coordination.
Each player observes an egocentric RGB view of the world, and every step can perform one of six actions: stand still, move {up, down, left, right), interact. The behavior of interact varies based on the cell which the player is facing (e.g. place tomato on counter).
Delivery
Collect and cooking
Chefs
Cooking pots
location
Counter
twice
Dish station
get
Deliver
Tomato
stations
Delivered! global
Figure 3: The Overcooked environment: two-player common-payol game in which players must coordinate to cook and deliver soup.
Cramped Room
Asymmetrio Advontages
Coordinaionn Ring
Counter Cirouit
Coordmation
Figure 4: Layouts: the kitchens which agents and humans play in, each emphasizing different
coordination strategies. Highlighted in bold are the terms used refer each in the rest of this paper.
2.4
Implementation details
Here we highlight several key implementation details for training methods. For full details, including the architectures, hyperparameters, and compute used, please see Appendix B
For our reinforcement learning agents, plus LSTM [29] architecture which we found led to optimal behavior across all layouts. Agents are trained using distributed set of environments running in parallel [17]. each sampling two agents from the training population to play together every episode.
the V-MPO 65 algorithm along with ResNet 2
Both PP and FCP are trained with population size of N = 32 agents which are sampled uniformly. For FCP, we use 3 checkpoints for each agent, therefore incurring no additional training burden: (1) at initialization (i.e. low-skilled agent), (2) at the end of training (i.e. fully-trained expert agent), and (3) the middle of training, defined as when the agent reaches 50% of its final reward (i.c. average-skilleds agent). When varying architecture for the training partners of the FCP+A + and

---

FCP-7,+4 variants, we vary whether the partners use memory (i.e. LSTM vs not) and the width of their policy and value networks (i.e. 16 vs 256). In total, we train 8 agents for each of the combinations, leaving the total population size of N = 32 unchanged, ensuring a fair comparison.
To train agents via behavioral cloning 55 we use the open-source Acme 301 to learn a policy from human gameplay data. Specifically, we collected 5 human-human trajectories of length 1200 time steps for each of the 5 layouts, resulting 60k total environment steps. We divide this data in half and train two BC agents: (1) a partner for training BCP agent, and (2) a "human proxy" partner for aggntt-aeent evaluation. Following Carroll al. [12]. we use a set of feature- -ased observations for the agents (as opposed to RGB) and generate comparable results: performance is higher on 3 layouts (asyumetric, cramped, and ring) but poorer on the other (circuit and forced).
Related work
Ad-hoc team play There is a large and diverse body of literature on ad-hoc team-play -lay S 660, also known zero-shot coordination [T Prior work based in same-theoretic settings has suggested the benefits of planning [71], online learning [S]]. and novel solution concepts (2]. to name a few examples. More recently, multi-agent deep reinforcement learning has provided the tools to scale to more complex gridworld or continuous control settings, leading to work on hierarchical social planning [6]. adapting to existing social conventions
40 (2). trajectory diversity [45). and theory of mind [4] Ad-hoc team-play among novel agent partners is also an object of active study in the emergent communication literature [TC] [1 43 This prior work has tended to focus on generalization
to held-out agent partners proxy for human co-players.
Collaborative play with novel humans has been evaluated more actively in the context of training agent assistants; see for instance (57] 68). To our knowledge, our FCP agents represent the state- of-the-art in coordinating with novel human partners on an equal footing of capabilities in rich gridworld environment, as measured by the challenge tasks in Carroll et al. [2].
Diversity in multi-agent reinforcement learning In multi-agent reinforcement learning, agents that train with behaviorally diverse populations of game partners tend to demonstrate stronger performance than their self-play counterparts. For example, across a range of multi-agent games, generalization to held-out populations can be improved by training larger and more diverse populations
I 22 50
In mixed-motive settings, cooperation among agents be encouraged through social Similarly, competitiveness can be
diversity, such as in player preferences and rewards 3
optimized through selective matchmaking between increasingly diverse agents 24 39 69].
Despite the increased focus on improving multi-agent performance, evaluation has typically been constrained agent-agent settings. High-performing agents have infrequently been evaluated with humans, particularly in non-competitive domains [16]. We add to this growing literature, showing that training with diversity is a powerful approach for effective human-agent collaboration.
-agent interaction In recent years, increased attention has been directed toward designing machine learning agents capable of collaborating with humans 41
57
68
72 (see also [6 for
broader review on Cooperative AI). Tylkin et al. 68] is particularly notable in also demonstrating that partially trained agents be useful learning targets for human helpers, although in different domain (cooperative Atari). Our method, FCP, can be seen as extending theirs by training with multiple "skill levels" and random seeds, rather than just one, which we demonstrate to be crucial to our agents' performance (Tables D and Indd Figure 7
A key preceding entry in this research area is Carroll et al. 12]. who similarly investigated human- agent coordination in Overcooked. We use their method (BCP) as baseline throughout our exper- iments (Section 2.2) Relative to BCP, our approach removes the need for the expensive step of human data collection for agent training. Furthermore, through our novel human-agent experimental design, we go beyond objective performance metries to compare the subjective preferences that agents generate. For a detailed comparison of methods and results, see Appendix
4 Zero-shot coordination with agents
In this section, we evaluate our FCP agent, its ablations, and the baselines with held-out agents.
5

---

4.1 Evaluation method: collaborative evaluation with agent partners
Our primary concern in this work is generalization novel human partners (as investigated in Section 55. However, just as collecting human human-human human data for behavioral cloning is expensive, so is evaluating agents with humans. Consequently, we instead use generalization held-out agent partners cheap proxy of performance with humans. This is then used to guide our model selection process, allowing us be more targeted with the agents we select for our numan-agent evaluations.
We evaluate with three held-out populations:
1. A BC model trained human data, Hproxy, intended as proxy of generalization to humans, as done by Carroll et al. [12].
2. A set of self-play agents varying in seed, architecture, and training time (specifically, held- out seeds of the . = 32 partners trained for the FCP+A agent, see Section 2.4). These are intended lest generalization
diverse yet still skillful population.
3. Randomly initialized agents intended to test generalization to low-skill partners.
For all results, we report the average number of deliveries made by both players within an episode, aggregated across the 5 different layouts from Figure 4 (with the per-layout results reported in Appendix (.2). We estimate mean and standard deviation across random seeds. For each seed, we evaluate the agent with all members of the held-out population for 10 episodes per agent-partner pair.
4.2 Results
Finding 1: FCP significantly outperforms all baselines
To begin, we compare our FCP agent and the basclines when partnered with the three held-out populations introduced above. As can be seen in Figure FCP significantly outperforms all baselines when partnered with all three held-out populations. Notably, it performs better than BCP with H proxy. even though BCP trains with such a model and FCP does not. Similar to Carroll et al. [12, we find that BCP significantly outscores SP.
When paired with a randomly initialized partner which behaves suboptimally, we see an even greater difference between FCP and the baselines. Given that FCP is trained with non-held-out versions of such agents, it may not be surprising that it does so well with partners that behave poorly. However, what is surprising is how brittle the other training methods are. This suggests that they may not perform well with humans who are not highly skilled players, which we will see in Section
Deliveries
10
Deliveries
10-
Deliveries
10-
5.
5-
SP
PP
BCP
FCP
SP
PP
BCP
FCP
SP
PP
BCP
FCP
With Hproxy-
(b) With diverse SP agents.
(c) With random agents.
Figure 5: Agent-agent collaborative evaluation: Performance of each agent when partnered with each of the held-out populations (Section 4.1) in episodes of length T 540. Importantly, FCP scores higher than all baselines with variety of test partners. Error bars represent standard deviation over five random training seeds. Plots aggregate data across kitchen layouts; results calculated by individual layout can be found in Appendix
Finding 2: Training with past checkpoints is the most beneficial variation for performance
Next, we investigate how the different training partner variations influence FCP's performance. In particular, separately ablate the past checkpoints (T) and architecture (A) variations, evaluating them with the same partners as in Figure 5 The results of this evaluation are presented in Table
Comparing the FCP and FCP- columns, we see that removing past checkpoints from training significantly reduces performance. Comparing the FCP and FCP+ columns, we see that adding architectural variation to the training population offers improvement over training with past

---

Partner
FCP
FCP-T
PCP+A
FCP-T,+A
Hproxy
0..5
4.7
9.6
7.0
Diverse SP
11.2
6.9 0.1
1..1
8.6
Random
8.6 0.2 0.2
1.0
8.4
3.5
Table 1: Ablation results:
Performance of each variation of FCP training with past partner checkpoints (T for time) and adding partner variation in architecture (A). Scores are mean deliveries with standard deviation over 5 random seeds. Notably, we find that the inclusion of past checkpoints is essential for strong performance (FCP > FCP-T). and additionally including architectural variation does not improve performance (FCP z FA)+ However, architectural variation better than no variation, improving performance when past checkpoints are not available (FCP (FCP-T++A> FCP-T).
checkpoints. However, comparing the FCP-T and FCP-T.+A columns, we see that without training with past checkpoints, architectural variation in the population does improve performance.
Zero-shot coordination with humans
Ultimately, goal is to develop agents capable of coordinating with novel human partners. In this section, we run an online study to evaluate our FCP agent and the baseline agents in collaborative play with human partners.
Assign agents chefs, then generate rounds (partners A & layout)
Human participant plays with agent partners in series on the same layout
Elicit preference from human "Which partner did you prefer this round?"
Round Partnor
Round Portner
Strongly prefer
Prefer
BCP
FOP
FCP
Strongly prefer
Figure 6: Human-agent collaborative study: For our human-agen collaboration study, we recruited participants online to play games with FCP and baseline agents. Participants played randomized sequence of episodes with different agent partners and kitchen layouts. After every two episodes, participants reported the direction and strength of their preference between their last two partners.
5.1 Evaluation method: collaborative evaluation with human participants
To test how effectively FCP's performance generalizes to human partners, recruited participants from Prolific 18 551 for an online collaboration study (N = 114; 37.7% female, 59.6% male, 1.8% nonbinary; median age between 25 years). We used within-partie cathin-participant design for the study: each participant played with a full cohort of agents (i.e. generated through every training method). This design allowed to evaluate both objective performance as well subjective preferences.
Participants first read game instructions and played short tutorial episode guiding them through the dish preparation sequence (see Appendix ID.I.T for instruction text and study screenshots). Participants then played 20 episodes with a randomized sequence of agent partners and kitchen layouts. Episodes lasted T 300 steps (1 minute) each. After every two episodes, participants reported their preference over the agent partners from those episodes on five-point Likert-type scale. After playing all 20 episodes, participants completed debrief questionnaire collecting standard demographic information and open- open-ended feedback on the study. Our statistical analysis below primarily relies upon the repeated-measee measures analysis of variance (ANOVA) method. See Appendix for additional details of our study design and analysis, including independent ethical review.
5.2 Results
Finding FCP coordinates best with humans, achieving the highest score across maps To begin, we compare the objective team performance supported by our FCP and baseline agents. The strong FCP performance observed in agent-agent play generalizes to human-agent collaboration:
7

---

the FCP-human teams significantly outperform all other agent-human teams, achieving the highest average scores across maps. every p 0.001 (Figure 7ap. while performing as well as better than the other teams on each individual map (see Appendix D.3 Echoing the results from our agent-agent ablation experiments (Table Dp. the inclusion of past checkpoints in training proves critical FCP's strong performance, 0.001 (Figure / Similar to Carroll et al. 122 we find that BCP outscores SP when collaborating with human players, p < 0.001.
Finding 2:
Participants prefer FCP over all baselines
FCP's strong collaborative performance carries over our participants' subjective partner preferences. Participants expressed a significant preference for FCP partners over all other agents, including BCP, with every < 0.05 (Figure 7c.. Notably, while human-BCP and human-PP teams did not significantly differ their completed deliveries, participants reported significantly preferring BCP over PP, 0.003, highlighting the informativeness of our subjective analysis.
0.35
4-
4
Deliveries
Deliveries
BCP-
0..35
0.42
0.52
0.26
3
3
0.42
0.18
2-
1
1
FCP--
0.48
-6.2
0.55
0.23
0
0
SP
.0.6
0.26
0.11
-0.23
SP
PP
BCP
FCP
FCP FCP
FCP
BCP
FCP.r
(a) Number of deliveries by partner (FCP and baselines).
(b) Number of deliveries by
(c) Participant preference for row
partner (FCP and FCP-r).
partner column partner.
Figure 7: Human-agent collaborative evaluation: Evaluation and preference metrics from human- agent play episodes of length 300. Error bars represents 95% confidence intervals, calculated over episodes. Plots aggregate data across kitchen layouts; results calculated by individual layout can be found in Appendix D.3
5.3 Exploratory behavioral analysis
To better understand how the human-agent scores and preferences may have arisen, here we analyze the resulting action trajectories of each human and agent player in our experiment.
Cramped
0.12
0.35
0.31
0.30
0.37
Asymmetric
0.14
0.01
Asymmetric
Ring
0.05
0.03
0.17
0.19
0.03
Ring
0.23
0.39
0.41
0.41
0.54
Circuit
Circuit
0.21
0.39
0.40
0.32
0.47
0.34
0.56
0.21
0.26
0.31
Forced
Average
0.13
0.28
0.25
0.23
0.35
Average
0.15
0.53
0.39
0.24
0.11
FCP
8CP
(a) Proportion of episode spent moving.
Differences in pot preference.
Figure 8: Behavioral analysis: (a) FCP is able to move most frequently (35% of the time), cor- responding the best movement coordination with human partners. (b) FCP exhibits the most equal preferences over cooking pots (0.11 difference), aligning with human preferences. Values are calculated as the absolute difference in preferences between the two pots; indicates that the player only uses one of the two available pots, while 0 indicates that the player uses both pots equally.
Finding 1: FCP exhibits the best movement coordination with humans
First, we investigate how much each player moves an episode (Figure Sa), where moving higher fraction of timesteps may suggest fewer collisions and thus better coordínation with partner. Notably, we observe two results: (1) humans rarely move, behavior which dusti---istio for typical training methods (c.g. SP, PP) but seen in the training distribution for BCP and FCP.

---

(2) FCP moves the most on all layouts other than Forced, suggesting it is better at coordinating its movement strategy with its partner. This result was also reported by human participants, for example: "I noticed that some of my partners seemed to know they needed to move around me, while others seemed to get 'stuck' until I moved out of their way" (see Appendix D for more examples).
Finding 2: FCP's preferences over cooking pots aligns best with that of humans
Next, we investigate whether there was a preference for a specific cooking pot in the layouts which included two cooking pots (Figure [b) To do this, we calculate the difference in the number of times each pot was used by each player, where a high value indicates a strong preference for one pot and a low value indicates more equal preference for the two pots.
As can seen the FCP column, our agent typically has the most aligned preferences with that of humans (0.11 for FCP to 0.14 for humans). Behaviorally speaking, this means that our agent prefers one cooking pot over the other 55.5% of the time (i.e, a 0.11 point difference). In contrast, all other agents have strong preference for single pot. This is non-adaptive strategy which generalizes poorly to typical human behavior of using both pots, leading to worse performance.
Discussion
Summary In this work, we investigated the challenging problem of zero-shot collaboration with humans without using human data in the training pipeline. To accomplish this, introduced Fictitious Co-Play (FCP)- - a surprisingly simple yet effective method based on creating diverse set of training partners. We found that FCP agents scored significantly higher than all baselines when partnered with both novel agent and human partners. Furthermore, through rigorous human-agent experimental design, we also found that humans reported strong subjective preference to partnering with FCP agents over all baselines.
Limitations and future work Our method currently relies on the manual process of initially training and selecting a diverse set of partners. This is not only time consuming, but also prone to researcher biases that may negatively influence the behavior of the created agents. Additionally, while we found FCP with a partner population size of N = 32 sufficient here, for more complex games, FCP may require an unrealistically large partner population size to represent sufficiently diverse strategies. To address these concerns, methods for automatically generating partner diversity for common-payonn games may be important. Possibilities include adaptive population matchmaking as been used in competitive zero-sum games 669. as well as auxiliary objectives that explicitly encourage behavioral diversity IT9
Our method requires a known and fixed reward function. We also focus on one domain in order to compare with prior work which has argued that human-in-the-loop training necessary. Consequently, the resulting agents are only designed to adaptively collaborate on a single task, and not to infer
human preferences in general 1 35 59). Moreover, if task's reward function is poorly aligned with how humans approach the task, our method may well produce subpar partners, as would any method without access to human data. Thus, additional domains and tasks should be studied to better understand how our method generalizes. Targeted experiments to test specific forms of generalization may be especially helpful in this regard [ل as could approaches that procedurally
generate environment layouts requiring diverse solutions
(2
Finally, it may be possible to produce even stronger agent assistants by combining the strengths of FCP (i.e. diversity) and BCP (i.e. human-like play). Indeed, Knott et al. (3 recently demonstrated that modifying BCP train with multiple BC partners produces more robust collaboration with held-out agents, a finding that would be interesting to test with human partners.
Societal impact A challenge for this line of work is ensuring agent behavior is aligned with human values (i.e. the AI value alignment problem 23 ).). Our method has no guarantees that the resulting policy aligns with the preferences, intentions,
welfare of its potential partners. It likewise does not exclude the possibility that the target being optimized for is harmful (e.g. if the agent's 's partner expresses preferences or intentions to harm others). This could therefore produce negative societal effects either if training leads to poor alignment or if agents are optimized for harmful metrics.
One potential strategy for mitigating these risks is the use of human preference data [5]] Such data could be used to fine-tune and filter trained agents before deployment, encouraging better alignment with human values. A key question in this line of research is how human preference data should be

---

aggregated-og or selected, in the case of expert preferences-wine -hen our aim is to create socially aligned agents (i.e. agents that are sufficiently aligned for everyone). Relatedly, targeted research on human beliefs and perceptions of Al 48 and how they steer human-agent interaction, would help inform agent design for positive societal impact. For instance, developers could incorporate specific priors into agents to reinforce tendencies for fair outcomes 20 32
Conclusion We proposed a method which is both effective collaborating with humans and simple to implement. We also presented a rigorous and general methodology for evaluating with humans and eliciting their preferences. Together, these establish a strong foundation for future research on the important challenge of human-agen collaboration for benefiting society.
Acknowledgements
The authors would like to thank Mary Cassin for creating the game sprite art; Rohin Shah, Thore Graepel, and lason Gabriel for feedback on the draft; Lucy Campbell-Gillinganne Tina Zhu, and Saffron Huang for support in evaluating agents with humans; and Max Neiman--Wennee Natasha Jaques, Marc Lanctot, Mike Bowling, and Dan Roberts for useful discussions.
Funding disclosure
This work was funded solely by DeepMind. The authors declare no competing interests.
References
[1] Abramson, A. Ahuja, I. Barr, A. Brussee, Carnevale, M. Cassin, Chhaparia, S. Clark, B. Damoc, A. Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXiv:20/2.05672, 2020.
[2] S. Albrechh and S. Ramamoorthy, A game- thee-theoreeic model and best-response learning method for ad hoc coordination in multiagent systems. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2013.
[3] B. Baker. Emergent reciprocity and team formation from randomized uncertain social prefer- ences. In Neural Information Processing Systems (NeurlPS), 2020.
[4] N. Bard, J. N. Foerster, S. Chandar, N. Burch, M. Lanctot, H. F. Song, E. Parisotto, V. Dumoulin,
S. Moitra, E. Hughes, et al. The Hanabi challenge: A new frontier for AI research. Artificial Intelligence, 280: 103216, 2020.
[5] Barrett, A. Rosenfeld, Kraus, and Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132-171 2017.
[6] A. Bauer, D. Wollherr, and M. Buss. Human roboot collaboration: A survey. International Journal of Humanoid Robotics. 5(01):47-66 2008
[7] G. W. Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374-376, 1951.
[8] N. Brown and T. Sandholm. Superhuman Al for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 4244,442422202 2018.
[9] N. Brown and T. Sandholm. Superhuman Al for multiplayer poker. Science, 365(6450))8838889
2019.
[10] K. Bullard, F. Meier, D. Kiela, J. Pineau, and J. Foerster. Exploring zero-shot emergent communication in embodied multi-agent populations. arXiv preprint arXiv:2010. 15896, 2020.
[11] K. Bullard, D. Kiela, Pineau, and J. Foerster. Quasi-equivalence discovery for zero-shot emergent communication. arXiv preprint arXiv:2103.08000 2021.
[12] M. Carroll, R. Shah, M. Ho, T. Griffiths, S. Seshia. P. Abbeel, and A. Dragan. On the utility of learning about humans for human-Al coordination. In Neural Information Processing Systems (NeurIPS), 2019.
[13] R. Charakorn, P. Manoonpong, and N. Dilokthanakul. Investigating partner diversification methods cooperative multi-agent deep reinforcement learning. International Conference on Neural Information Processing, 2020.
10

---

[14] R. Choudhury, G. Swamy, D. HadielldMMenlll, and A. D. Dragan. On the utility of model learning in HRI. In ACM/IEEE International Conference on Human-Rond Interaction (HRI), 2019.
[15] P. F. Christiano, J. Leike, T. B. Brown. M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4302 4310, 2017.
[16] A. Dafoe, E. Hughes, Y. Bachrach, Т. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open problems cooperative AI. arXiv preprint arXiv:2012.085: 08630, 2020.
[17] I.. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
[18] P. Eyal, R. David, G. Andrew, E. Zak, and D. Ekaterina. Data quality of platforms and panels for online behavioral research. Behavior Research Methods, pages 1-20, 2021.
[19] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations (ICLR), 2019.
[20] E. Fehr and K. M. Schmidt. A theory of fairness, competition, and cooperation. The Quarterly Journal of Economics, 114(3):817-86) 168, 1999.
[21] J. Foerster, F. Song. E. Hughes, N. Burch, I. Dunning, S. Whiteson, M. Botvinick, and M. Bowl- ing. Bayesian action decoder for deep mulli-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
[22] M. Fontaine, Y.-C. Hsu, Y. Zhang, B. Tjanaka, and S. Nikolaidis. On the importance of environments in human-robot coordination. In Robotics: Science and Systems (RSS), 2021.
[23] I. Gabriel. Artificial intelligence, values, and alignment. Minds and Machines, (003):11-4431
2020.
[24] M. Garnelo, W. M. Czarnecki, S. Liu, D. Tirumala, Oh. G. Gidel, H. van Hasselt, and D. Balduzzi. Pick your battles: Interaction graphs as population-level objectives for strategic diversity. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2021.
[25] Ghost Town Games. Overcooked, 2016.
[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual lcarning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[27] J. Heinrich and D. Silver. Deep reinforcement learning from self-play in imperfect-inffomaaiio games. In NIPS Deep Reinforcement Learning (DRL) Workshop, 2016.
[28] J. Heinrich, M. Lanctot, and D. Silver. Fictitious self-play in extensive-form games. In International Conference Machine Learning (ICML), 2015.
[29] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
[30] M. Hoffmann B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, Norman, A. Abdol- maleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforeement learning. arXiv preprint arXiv:2006.000 00900, 2020.
[31] H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. "Other-Play" for zero-shot coordination. In International Conference on Machine Learning (ICML), 2020.
[32] E. Hughes, J. Z. Leibo, M. G. Phillips, K. Tuyls, E. A. Duéiiez-Guzmán, A. G. Castañeda, I. Dunning, T. Zhu, K. R. McKee, R. Koster, H. Roff, and T. Graepel. Inequity aversion improves cooperation in intertemporal social dilemmas. In Neural Information Processing Systems (NeurIPS), 2018.
[33] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in Atari. Neural Information Processing Systems (NeurIPS), 2018.
11

---

[34] M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. Morcos, A. Ruderman, et al. Human-level performance in 3D mul- tiplayer games with population-based reinforcement learning. Science, 364(6443):859-866) 2019.
[35] S. Javdani, S. S. Srinivasa, and A. Bagnell. Shared autonomy via hindsight optimization. Robotics Science and Systems, 2015.
[36] M. Kleiman- Weiner, M. K. Ho, J. L. Austerweil, M. L. Littman, and J. B. Tenenbaum. Co- ordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In Conference of the Cognitive Science Society (CogSci), 2016.
[37] G. Klien, D. Woods, J. M. Bradshaw, R. R. Hoffman, and P. J. Feltovich. Ten challenges for making automation "team player" in joint human-agent activity. IEEE Intelligent Systems, 19 (6):91-95, 2004.
[38] P. Knott, M. Carroll, S. Devlin, K. Ciosek, K. Hofmann, A. Dragan, and R. Shah. Evaluating the robustness of collaborative agents. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2021.
[39] M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, and
T. Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Neural Information Processing Systems (NIPS), 2017.
[40] A. Lerer and A. Peysakhovich. Learning existing social conventions via observationally augmented self-play. In AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES), 2019.
[41] E. Lockhart, N. Burch, N. Bard, S. Borgeaud, T. Eccles, L. Smaira, and R. Smith. Human-agent cooperation in bridge bidding. arXiv preprint arXiv:2011.14124, 2020.
[42] R. Lowe. Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch. Multi- aggent actor- for mixed cooperative-competiiee environments. In Neural Information Processing Systems (NIPS), 2017.
[43] R. Lowe. A. Gupta, J. Foerster, D. Kiela, and J. Pineau. Learning to learn to communicate. In ICML Adaptive Multitask Learning Workshop, 2019.
[44] R. Lowe, A. Gupta, J. Foerster, D. Kiela, and J. Pineau. On the interaction between super- vision and self-play in emergent communication. In International Conference on Learning Representations (ICLR), 2020.
[45] A. Lupu, H. Hu, and Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2021.
[46] A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. Maven: Multi-agent variational exploration. In Neural Information Processing Systems (NeurIPS), 2019.
[47] K. R. McKee, Gemp, B. McWilliams, E. A. Duéñezz-uzzmiae E. Hughes, and J. Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.
[48] K. R. McKee, X. Bai, and S. Fiske. Understanding human impressions of artificial intelligence. PsyArXiv, 2021.
[49] K. R. McKee, E. Hughes, T. O. Zhu, M. Chadwick, R. Koster, A. G. Castaneda, C. Beattie, T. Graepel, M. Botvinick, and J. Z. Leibo. Deep reinforcement learning models the emergent dynamics of human cooperation. arXiv preprint 1iXii:22133.04822 2021.
[50] K. R. McKee, J. Z. Leibo, C. Beattie, and R. Everett. Quantifying environment and population diversity in multi-agent reinforcement learning. arXiv preprini arXiv: (tt::2/022087700 2021. [51] F. Melo and A. Sardinha. Ad hoc teamwork by learning teammates' task. International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). 2015.
[52] B. Mutiu, A. Terrell, and C.-M. Huang. Coordination mechanisms human-robot collaboration.
In HRI Collaborative Manipulation Workshop, 2013.
[53] S. Nikolaidis and J. Shah. Human-robot cross-training: Computational formulation, modeling
and evaluation of human team training strategy. In ACM/IEEE International Conference on Sluman-Robo Interaction (HRI), 2013.
12

---

[54] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and I.. Zhang. Solving Rubik's Cube with robot hand. arXiv preprint arXiv: 1910.07113, 2019.
[55] E. Peer, I.. Brandimarte, S. Samat, and A. Acquisti. Beyond the Turk: Alternative platforms for crowdsoureing behavioral research. Journal of Experimental Social Psychology, 10:153-, 2017.
[56] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[57] P. M. Pilarski, A. Butcher, M. Johanson, M. M. Botvinick, A. Bolt, and A. S. Parker. Learned human-agent decision-making, communication and joint action in virtual reality environment. In Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2019.
[58] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88 977888897,11 1991.
[59] S. Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin, 2019.
[60] D. Sadigh, N. Landolfi, S. S. Sastry, S. A. Seshia, and A. D. Dragan. Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state. Autonomous Robots, 42(7)) 405-6422, 2018
[61] N. Schurr, J. Marecki, M. Tambe, and P. Scerri. Towards flexible coordination of tugen-agent teams. Multiagent and Grid Systems, 1(1):3-16, -6, 2005.
[62] A. Shih, A. Sawhney, J. Kondic, S. Ermon, and D. Sadigh. On the critical role of conventions in adaptive human-Al collaboration. In International Conference on Learning Representations (ICLR), 2021.
[63] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 55047676):554-35,, 2017.
[64] D. Silver, T. Hubert, J. Schrittwieser, I. Antonogiou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140- 1144, 2018
[65] H. F. Song, A. Abdolmaleki, J. T. Springenberg, A. Clark, H. Soyer, J. W. Rae, S. Noury, A. Ahuja, S. Liu, D. Tirumala, N. Heess, D. Belov, M. Riedmiller, and M. M. Botvinick. V- MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations (ICLR), 2020.
[66] P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein. Ad hoc autonomous agent teams: Collabo- ration without pre-coordination. In AAAI Conference on Artificial Intelligence, 2010.
[67] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RS International Conference on Intelligent Robots and Systems (IROS), 2017.
[68] P. Tylkin, G. Radanovic, and D. C. Parkes. Learning robust helpful behaviors in two-player cooperative Atari environments. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2021.
[69] O. Vinyals. I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu,
D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent
reinforcement learning. Nature, 575(7782)350-334, 2019.
13

---

[70] R. E. Wang, S. A. Wu, J. A. Evans, J. B. Tenenbaum, D. C. Parkes, and M. Kleiman Weiner. Too many cooks: Coordiaatin multi-agent collaboration through inverse planning. International Conference Autonomous Agents and Multiagent Systems (AAMAS), 2020.
[71] F. Wu, S. Zilberstein, and X. Chen. Online planning for ad hoc autonomous agent teams. In International Joint Conference Artificial Intelligence (IJCAI), 2011.
[72] S. Zheng, A. Trott, S. Srinivasa, N. Naik, M. Gruesbeck, D. C. Parkes, and R. Socher. The Al economist: Improving equality and productivity with Al-driven tax policies. arXiv preprint a3X13 2020.
14


Reviews:
[{"review_id": "uy_jQ_OEyf", "reviewer": "Reviewer_6sVb", "summary": "The authors present a new training approach for human-AI collaboration (Fictitious Co-Play, FCP) which in a tested environment enables higher zero-shot coordination performance with humans than state-of-the-art. The idea behind FCP is to first train many self-play agents with different seeds and save snapshots from these training runs; then one can train the actual collaborative agent to perform well with all the previously saved snapshots. An important aspect of this approach is that it requires no human data, which can be expensive or difficult to come by. Experimentally, they show that this approach is able to perform both better than previous self-play or population-play approaches, and human aware agents (trained with a human model) trained with some amount of human data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nThe method proposed is new, although it does have connections to previous work which also tries to construct populations of agents to improve collaboration performance [Knott, Evaluating the Robustness of Collaborative Agents, 2021] (not currently cited). It adapts similar ideas to those common in zero-sum settings for creating diverse populations to train high-performing agents, applying them in the collaborative setting.\n\n## Quality / Clarity\n\nThe papers' claims are well supported, although in my opinion there is not enough emphasis on the limitations (as mentioned below). The experimental setup and the user study were well designed. Overall the paper was very well written and clear, and was a pleasure to read!\n\n## Significance\n\nThese results constitute a significant contribution to the field of human-AI collaboration, which add to the conversation as to what are possible approaches to create AI agents able to collaborate with humans effectively. Through their results, the authors demonstrate that even without human data, in some domains we might be able to achieve better collaboration with humans than previous state of the art methods (which might even be using human data). Although their method is quite simple and not very dissimilar from previous population approaches in zero-sum settings, the results are strong enough to still make this a very worthwhile contribution that adds to the conversation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While FCP clearly performs better than SP, PP, and BCP (at least in Overcooked), I am skeptical about the claim that the improvement is mostly due to improved coordination-protocol symmetry-breaking enabled by the FCP training.\n\nMuch of my skepticism originates in the fact that FCP is a suboptimal method even in the limit of perfect RL training (which I think should be emphasized more in the paper): to achieve best human-AI collaboration performance, one would optimally train an agent directly with humans (or with a perfect human model obtained with infinite data). Training with a diverse population might help increase the robustness of the trained agents, but will actually be biasing the method away from the true human-AI optimum. More on this below.\n\nAs mentioned in [Knott 2021], if one were to train with a population of adversarial agents (although we are in a collaborative setting), that would be far too conservative: the best thing in a self-driving domain might be to never leave the garage, because going outside you would almost inevitably crash (if everyone was trying to crash into you). Because of this, simply increasing diversity (the Diversity is All You Need approach, cited by this paper) will not necessarily increase performance (one instance of this might also be present in this paper itself, with the ablation introducing diversity of network architectures).\n\nWhat is really needed for maintaining the theoretical claim to optimality is a population of _human-like_ agents. The more the population differs from human-likeness, the more the trained agents will be biased to act more conservatively or optimistically about their partner's capabilities than they possibly could.\n\nAgents created by FCP will likely not be very human-like, both in Overcooked but especially so in environments like Hanabi. While you claim that \"The ﬁnal checkpoint represents a fully-trained “skillful” partner, while earlier checkpoints represent less skilled partners.\", Converged SP agents are not like skillful partners (the coordination protocols they settle on can be extremely uninterpretable and non-human-like), as you claim yourself in the intro and cite: [9, 10, 11, 18, 41, ...]. Earlier checkpoints too are likely not going to be similar to humans mid-way through learning about the game, as V-MPO agents and humans likely learn in very different ways: a beginner SP agent will be very different from a beginner human, and so doing well with the first will not guarantee doing well with humans. Additionally, humans definitely do not act randomly, but random agents (random initialization networks) are added to the population of FCP. This will definitely bias the obtained agents to be more independent and not rely on the other agent at first (under the assumption that the human might be random), more so than would be warranted by real humans.\n\nRelatedly, as another source of bias, you are assuming that the pool of agents will be representative of all the conventions and symmetry breaking which humans perform too, and human biases (this is necessary for obtaining \"a good agent partner [that] will adaptively switch between these conventions if a human clearly prefers one over the other\"). Systematic myopia / cognitive biases would likely not be represented in the FCP population and could not be adapted to at test time. As an example, human-like ways of hinting in Hanabi would not necessarily naturally emerge among the SP trained agents. Generally, it might be very unlikely to stumble upon human-like protocols among the space of possible coordination protocols (this is most clear with languages, where training SP conversation bots with no grounding in languages would essentially never \"stumble upon\" english or any other human language – there is no way to get around this without human data / inductive biases).\n\nHowever, as acknowledged above, clearly FCP is working better than just training with a SP agent or even an approximately learned human model (with limited human data). I believe that this is because actually FCP is improving the overall _robustness_ of the agents rather than simply their ability to adapt to different symmetry-breaking strategies (although this is probably improved too to some degree, as supported by Figure 8.b). I suspect that much of the improvement is due to decrease in brittleness relative to the baselines, which all overfit to the single agent they were trained with (the extreme brittleness is documented for Overcooked in [Knott 2021]). Many of the results are also consistent with the robustness hypothesis – for example also the fact that \"FCP is able to move most frequently (35% of the time), corresponding to the best movement coordination with human partners\". While this might be hard to verify (I can't think of an experiment in this setup off the top of my head), I would use the conversational AI example above as a cautionary tale: even in the limit of perfect RL convergence and compute, I doubt that doing FCP in a human-AI collaborative conversation task would work; English would never be \"stumbled upon\", and the agent – although very capable of adapting to optimal obscure languages that SP tends to come up with – would have no idea what to make of it. FCP in my mind might work empirically in many settings better than the alternatives, but is not guaranteed to and likely its performance will depend critically on the combination of domain and how the checkpointing is done (i.e. the population is formed).\n\nConditional on the authors being more clear about these limitations upfront and throughout the paper (i.e. 1. the inevitable bias that comes from not assuming access to human data, and 2. the difficulty of disambiguating robustness effects from increased coordination), I would be in favor of acceptance. This is a well executed work that sheds important insights on how one might build populations of agents that one can train with to improve human-AI performance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "fgmsScodKsY", "reviewer": "Reviewer_5uRi", "summary": "This paper proposes using Fictitious Co-Play as a way to train agents so that they will be able to cooperate with humans. The idea is to avoid needing human data and rather to train a diverse set of self-play agents and then train an RL agent so it can play well with this diverse set of human surrogates. The paper is very well written, the method is simple and effective, and the results clearly illustrate the benefits of the proposed approach.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very nice paper. I like the simplicity of the method and the thoroughness of the results and analysis. The authors do a good job of motivating the need for diversity and the fact that the method works well with humans despite not using any human data is seems very promising.  The experiments are well designed and the results are impressive.\n\nQuestions: \nLine 125: Why was V-MPO chosen for RL? Would other methods work as well?\nLine 137: Was this data collected from two different people playing for the first time or were they expert demos? Were the people cooperating well?\n\nFigure 5a. It seems like BCP is the oracle here why does FCP perform so much better?\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "eU5wbcwc4Mx", "reviewer": "Reviewer_bYPA", "summary": "The paper introduces a practical method, named Fictitious Co-Play, for agents to learn to collaborate with humans / new agents in common-payoff games. The key idea is to first collect a diverse set of simulated agents, and then train the primary agent such that it can collaborate with the whole set of simulated partners. To promote the simulation diversity, they propose to gather the simulated agents at different learning stages, namely checkpoints, and from different runs, namely random seeds. The method is evaluated in the two-player cooking simulator with either held-out agents or human participants. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall: \nThe technical contribution seems limited. The experimental results are not fully convincing. \n\nOriginality: \nThe core idea of the paper seems very similar to domain randomization, an already popular technique for transferring policy from simulation to the real world. The task, collaborating with a human partner without human data, can be viewed as a particular instance of learning a policy for a real-world environment by using only simulation data. It's already known that diversifying the simulated domains can promote generalization. It would be good to see a clear discussion about how the proposed method differs from the previous domain randomization approach. \n\nQuality: \nThe proposed method is highly intuitive. It would be good to see when it works and fails. Unfortunately, this is not clear in the experimental evaluations. In fact, half of the experiments are performed over held-out simulated agents, which are likely well represented by the simulated sets of agents during training. \n\nClarity: \nThe paper is well written and easy to follow. \n\nSignificance: \nI'm not sure if the paper provides any significant insight into the community. Nevertheless, the simulation environment, which the author promised to open-source, might be a good contribution to the field. \n\n[1] Tobin, Josh, et al. \"Domain randomization for transferring deep neural networks from simulation to the real world.\" 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017.\n\n------------------\n\nPost rebuttal:\n\nI've carefully read all responses from the authors and comments from other reviewers. \n\nThe authors addressed part of my concerns about experimental evaluations, and I, therefore, increase my rating from 4 to 5.\n\nThe limitation that prevents me from raising it to 6 or higher is that the technical contribution and insight seem quite marginal. After discussions with the authors and other reviewers, I still feel that the proposed method is a very natural instantiation of domain generalization in the considered human-AI collaboration context.\n\nI suggest the authors add a detailed discussion about the connection between their method and the domain generalization framework, which in my opinion is essential.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Generally, I think the proposed method may inherit all the limitations of the domain randomization approach, e.g., the method may fail when the simulated sets of agents cannot cover the real-world human partner. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "Vi-38jmsdGr", "reviewer": "Reviewer_sJ2R", "summary": "The paper studies the problem of cooperating with humans (without human data) in the overcooked domain, by proposing a new training procedure.  To cooperate with humans, they study making a training procedure for agents which robustly cooperate. The proposed procedure Fictitious Co-play (FCP) allows agents to robustly cooperate with other agents of varying skill-levels.  This is done by saving checkpoints of agents along training trajectories, and then training a final agent against a distribution of checkpoints.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge, the proposed method is novel.  The differences from other competing methods -- ex., self-play, population-play, behavioral cloning -- is clear.  My experience is primarily in multi-agent learning and the related work looks adequately cited there.  I am less able to comment on related work for (a) zero-shot coordination, (b) reinforcement learning, or (c) human-agent interaction.\n\nThe paper does not provide a theoretical analysis of the method, though I am not familiar with any machinery the authors could use for analysis in zero-shot coordination, or coordination with humans.  A strength is the paper provides strong intuitive arguments for why the method works, which are supported by empirical results.  However, a weakness is the paper is highly focused on the Overcooked environment. Do these engineering choices generalize to other environments?  For example, do we ever need different choices about when to save checkpoints and how to select the checkpoints we train against, or can we simply use their 3 checkpoint heuristic?\n\nFor the most part, the paper is clearly written and the key ideas are easy to understand. The authors have excellent descriptions and visualizations for the overcooked environment. Some details in the appendix, like Algorithm 1 or a discussion of how checkpointing is handled, are important details and might be better suited to the main body. The tables are difficult to parse. For example, FCP, FCP_{-T, +A}, etc. are confusing notations for the idas they represent. Also, in figures (where space allows) I would appreciate it if you re-defined the acronyms that are used, so readers don’t have to look through the paper to find what they mean.  For example, H_proxy, notable baselines, or what the FCP subscripts intuitively represent. Alternatively, a notation table would be useful.\n\nI believe the results are reasonably significant because zero-shot coordination, particularly with humans and limited data, will become increasingly important as machine learning models have to coordinate with people. Practitioners are likely to use the method, because it is simple to understand, does not look difficult to implement, does not introduce excessive computational burden, and seems to work reasonably well.  The paper claims to advance the state-of-the-art in coordinating with human partners in a set of gridworld tasks, which is supported by their human-agent experiments. The significance would be higher if the other domains besides Overcooked were evaluated.\n\nMy score is 8, because the authors provide a novel method with well-supported claims in a (for the most part) clearly written paper, which advances state of the art in the significant problem of coordinating with humans. My confidence 4, because I have experience with multi-agent learning, but less so with zero-shot coordination and human-agent interaction.\nThe score would be improved if authors investigated the method in domains besides overcooked, or clarity in the method, ablations and comparisons to baselines is improved.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the potential negative societal impacts of their work.\n\nThe authors present reasonable limitations.  A limitation that is not discussed, is that they introduce hyperparameters which are glossed over in the main paper.  Specifically, they must choose when to save checkpoints of the agents, and how to select which checkpoints the final agent will be trained against.  A (single) heuristic choice is made for Overcooked, but how well these choices will generalize to other games is unclear.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical issues with the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple trick to get multi-agent RL agents to cooperate effectively with a previously unseen agent (such as a human). The idea is to first train multiple agents with self-play and then train the final agent to cooperate with all of those agents as well as their past checkpoints. Experiments show successful results in the Overcooked game.

This submission was a pleasure to read. Three of the reviewers enthusiastically recommend acceptance, while the remaining reviewer recommends rejection because they consider it an incremental extension of domain randomization methods. In my opinion, this setting is substantially different from traditional uses of domain randomization, and while this work is certainly inspired by the idea of randomization helping generalization, the details are (as far as I know) novel. Apart from this, the reviewers didn't have any major objections. Since the submission seems high quality and ought to be of broad interest, I recommend it for a spotlight.

Decision: Accept (Spotlight)