Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a method of training collaborator agents that cooperates with humans without the need for human behaviour data. It proposes that the main drawback of "behaviour cloning play"(BCP) is that it is expensive to collect sufficient human data. The method proposed in this paper is called Fictitious Co-Play(FCP), and is based on the prior methods self-play(SP). Through different settings, initialization and additional checkpoints, we can get a collection of self-play agents of different degrees of skill. Via training with the collection of self-play agents, a collaborator agent which collaborates with novel human well in common-payoff games can be obtained. Self-play agents and their checkpoints during training replace a large amount of human data.
main_review: Quality: This paper makes possible the zero-shot coordination with humans. Yet, this paper does not give a theoretical basis for this, nor does it give a proof. The experimental environment to prove the effectiveness of this method is relatively simple.

Clarity: This paper illustrates the idea and logic in a very clear and easy-to-understand way.

Originality: The main innovation of this paper is to use the checkpoint in the training process, so as to improve the diversity of the behavior of the collaborators who train the collaborator agent, and make the collaborator agent have strong generalization ability. The application of this method in the field of agent human cooperation is an innovation, but in the field of deep computer vision, there has been the realization of similar idea.

Significance: This paper is of great significance to the research field of human agent cooperation. It proposes that human data can be unnecessary for training agents which collaborate with humans. This attempt is of significance.

The main weakness of this paper is that the author does not give the theoretical connection between the method and the effect, and does not give a large number of results under different experimental environment.

Yet, it is an interesting problem to train agents that can cooperate well with human beings without using human data. The author's attempt is worthy of affirmation.


limitations_and_societal_impact: Question:
1.Can the authors give theoretical proof of what the thought is effective?
2.Can the experiment be carried out in more experimental environments to obtain more convincing experimental results?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper seeks to build agents that can effectively collaborate with humans when playing "common payoff games." They propose to modify Fictitious Self-Play (FSP) methods to work in this setting, hence the name Fictitious Co-Play (FCP). Specifically, a final agent is trained to collaborate with a diverse range of agents (generated w/ varying random seed) alongside *earlier checkpoints* of those agents. This simulates a self-play pool with varying skill levels and varying strategies. Thanks to this more diverse training experience, the final FCP agent is more robust to unseen strategies and more amenable to human interaction.
main_review: The idea is strikingly simple and presented in an easy to follow manner. Furthermore, it shows significant gains when compared to prior work that relies simply on self-play or examples of optimal human behavior. The ablations due a good job justifying the use of previous model checkpoints, as well as the network architectures changes made v.s. prior work. Finally, the human study convincingly shows that this agent excels (v.s. baselines) at its stated goal - collaborating with humans.
limitations_and_societal_impact: The authors do a good job discussing their method's limitations as well as the potential negative social implications. However, I believe they should also mention that the computational/time burden of training N separate agents could prove overwhelming in more complicated settings than grid world (e.g. real world robotics, Atari, etc.). Future work should definitely focus on more efficiently learning a diverse agent set.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a model-free RL training method to improve robustness to partner/teammate changes, with a focus on human partners. The key finding is that diversity in behavior appears sufficient; as long as the agent has been exposed to similar partner behaviors during training, performance is good. The authors propose to incorporate such behaviors by using agents trained using different random seeds or from previous checkpoints in the training process. Experiments in the Overcooked domain suggest this fictitious co-play (FCP) setup enables agents to better cooperate with both artificial and human agents. 
main_review: This paper is well-written and timely given growing interest in applying AI to work with humans. The results show that FCP results in more robust models and not having to use collected human data is a key advantage. I can see others adopting this technique for their own domains. Although I am generally supportive of the work, I have concerns about the novelty and generalizability of the results. 

First, using prior agent "checkpoint" models to capture different skill levels is similar to previous work [Tylkin, Radanovic & Parkes, 2020]. There, the domain is different (Atari games) and the goal was to train a helper agent, but there are similarities in motivation and technique. Perhaps the authors can clarify the key differences? Also, the idea that diversity is important for training agents to collaborate is not exactly new; there is also prior work in using hand-coded policies or past teammate models, e.g., PLASTIC [Barrett, Rosenfeld, Kraus, Stone. 2017]. 

I like the fact that different layouts were engineered to test different collaborative behaviors and robustness to different problem settings. However, it remains unclear how well the findings will generalize to other domains (e.g., Starcraft, Hanabi). A more thorough analysis would help support the main claims and to provide more insights. I also have some specific questions about the experiments:

- it would help to quantify how different the held-out agent populations are. Although the model structures may be different, the policies could be functionally very similar for this task?
- For the experiment with humans, there appear to be performance differences on specific layouts? Figure 20 in the Appendix shows that on the asymmetric and forced layouts, FCP's performance is fairly similar to BCP and PP, respectively. Some elaboration is needed here, especially given the claim that "FCP coordinates best with humans, achieving the highest scores across all maps". Perhaps I am missing something here? 
- In Fig 8, why are the BCP metrics for pot preference so dissimilar from the human data? Presumably, BCP was trained using human data and would be able to replicate the preference distribution? 

Overall, this work contributes an interesting study and the findings (if generally applicable) are meaningful for human-AI collaboration. The method is simple to implement and works well. That said, I am on the fence given my concerns above. If the authors can address them, I am inclined to increase my score. 

### Post-Reponse Comments
Thank you for the clear response. I've read through the other reviews and responses. Overall, I am in favor of the topic and questions, but I find the proposed method/approach lacks clear novelty. Training the model with multiple seeds and for different skill levels appears to be an incremental change. Stronger experiments with analysis into the policy difference/disagreement scores would help. While these scores are "low-level", they can help to at least indicate that the policies are different to some degree. As such, I am keeping my score. 
 

limitations_and_societal_impact: Limitations and societal impact are adequately discussed.
review_title: null

Meta-review: The paper proposes a new approach for zero-shot coordination with humans in a multi-agent RL setting. The problem studied in the paper is of great significance to the research on human-AI collaboration, and the reviewers acknowledged several strengths of the paper, including: (a) the key ideas for zero-shot coordination are intuitive and show significant gains; (b) the experiments include an evaluation with human subjects; (c) the paper is clearly written and easy to follow.  

However, several reviewers also raised concerns about the paper, including: (a) limited novelty as the work appears incremental given prior work by [Tylkin et al. 2020]; (b) no theoretical connection between the method and the effect is provided; (c) experimental evaluation is limited to a single environment. Given these strengths and concerns, we collected additional feedback from an emergency reviewer. This additional reviewer shared the same concerns as mentioned above. Based on this input and the discussions, unfortunately, the final decision is a rejection. I want to thank the authors for their detailed responses and for engaging with the reviewers during the discussion phase. The reviewers have provided detailed and constructive feedback to the authors. Apart from the reviewers' comments in their initial reviews, we would like to share the following comments that came up during discussions:

- The paper should try to disentangle the contributions from the existing ideas in multi-agent RL. The proposed training approach, FCP, closely relates to fictitious self-play, but also to the line of work on training robust cooperative AI agents. In particular, the paper could benefit from a more detailed discussion on how its results relate to those in [62, 63] since FCP appears to be a combination of fictitious self-play and training approaches in [62, 63]. 

- Experimental results could be extended and more thoroughly discussed. For example, while it's clear that partly trained partner models (past checkpoints) help, it's less clear how much fully trained partner models (final checkpoints in self-play) contribute in the FCP method. Here additional baselines would help, e.g., those trained with random agents as partners or with only past checkpoints (low/medium-skilled partners). The latter could also be beneficial for relating the experimental findings to [62, 63].

- It is not clear under which condition these results generalize to other domains/environments. More in-depth analysis with a more diverse set of environments would be beneficial for understanding when the proposed approach will likely outperform the baselines.

- While the paper reads well, its structure could be improved. For example, the figures take a lot of space, and some of them do not appear to be needed (e.g., Figure 4, since these environments seem to be taken from [11]). On the other hand, some parts that are in the appendix (e.g., per-layout results) could be in the main part of the paper.

This is exciting and potentially impactful work, and we encourage the authors to incorporate the reviewers' feedback when preparing future revisions of the paper.

Decision: Accept (Spotlight)