Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The current work proposes to use dynamic population based meta-learning in referential games between agents using natural language. In particular, it uses meta-learning to capture the diverse speaker/listener behaviors in a population and then initialize new agents by interacting with these meta-agents across iterations.  They also add supervised learning iterations to ensure there is no language drift. Experiments show that agents trained through the proposed approach have higher accuracy in the referential game interacting within themselves and also with humans, while retaining the naturalness of the language.
main_review: **Strengths**  
(S1) The idea to use a meta-listener and meta-speaker across different iterations in dynamic populations is indeed an interesting and novel direction. It is also well motivated in the paper.

(S2) Experimental evaluation is solid and backs up the claims in the earlier part of the text. The increase in referential game accuracy (with statistical significance) and improved BLEU score when compared to ground truth captions both evidence the effectiveness of the proposed solution. The human studies are the cherry on top where the human-agent teams trained using the proposed approach outperform other existing approaches.

(S3) The authors did a good job writing the manuscript as it is easy to read and understand. The work is well placed in the context of related work and all the experimental details are clearly explained.

**Weaknesses**  
(W1) The manuscript does not contain any qualitative examples in the main paper. Some have been included in the appendix. Further, it would be interesting to see where the agents fail to better understand the strengths and weaknesses of the proposed approach. For instance, categorizing the errors due to imperfect vision, incorrect language usage by the speaker, or language understanding by the listener would be useful.

**Comments**  
(C1) How important is the role of supervised learning iterations in order to retain the naturalness of the language? The current work is missing this ablation, where the amount of supervised learning is varied.

(C2) To check for robustness against any implicit bias in image datasets, how do agents trained on MSCOCO perform on a different image dataset at test time, e.g, PASCAL?

**Typos**  
* Algorithm 1: After the meta learning stage, is there a typo in the initialization of \phi’_0?
* L15: your -> our

limitations_and_societal_impact: The authors briefly discuss the limitations of the proposed approach but do not discuss broader impact in the main paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors aim at training agents that can coordinate with seen, unseen/human partners in a multi-agent communication environment. Instead of using static populations, the authors propose a dynamic population-based meta-learning approach that dynamically builds the population. Such an approach enables the trained agents to generalize to seen and unseen/humans partners. On the other hand, the authors try to use a limited amount of human data to fine-tune the agents to coordinate with humans using natural language. They also compare the proposed method and baselines on two referential games.
main_review: The paper is well written and easy to follow. The explanation of the problem setup and model is clear. 

Different from previous works that use static populations, this paper proposed a dynamic population-based meta-learning approach. The dynamic population for multi-agent communication sounds interesting. 

The experiments are good. The authors compare the proposed approach and baselines on two referential games. They also have some ablation experiments to evaluate each component of our proposed algorithm.

The authors show the results of agents interacting with humans. It is good to show the generalization ability of the proposed method. However, in this experiment, it is hard to evaluate the generalization ability of the proposed method when communicating with humans. Please see the detailed limitations below.

There is no explanation of why does Behavior Grounding enable agents to communicate with humans using a limited amount of human data? See detailed limitations below.
limitations_and_societal_impact: It is good that the authors could provide more details about the human experiment, such as some example conversations between the agent and human. How much does human message differ from the trained agent? What if the human is not optimal and speaks some random messages? How do you filter bad examples? Without this information, it is hard to evaluate the generalization ability of the proposed method in the human experiment.

Why does Behavior Grounding enable agents to communicate with humans using a limited amount of human data? If the human data is limited, I don't think the proposed method can generalize well on human experiments. How much does the collected human data different from the messages that appeared in the human experiments. 

It is not clear how do the authors update the speaker population. Does the population get larger and larger? 

The introduction of baselines is not clear as well. Which baselines corresponding to using the static population? 

Figure 2 is a little bit confusing in the Dynamic Population part.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work deals with the problem of multi-agent communication where the agents can be artificial or human. The latter means that the conversation has to be in natural language. New agents can be added or removed from conversation so the environment is not static. The authors propose a dynamic population-based meta-learning approach. They analyse the language skills of agents using BLEU, the quality of interaction in a human trial, as well as cross-play with unseen partners achieving overall good results.
main_review: It would be nice to give some more context to this work. Where do authors see this technology to be used? The test-beds are clearly toy examples and not very relevant. Do they see the use of these agents as bots that engage in eg Twitter communication? Or is this meant to improve two-party dialogue?

When we talk about communication and dialogue what we have in mind is multi-turn communication. The test-beds considered here focus only require singe turns so it’s not quite clear to me how much of the communication management behaviour is learned or can be learned. Note the vocabulary of the test-beds are very small, only 100 words. Can this approach scale?

The issue of an unseen and seen interlocutor is incredibly important and I would be keen to see how this method generalises to (or improves) RL approaches to task-oriented dialogue. Ablation studies show that removing meta-agents does not cause a drastic decrease in performance and I wonder if the reason is the simplicity of the task or limited use of this method. In any case it would be important to investigate this.

A lot of details are in the Appendix that would be better suited in the main text, in particular the qualitative samples.

----

The authors addressed my concerns to some extent so I'm willing to increase my score to 6.
limitations_and_societal_impact: This is difficult to judge as it is not elaborated what this technology can be used for. The test cases presented do not pose any negative social impact as far as I can see.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper is motivated by group coordination through natural language and creates a goal of developing agents that can communicate effectively with other agents and humans. They build off of work on emergent communication and their method is most related to L2C [Lowe19], which uses a meta-learning approach to learn communication protocols.

Their training method is as follows:
```
Train with some human data
while not_converged:
    for n_meta steps:
        Learn a meta-listener on a population of speakers
    for n_selfplay steps:
        Learn new speakers through self-play with the meta-listener
    for n_supervised:
        Fine-tune on human data
train 
```

And their evaluations are:
1. Image based referential games
2. A text based referential game that they design where agents are trained to communicate in German about English sentences

Their experiments show that:
1. They outperform baselines on referential accuracy in both modalities
2. They have a high sentence quality (BLEU, caption score) wrt baselines
3. The meta-speaker in the text task works reasonably effectively as an English to German translation system (BLEU)
4. Their speakers and listeners perform better on human evaluation in the image task

Learning to learn to communicate. Lowe et. al., 2019.
main_review: Originality: The paper directly extends past work. The novelty in their training procedure comes from repurposing meta-speakers and -listeners for population generation. Their experiment design is creative: they introduce a new task that serves allows them to evaluate speakers in a novel way (as translation engines).

Quality: The thoroughness and rigor of the experiments are high. They include p-values, extensive ablations, and reasonable comparison to past work.

Clarity: The literature was well presented. By integrating related work into introduction/background on past methods (sections) their method was easier to understand. Some training details were less straightforward. E.g., in the prose it could state more explicitly how the meta-speaker and meta-listener are used together and to my knowledge the initialization is only mentioned in the algorithm box.

Significance: This paper brings studies on emergent communication in line with tasks of interest (communication with humans because of the evaluation, nlp tasks because of the translation system evaluation). One open ended question that the authors may be interested in addressing (perhaps verbally): what is the advantage of conceptualizing these training methods as emergent communication among a population of agents? E.g., a different conclusion from this paper is that using meta-learning as a distillation method across re-initializations is an effective method for solving various language tasks.

Minor comments to authors:
- L15 your -> our?
- L26 like these agents -> these agents
- L52 the use -> the use of
- It would be helpful to label Figure 1 with the math in the prose
- L162: Out -> Our
limitations_and_societal_impact: I could not find where the limitations were addressed. It would be helpful to see a discussion that describes in more detail the benefits of their framing (emergent communication instead of something drier (see main review).
review_title: null

Meta-review: This paper introduces an interesting approach to combating semantic drift in communication-based training for language agents. The biggest concern on reviewers was the need to more clearly describe the details of methods and results for the human experiments. It will be very important to add methods, ethics, and results details in the revision. Some reviewers felt that the domains considered were too artificial, but I believe (along with other reviewers) that these are an appropriate first step given previous work in this area.
Decision: Accept (Poster)