Dynamic population-based meta-learning for multi-agent communication with natural language
Authors: Abhinav Gupta
Marc Lanctot
Angeliki Lazaridou
Venue: NeurIPS 2021 Poster (2021)
Keywords: emergent communication
meta-learning
population-based methods
multi-agent reinforcement learning
few-shot generalization
human-AI coordination

Abstract: In this work, our goal is to train agents that can coordinate with seen, unseen as well as human partners in a multi-agent communication environment involving natural language. Previous work using a single set of agents has shown great progress in generalizing to known partners, however it struggles when coordinating with unfamiliar agents. To mitigate that, recent work explored the use of population-based approaches, where multiple agents interact with each other with the goal of learning more generic protocols. These methods, while able to result in good coordination between unseen partners, still only achieve so in cases of simple languages, thus failing to adapt to human partners using natural language. We attribute this to the use of static populations and instead propose a dynamic population-based meta-learning approach that builds such a population in an iterative manner. We perform a holistic evaluation of our method on two different referential games, and show that our agents outperform all prior work when communicating with seen partners and humans. Furthermore, we analyze the natural language generation skills of our agents, where we find that our agents also outperform strong baselines. Finally, we test the robustness of our agents when communicating with out-of-population agents and carefully test the importance of each component of our method through ablation studies.  

Dynamic population-based meta-learning for multi-agent communication with natural language
Abhinav Gupta' MILA abhinavg@nyu.edd edu
Marc Lanctot DeepMind lanctot@despmind.c com
Angeliki Lazaridou DeepMind angeliki@deepnin com
Abstract
In this work, our goal is to train agents that can coordinate with seen, unseen well as human partners multi-agent communication environment involving natural language. Previous work using single set of agents has shown great progress in generalizing to known partners, however struggles when coordinating with unfa- miliar agents. To mitigate that, recent work explored the use of population- based approaches, where multiple agents interact with each other with the goal learning more generic protocols. These methods, while able to result in good coordination between unseen partners, still only achieve so in cases of simple languages, thus failing adapt to human partners using natural language. We attribute this to the use of static populations and instead propose a dynamic population- based meta- learning approach that builds such population in iterative manner. We perform a holistic evaluation our method on two different referential games, and show that agents outperform all prior work when communicating with seen partners and humans. Furthermore, we analyze the natural language generation skills of our agents, where we find that our agents also outperform strong baselines. Finally, we test the robustness of our agents when communicating with out-of outoof-oopulation agents and carefully test the importance of each component our method through ablation studies.
Introduction
Humans excel large-group coordination, with natural language playing key role in their problem solving ability 49 g Inspired by these findings, in this work goal is to endow artificial agents with communication skills in natural language, which can in turn allow them to coordinate effectively in three key situations when playing referential games (i) with agents they have been paired with during their training (ii) with new agents they have not been paired with before and (iii) with humans.
Our starting point is recent work in the emergent communication literature, which proposes to learn protocols "from scratch". allowing like these agents that have been trained together (and for large number of iterations) to form conventions 47 24 12 19). While such communication approaches result in agents achieving higher rewards at test
when paired with their training partners 21 14 23 there is growing evidence that these agents do not acquire communication skills with but rather learn to form communication protocols that are based solely idiosyncratic conventions.
general properties BI II & &
Interestingly, these type of communication pathologies are also observed in natural languages, where they are linked to the size of linguistic community 42
34
50
35 i.c. the smaller the
Work partially done during an internship DeepMind.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

community the more complex the languages get. This has recently inspired the use of population- based approaches multi-agent communication in which agents within a population interact with
One such approach is L,2C 32 which uses meta-learning approaches like MAMI TC combined with a static population of agents. Indeed, L.2C results in communication protocols with some primitive forms of compositionality. allowing agents to successfully communicate with unseen partners. Nevertheless, even in this case, the induced protocols fail to adapt to natural language, leaving open questions with regard to coordination with humans. We hypothesize that potential bottleneck of this and related approaches lies in the use of static populations. Intuitively, the initial diversity induced by different random initializations of the population gradually vanishes when given enough capacity and training iterations, thus allowing all agents in the population to co-adapt to each other and converge to the same ad-hoo conventions. Simply put, it is challenging to maintain diversity in static populations, which is what drives the learning of generalizable protocols.
one another 48 5
13
( With this in mind, we propose to address this issue by dynamically building the population to gradually induce constructive diversity within the population, so as to enable population- based algorithms (like meta-learning) to train agents that can then generalize to seen and unseen partners as well as humans. Since communication with humans is one of our goals, grounding to natural language is of paramount importance. Consequently, we want the agents coordinate with humans using limited amount of human data while not drifting away from human behavior. In order to maximize efficiency and minimize the use task- speciffc language data, we combine two learning signals 25 33 each bootstrapping the other 2 First, maximizing task rewards allows the agents to learn to communicate with one another in purely self-play manner, but using emergent protocols not grounding in natural language. Hence, combining this signal with supervision from a limited dataset of task-specific language data, allows grounding of the emergent protocols in natural language. All in all, our proposed approaches consists of three different phases (i.e., interactive reward- based learning, supervised learning, and meta-learning), each imposing an inductive bias.
We present extensive experiments in two referential game environments (an image-based and a text- based defined in $ using natural language communication and compare our model against previous approaches, strong baselines and ablations of our method. Moreover, we follow a holistic approach to evaluating multi-agent communication, reporting results on: (a) task performance measured by (b) the agents' natural language generation skills by computing BLEU (d) cross-play evaluation with unseen partners in 4.5 and (e) robustness against implict bias in the dataset ($4.6).
referential accuracy in ($4.T) I
score with language data ($4.7
(c) human evaluation of both agents in ( 4.33
2 Learning in multi-agent games with natural language: The case of referential games
Referential Games Referential games are a type of Lewis Signaling games 28 that have been used in hu- man as well as artificial language experiments [5 23 9 17 29 13 This fully cooperative game consists of two players, a speaker S and a listener L who interact with each other solve common task. In our setup, both agents are parameterized using deep neural networks, where the speaker's 's and listener's parameters are denoted by 0 and é respectively. The speaker gets as input a target object f, encodes it in its own embedding space and sends a discrete message m S(t) to the listener. The listener receives two inputs, the message m and set of distractor objects D (D) K) and the target object t. The listener embeds both of these into shared vector representation to compute similarity score between the message and the objects and outputs a prediction l' about the target object. Both agents are given reward if the listener is able to correctly predict the target object. We denote the maximum length of the message m with 1 and the vocabulary set with V. our work, we use two different types of objects: images and sentences as described in S
Araa aboyds
Figure 1: Referential Game with images.
The most common approach to train agents to solve the main communicative task in the multi-agent communication framework is by using interactive (reward-driven) learning
Interactive Learning
p 3W
15
29).
The reward function for both agents is the same and given by: 1 if

---

Tral popuidiioss
Test
Learnny the ceete-boveer
ADB
DOD
P
OO
D
O
OO
O
O
O
0
0
0.0
0
(a) Stain:Static Population
(b) Train:Dynamis Population
(c) Test:Humans
Meta-learning listener using two types of population. In Fig 2a fixed set of speakers and listeners are trained using self-play. Then meta-listener trained by interacting with the fixed population of speakers. In Fig 26 the meta- meta-listener is trained using the current population of speakers time in top-phase. Speakers are added to this population iteratively by training the latest speaker (obtained at t - 1) with the updated meta-listener (denoted by dashed arrow) in the bottom-phase. In the final trained meta-listener plays with human speakers during test time.
Figure 2:
Fig 2c
and - 0.1 otherwise. We optimize the agents" parameters using reinforcement learning, and specifically, using policy gradients (REINFORCE (53)) similar 25. The listener additionally optimized using supervised learning ccross-entropy loss since we know the ground truth label (which is the target object). The corresponding interactive loss functions for the speaker (J and the listener (Jh"") are given by:
TS -
r
Σ log p(n, mm,,, 1;0) AnoHs(()
(1)
JE, D: = -r log pt [m, D: + X, logp(t' = t/m, D: + AmHo((o)
(2)
where Hs and He denote entropy regularization for the speaker and listener policies respectively. Ahe and An are non-negative regularization coefficients and X 2 O a scalar quantity.
Behavior Grounding If we wish to enable agents to communicate with humans, it is essential to ground their behavior into the corresponding human distribution. Re- achieves this by performing super- vised learning with the use of natural language dataset. Consequently, the speaker needs to send messages that numan-interrpretbble while the listener should be able to understand natural language. To achieve this, we let humans provide descriptions of the objects and collect such (object, description) pairs to train our agents using supervised learning. Let us denote this dataset by D with m being the description for the target object present in D. Then the corresponding cross-entropy losses for the speaker (JSup) and the listener (Teup) are:
legegrragu
cent work 25 33 34
rure bouta WaTEr
inouth baye
Figure 3: Limited human dataset DD
IV
JSEP((M (m=,;tt))) =
Σ
Σ
mi log p(mm,,,lmms
(3)
j=1c=1
K+1
T J ,t, D) = P(,= o)
Σ
(4)
1=1
3

---

3 Dynamic Population-based Meta-Learning
The previous two approaches help in learning protocols that are either closer to human prior or are ad-hoc conventions that get emerged during training. A common pitfall of training a single set of speaker/listneer is that both agents tend to overfit to their partners resulting in loss of generalization across multiple partners including humans I1 S [0] Since our objective is for the agents to learn robust protocols that match the human distribution and are high rewarding, we adopt a population- based approach to train agents that generalize across diverse agents in a population.
Previous approaches have used population as means regularize the behavior of the agents using a community of fixed number of agents [+ or by meta-leaming an agent on this fixed population of agents. In given population, each agent learns best-response to the other agent with the underlying hypothesis being the meta-agent learning the overall best-response to all agents in the population.
The main aim of meta-learning techniques [S1 45 T0] is to learn meta-agent that is able to achieve good generalization across multiple tasks. In the reinforcement learning set- ting, these tasks can be different sets of goals in the same or different environments. In multi- agent interactive setting, we can reformulate these goals generalizing different agents in a population where the agent's 's partner is its environment. Consequently, meta-learning can only aim to learn generic protocols if there is rich signal emerging from a diverse set of pop- ulations. A previous approach (32 (L2C) in- vestigating learning diverse agents using differ- ent seeds for random initialization of the agent parameters. However, this simplistic way of in- ducing diversity is limited and only works with more toy (artificial) languages, thus leaving big performance gap when agents communicate
Algorithm 1: Algorithm
Input
Dataset D, collection of objects O randomly initialized speaker parameters 0.. listener parameters Φu. meta-speaker parameters 9o, and meta-listener
parameters yoo empty buffers B. and B2
01 4 SuparvisedLearning// {m, t ~ D; 00)
b Eq B
Φ1 SuperviseddLeammingg/
(m. P)
D: do)
Eq
i <-> 1
repeat
B's t- U
Bi 1---"- U
0 + 1, 01-1: + pi-1
for E (1.2...., Rmeta do
8f KetaLearning(O 1 -1 (Vo
B¹2)
Eq
in natural language (6 We hypothesize that
4/ f- MetaLearring(
O (A8 E
the desired useful diversity (for meta-learning) is absent in these static populations, hence re- sulting in lower performance of the meta-agent,
B)²))
Eq
end
0/ f- 0g 0 +- 8/: 02 t 83: f 42
for 1 E {1, 2, Ment do
We aim to tackle this issue by building dy- namic population in an iterative manner, similar Policy-Space Response Oracles (PSRO) 22 which uses empirical game-theoretic analysis in a non-cooperative game to learn a meta-strategy for policy selection. An agent learns best- response to an Oracle which is then added to buffer that stores this growing population of agents. This buffer is used to obtain an improved Oracle which is used in subsequent iteration. We split the training into two phases as depicted in Fig 2b We will describe the method for training a meta-listene (an equivalent method is used for the meta-speaker):
01 + InteraattivLeerminng
(t. D}
O: 0.0011.900 Eq
of + InIneraccivveeerninn (t, D}
0 1 0,)
Eq
end
0 00' Φ1 for m E {1, 2, ....., neap do
0%
0% + SuperrieddLearnnngg... ~
D: 8
Eq
An SupervisedLearminng( ((m, t, D3 ~
D; (- P Eq
end
O.11 +- 0": 01+1 6%
i < i+1 until performance of i and up converge
Learning a meta-listener We train a meta- listener that plays with all speakers present in the current population buffer. At each training iteration, the meta-listener aims to distill the behavior of all past listeners that interacted with the current speaker population.
Building the speaker population After obtaining the newly trained mita-listener, we expand the speaker population by playing the meta-listener with the speaker that was added to the buffer in

---

the previous iteration. This encourages new behavior to emerge result of interacting with the distilled meta-listener. Our hypothesis is that by adopting this iterative mechanism of distillation and expansion, we are able to obtain diverse population where the behavior of each consecutive agent in the buffer changes gradually.
In this work, we use techniques that use gradient descent for optimizing the meta-agent. In particular, we use the popular Model Agnostic Meta Learning (MAML) 10 train the meta-agents. MAML based approaches provide a good initialization over the parameters to be able to quickly adapt to new tasks. An important point to note here is that in our setup, the task distribution changing over time and is determined as a result of multi-agent learning. We also show some results using other algorithms that are derived from MAML in the Appendix. We denote the parameters of meta-speaker S by & and meta- listener L by ş. We assume a buffer of speakers denoted by Bg and listeners by Be. We split the collection of objects O into two sets O2 and O₂ order to compute the inner and outer loop losses in MAML, respectively. Now, we can define the objective functions for the meta-speaker (Igm and the meta-listener (Tgal as follows:
Jasta (O; 0) = Σ
Jint
11:0 VoJ (P::0)
(5)
LEBe
T Jeats (0:4)=
Σ
Tim
m², Do: a a a. (m'; D':p))
(6)
SEBs
where e' E O,, 1o E O,, m = S(1)), m = S(t'), and D' and D are sets of distractor objects sampled from O and O. respectively.
Finally, the trained meta-agents are fine-tuned using the dataset D before the testing phase. The fine-tuning losses are the same as the supervised losses Eq 3 described in the previous section. Since the number of training iterations is unknown and could be potentially much larger than the size of the buffer the memory can hold, we use reservoir sampling to keep a uniform sample of past agents in the buffer. The detailed algorithm can be found in Algorithm
3.1 Prior approaches
Recent work has investigated combining the two objective functions of self-play and imitation learning, with the goal of training agents that use natural language and perform well on given emergent communication task. This transforms the problem into training task conditional language model multi-agent setup. S2P [3] proposes methods that devise a curriculum between the two training phases updating the speaker and the listener in an iterative manner. SII KE follows a ctudent-teacher paradigm that is trained sequentially, where the teacher agents, initialized from the student agents, trained using interactive learning. Then the student agents are trained to imitate the teacher agents by sampling data at every training iteration. Another work by [ investigates different types of language (semantic and structural) and pragmatic drifts. They propose reranking module that first samples multiple messages from the speaker and then ranks them according to the task performance Crucially, the reranking module presents orthogonal axis of progress and thus can be combined with other approaches presented here.
propose learning method that uses community of fixed number of agents where speakers and listeners are randomly paired with each other at every training iteration. L2C proposes meta-training method fixed population of agents. They first train different populations of agents via self-play with each population initialized with a different random seed. Then meta-learner interacts with these agents and learns to adapt to each population simultaneously. Another similar method in [5] aims at learning a community of agents, where groups of speakers and listeners are used to sample pair uniformly at random who then play the game and jointly optimize for better task performance. During learning, few agents reinitialized periodically/at random from group of agents. The idea to promote cultural transmission to induce compositionality over multiple generations of language transfer. For our experiments, reinitialize agents to the pretrained agents that trained using the human dataset. We denote this method as TRANS. in the following sections.
With respect to training agents using population- methods, 48
2 Theii approach different setup that involves giving speaker access the distractor objects.
5

---

which a common environment used to analyze emergent protocols involving multimodal inputs. We use the MSCOCO dataset BD to obtain real images and the corresponding ground truth English captions annotated by humans. Since this dataset has multiple gold captions for each image, randomly select one from the available set and keep | = 7000. Following 226 33 both speaker and listener are parameterized with recurrent neural networks (GRU 4 of size 512 with embedding layer of size 256. We embed images using a Resnet-50 model [16 (pretrained on ImageNet [7]. We set the vocabulary size to 100 and the maximum length of the sentences at 15. We use the same speaker and listener Experiments and Results
In this work, we conduct experiments on two referential games in two different modalities, i.e., having agents communicating in English about images, process akin to image captioning. and having agents communicating in German about English sentences, a process akin to machine translation.
Images We use the image-hased referential game 27 23
buffer size of 200 for reservoir sampling. Other implementation details are given in the Appendix.
Text To further the robustness of our approach to different types of structural priors found on the input data [А]. we also play a referential game using text as the input modality [26]. We use the publicly available IWSLTT 14 English-Gennnn target language. Instead of randomly selecting the distractors from the set of English sentences, we create harder distractors by picking English sentences that are more similar to the source English sentence. For this, use BeRT model [ (pretrained on SNLI 2 and MultiNLI (52)) to embed all English sentences in the dataset and filter them using a threshold criteria of cosine similarity to systematically choose the distractors. The game complexity increases with increase in cosine similarity, and we fix this to 0.85. We use a vocabulary size of 100 and maximum sentence length of 20. All other details the same the game with images.
dataset with English as the source and German as the
Throughout this section, we report results using 3 random seeds and K 9. D 5000 in the image game and K = 14, (D) 5000 in the text game. In the Appendix, we show that our findings are robust and include results with [D] 2000 and 9 on the image and text game respectively.
4.1 Referential Accuracy
We report referential accuracy as the final task performance the meta-agents, i.e., how ac- curately the (meta-) listener is able to predict the target using the (meta-) speaker's messages. Fig plots the referential accuracy the test set of 1000 images/sentences for the image/text game. In both plots, the pretrained baseline is obtained by separately pretraining the speaker and the listener the language data without any interactive or meta-learning signal, i.e.,
Pretrained EmeCom
SIL L2C
Gen Trans.
Ours
Text scc %
K0.1
62.6
G2.5
only using Eq 3 and Eq 4 and then pairing them together. The emergent communication (EMECOM) baseline is computed by training agents purely via interactive learning and with- out any supervision from the dataset, i.c., only using Eq [ and Eq 2 In both image (left plot, 10% random baseline) and text (right plot, 6.7% random baseline), our method outperforms al- ternative approaches. Specifically, TRANS. and our method outperform S2P and SIL, both of which can be thought of as single-agent population, indicating the importance of diversity among the agent population. Moreover, our method surpassing GEN.TRANS ARS.S. and L.2C indicates the importance of using the proposed meta-learning approach in conjunction with an adaptive population instead of using a static set of agents. See 4.4 we conduct more ablations to better assess the importance of the different components for approach.
57.2
Image
Referential accuracy of the (meta-) agents on the test set (Left): Image game and (Right) Text game (p 0.05, t-test).
Figure 4:

---

4.2 Evaluating Natural Language Skills of (Meta-) speakers
Previous work on multi-agent communication with language has showed that even when agents receive supervision using natural language data, language drift phenomena result in agents no- ticeably diverging from natural language while still achieving high referential accuracy 2 26
As such, we proceed with directly evaluating the nat-
ural language capabiliiiies of the meta-speakers We
Caption Score
BLEU
start with the image game, in which we view the ut-
terances generated by the meta-speaker as captions
PRETRAINED
5.2 0.02
24.2 + 0.1
and hence evaluate their alignment of the generated captions with human generated ones. We evaluate all methods using 1000 generated captions from the test set. We report BLEU score B9 between the gener- ated captions and the ground-trun truth English caption. We also report alternative caption score proposed by [d. which uses a pre-trained model on MSCOCO to compute similarity score between the generated caption and the context (image and ground-truth cap- tion). The results are shown in Table where we see that our method outperforms alternatives indicating that the use of meta-agents is effective in getting higher referential accuracy but not on the expense of worse language skills.
S2P
5.5
26.1 +: 0.3
SIL
5.5 0.04
25.6 0.24
L2C
5.0
25.4
GEN. TRANS.
6.1 0.05
28.1 0.32
OURS
6.6 1 0.04
29.3 0.25
Table I: Evaluating the (meta-) speaker in the image game using the metric proposed in 6 (p < 0.05, t-test).
In Table 2 we perform a similar analysis for the (meta-) speaker on the text game. Here, the ut- terances of the meta-speaker could be thought of as translating the English input sentences to Ger- man. Hence, we evaluate the (meta-) speaker as an Anglish-Germar machine translation system computing BLEU score with the German ref- erence sentences. Apart from comparing our model against different alternatives, we also create stronger baselines by using the English- German pretrained model in the text game with different decoding strategies (all others methods including ours will use greedy decoding). We find that our method, despite using greedy de- coding, is able to outperform all variables of the PRETRAINED model, indicating that our tech- nique has somewha what different effect than the one introduced by simply changing the decoding strategy.
BLEU
S2P
21.7 + 0.22
SIL
21.6 I 0.21
T.2C
21.1 + 0.29
GEN. TRANS.
23.9 + 0.21
PRETRAINED
Greedy Decoding
20.8 1 0.06
Beaam Search (n 2)
21.1 0.06
+
Beaa Search (n = 4)
22.4 = 0.05
Top-k Sampling (k 40)
24.6 1 0.05
OURS (using greedy decoding)
25.2 - 0.16
Table 2: BLEU score on German sentences for (meta-) speaker in the text game across various baselines and decoding strategies. n denotes num- ber of beams in beam-search (p < 0.05, t-test).
All all, we find that even though our objective task was never to just simply maximize captioning (or machine translation) performance, our dynamic population-bal based meta-learning resulted in meta- speakers with better language skills than the pretrained models, which use the very same number of (limited) language data. While our results are far from being state-of-the-art on captioning or translation, we nevertheless see that our technique can be thought of as a form of semi-supervised learning through self- self-play, and can perhaps be used in combination with specialized models of the respective tasks. We also include some qualitative samples generated by different speaker models alongside the ground truth captions in Fig & In the Appendix, we show some additional corpus-level statistics that further corroborates this claim.
4.3 Agents interacting with Humans
Although BLEU score is able to capture some form of syntactical and semantic drift, it still fails to counter the phenomenon of pragmatic drift, as introduced in [25] For this reason, we evaluate the performances of our agents, and all other alternative methods, by having agents interact with humans and report referential accuracy. As such, we play games where the (meta-) speaker interacts with a human listener and the (meta-) listener with human speaker. Thus, in the first case we evaluate the natural language generation abilities of the (meta-) speaker, whereas in the second
7

---

case we evaluate the natural language understanding skills of the (meta-) listener. The (meta speaker evaluated using 1000 games with humans and the (meta-) listener using 400 games
Moreover, we compute an upper-bound by computing the referential accuracy when pairing two humans to play the game. In Fig 5a we show the results for meta-speaker in the image game where our method out- performs other approaches by significant margin.
Hume Speaber
90
71.0
$2.1
72.1
70.9
74.3
69.5
154
70-
50-
60
Fig 5b compare the performance the meta- listener in the image game with other baselines and found similar observations to the meta-speaker. We find that our method is able better understand human descriptions by learning diverse caption representations. The results for the game with text follow similar pattern and can be found in Appendix along with other details of the experimental setup. Overall we infer that our method suffers the least amount of pragmatic drift as compared to other baselines, as measured by the performance gap with the human-human gameplay.
(a) Agent speaker Human listener (b) Agent listener Human speaker
Figure 5: Human evaluation on the image game. The black line in both plots represent the performance of the (Human-speaker, Human- listener) pair.
4.4 Ablation studies
We further analyze the importance of each component of our proposed algorithm by introducing the following ablations.
no meta-agents In this ablation, we test the importance of building populations agents by learn- ing speakers (or listeners) through interaction with the meta-agent listener (or speaker respectively), as opposed to just interacting with a randomly chosen listener (or speaker) from the buffer. Our hypothesis is that this promotes diversity within the population, resulting in learning communication protocols. Before commenting on the performance of this ablation, we note that this type of training is in fact more unstable, which attribute to making the training of the two meta-agents interdependent of each other. We leave further analysis of this for future work.
general
no adaptive-meta In this ablation, we test the impor- tance of training the meta- in an iterated manner, opposed to training it from scratch by resetting the weights of the meta-agent before interacting with the population. We build the population of agents that interact with the meta-agent (whose weights have been reset) in two dif- ferent ways: (i) by following the approach illustrated Alg 1 and (ii) by following the approach introduced in No meta-agents above.
023 mata-grrtt
caaatteeeea expaapee grandig
Acc
no separate grounding In this ablation, we test the im- portance of avoiding catastrophic forgetting of language skills by conducting interactive and supervised learn- till in Algorithm I in separate phases, in line with previous work [33]. Recently, other works 25 [4] have proposed using KL divergence be- tween the agents' policies with a pre-trained policy on
Figure Referential accuracy for differ- ent ablations in the image game.
ing (i.c., Equations [
3 We note that the participants were neither aware of identity of agent they were playing against nor they played consecutive games with the same agent. This allows for more fair comparison of the language skills of agents since participants did not have the chance to adapt to ad hoc conceptions potentially used by agents (e.g.. consistently referring cats onions).

---

60
52
$
53
$5
%
46
$7
52
44
60-
58
#Training Reration
Figure 7b: Cross-play evaluation using referential accuracy for our methad (Left) that dynamically builds the population and 1.2C (Right) that uses static population, across sets of meta-speakers and lettalistennrs. Left: Our meta-agents have much less variance across various partners. Right: L2C meta-agents suffer from high variance and overfitting to their own partners (diagonal). The scale of the colorbar is calibrated based the highest variance interval in both plots to allow fair comparison between the two methods.
Figure 7a: Average referential accuracy on the test set when trained meta-listener individually plays with each speaker present in the buffer at the corresponding training iteration. show the standard deviation across all speakers in the buffer. The gray bars show the interval between the minimum and maximum performing speaker.
blue bars
language data, and as such penanalize policies that diverge from language. Concretely, the corre- sponding speaker objective is Ain + K [OOprettnineal)) (0).
In Fig we compare the performance of full approach against all ablations by reporting the referential accuracy on the test set in the image game. The comparatively lower performance of no meta-agents against our method indicates that the agents that interact with the corresponding meta-agent learn more robust and diverse strategies as compared to the ones that interact with any agent in the buffer, Both variants of the no adaptive-meta agents underperform against our approach as well as against no-meta agents. This suggests that the adaptive way of training the meta-agent is crucial in getting higher performance as compared to training it from scratch. We hypothesize that our iterated meta-agent has a slight advantage of being able to learn from agents that get created during the training in turn providing a better initialization meta-agent captures useful information from past iterations of other agents and provides richer learning signal to train the next agent be added. No separate grounding agents perform close alternating updates used in our method indicating that one can use multiple ways to integrate the two loss functions in combination with our meta-learning approach
adapt the new population. The
4.5 Induced diversity and out-of iil-of-populationn evaluation
In Fig 7a we show the average performance of the meta-listenee obtained after training, playing with different speakers at different stages of their training in the image game. We show that as the training of the speakers progresses, the standard deviation (denoted by blue bars) of the referential accuracy across all the speakers present in the buffer up that training iteration increases, together with the difference between the best and worst performing agents (denoted by gray bars). This indicates that as the population grows the diversity among the agents also improves, thus resulting richer training signal for the meta-agents. In the Appendix, we include a similar plot for the meta-speaker reporting average BLEU score.
We also perform cross-play evaluation to test the generalization during out- oft-of-population commu- nication. For doing so, we obtain meta-agents meta-speakers and 5 meta-listeners) by using Algo D with 5 different seeds and report referential accuracy on the test set when pairing all possible combinations of meta-speakers and meta- leta--isteners. On the left plot of Fig 7b we observe that, as expected, the in in-population communication found on the diagonal is the best, and performance barely degrades when pairing agents that were not initialized with the same seed, indicating that our agents have learn more generalizable conventions. On the other hand, when conducting similar evaluation with L2C (right plot), we observe that the protocols learnt in this ease overfit to their own partners
'Note that our choice of using alternating updates for interactive learning and supervised learning is arbitrary and based on [3 We believe that any multi-objective learning method could work here.

---

Human: boy dress sitting black couch while holding pizza.
S2P: boy white eating pizza while sitting.
Sil: A boy white dress sitting bed with pizza.
L2C: boy white dress eating and pizza.
Gen.Trans A boy white dress sitting bed and bed and bed...
Ours: boy white dress holding pizza sitting black couch.
Human: People on roller-skates riding on the road with white car in the background.
S2P: A group of people with coller-skates on the road.
SIL: People running with roller-skates on the road.
L2C: A group of people walking on the road. Gen.Trans:: A group people with roller skates road car road car... Ours: A group of people with roller-skates on the road and a white car.
Figure 8: Qualitative samples generated by the (meta-) speaker in the game with images.
(diagonal) resulting in high variance across other players, and thus indicating the adoption of more ad-hoc protocols. Consequently, we posit that our method be used for few-shot generalization to unseen partners multi-agent interactive environments.
4.6 Robustness against implicit bias in the dataset
As agents trained using single dataset may overfit to the objects present in that particular dataset, in this section we design an experiment to test the agents' robustness against implicit dataset biases. We conduct cross-task evaluation experiment compute the referential accuracy using images from the Flickr8k dataset [[77] while the agents were trained using the images present in the MSCOCO dataset. Similarly, also perform within-tass evaluation where we train and then evaluate our agents using the same Flickr8k dataset. We use 5000 images training and 1000 images as validation/test set.
the referential game with K distractors where
Results are presented Table While we observe drop in the performance when com- pared to the within-task performance (70.2 v/s 78.9), across-task task model still outperforms the within-task PRETRAINED baseline (70.2 v/s 65.3). This observation correlates with our sults using human players and out-of oottoo--oopulation evaluation shown in Sec 4.3 and Sec 4.5
PRETRAINED
OURS
Cross-task
59.7 + 0.9
70.2 = 1.8
Within-task
65.3 0.3
78.9 = 1.4
Table 3: Referential accuracy on the test set show- ing robustness against implicit bias in the dataset.
respec-
tively.
Conclusion
We presented dynamic population-based method to train agents in a cooperative multi-agent reinforcement learning setup. We showed that our method induces useful diversity into a population of agents which helps in learning robust meta-agent. Empirically, we show that our agents outperform prior work on the task performance, auxiliary tasks and even human- based evaluation. In the future, we plan to extend this approach in cases where agents are performing temporally extended physical actions an environment with perceptual observations. One limitation of our approach pertains to maintaining a buffer of past agents that are used to train the meta-agent. Moreover, we would also like to analyze the effect of using task-independent language data (as opposed to task-dependent that we now) within multi-agent reinforcement learning task.
10

---

Funding Transparency Statement
AG supported by IBM scholarship. We thank Compute Canada for providing the compute resources to run the experiments.
References
[1] Diane Bouchacourt and Marco Baroni. Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances. arXiv:1905.1187 [cs], May 2019. arXiv: 1905.11871.
[2] Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642 Lisbon, Portugal, September 2015, Association for Computational Linguistics.
[3] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4427 4442. Association for Computational Linguistics, 2020.
[4] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods Natural Language Processing (EMNLP), pages 1734,,,,, Doha, Qatar, October Association for Computational Linguistics.
2014.
[5] Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of Compositional I Language with Deep Generational Transmission. arXiv:/904.099 [cs, stat], April 2019. arXiv: 1904.09067.
[6] Yin Cui, Guandao Yang. Andreas Veit, Xun Huang, and Serge Belongie. Learning evaluate image captioning. In CVPR, 2018.
[7] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[8] Tamás Dávid -arieBBrreettt and Robin Dunbar. Language as a coordination tool evolves slowly. Royal Society Open Science. 3:160259, 12 2016.
[9] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communi- cation in a multi-modal, multi-step referential game. In International Conference on Learning Representations, 2018.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv:/703.034000 [cs]. March 2017. arXiv: 1703,03400.
[11] Nicole Fitzgerald. To populate is to regulate, 2019.
[12] Jakob Foerster, Ioannis Alexandros Assacl, Nando de Freitas, and Shimon Whiteson. Learning Communicate with Deep Multi-Agent Reinforcement Learning. D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2137-2145 Curran Associates, Inc 2016.
[13] Abhinav Gupta, Ryan Lowe, Jakob Foerster, Douwe Kiela, and Joelle Pineau. Seeded self-play for language learning. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real- world kNowledge (LANTERN), pages 62-66, Hong Kong, China, November 2019. Association for Computational Linguistics.

---

[14] Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural I. Language Processing (EMNLP-IJCNIP), pages 3691-3701, Hong Kong, China, November 2019. Association for Computational Linguistics.
[15] Serhii Havrylov and Ivan Titov. Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems 30, pages 2149 -459. Curran Associates, Inc., 2017.
[16] K. He, Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[17] Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics (extended abstract). Journal of Artificial Intelligence Research, 47:853 8993389992088 2013.
[18] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob N. Foerster. "other-play" for zero-shot coordination. CoRR, abs/2003.0290 0.099, 2020.
[19] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Information Minimization In Emergent Languages. arXiv:1905. 13687 [cs], May 2019. arXiv: 1905.13687.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[21] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2967, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[22] Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 4190-4203. Curran Associates, Inc., 2017.
[23] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Conference on Learning Representations, 2018.
[24] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-Agen Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations, 2017.
[25] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674, Online, July 2020. Association for Computational Linguistics.
[26] Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages 4376-4386, Hong Kong, China, November 2019. Association for Computational Linguis- tics.
[27] Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi- agent communication. International Conference on Learning Representations, 2018
[28] David Lewis. Convention: A philosophical study. Harvard University Press, 1969.
12

---

[29] Fushan Li and Michael Bowling. Case-of-ttachhing and language structure from emergent communication. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Bos Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 15825-15835. Curran Associates, Inc., 2019.
[30] Tsung-Y Yi Lin, M. Maire, Serge Belongie, James Hays, Perona, Ramanan, Piotr Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[31] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pages 693-701. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
[32] Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kicla, and Joelle Pineau. Learning to learn to communicate. Adaptive and Multitask Learning Workshop at ICML, 2019.
[33] Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kicla, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In International Conference on Learning Representations, 2020.
[34] Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron C. Courville, Counter- ing language drift with seeded iterated learning. CoRR, abs/2003. 12694, 2020.
[35] Gary Lupyan and Rick Dale. Language structure is partly determined by social structure. PloS one, 5(1):e8559, 2010.
[36] Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsupervised learning rules. In International Conference on Learning Representations, 2019.
[37] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. In AAAI Conference on Artificial Intelligence, 2018.
[38] Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Leaming Algorithms. arXiv:1803.02999 [es], March 2018. arXiv: 1803.02999.
[39] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA, 2002. Association for Computational Linguistics.
[40] Adam Paszke, Sam Gross, Francisco Massa, et al. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 8024 8035. Curran Associates, Inc., 2019.
[41] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[42] Florencia Reali, Nick Chater, and Morten H Christiansen. Simpler grammar, larger vocabulary: how population size affects language. Proc. R. Soc. B, 285((8811:200175562) 2018
[43] Nils Reimers and Iryna Gurevych. BERT-networks, In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics.
Biitence-BERT: Sentence embeddings using Siamese
[44] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal meta-policy search. In International Conference on Learning Representations, 2019.
13

---

[45] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-leariing with memory aaggonee neural networks. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pages 1842-1850, New York, New York, USA, 20-22 Jun 2016. PMLR.
[46] Agnieszka Slowik, Abhinav Gupta, William L. Hamilton, Mateja Jamnik. Sean B. Holden, and Christopher Pal. Exploring structural inductive biases in emergent communication. CoRR, abs/2002.01335, 2020.
[47] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances Neural Information Processing Systems, pages 2244-2252. Curran Associates, Inc., 2016.
[48] Olivier Tieleman, Angeliki Lazaridou, Shibl Mourad, Charles Blundell, and Doina Precup. Shaping representations through communication: community size effect in artificial learning systems. NeurlPS Workshop on Visually Grounded Interaction and Language, 2019.
[49] Michael Tomasello. Origins of human communication. MIT press, 2010.
[50] Peter Trudgill. Sociolinguistic typology: Social determinants of linguistic complexity. Oxford University Press, 2011.
[51] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 0630-3638. Curran Associates, Inc., 2016.
[52] Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pages 1112-11223 New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[53] Ronald J Williams. Simple statistical
algorithms for connectionist reinforce-
ment learning. Machine learning, 8(3-4):229-255 1992.
[54] Alison Wray and George W Grace. The consequences of talking strangers: Evolutionary corollaries of socio-cultural influences on linguistic form. Lingua, 117(3)543--77,, 2007.
14


Reviews:
[{"review_id": "yxU5JnlMJ5", "reviewer": "Reviewer_rond", "summary": "The paper is motivated by group coordination through natural language and creates a goal of developing agents that can communicate effectively with other agents and humans. They build off of work on emergent communication and their method is most related to L2C [Lowe19], which uses a meta-learning approach to learn communication protocols.\n\nTheir training method is as follows:\n```\nTrain with some human data\nwhile not_converged:\n    for n_meta steps:\n        Learn a meta-listener on a population of speakers\n    for n_selfplay steps:\n        Learn new speakers through self-play with the meta-listener\n    for n_supervised:\n        Fine-tune on human data\ntrain \n```\n\nAnd their evaluations are:\n1. Image based referential games\n2. A text based referential game that they design where agents are trained to communicate in German about English sentences\n\nTheir experiments show that:\n1. They outperform baselines on referential accuracy in both modalities\n2. They have a high sentence quality (BLEU, caption score) wrt baselines\n3. The meta-speaker in the text task works reasonably effectively as an English to German translation system (BLEU)\n4. Their speakers and listeners perform better on human evaluation in the image task\n\nLearning to learn to communicate. Lowe et. al., 2019.", "questions": "", "limitations": "", "rating": 9, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper directly extends past work. The novelty in their training procedure comes from repurposing meta-speakers and -listeners for population generation. Their experiment design is creative: they introduce a new task that serves allows them to evaluate speakers in a novel way (as translation engines).\n\nQuality: The thoroughness and rigor of the experiments are high. They include p-values, extensive ablations, and reasonable comparison to past work.\n\nClarity: The literature was well presented. By integrating related work into introduction/background on past methods (sections) their method was easier to understand. Some training details were less straightforward. E.g., in the prose it could state more explicitly how the meta-speaker and meta-listener are used together and to my knowledge the initialization is only mentioned in the algorithm box.\n\nSignificance: This paper brings studies on emergent communication in line with tasks of interest (communication with humans because of the evaluation, nlp tasks because of the translation system evaluation). One open ended question that the authors may be interested in addressing (perhaps verbally): what is the advantage of conceptualizing these training methods as emergent communication among a population of agents? E.g., a different conclusion from this paper is that using meta-learning as a distillation method across re-initializations is an effective method for solving various language tasks.\n\nMinor comments to authors:\n- L15 your -> our?\n- L26 like these agents -> these agents\n- L52 the use -> the use of\n- It would be helpful to label Figure 1 with the math in the prose\n- L162: Out -> Our", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I could not find where the limitations were addressed. It would be helpful to see a discussion that describes in more detail the benefits of their framing (emergent communication instead of something drier (see main review).", "review_text": "", "extra_scores": {"ethical_concerns": "I do not think their work presents ethical concerns at this time.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "OyGWghjxk7o", "reviewer": "Reviewer_YWdX", "summary": "In this paper, the authors aim at training agents that can coordinate with seen, unseen/human partners in a multi-agent communication environment. Instead of using static populations, the authors propose a dynamic population-based meta-learning approach that dynamically builds the population. Such an approach enables the trained agents to generalize to seen and unseen/humans partners. On the other hand, the authors try to use a limited amount of human data to fine-tune the agents to coordinate with humans using natural language. They also compare the proposed method and baselines on two referential games.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The explanation of the problem setup and model is clear. \n\nDifferent from previous works that use static populations, this paper proposed a dynamic population-based meta-learning approach. The dynamic population for multi-agent communication sounds interesting. \n\nThe experiments are good. The authors compare the proposed approach and baselines on two referential games. They also have some ablation experiments to evaluate each component of our proposed algorithm.\n\nThe authors show the results of agents interacting with humans. It is good to show the generalization ability of the proposed method. However, in this experiment, it is hard to evaluate the generalization ability of the proposed method when communicating with humans. Please see the detailed limitations below.\n\nThere is no explanation of why does Behavior Grounding enable agents to communicate with humans using a limited amount of human data? See detailed limitations below.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is good that the authors could provide more details about the human experiment, such as some example conversations between the agent and human. How much does human message differ from the trained agent? What if the human is not optimal and speaks some random messages? How do you filter bad examples? Without this information, it is hard to evaluate the generalization ability of the proposed method in the human experiment.\n\nWhy does Behavior Grounding enable agents to communicate with humans using a limited amount of human data? If the human data is limited, I don't think the proposed method can generalize well on human experiments. How much does the collected human data different from the messages that appeared in the human experiments. \n\nIt is not clear how do the authors update the speaker population. Does the population get larger and larger? \n\nThe introduction of baselines is not clear as well. Which baselines corresponding to using the static population? \n\nFigure 2 is a little bit confusing in the Dynamic Population part. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are human subject experimentations in Section 4.3. Do the experiments have been reviewed and approved by a relevant oversight board? I didn't find such information in the paper.\n", "needs_ethics_review": "Yes", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "CkdAdLAQvmM", "reviewer": "Reviewer_ctGD", "summary": "The current work proposes to use dynamic population based meta-learning in referential games between agents using natural language. In particular, it uses meta-learning to capture the diverse speaker/listener behaviors in a population and then initialize new agents by interacting with these meta-agents across iterations.  They also add supervised learning iterations to ensure there is no language drift. Experiments show that agents trained through the proposed approach have higher accuracy in the referential game interacting within themselves and also with humans, while retaining the naturalness of the language.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**  \n(S1) The idea to use a meta-listener and meta-speaker across different iterations in dynamic populations is indeed an interesting and novel direction. It is also well motivated in the paper.\n\n(S2) Experimental evaluation is solid and backs up the claims in the earlier part of the text. The increase in referential game accuracy (with statistical significance) and improved BLEU score when compared to ground truth captions both evidence the effectiveness of the proposed solution. The human studies are the cherry on top where the human-agent teams trained using the proposed approach outperform other existing approaches.\n\n(S3) The authors did a good job writing the manuscript as it is easy to read and understand. The work is well placed in the context of related work and all the experimental details are clearly explained.\n\n**Weaknesses**  \n(W1) The manuscript does not contain any qualitative examples in the main paper. Some have been included in the appendix. Further, it would be interesting to see where the agents fail to better understand the strengths and weaknesses of the proposed approach. For instance, categorizing the errors due to imperfect vision, incorrect language usage by the speaker, or language understanding by the listener would be useful.\n\n**Comments**  \n(C1) How important is the role of supervised learning iterations in order to retain the naturalness of the language? The current work is missing this ablation, where the amount of supervised learning is varied.\n\n(C2) To check for robustness against any implicit bias in image datasets, how do agents trained on MSCOCO perform on a different image dataset at test time, e.g, PASCAL?\n\n**Typos**  \n* Algorithm 1: After the meta learning stage, is there a typo in the initialization of \\phi’_0?\n* L15: your -> our\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly discuss the limitations of the proposed approach but do not discuss broader impact in the main paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0quBPRIvyMc", "reviewer": "Reviewer_V9jU", "summary": "This work deals with the problem of multi-agent communication where the agents can be artificial or human. The latter means that the conversation has to be in natural language. New agents can be added or removed from conversation so the environment is not static. The authors propose a dynamic population-based meta-learning approach. They analyse the language skills of agents using BLEU, the quality of interaction in a human trial, as well as cross-play with unseen partners achieving overall good results.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It would be nice to give some more context to this work. Where do authors see this technology to be used? The test-beds are clearly toy examples and not very relevant. Do they see the use of these agents as bots that engage in eg Twitter communication? Or is this meant to improve two-party dialogue?\n\nWhen we talk about communication and dialogue what we have in mind is multi-turn communication. The test-beds considered here focus only require singe turns so it’s not quite clear to me how much of the communication management behaviour is learned or can be learned. Note the vocabulary of the test-beds are very small, only 100 words. Can this approach scale?\n\nThe issue of an unseen and seen interlocutor is incredibly important and I would be keen to see how this method generalises to (or improves) RL approaches to task-oriented dialogue. Ablation studies show that removing meta-agents does not cause a drastic decrease in performance and I wonder if the reason is the simplicity of the task or limited use of this method. In any case it would be important to investigate this.\n\nA lot of details are in the Appendix that would be better suited in the main text, in particular the qualitative samples.\n\n----\n\nThe authors addressed my concerns to some extent so I'm willing to increase my score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is difficult to judge as it is not elaborated what this technology can be used for. The test cases presented do not pose any negative social impact as far as I can see.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any ethical concernes.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an interesting approach to combating semantic drift in communication-based training for language agents. The biggest concern on reviewers was the need to more clearly describe the details of methods and results for the human experiments. It will be very important to add methods, ethics, and results details in the revision. Some reviewers felt that the domains considered were too artificial, but I believe (along with other reviewers) that these are an appropriate first step given previous work in this area.
Decision: Accept (Poster)