Emergence of Compositional Language with Deep Generational Transmission
Authors: Michael Cogswell
Jiasen Lu
Stefan Lee
Devi Parikh
Dhruv Batra
Venue:  (2020)
Keywords: Cultural Evolution
Deep Learning
Language Emergence

Abstract: Recent work has studied the emergence of language among deep reinforcement learning agents that must collaborate to solve a task. Of particular interest are the factors that cause language to be compositional---i.e., express meaning by combining words which themselves have meaning. Evolutionary linguists have found that in addition to structural priors like those already studied in deep learning, the dynamics of transmitting language from generation to generation contribute significantly to the emergence of  compositionality. In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language. We show that this implicit cultural transmission encourages the resulting languages to exhibit better compositional generalization.

Under review as a conference paper at ICLR 2020
EMERGENCE OF COMPOSITIONAL G WITH DEEP GENERATIONAL TRANSMISSION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has studied the emergence of language among deep reinforcement learning agents that must collaborate solve a task. Of particular interest are the factors that cause language to be compositional-ii. express meaning by com- bining words which themselves have meaning. Evolutionary linguists have found that in addition to structural priors like those already studied in deep learning, the dynamics of transmitting language from generation to generation contribute sig- nificantly to the emergence of compositionality. In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replac- ing agents in a population to create knowledge gap, implicitly inducing cultural transmission of language. We show that this implicit cultural transmission en- courages the resulting languages exhibit better compositional generalization.
INTRODUCTION
Compositionality is an important structure of language that reflects a disentangled understanding of the world enabling the expression of infinitely many concepts using finitely many elements. Agents that have compositional understandings of the world generalize in obviously correct ways even in the face of limited training examples (Lake & Baroni 2018). For example, an agent with a compositional understanding of blue squares and purple triangles should also un- derstand purple squares without directly observing any of them. Developing artificial agents that can ground, understand, and produce compositional (and therefore more interpretable) language could greatly improve generalization to new instances and ease human-Al interactions.
In building theories of how compositionality emerges in human languages, work in evolutionary linguistics looks to the process of cultural transmission (Kirby) 2001 Kirby et al. 2008). Cultural new group
transmission of language occurs when a group of agents pass their language of agents, e.g. parents who teach their children to speak as they do. Because this education is incomplete and biased, it allows the language itself to change over time via process known as cultural evolution. This paradigm (Kirby et al.) 2014) explains the emergence of compositionality result of expressivity and compressibility i.e. be most effective, language should be expressive enough to differentiate between all possible meanings (e.g.. objects) and compressible enough to be learned easily. Work in the evolutionary linguistics community has shown that over multiple "generations' these competing pressures result in the emergence of compositional languages both in simulation (Kirby 200T) and with human subjects (Kirby et al. 2008). These studies aim to understand humans whereas we want to understand and design artificial neural networks.
Approaching the problem from another direction, recent work in Al has studied language emergence in such multi-agent, goal-driven tasks. These works have demonstrated that agent languages will emerge to enable coordination-ceenric tasks to be solved without direct or even indirect language supervision (Foerster et al. 2016 SSukhbattar et al. 2016 Lazaridou et al.) 2017 Das et al. 2017 However, the resulting languages are usually not compositional (Kottur et al. 2017) and are difficult to interpret, even by other machines (Andreas et al. (017). Some existing work has studied means to encourage compositional language formation (Mordatch & Abbeel, 2018 Kottur et al. (017). but these settings study fixed populations of agents -iee examining language within single generation.
In this work we bridge these two areas - examining the effect of generational cultural trans- mission the compositionality of emergent languages in multi-agent, goal-driven setting.

---

Under review as a conference paper at ICLR 2020
fcolor Shape)
Rowflales U. anc
Prediction (purple, square)
O
Q2
Aq
Ay
(a) Example agent dialog
Coneration Nil
Generaaion
and By lesnniogguge
(b) Implicit cultural transmission
We introduce cultural transmission into language emergence between neural agents. The starting point of our study goal-oriented dialog task (similar that of Kottur al.] (2077). summarized in Fig. la During learning we periodically replace some agents with new ones (gray agents). These new agents do know any language, but instead of creating one they learn from older agents. This creates generations of language that become more compositional over time.
Figure I:
We study this in the context of a cooperative dialog- based reference game involving two agents communicating in discrete symbols (Kottur et al. 2017); an example dialog is shown in Fig. la To examine cultural transmission, introduce a simple mechanism to induce the expressivity and compressibility pressures inherent in cultural transmission. Specifically, we periodically re-initialize some subset of the agents in the population. In order to perform well at the task, the population's emergent language must be sufficiently expressive to reference all the objects (expressivity) and must be easily learnable by these 'new' agents (compressibility) The new agents have randomized language whereas the surviving agents already know a grounded language. This "knowledge gap" creates an implicit 'teaching' setting that is analogous to the explicit transmission stage in models of iterative learning (Kirby 2001).
extend this setting
population of agents (Fig. 1b and
Through our experiments and analysis, we show that periodic agent replacement is an effective way to induce cultural transmission and yields more compositionally generalizable language in our setting. To summarize, our contributions are:
We propose method for inducing implicit cultural transmission in neural language models. We introduce new metries to measure the similarity between agent languages and verify cultural
transmission has occurred as a result of our periodic agent replacement protocol.
We show our cultural transmission procedure induces compositionality in neural language models, going from 13% accuracy on compositionally novel test set to 46% in the best configuration. Further, show this is complementary with previous priors which encourage compositionality.
2 TASK & TALK: TESTBED FOR COMPOSITIONAL LANGUAGE EMERGENCE
We consider the cooperative Task & Talk reference game introduced in Kottur et al. (0117) Shown in Fig. Tal the game played by two agents one who observes an attributed object - e.g. (purple, solid, square) and another who is given task retrieve subset of these attributes over the course of the dialog- - e.g. (color, shape). . The dialog itself consists of two rounds of agents exchanging single-token utterances from fixed vocabularies. At the end of the dialog, the task-aware agent must report the requested attributes and both agents are rewarded for correct predictions. This causes language grounded in the objects to emerge because there is no other way to solve the task.
A compositional solution to this task can look like a crestion-answers style dialog where the task- aware agent queries the other for specific attributes (Fig. la] e.g. uttering "X" requesting the co color to which the other agent replies "1" indicating purple Importantly, this pattern would persist regardless of the other attribute values of the object (e.g. for all (purple, *, *) ob- jects). However, as there is no grounding supervision provided, agents must learn to associate specific meanings specific words and it is unlikely for compositional languages to emerge purely by chance. Given color task, an agent might use "1" for (purple, solid, square) and then "2" for (purple, solid, circle). It is impossible for other agents to know that "2" means purple without having seen (purple, solid, circle), compositional language is essential for generalization to compositionally novel instances.
2

---

Under review as a conference paper at ICLR 2020
Models. To formalize this setting, let Q-bot and A-bot be agent policies parameterized by neural networks Q and A respectively. At each round t, Q-bot observes the task z and it's memory of the dialog so far t--1 and produces a single-token utterance m/a E V from the vocabulary V. Function- ally, mmg M/ Qmm'r' =q:"a 1Q where m is A-bot's reply in the previous round. Likewise, A-bot responds by computing m' N1A A(mb: "A," where I is the object instance repre- sented symbolically by concatenating one-hot vectors, one per attribute. After two rounds, Q-bot must respond to the task, predicting the requested attribute pair u U Ux h22) as a function of the task and Q-bot's final memory state. Both agents are rewarded if both attributes are correct (no partial credit). We follow the neural network architectures of Q, A, and U from Kottur et al. (017).
Measuring Compositional Generalization, Kottur al. (017)) generated a synthetic dataset con- sisting of three attribute types (color, shape, style) each with four values (e.g. red, blue, square, star, dotted, solid, ...) and six tasks, one task for each ordered pair of different attribute types. total, this results in 64 unique instances and 384 task-instance pairs. To evaluate compositionality, Kottur iet al. (2077) held out 12 random instances for testing. Given the closed-world set of instances, these 12 are guaranteed to be never-before-seen triplets of attribute values; however, each individual value has been seen in training in other triplets. As such, accuracy on this set is a measure of compositional generalization.
Shorteomings Kottur et al. (017)) Evaluation. In our investigations, we found some shortcom- ings in the evaluation protocol of Kottur et al. (2017). First, the authors do not report variance over multiple runs or different random test-sets which we found to be significant. Second, the strategy of randomly selecting the test instance, suppose agents develop a language that uses single words to refer to attribute pairs like (red, *, triangle) and (red, filled, *). Such agents might generalize an un- seen instance (red, filled, triangle) by composing the 'paired' words above instead of disentangling individual attributes.
still reward some only partially compositional strategies. For
We make two modifications to address these issues. Our results are reported as means and variances estimated from multiple training runs with different random seeds evaluated with 4-way cross- validation. We also introduce harder dataset where instead of withholding random individual instances (e.g., (green, dotted, triangle),.... as in Kottur et al. (2017), we withhold all instances for set of attribute pairs (e.g., (green, dot dotted, *), , (red, so solid, .....). We will refer to datasets generated in this fashion as novel pair and the original dataset as novel instance. We report on both settings for comparison, but find our new setting to be significantly more chal- lenging in practice- - requiring a stricter notion of compositionality that is more closely aligned with human intuitions about these attributes.
3 COMPOSITIONAL LANGUAGE EMERGENCE WITH CULTURAL TRANSMISSION
In iterative learning models of cultural transmission from evolutionary linguistics, competing pres- sures towards expressivity and compressibility have been shown to induce compositionality over multiple 'generations' of language transfer (Kirby, 2001 Kirby et al., 2008). The roal-driven nature of our reference game already encourages expressivity agents must be able to refer to the objects in order to succeed. To introduce compressibility pressure and parallel literature in evolutionary lin- guistics, we introduce a population of agents which regularly has members replaced by new agents that lack any understanding of the remaining population's 's language. As this paradigm lacks explicit teaching steps where new agents are trained to ground existing words, we consider this approach as means of implicit cultural transmission.
Populations of Agents. We consider a population of Q-bots {Q'....,Q P..,,NNGN QNo] } and a population of A- bots (At,... ...,.... A with each agent having different set of parameters. At each iteration during learning, we sample a random Q-bot-A-bot pair to interact and receive updates i.e. the red line (2) in Alg. [ As any Q-bot may be made to communicate with any A-bot, there pressure for the population to adopt a unified language. Likewise, when an agent is reinitialized it will receive positive reward much more quickly when happens to use language that its conversational partners understand. Furthermore, 'compressible" languages that are easier to learn will result in greater reward for the population in the face of periodic re-initialization of agents.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Training with Replacement and Multiple Agents
for epoch 1,..., Nepocks
Sample Q-bot iq from U (1, No} and A-bot ia from (1,
2
for zQ; T.A. in each batch do
for dialog rounds 1 == 1
T do
mg. hig = (m' 1-1 zq. h111 m A A f- IA ha 1)
0 (1r. (2))
7
8
Policy gradient update w.r.t. both Q-bot and A-bot parameters
if mod E 0 then
Sample replacement set B under policy v and re-initialize all agents in B
return all Q-bots and A-bots.
Introducing multiple agents may in itself add compressibility pressure and improve generalizations even without replacement (Raviv al. 2018). Agents in a population have model minor linguistic differences between conversational partners given the same memory capacity. Further, each agent provides another potential language variation that can mimicked and sespetuatediincraast guage diversity early in training. We examine these effects through no-replacement baselines, but find that generational pressure where some agents know less than others can also be important for compositionality in setting.
lan-
Replacement. In order to create a notion of "generations' we replace agents periodically. Let æ be some replacement strategy that returns subset of the population. Every E epochs, call and reinitialize the parameters and optimizers for the corresponding agents (blue lines 9-10 in Alg. We investigate three settings of (appendix A.2 for more details):
Uniform Random. Sample an A-bot and Q-bot from uniform random distributions.
Epsilon Greedy. With probability 1-E replace the A-bot and Q-bot with the lowest validation accuracy. We use = 0.2 in our experiments. Oldest. Replace the oldest A-bot and Q-bot, breaking ties with uniform random sampling.
4 EXPERIMENTAN SETTING
Experimental Setting, We evaluate on both our modified Task & Talk dataset and the original from Kottur et al. (2117). as described in Section [ All results are reported means and variances computed from a total of 16 trials (four random seeds each with 4-way vross-validation. We report accuracy based on Q-bot getting both elements of the task correct corresponding to the more restrictive "Both" setting from Kottur et al. (2017).
Kottur al. (2017) examined series of increasingly restrictive settings in order to study conditions under which compositionality emerges. The primary variables are whether A-bot has memory (ab- lated by setting h², =0) and the vocabulary sizes Vo and VA for Q-bot and A-bot respectively. For comparison also evaluate in these settings: Minimal Vocab (V=3,, VA=4). Memoryless Minimal Vocab (V (Va=3, V4=4, ht, =0), Overcomplete (Vq=Vı=Gle We also introduce Memo- ryless + Overcomplete (Vg=V,=64, h', -0) complete the cross product of settings and examine the role of memory restriction in overcomplete vocabularies.
The Memoryless Minimal Vocabulary setting results in the best compositional generalization; however, this is an extreme setting requiring not only that the minimum number of groundable symbols be known but also that A-bot not be able remember it's previous utterance. While do report these settings and see quite large performance gains due to cultural transmission, we are mainly interested the more realistic Overcomplete setting where large pool of possible tokens is provided and both dialog agents remember the conversation.
Model and Training Details. Our A-bots and Q-bots have the same architectur in Kottur et al. (217)). All agents are trained with E = 25000, batch size of 1000, and the Adam (Kingma & Ba 2015) optimizer (one per bot) with learning rate 0.01. the Multi Agent setting we use
'There is batch per epoch hecause there only 384 instances (64 objects tasks).

---

Under review as a conference paper at ICLR 2020
12-
Peposcement
Sing Olinst
12
16
Seplact menere
17
Gruady
MAA Clisst
Moceel
Test set accuracies (with standard deviations) are reported against our new harder dataset using models similar to those in Kottur et al. (2017). Our variations on cultural transmission (darker blue bars) outperform the baselines where language does not change over generations.
Figure 2:
NA No 5. We stop training after generations (199000 epochs Multi Agent; 39000 epochs Single Agent). This differs from Kottur et al. (2017), which stopped once train accuracy reached 100%. Further, we do not perform negative mining.
Baselines.
We consider set of baseline setting to isolate the effect of our approach.
Single Agent Populations. We ablate the effect multi-agent populations by training individ- ual A-bot-Q-bot pairs (i.e. populations with NA No 1). We apply the uniform random (either A-bot or Q-bot at random) and oldest (alternating between A-bot and Q-bot) replacement strategies to these agents; however, the epsilon greedy strategy is not well-defined here. In this setting we decrease E from 25000 to 5000 to keep the average number of gradient updates for each agent constant with respect the multi-agent experiments.
No Replacement. We also consider the effect of replacing no agents at all, but still allowing the agents to train for the full 199,000 (39,000) epochs. Improvement over this baseline shows the gains from our replacement strategy under identical computational budgets.
RESULTS AND ANALYSIS
5.1 IMPACT OF CULTURAL. TRANSMISSION ON COMPOSITIONAL. GENERALIZATION
Results with standard deviations against our harder dataset are reported in Fig. 2 We compared methods and models using dependent paired t-tests and reported the resulting p-values in Section
A.3 Result on the original Task & Talk dataset are in Section A.
Cultural transmission induces compositionality. Our main result is that cultural transmission ap- proaches outperform baselines without cultural transmission. This can be seen by noting that for each model type Fig. 2 the 3 darker blue bars (Multi Agent Replacement approaches) are largest. After running a dependent paired t-test against all pairs of baselines and cultural transmission ap- proaches we find meaningful difference in all cases (p S 0.05). This is strong support for our claim that our version of cultural transmission encourages compositional language because it causes better generalization to novel compositions of attributes.
Next go to discuss some additional trends we hope the community will find useful.
Population dynamics without replacement usually lead some compositionality. The Multi Agent No Replacement policies usually outperform than the Single Agent No Replacement policies, though the difference isn't very significant the except the Overcomplete and Minimal Vocab settings. This agrees with recent work from evolutionary linguistics, where multiple agents can lead to compositionality without generational transmission Raviv et al. (011)).
Variations in replacement strategy tend to not affect performance. The Multi Agent Uniform Random/Eppsilon Greedy/Oldes/ replacement strategies are not largely or consistently different from one another across model variations. This suggests that while some agent replacement needs to occur, it is not critical whether agents with worse language are replaced or whether there is a pool of similarly typed agents to remember knowledge lost from older generations. The main factor is that new agents learn in the presence of others who already know a language.
Cultural transmission is complementary with other factors that encourage compositionality. As in Kottur et al. 2017). we find the Memoryless + Small Vocab model is clearly the best. This agrees with factors noted elsewhere Kottur et al. (2017); (2000) and shows how many different factors can affect the emergence of compositionality.
& Abbeel (2018); Nowak et al.
5

---

Under review as a conference paper at ICLR 2020
Removing memory makes only minor differences. Removing memory makes difference (neg- ative or positive) in Single Agent settings, but it can have a relatively small effect in Multi Agent settings, helping Small Vocab models and hurting Overcomplete models. While our approach is complementary with minimizing vocab size to increase compositionality. its makes memory re- moval less useful. As the Memoryless Overcomplete setting has not been reported before, these results suggest that the relationship between inter-round memory and compositionality is not clear.
Overall, these results show that adding cultural transmission to neural dialog agents improves the compositional generalization of the languages learned by those agents in a way complementary to other priors. thereby shows how to transfer the cultural transmission principle from evolutionary linguistics to deep learning.
5.2 Is GENERATIONAL TRANSMISSION OCCURRING?
Because it is implicit, cultural transmission may not actually be occurring; improvements may be from other sources. How can we measure cultural transmission? We focus on A-bots and take a simple approach. We assume that if two A-bots 'speak the same language" then that language was culturally transmitted. There is a combinatorial explosion of possible languages that could refer to all the objects of interest, so if the words that refer to the same object for agents are the same then they were very likely transmitted from the other agents, rather than similar languages emerging from scratch just by chance. This leads a simple approach: consider pairs of bots and see if they say similar things in the same context. If they do, then their language was likely transmitted.
More formally, consider the distribution of tokens A-bot A' might use describe its object a when talking to Q-bot Qt: (m, () or pk. for short. We want to know how similar A's language to that of another A-bot A'. We'll start by comparing those two distributions by computing the KL divergence between them and then taking an average over context (objects, Q-bots, and dialog rounds) to get our pairwise agent language similarity metric Diy:
Dij- Ex k. k. [DEL (px,((m'A1z.) Pk., (=))
(1)
Taking another average, this time over all pairs of bots (and also random seeds and cross-val folds), gives our final measure of language similarity reported in Fig.
D Eij st. inf [Diy]
D is smaller the more similar language is between bots. Note that even though D.y is not symmetric (because KL divergence not). D is symmetric because it averages over both directions of pairs.
We compute D by sampling an empirical distribution over all messages and observations, taking 10 sample dialogues in each possible test state (Ir,, #Q) of the world using the final populations of agents as in Fig. 2 Note that this metric applies to group of agents, so we measure it for only the Multi Agent settings, including two new baselines colored red Fig.3) The Single Agents Combined baseline trains 4 Single Agent No Replacement models independently then puts them together and computes D for that group. These agents only speak similar languages by chance, so D is high. The Random Initialization baseline evaluates language similarity using newly initialized models. These agents have about a uniform distribution over words at every utterance, so their languages are both very similar and useless. For each model these baselines act like practical (not striet) upper and lower bounds on D, respectively.
Fig. 3 shows this language dissimilarity metric for all our settings. As we expect, the paired Single Agents are highly dissimilar compared to agents from Multi Agent populations. Further, all the replacement strategies result in increased language aimilarty-although the degree of this effect seems dependent on vocabulary setting. This provides some evidence that cultural transmission is occurring in Multi Agent settings and encouraged by the replacement strategy in our approach. While all Multi Agent settings resulted in language transmission, our replacement strategies results in more compositional languages due to repeated teaching of new generations of agents.
5.3 VISUALIZING EMERGENT LANGUAGES
In this section we visualize the language learned by agents at various stages of training to reinforce previous conclusions and build intuition. Each of the three sub-figures in Fig. 4 summarizes all of the conversations between a particular pair of bots for the (shape, color) task. To see how these summaries work, consider Fig. 4a That sub-figure is divided into a 4 x 4 grid with 4 elements

---

Under review as a conference paper at ICLR 2020
22
Agents Combined
Hade ha Raplocemati
Us Bunson
Macel
Figure 3: Do bots in population learn similar languages? On the y-axis (eq. lower values indicate similar language. Populations evolved with our method speak similar languages, but inde- pendently evolved agents do not. Thus our implicit procedure induces cultural transmission.
in each cell, so there is element for each of the 64 possible objects. For this task, objects in each row of the 4 x 4 grid have the same shape and objects in each column have the same color. To the right each object are the two tokens A-bot used respond to Q-bot the dialog rounds. Ideally they should indicate the color and shape of the object. Finally, the check-marks or Xs to the right of A-bot's utterances indicate whether Q-bot guessed correctly.
20
21
22 22 2
3
02
000 O01
0030
02
20
21
211
O220
02X
Goo
02X
Dot
20
21
21
22
02X
01
00
02X
01
000
Ozo
2218
220
Oos
x 101X
000
022
113
2.11
3118
1128
23
2
122251211
200
2:23
222
2220
130
230
110
122
2
22
22
21
200
20
.32
20
13
23
11
120
23
22
21
20
23
22
20
136
230
11
12
23
21
20
23
22
21
20
22
21
21
02
13
1
10
5 12
11
21
21
X1
02
13
10
13
X 1
1
10
21
21
21
22X
130
100
13
12
10
x22
22
22X
22X
113
13
A
11
10
23
2.31
2.01
21 03X
23.33 K7
2.32
30X
3.33
32
2.31
30
2 03
33
à 01
02X
33 77
A 32
à 31
30X
3 32
31
30X
03 87
33
02X
3:3 87
3
30X
33
32
31
30X
0.3
33
0.0
022X
3.3
30
31
300
[shape, color)
Train
(shape, color)
Train
(shapo, color)
Train Test
(a) Gen 1 (Single) New A-bot (b) Gen 8 (Multi) New A-bot
(c) Gen 8 (Multi) Old A-bot
Figure 4: Each sub-figure summarizes an A-bot's language, as described in Section 5.3
By com-
paring the baseline of Fig. 4a to a similar pair of bots from our approach Section 4b we can see that our approach encourages compositional language to emerge. Furthermore, the similarity between Fig. 4b and Fig. 4c suggests language is indeed transmitted in our approach.
From left to right: Fig. 4a summarizes the single pair from Single Agent No Replacement
(3000 iterations old); Fig. 4b summarizes dialogs between an old Q-bot (about 23000 iterations) and recently re-initialized A-bot (about 3000 iterations) the 8th and final generation of Multi Oldest run; Fig. 4c summarizes dialogs between the same old Q-bot as in Fig. 4..44bjnnd an old A-bot (13000 iterations) from the same Multi Oldest experiment. Even though the A-bots Fig. and Fig. 4b have trained for abou same number of iterations, the A A-bot trained in the presence of other bots which already know functional language has already learned somewhat compositional language whereas the Single Agent A-bot has not. Furthermore, by comparing the old A-bot's language Fig. 44 with the new one Fig, 4 we can see that they are extremely similar. They even lead to the same mistakes. This again suggests that language is transmitted between bots, in agreement with our previous experiments.
RELATED WORK
Language Evolution Causes Structure. Researchers have spent decades studying how unique properties of human language like compositionality could have emerged. There is general agree- ment that people acquire language using combination of innate cognitive capacity and learning from other language speakers (cultural transmission), with the degree each being widely dis- puted Perfors 2002): Pinker Bloom 1990). Both innate cognitive capacity and specific modern
DDue the stochastic nature of Multi Agent approach.

---

Under review as a conference paper at ICLR 2020
human languages like English co-evolved Briscoe (0000) via biological Pinker & Bloom (1990) and cultural Tomasello (1999); Smith (2006) evolution, respectively.
In particular, explanations of how the cultural evolution of languages could cause structure like compositionality are in abundance Nowak & Krakauer (1999); Nowak et al. (2000;; |mmthh et al.
20033) ; BBighton (2002); Vogt :000)]]Kibby al. (2014 [Spike al. (017). An important piece of
the explanation of linguistic structure the iterated learning model Kirby et al. (2014); :001)::Kirb))) Kirby (200T);
Kirby et al. (2008) used to motivate our approach. Indeed it shows that cultural transmission causes
structure in computational Kirby (200T, 2002); Christiansen & Kirby (2003); Smith et al. (2003) and human Kirby al. (2008); Comish et (2009): Scott-Phillips
Kirby (2010) experiments.
Even though cultural transmission may aid the emergence of compositionality. recent results in
evolutionary linguistics Raviv et al.) (2018) and deep learning Kottur et al. (2017) Mordatch &
Abbeel (2018) also emphasize other factors.
While existing work in deep learning has focused on biases that encourage compositionality, it has not considered settings where language is permitted to evolve over generations of agents. We have shown such an approach is viable and even complementary with other approaches.
Language Emergence in Deep Learning. Recent work in deep learning has increasingly focused on multi-agent environments where deep agents learn to accomplish goals (possibly cooperative or competitive) by interacting appropriately with the environment and each other. Some of this work has shown that deep agents will develop their own language where none exists initially if driven by a task which requires communication |oerster al. (2016); (Sukhbaatar et al. (2016); (agg):][aaaa Lazaridou al.) 2017). Most relevant is work which focuses on conditions under which compositional language emerges deep agents learn to cooperate Mordatch & Abbeel (2018); Kottur et al. (2017). Both Mordatch & Abbeel (2018) and Kottur et al. (2017) find that limiting the vocabulary size so that there aren't too many more words than there are objects to refer encourages compositionality, which follows earlier results in evolutionary linguistics /ooiiittic//oakkk Nowak et al. (2000). Follow up work has continued to investigate the emergence of compositional language among neural agents, mainly focusing on perceptual as opposed to symbolic input and how the structure of the input relates to the tendency for compositional language to emerge Choi et al. (2018); Havrylov & Titov (2017): Lazaridou et al. (2018). Other work has shown that Multi Agent interaction leads to better emergent translation Lee et al. 2018), but it does not measure compositionality.
Cultural Evolution and Neural Nets. Somewhat recently, |Beggio 2012) suggested that cultur- ally transmitted ideas may help in escaping from local minima. Experiments in Gülçehre & Bengio (2016) support this idea by showing that supervision of intermediate representations allows more complex toy task to be learned. Unlike our work, these experiments use direct supervision provided by the designed environment rather than indirect and implicit supervision provided by other agents.
Two concurrent works examine the role of periodic agent replacement on language emergence albeit in different environments. In & Bowling (2019) replacement is used to encourage languages to be easy to teach, and this in turn causes compositionallty. In Dagan et al. (2019) neural language transmitted through a bottleneck caused by replacement. The resulting language has increased efficiency and effectiveness, with further results showing that co-evolving the agents themselves with the language amplifies the effect. Both of these works support our central observations.
7 CONCLUSION
In this work we investigated cultural transmission in deep neural dialog agents, applying it to lan- guage emergence. The evolutionary linguistics community has long used cultural transmission to explain how compositional languages could have emerged. The deep learning community, having recently become interested in language emergence, has not investigated that link until now. Instead of explicit models of cultural transmission familiar in evolutionary linguistics, we favor an implicit model where language transmitted from generation to generation only because it helps agents achieve their goals. We show that this does indeed cause cultural transmission and compositionality.
Future work. While our work used an implicit version of cultural transmission, we are interested in investigating the effect of explicit versions of cultural transmission on language structure. In another direction, cultural transmission may also provide an appropriate prior for neural representa- tions of nan-language information.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jacob Andreas, Anca D. Dragan, and Dan Klein. Translating neuralese. In ACL, 2017.
Yoshua Bengio. Evolving culture local minima. CoRR, abs//103. 2012.
Henry Brighton. Compositional syntax from cultural transmission. Artificial Life, 8:25-54, 2002.
Ted Briscoe. Grammatical acquisition: Inductive bias and coevolution of language and the language acquisition device. In Language, volume 76. Linguistic Society of America, 2000.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Compositional obverter communication learning from raw visual input. In International Conference on Learning Representations (ICLR), 2018.
Morten H. Christiansen and Simon Kirby, Language evolution: consensus and controversies. Trends in cognitive sciences, 7:300-307, 2003.
Hannah Comish, Monica Tamariz, and Simon Kirby. Complex adaptive systems and the origins of
adaptive structure: What experiments can tell us. Language Leurming, 59(s1):187-205, 2009, doi: URL httttt:tt//oo//
(41111/j.467--922222205 1477 9922. 2009. 00540.x
wi ley. com/doi/
Gautier Dagan, Dieuwke Hupkes, and Elia Bruni Co-evolution language and agents in referen- tial games. pre-print, 2019. URL
dagan201 9coevolut.ii... .ediioo.p..... fd
Abhishek Das, Satwik Kottur, José M.F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. Proceedings of the IEEE International Conference Computer Vision (ICCV), 2017.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson, Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Çağlar Gülçehre and Yoshua Bengio. Knowledge matters: Importance of prior information for optimization. Journal of Machine Learning Research, 17:8:1-8:32, 2016.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning com- municate with sequences of symbols. In NIPS, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emer- gence of regularity and irregularity. IEEE Trans. Evolutionary Computation, 5:102-110, 2001.
Simon Kirby. Natural language from artificial life. In Artificial Life, 2002.
Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences, 2008 ISSN 0027-8424. doi: htw.0////aa.00078335551111 www pnas. http://wwwww......ooooooooo/////
URL
Simon Kirby, Tom Griffiths, Kenny Smith. Iterated learning and the evolution of language.
Current Opinion in Neurobiology, 28: 18:108-114, 2014.
Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally" in multi-agent dialog. In EMNLP, 2017.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequene recurrent networks. In ICMI., 2018.

---

Under review as a conference paper at ICLR 2020
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agems cooperation and the emergence of (natural) language. In International Conference on Learning Representations (ICLR), 2017.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations (ICLR), 2018.
Jason D. Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi- agent communication. CoRR, abs/1710.06922, 2018.
Fushan Li and Michael Bowling. Case-of-teachinn and language structure from emergent commu-
nication. CoRR, abs/1906.02203, 2019. Ut./sttpp//arrtv..rr/////////00 02403
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent
populations. In AAAI, 2018.
Martin A. Nowak and David C. Krakauer. The evolution of language. Proceedings of the National Academy of Sciences of the United Stutes of America, 96 14:8028-33 10002833,,1 1999.
Martin A. Nowak, Joshua B. Plotkin, and Vincent A A Jansen. The evolution of syntactic commu- nication. Nature, 404:495-4444 49899999999-- 2000.
Amy Perfors. Simulated evolution of language: a review of the field. J. Artificial Societies and Soccial Simulation, 5, 2002.
Steven Pinker and Paul Bloom. Natural language and natural selection. Behavioral and brain sciences, 13(4):707-727, 1990.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Compositional structure emerge without genera- tional transmission. Cognition, 182:151-164 2018.
Thomas C. Scott-Phillips and Simon Kirby. Language evolution in the laboratory. Trends in cogni- five sciences, 14 9:41 1-7,
Kenny Smith. Cultural evolution of language. Encyclopedia of Language and Linguistics Edition, 2:315-3222 2006.
Kenny Smith, Simon Kirby, and Henry Brighton. Iterated learning: A framework for the emergence of language. Artificial Life, 9:371-386, 2003.
Matthew Spike, Kevin Stadler, Simon Kirby, and Kenny Smith. Minimal requirements for the emer- gence of leareed signaling. In Cognitive Science, 2017.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NIPS, 2016.
Michael Tomasello. The cultural origins of human cognition. Harvard university press, 1999.
Paul Vogt. The emergence of compositional structures in perceptually grounded language games. Artificial intelligence, 167((-2):206- 242, 2005.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 RESULTS ON SINGLE HELD OUT ATTRIBUTE DATASET OF KOTTUR ET AL. 2017
In Section 4 we proposed a new harder compositional dataset different from the one in Kottur al. (2017). For comparison, in this section we train and evaluate our models on the original dataset from Kottur al. 2017) to show that our approach also improves still improves compositionality in this setting and to show that our new dataset is indeed harder.
332
Restaccmion
Mult
Multt Unitorm fardom
Groudy
Mirsayyes: Minina Vocas
DeinCinplee
Menweyles Overonggies
Figure 5: Test set accuracies (with standard deviations) are reported by training and evaluating the same models as our main results Fig. Dagainst the dataset from Kottur al. (2017). These results do not perform cross-validation, following Kottur et al. (017). They only vary across 4 different random seeds. This dataset significantly easier than our new dataset, as indicated by the Our proposed approach still outperforms models without replacement multiple agents.
A.2 REPLACEMENT STRATEGIES
Our approach to cultural transmission periodically replaces agents by re-initializing them. The ap- proach section outlines various replacement strategies (policy т), but does not detail their imple- mentation. We do here.
These strategies depend on a number of possible inputs:
the current epoch
the period agent replacement
5 /u/l the validation accuracy of agent for Q-bots/A-bots. For Q-bots this is averaged over all potential A-bot partners, and vice-versa for A-bots.
af /a the age in epochs of agent i for Q-bots/A--bots
Single Agent strategies given in Alg. 2 and Alg. 3 Multi Agent strategies are given in Alg, 4
Alg. 5] and Alg. [ Note that Single Agent strategies always replace one agent while Multi Agent strategies always replace one Q-bot and A-bot. An additional Replace All baseline strategy is given in Alg. ?? and generalizes to both Single and Multi Agent cases.
Algorithm 2: Single Agent Random Replacement
~ 4 1}
if d = 0 then
L return { A-bot]
else
return { Q-bot
A.3 DETAILED RESULTS
In our experiments we compare models and we compare replacement strategics. We ran dependent paired t-tests across random seeds, cross-val folds, and replacement strategies to compare models. We ran dependent paired t-tests across random seeds, cross-val val folds, and models to compare re- placement strategies. The p-values for all of these t-tests reported here.
Replacement strategy comparisons are in Fig. (Single Agent) and Fig. (Multi Agent). Model comparisons in Fig.

---

Under review as a conference paper at ICLR 2020
5-10
F172
F1172
10:00
11:07
Figure 6: Replacement strategy comparison p-values.
12

---

Under review as a conference paper at ICLR 2020
Single No Replacement
Single Random
0.0069
0:21
0.8
0.85
0.81
0.35
0.75
0.0057
0:13
0.50
0.85
0.55
0.042
0.0069
0.0057
0.3
0.4
0.81
0.55
0.26
0.30
0.13
0.042
0.26
0.15
(MMmorryees- Mininal Vocaal, 'Single Repascementt
('Mimial Vacabl, "Single No Roplazement)
('Overcomplete "Single Replecement))
('femrryleess Overcomplete, "Sinate No Replarement).
(MMemoyylss ylass Minimaal Vocall 'Single Random")
Random')
('Oventoppee kenee, "Single Random")
I'Memoryless Overcomplesee meleeeeeeeee 'Single Randam)).
('Ainimal
Single Oldest
0,0055
0.024
0.45
0.30
0.0055
0.024
0.68
(Memoryyeess Minimaal Vocao:, 'Single Oldest))
Single Oldest")
('Overcompletee SSingle Oldest')
(MMemryfeess Overcompiete: Siingee Oroest)))
Figure Single Agent model comparison v-vauues.
13

---

Under review as a conference paper at ICLR 2020
Replacement
0.40
0.48
0.32
0.32
0.013
1.1e-00
3.4e-on
0.00027
0.027
0.055
0.16
0.16
0.08
0.08
0.2
3.4e-06
5.053
0,00052
0.005
0.42
(Memrcyy woe Revimat Vooae, Melt ingtaemmeeee
(Niemmal Melt taplecament)
rOversomplee: No Reglacomnne)
["tentoryyee Overrampiene Melh to tegtlemmmne
Wainal Murt Uniform Randon")
("Anmm VocKo: Purt unttom Bundony)
(Memmnyiee Diszoopaa Madt Unterm Bandowe
(Overo rpptee
Multi Epsilon Greedy
0.40
Multi Oldest
0.02
0.073
0.038
0.40
0.00048
0.0064
0.00036
0.02
0.097
0.49
0.00048
0.021
0.92
0.24
0.073
0.097
0.13
0.16
0.0064
0.0022
0.038
0.49
6.13
0..00
(Nemmoyylss Misimal Vocals, 'Multi Epston Growdy))
('Hinimml Epsito Greedy)
['vercompleter ete." Multi tpsiton Greedy))
(Memaryless Overcomp eoe, MMlti Epston Greedy).
Minimal Vecab', "Murd Oldest)
('iniin Vocab: 'Muts Oldest)
('veeccopp "Muid Oldest)
('Wemorytess Overcompiete: "Mund Oldess")
Figure 8: Multi Agent model comparison p-values.
14

---

Under review as a conference paper at ICLR 2020
Algorithm 3: Single Agent Alternate Replacement
Input:
if [e/E| = 0 then
return { A-bot
else
L
return { Q-bot]
Algorithm 4: Multi Agent Uniform Random Replacement
iA ~ 1,,1, N
iQ ~ (1, NQ)
return { AA-bot iA. Q-bot iQ }
Algorithm 5: Multi Agent Epsilon Greedy Replacement
Input: of of v/Vi, [0, 1) (usually 0.2)
~ U(0
if d < then
iA NA) iq NQ)
else
i = argmin, of (unique in our experiments)
iQ argmin, of (unique in our experiments)
return { A-bot i,, Q-bot iq }
Algorithm 6: Multi Agent Oldest Replacement
Input: qvi, a Vi i = 24 (frgmax, a²
ig [argmax, 0 Q return { A-bot iA. Qob }
15


Reviews:
[{"review_id": "rkx4Qvul9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies whether language composition may emerge by partially re-sampling new agents inside a pool of language agents. They set up a consistent experimental setting for assessing compositionality,  assess different agent architectures, e.g., memory vs. memoryless agents,  and explore how the language remains close to each other by re-sampling new agents.\n\nThe paper is well-motivated with substantial background literature on the cognitive science and emergent communication side. The claim is clear, the hypotheses are well-stated, and the experiments look solid (I particularly appreciated the paragraph on shortcoming evaluation). In the end, I enjoy reading the paper despite its density, and I could see that the authors made quite some effort in that direction.\n\nImprovement direction, questions:\n - The authors made were careful not to take ownership of Kottur et al. 's works. Yet, the writing sometimes gives the feeling that their work is solely an extension of Kottur's work, which is not the case. It also gives the feeling that Kottur et al. is the only valid experimental setting, which is not the case (at the authors pointed out in the related work section). Thus, I would recommend to summarise at some point the similarity/difference between the two papers, or at least stop referring the paper every two lines! \n - Replacement strategy: the authors use simple replacement strategies, and conclude that it has little impact. Althought It sounds reasonable in the current setting, the conclusion may be a bit premature. I would recommend to discuss further this result with complementary experiments could be the following: see the impact of epsilon, trying tournament strategies, why 8 populations (this sounds a bit arbitrary too). I would also like to put those observations in perspective with the evolutionary literature [1], and even provide a full paragraph in the related work section.\n - Population dynamics: I am missing a key element in the paper: an analysis of the population dynamics. Although the paper deals with generational transmissions, there are no experiments that analyze the evolution of language generations after generations. Most of the experiments deal with the final convergence state. Again, I would recommend having a look at the evolutionary literature to see which protocol they use to analyze such behavior.\n - Literature side: The authors did an excellent job on the emergent communication and cognitive science side. I think that it is worth extending the comparison further. For instance: \n    * generational transmission can be studied in the light of game theory [2] where compositionality can be seen as a Nash Equilibrium between agent. \n    * generational transmission is a form of dynamic distillation [3]\n    * and evolutionary algorithms!\n - I had some difficulties in understanding Fig4, and the final-take away correctly. Would it be possible to give me one or two examples to correctly parse the table? More generally, I would recommend to add a few lines with some concrete and cherry-picked examples from the experiments to help the reader to have more intuition). \n - In a similar spirit, it is hard to interpret the distance in Figure 3. What would correspond to an increase of 1pt of distance? Having said that, the experiment is sound, and it is insightful.\n - reproducible: having a final table in the array in the appendix could be very helpful \n - crazy experiment: even if I am also a DRL addict, I would be curious to train one of the models with evolutionary algorithms (CMA-ES over parameters, for instance) to assess whether RL has an impact on compositionality (or it is solely the experimental protocol that matters). \n - I may have missed this point, but how many seeds did you use to run your experiments? \n - I may have also missed this point, what is the average length of the dialogue. Can you upload (non-understandable) dialogue example? \n\nLast point... but it does not undermine the soundness of the experimental protocol! \n - In the end, Is 25 accuracy points really compositionality? What would be the score of simple strategies with overcomplete tokens? What is the score of the minimal vocab if we are only correct with one modality, two modalities?\n\n\n\nRemarks:\n - in the introduction, you mention that previous old agents have grounded language, I am not sure whether we can speak of grounded language here, they have a predefined language, but it is not grounded. \n - Please remove the bold sentence in the introduction :) The claim is clear!\n - P11: Alg undefined\n - P12: the legend cannot be read\n \n\nConclusion\nI am familiar with this type of experimental protocols, and I am well aware that they are never-ending works. There are always more experiments to do, more parameters to analyze. The final question is the following: is this paper have enough of these never-ending experiments? I think that this paper is just above this threshold by a short margin, and I vouch for weak accept.\n\nHowever, I am missing at least one dynamic figure (to see the impact of the population along time, which is one of the core concepts of the paper), and there are several links with other ML communities that still have to be highlighted (especially evolutionary algorithms). \nBesides, I somehow feel that the authors pursue two different goals in this paper: they both analyze memory/memoryless complete/overcomplete agents, which is somehow orthogonal to the general transmission hypothesis. Maybe, It would have made more sense to focus on one (or two) of the models and change the experimental setting on them (population size, training time, etc.) \n\nIn the end, I would favor a weak accept. \nI am open to discussion regarding this scoring.\n\n[1] Bäck, Thomas, and Frank Hoffmeister. \"Extended selection mechanisms in genetic algorithms.\" (1991).\n[2] Lanctot, Marc, et al. \"A unified game-theoretic approach to multiagent reinforcement learning.\" Advances in Neural Information Processing Systems. 2017.\n[3] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. \"Distilling the knowledge in a neural network.\" arXiv preprint arXiv:1503.02531 (2015).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkgQjoTV9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple extension to the training of emergent communication protocols in multi-agent settings. \nThe central hypothesis is that learnability will favor more 'compressible' and therefor more compositional languages to emerge. \n\nThis hypothesis is tested by training a population of listens and speakers in an emergent communication task and comparing a number of different strategies for reinitializing agents in the population. Since the new agents start from a random initialization, they provide a learning signal that reinforces protocols which can be learned quickly. \n\nExperiments:\nWhile the experimental results largely confirm the hypothesis there are a few issues:\n-all of the plots show mean and standard deviations, rather than error of the mean. This makes it difficult to understand which differences are statistically significant and which ones are not. \n- The replacement strategy 'epsilon-greedy' replaces agents based on their validation loss, which seems like an unfair advantage. \n- It is currently unclear how much we can learn from Section 5.2: Naturally, agents trained in the same population will develop more similar protocols than those trained independently. This result is obvious and it is unclear whether reinitializing the agents makes any significant difference to the similarity. Again, confidence intervals would help.  \n- The experiments are also extremely toy. I would be more convinced if the authors tested their method on a more challenging task, though I am aware this is a common problem in this field. \n\nNovelty:\nThe single biggest issues with the current form of the paper is the related work section. In this section two papers [1,2] are mentioned as \"concurrent\" when at least one of them [1] has been available online since June 2019. I think it is important to clearly point out the novelty of the current work compared to those two previous papers. \nIn particular [1] seems to be extremely close to the ideas and methods proposed here. Saying that these papers \"confirm the hypothesis\" simply is not enough. \n\nReferences:\n[1]: \"Ease-of-Teaching and Language Structure from Emergent Communication\", Funshan Li et al\n[2]: \"Co-evolution of language and agents in referential games\", Gautier Dagan et al \n\n[Updated score based on the rebuttal]\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeUQfre5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper continues the line of work on emergent compositionality in dialogs, and here it is extended to handle groups of interacting agents that pass language in an evolutionary way from one generation to another. The key idea is that with group of interacting agents, if some agents are replaced with new ones, then the newbies would learn the same language as the group. \n\nGeneral assessment: \n\nThe setup of the paper is is interesting, and the paper covers a lot of ground. The paper makes bold claims like \"cultural transmission induces compositionality\" and \"Variations in replacement strategy tend to not affect performance\". Unfortunately, the paper does not systematic provide experimental evidence to adequately support its claims.\n\nIn my experience, the emergence of compositionality (or lack of) in the setup learned here is very sensitive to various aspects of the learning setup, hence are hard to reproduce. Specifically, they the task-and-talk paper by Kottur et al may yield different conclusions, if parameters of the original experiments are modified, even slightly. The current paper does improve the evaluation protocol of Kottur 2017 by reporting variance over runs, but does not explore the parameter space more systematically, hence I am concerned that it may suffer from similar fragility. \n\nIn this light, significantly more evidence should be provided to convince that the experimental results are stable and can be reproduced.  First, one needs to show the experiments repeated over the full range of setup parameters. Including, the vocabulary sizes V_Q and V_A, the number of tasks, the number of attributes per task, and number of agents etc. Similarly, the current paper introduces a new compositional split, generated in one specific way. The effect of the split on what aspects of language emerge should be studied systematically, instead of using one \"hard\" and one random split. \nThe \"evolutionary\" part has a similar issue. The paper draws conclusions from three anecdotal rules for replacing the population. There is no systematic analysis of the \"evolution\" process, not even studying a range of the parameter epsilon, which is set 0.8 (arbitrarily? to fit the story? we do not know).  Drawing conclusions based on anecdotal evidence is bad scientific practice, that ICLR should discourage.\n\nWhile the ideas in this paper are innovative and exciting, the paper promises much more than its analysis supports, and the paper is not ready for publication.\n\nOther comments: \n-- The paper states that \"darker blue bars\" in figure 2 are higher. The statistical analysis is not well explained, not even in the supplemental, so it is hard to tell which differences are significant. If data is paired, it would be useful to view data as a scatter plot, instead of a barplot which hides the pairing. BTW, p<0.05 is not a \"strong support\", but rather is the most permissive threshold. The results in the supplemental may be stronger. \n-- \"Variations in replacement strategy tend to not affect performance.\" This is a key result of the paper and mush be quantified and analyzed. Authors should define some space of replacements strategies (e.g. in in parametric ways like  how often and how many agents are replaced), then compute performance difference as a function of grid search over the parameter space and show a figure. \n-- \"We stop after 8 generations\". Justify with data. \n-- Other parts of the paper make additional claims, that should similarly be systematically analyzed and supported with data-driven evidence.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the emergence of language in environments that demand agents communicate, focusing on the compositionality of language, and the cultural transmission of language.

Reviewer 1 has several suggestions about new experiments that are possible. The AC does think there is value in many of the suggested experiments, if not to run, then just to acknowledge their possibility and leave for future work. The reviewers also point to some previous work that is very similar.  E.g. "Ease-of-Teaching and Language Structure from Emergent Communication", Funshan Li et al
Decision: Reject