Title: EMERGENCE OF COMPOSITIONAL LANGUAGE WITH DEEP GENERATIONAL TRANSMISSION

Abstract: Recent work has studied the emergence of language among deep reinforcement learning agents that must collaborate to solve a task. Of particular interest are the factors that cause language to be compositional-i.e. express meaning by combining words which themselves have meaning. Evolutionary linguists have found that in addition to structural priors like those already studied in deep learning, the dynamics of transmitting language from generation to generation contribute significantly to the emergence of compositionality. In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language. We show that this implicit cultural transmission encourages the resulting languages to exhibit better compositional generalization.

1. INTRODUCTION:

Compositionality is an important structure of language that reflects a disentangled understanding of the world -enabling the expression of infinitely many concepts using finitely many elements. Agents that have compositional understandings of the world generalize in obviously correct ways even in the face of limited training examples (Lake & Baroni, 2018) . For example, an agent with a compositional understanding of blue squares and purple triangles should also understand purple squares without directly observing any of them. Developing artificial agents that can ground, understand, and produce compositional (and therefore more interpretable) language could greatly improve generalization to new instances and ease human-AI interactions.

In building theories of how compositionality emerges in human languages, work in evolutionary linguistics looks to the process of cultural transmission (Kirby, 2001; Kirby et al., 2008) . Cultural transmission of language occurs when a group of agents pass their language on to a new group of agents, e.g. parents who teach their children to speak as they do. Because this education is incomplete and biased, it allows the language itself to change over time via a process known as cultural evolution. This paradigm (Kirby et al., 2014) explains the emergence of compositionality as a result of expressivity and compressibility -i.e. to be most effective, a language should be expressive enough to differentiate between all possible meanings (e.g., objects) and compressible enough to be learned easily. Work in the evolutionary linguistics community has shown that over multiple 'generations' these competing pressures result in the emergence of compositional languages both in simulation (Kirby, 2001) and with human subjects (Kirby et al., 2008) . These studies aim to understand humans whereas we want to understand and design artificial neural networks.

Approaching the problem from another direction, recent work in AI has studied language emergence in such multi-agent, goal-driven tasks. These works have demonstrated that agent languages will emerge to enable coordination-centric tasks to be solved without direct or even indirect language supervision (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2017) . However, the resulting languages are usually not compositional (Kottur et al., 2017) and are difficult to interpret, even by other machines (Andreas et al., 2017) . Some existing work has studied means to encourage compositional language formation (Mordatch & Abbeel, 2018; Kottur et al., 2017) , but these settings study fixed populations of agents -i.e. examining language within a single generation.

In this work we bridge these two areas -examining the effect of generational cultural transmission on the compositionality of emergent languages in a multi-agent, goal-driven setting. 

! % $ " $ # $ % $" $# $ % ! % ! # ! " (b) Implicit cultural transmission

Figure 1 : We introduce cultural transmission into language emergence between neural agents. The starting point of our study is a goal-oriented dialog task (similar to that of Kottur et al. (2017) ), summarized in Fig. 1a . During learning we periodically replace some agents with new ones (gray agents). These new agents do not know any language, but instead of creating one they learn it from older agents. This creates generations of language that become more compositional over time.

We study this in the context of a cooperative dialog-based reference game involving two agents communicating in discrete symbols (Kottur et al., 2017) ; an example dialog is shown in Fig. 1a . To examine cultural transmission, we extend this setting to a population of agents (Fig. 1b ) and introduce a simple mechanism to induce the expressivity and compressibility pressures inherent in cultural transmission. Specifically, we periodically re-initialize some subset of the agents in the population. In order to perform well at the task, the population's emergent language must be sufficiently expressive to reference all the objects (expressivity) and must be easily learnable by these 'new' agents (compressibility) . The new agents have a randomized language whereas the surviving agents already know a grounded language. This "knowledge gap" creates an implicit 'teaching' setting that is analogous to the explicit transmission stage in models of iterative learning (Kirby, 2001) .

Through our experiments and analysis, we show that periodic agent replacement is an effective way to induce cultural transmission and yields more compositionally generalizable language in our setting. To summarize, our contributions are:

-We propose a method for inducing implicit cultural transmission in neural language models. -We introduce new metrics to measure the similarity between agent languages and verify cultural transmission has occurred as a result of our periodic agent replacement protocol. -We show our cultural transmission procedure induces compositionality in neural language models, going from 13% accuracy on a compositionally novel test set to 46% in the best configuration. Further, we show this is complementary with previous priors which encourage compositionality.

2. TASK & TALK: A TESTBED FOR COMPOSITIONAL LANGUAGE EMERGENCE:

We consider the cooperative Task & Talk reference game introduced in Kottur et al. (2017) . Shown in Fig. 1a , the game is played by two agents -one who observes an attributed object -e.g. (purple, solid, square) -and another who is given a task to retrieve a subset of these attributes over the course of the dialog -e.g. (color,shape). The dialog itself consists of two rounds of agents exchanging single-token utterances from fixed vocabularies. At the end of the dialog, the task-aware agent must report the requested attributes and both agents are rewarded for correct predictions. This causes a language grounded in the objects to emerge because there is no other way to solve the task.

A compositional solution to this task can look like a question-answer style dialog where the taskaware agent queries the other for specific attributes (Fig. 1a ) -e.g. uttering "X" requesting the color to which the other agent replies "1" indicating purple. Importantly, this pattern would persist regardless of the other attribute values of the object (e.g. for all (purple, * , * ) objects). However, as there is no grounding supervision provided, agents must learn to associate specific meanings to specific words and it is unlikely for compositional languages to emerge purely by chance. Given a color task, an agent might use "1" for (purple, solid, square) and then "2" for (purple, solid, circle). It is impossible for other agents to know that "2" means purple without having seen (purple, solid, circle), so compositional language is essential for generalization to compositionally novel instances.

Models. To formalize this setting, let Q-bot and A-bot be agent policies parameterized by neural networks Q and A respectively. At each round t, Q-bot observes the task x Q and it's memory of the dialog so far h t-1 Q and produces a single-token utterance

m t Q ∈ V from the vocabulary V. Function- ally, m t Q , h t Q = Q(m t-1 A , x Q , h t-1 Q ) where m t-1

A is A-bot's reply in the previous round. Likewise, A-bot responds by computing m t A , h t A = A(m t Q , x A , h t-1 A ) where x A is the object instance represented symbolically by concatenating 3 one-hot vectors, one per attribute. After two rounds, Q-bot must respond to the task, predicting the requested attribute pair û = U (x Q , h T Q ) as a function of the task and Q-bot's final memory state. Both agents are rewarded if both attributes are correct (no partial credit). We follow the neural network architectures of Q, A, and U from Kottur et al. (2017) .

Measuring Compositional Generalization. Kottur et al. (2017) generated a synthetic dataset consisting of three attribute types (color, shape, style) each with four values (e.g. red, blue, square, star, dotted, solid, ...) and six tasks, one task for each ordered pair of different attribute types. In total, this results in 64 unique instances and 384 task-instance pairs. To evaluate compositionality, Kottur et al. (2017) held out 12 random instances for testing. Given the closed-world set of instances, these 12 are guaranteed to be never-before-seen triplets of attribute values; however, each individual value has been seen in training in other triplets. As such, accuracy on this set is a measure of compositional generalization. Kottur et al. (2017) Evaluation. In our investigations, we found some shortcomings in the evaluation protocol of Kottur et al. (2017) . First, the authors do not report variance over multiple runs or different random test-sets which we found to be significant. Second, the strategy of randomly selecting the test set can still reward some only partially compositional strategies. For instance, suppose agents develop a language that uses single words to refer to attribute pairs like (red, * , triangle) and (red, filled, * ). Such agents might generalize to an unseen instance (red, filled, triangle) by composing the 'paired' words above instead of disentangling individual attributes.

Shortcomings of:

We make two modifications to address these issues. Our results are reported as means and variances estimated from multiple training runs with different random seeds evaluated with 4-way crossvalidation. We also introduce a harder dataset where instead of withholding random individual instances (e.g., (green,dotted,triangle),. . . ) as in Kottur et al. (2017) , we withhold all instances for a set of attribute pairs (e.g., (green,dotted, * ),(red,solid, * ),. . . ). We will refer to datasets generated in this fashion as novel pair and the original dataset as novel instance. We report on both settings for comparison, but find our new setting to be significantly more challenging in practice -requiring a stricter notion of compositionality that is more closely aligned with human intuitions about these attributes.

3. COMPOSITIONAL LANGUAGE EMERGENCE WITH CULTURAL TRANSMISSION:

In iterative learning models of cultural transmission from evolutionary linguistics, competing pressures towards expressivity and compressibility have been shown to induce compositionality over multiple 'generations' of language transfer (Kirby, 2001; Kirby et al., 2008) . The goal-driven nature of our reference game already encourages expressivity -agents must be able to refer to the objects in order to succeed. To introduce compressibility pressure and parallel literature in evolutionary linguistics, we introduce a population of agents which regularly has members replaced by new agents that lack any understanding of the remaining population's language. As this paradigm lacks explicit teaching steps where new agents are trained to ground existing words, we consider this approach as a means of implicit cultural transmission.

Populations of Agents. We consider a population of Q-bots {Q 1 , . . . , Q N Q } and a population of Abots {A 1 , . . . , A N A } with each agent having a different set of parameters. At each iteration during learning, we sample a random Q-bot-A-bot pair to interact and receive updates -i.e. the red line (2) in Alg. 1. As any Q-bot may be made to communicate with any A-bot, there is pressure for the population to adopt a unified language. Likewise, when an agent is reinitialized it will receive positive reward much more quickly when it happens to use language that its conversational partners understand. Furthermore, 'compressible' languages that are easier to learn will result in greater reward for the population in the face of periodic re-initialization of agents. 

Sample Q-bot i Q from U{1, N Q } and A-bot i A from U{1, N A } 3 for x Q , x A , u in each batch do 4 for dialog rounds t = 1, . . . T do 5 m t Q , h t Q = Q i Q (m t-1 A , x Q , h t-1 Q ) 6 m t A , h t A = A i A (m t-1 Q , x A , h t-1 A ) 7 û = U i Q (x Q , h T Q ) 8

Policy gradient update w.r.t. both Q-bot and A-bot parameters Introducing multiple agents may in itself add compressibility pressure and improve generalizations even without replacement (Raviv et al., 2018) . Agents in a population have to model minor linguistic differences between conversational partners given the same memory capacity. Further, each agent provides another potential language variation that can be mimicked and perpetuated-increasing language diversity early in training. We examine these effects through no-replacement baselines, but find that generational pressure where some agents know less than others can also be important for compositionality in our setting.

Replacement. In order to create a notion of 'generations' we replace agents periodically. Let π be some replacement strategy that returns a subset of the population. Every E epochs, we call π and reinitialize the parameters and optimizers for the corresponding agents (blue lines 9-10 in Alg. 1).

We investigate three settings of π (appendix A.2 for more details):

-Uniform Random. Sample an A-bot and Q-bot from uniform random distributions.

-Epsilon Greedy. With probability 1-ε replace the A-bot and Q-bot with the lowest validation accuracy. We use ε = 0.2 in our experiments. -Oldest. Replace the oldest A-bot and Q-bot, breaking ties with uniform random sampling.

4. EXPERIMENTAL SETTING:

Experimental Setting. We evaluate on both our modified Task & Talk dataset and the original from Kottur et al. (2017) , as described in Section 2. All results are reported as means and variances computed from a total of 16 trials (four random seeds each with 4-way cross-validation). We report accuracy based on Q-bot getting both elements of the task correct -corresponding to the more restrictive "Both" setting from Kottur et al. (2017) . Kottur et al. (2017) examined a series of increasingly restrictive settings in order to study conditions under which compositionality emerges. The primary variables are whether A-bot has memory (ablated by setting h t A =0) and the vocabulary sizes V Q and V A for Q-bot and A-bot respectively. For comparison we also evaluate in these settings:

Minimal Vocab ( V Q =3, V A =4). Memoryless + Minimal Vocab (V Q =3, V A =4, h t A =0), Overcomplete (V Q =V A =64). We also introduce Memo- ryless + Overcomplete (V Q =V A =64, h t A =0

) to complete the cross product of settings and examine the role of memory restriction in overcomplete vocabularies.

The Memoryless + Minimal Vocabulary setting results in the best compositional generalization; however, this is an extreme setting -requiring not only that the minimum number of groundable symbols be known but also that A-bot not be able to remember it's previous utterance. While we do report these settings and see quite large performance gains due to cultural transmission, we are mainly interested in the more realistic Overcomplete setting where a large pool of possible tokens is provided and both dialog agents remember the conversation.

Model and Training Details. Our A-bots and Q-bots have the same architectur as in Kottur et al. (2017) . All agents are trained with E = 25000, a batch size of 1000, 1 and the Adam (Kingma & Ba, 2015) optimizer (one per bot) with learning rate 0.01. In the Multi Agent setting we use Figure 2 : Test set accuracies (with standard deviations) are reported against our new harder dataset using models similar to those in Kottur et al. (2017) . Our variations on cultural transmission (darker blue bars) outperform the baselines where language does not change over generations. N A = N Q = 5. We stop training after 8 generations (199000 epochs Multi Agent; 39000 epochs Single Agent). This differs from Kottur et al. (2017) , which stopped once train accuracy reached 100%. Further, we do not perform negative mining.

Baselines. We consider a set of baseline setting to isolate the effect of our approach.

-Single Agent Populations. We ablate the effect of multi-agent populations by training individual A-bot-Q-bot pairs (i.e. populations with N A = N Q = 1). We apply the uniform random (either A-bot or Q-bot at random) and oldest (alternating between A-bot and Q-bot) replacement strategies to these agents; however, the epsilon greedy strategy is not well-defined here. In this setting we decrease E from 25000 to 5000 to keep the average number of gradient updates for each agent constant with respect to the multi-agent experiments. -No Replacement. We also consider the effect of replacing no agents at all, but still allowing the agents to train for the full 199,000 (39,000) epochs. Improvement over this baseline shows the gains from our replacement strategy under identical computational budgets.

5. RESULTS AND ANALYSIS:



5.1. IMPACT OF CULTURAL TRANSMISSION ON COMPOSITIONAL GENERALIZATION:

Results with standard deviations against our harder dataset are reported in Fig. 2 . We compared methods and models using dependent paired t-tests and reported the resulting p-values in Section A.3. Result on the original Task & Talk dataset are in Section A.1.

Cultural transmission induces compositionality. Our main result is that cultural transmission approaches outperform baselines without cultural transmission. This can be seen by noting that for each model type in Fig. 2 , the 3 darker blue bars (Multi Agent Replacement approaches) are largest.

After running a dependent paired t-test against all pairs of baselines and cultural transmission approaches we find a meaningful difference in all cases (p ≤ 0.05). This is strong support for our claim that our version of cultural transmission encourages compositional language because it causes better generalization to novel compositions of attributes.

Next we go on to discuss some additional trends we hope the community will find useful.

Population dynamics without replacement usually lead to some compositionality. The Multi Agent No Replacement policies usually outperform than the Single Agent No Replacement policies, though the difference isn't very significant in the except in the Overcomplete and Minimal Vocab settings. This agrees with recent work from evolutionary linguistics, where multiple agents can lead to compositionality without generational transmission Raviv et al. (2018) .

Variations in replacement strategy tend to not affect performance. The Multi Agent Uniform Random/Epsilon Greedy/Oldest replacement strategies are not largely or consistently different from one another across model variations. This suggests that while some agent replacement needs to occur, it is not critical whether agents with worse language are replaced or whether there is a pool of similarly typed agents to remember knowledge lost from older generations. The main factor is that new agents learn in the presence of others who already know a language.

Cultural transmission is complementary with other factors that encourage compositionality.

As in Kottur et al. (2017) , we find the Memoryless + Small Vocab model is clearly the best. This agrees with factors noted elsewhere Kottur et al. (2017) ; Mordatch & Abbeel (2018) ; Nowak et al. (2000) and shows how many different factors can affect the emergence of compositionality.

Removing memory makes only minor differences. Removing memory makes no difference (negative or positive) in Single Agent settings, but it can have a relatively small effect in Multi Agent settings, helping Small Vocab models and hurting Overcomplete models. While our approach is complementary with minimizing vocab size to increase compositionality, its makes memory removal less useful. As the Memoryless + Overcomplete setting has not been reported before, these results suggest that the relationship between inter-round memory and compositionality is not clear.

Overall, these results show that adding cultural transmission to neural dialog agents improves the compositional generalization of the languages learned by those agents in a way complementary to other priors. It thereby shows how to transfer the cultural transmission principle from evolutionary linguistics to deep learning.

5.2. IS GENERATIONAL TRANSMISSION OCCURRING?:

Because it is implicit, cultural transmission may not actually be occurring; improvements may be from other sources. How can we measure cultural transmission? We focus on A-bots and take a simple approach. We assume that if two A-bots 'speak the same language' then that language was culturally transmitted. There is a combinatorial explosion of possible languages that could refer to all the objects of interest, so if the words that refer to the same object for two agents are the same then they were very likely transmitted from the other agents, rather than similar languages emerging from scratch just by chance. This leads to a simple approach: consider pairs of bots and see if they say similar things in the same context. If they do, then their language was likely transmitted.

More formally, consider the distribution of tokens A-bot A i might use to describe its object x A when talking to Q-bot Q k : p k,i (m t A |x A ) or p k,i for short. We want to know how similar A i 's language is to that of another A-bot A j . We'll start by comparing those two distributions by computing the KL divergence between them and then taking an average over context (objects, Q-bots, and dialog rounds) to get our pairwise agent language similarity metric D ij :

EQUATION

Taking another average, this time over all pairs of bots (and also random seeds and cross-val folds), gives our final measure of language similarity reported in Fig. 3 .

D = Êi,j s.t. i =j [D ij ]

(2) D is smaller the more similar language is between bots. Note that even though D ij is not symmetric (because KL divergence is not), D is symmetric because it averages over both directions of pairs. We compute D by sampling an empirical distribution over all messages and observations, taking 10 sample dialogues in each possible test state (x A , x Q ) of the world using the final populations of agents as in Fig. 2 . Note that this metric applies to a group of agents, so we measure it for only the Multi Agent settings, including two new baselines colored red in Fig. 3 . The Single Agents Combined baseline trains 4 Single Agent No Replacement models independently then puts them together and computes D for that group. These agents only speak similar languages by chance, so D is high. The Random Initialization baseline evaluates language similarity using newly initialized models. These agents have about a uniform distribution over words at every utterance, so their languages are both very similar and useless. For each model these baselines act like practical (not strict) upper and lower bounds on D, respectively. Fig. 3 shows this language dissimilarity metric for all our settings. As we expect, the paired Single Agents are highly dissimilar compared to agents from Multi Agent populations. Further, all the replacement strategies result in increased language similarity-although the degree of this effect seems dependent on vocabulary setting. This provides some evidence that cultural transmission is occurring in Multi Agent settings and is encouraged by the replacement strategy in our approach. While all Multi Agent settings resulted in language transmission, our replacement strategies results in more compositional languages due to repeated teaching of new generations of agents.

5.3. VISUALIZING EMERGENT LANGUAGES:

In this section we visualize the language learned by agents at various stages of training to reinforce our previous conclusions and build intuition. Each of the three sub-figures in Fig. 4 summarizes all of the conversations between a particular pair of bots for the (shape, color) task. To see how these summaries work, consider Fig. 4a . That sub-figure is divided into a 4 × 4 grid with 4 elements Figure 3 : Do bots in a population learn similar languages? On the y-axis (eq. ( 2)) lower values indicate similar language. Populations evolved with our method speak similar languages, but independently evolved agents do not. Thus our implicit procedure induces cultural transmission.

in each cell, so there is one element for each of the 64 possible objects. For this task, objects in each row of the 4 × 4 grid have the same shape and objects in each column have the same color. To the right of each object are the two tokens A-bot used to respond to Q-bot in the two dialog rounds. Ideally they should indicate the color and shape of the object. Finally, the check-marks or Xs to the right of A-bot's utterances indicate whether Q-bot guessed correctly. 4a to a similar pair of bots from our approach Section 4b we can see that our approach encourages compositional language to emerge. Furthermore, the similarity between Fig. 4b and Fig. 4c suggests language is indeed transmitted in our approach.

From left to right: Fig. 4a summarizes the single pair from a Single Agent No Replacement run (3000 iterations old); Fig. 4b summarizes dialogs between an old Q-bot (about 23000 iterations) and a recently re-initialized A-bot (about 3000 iterations) at the 8th and final generation of a Multi Oldest run; Fig. 4c summarizes dialogs between the same old Q-bot as in Fig. 4b and an old A-bot (13000 iterations) from the same Multi Oldest experiment. Even though the A-bots in Fig. 4a and Fig. 4b have trained for about2 the same number of iterations, the A-bot trained in the presence of other bots which already know a functional language has already learned a somewhat compositional language whereas the Single Agent A-bot has not. Furthermore, by comparing the old A-bot's language Fig. 4c with the new one Fig. 4b we can see that they are extremely similar. They even lead to the same mistakes. This again suggests that language is transmitted between bots, in agreement with our previous experiments.

6. RELATED WORK:

Language Evolution Causes Structure. Researchers have spent decades studying how unique properties of human language like compositionality could have emerged. There is general agreement that people acquire language using a combination of innate cognitive capacity and learning from other language speakers (cultural transmission), with the degree of each being widely disputed Perfors (2002) ; Pinker & Bloom (1990) . Both innate cognitive capacity and specific modern human languages like English co-evolved Briscoe (2000) via biological Pinker & Bloom (1990) and cultural Tomasello (1999) ; Smith (2006) evolution, respectively.

In particular, explanations of how the cultural evolution of languages could cause structure like compositionality are in abundance Nowak & Krakauer (1999) ; Nowak et al. (2000) ; Smith et al. (2003) ; Brighton (2002) ; Vogt (2005) ; Kirby et al. (2014) ; Spike et al. (2017) . An important piece of the explanation of linguistic structure is the iterated learning model Kirby et al. (2014) ; Kirby (2001) ; Kirby et al. (2008) used to motivate our approach. Indeed it shows that cultural transmission causes structure in computational Kirby (2001; 2002) ; Christiansen & Kirby (2003) ; Smith et al. (2003) and human Kirby et al. (2008) While existing work in deep learning has focused on biases that encourage compositionality, it has not considered settings where language is permitted to evolve over generations of agents. We have shown such an approach is viable and even complementary with other approaches.

Language Emergence in Deep Learning. Recent work in deep learning has increasingly focused on multi-agent environments where deep agents learn to accomplish goals (possibly cooperative or competitive) by interacting appropriately with the environment and each other. Some of this work has shown that deep agents will develop their own language where none exists initially if driven by a task which requires communication Foerster et al. (2016) ; Sukhbaatar et al. (2016) ; Lazaridou et al. (2017) . Most relevant is work which focuses on conditions under which compositional language emerges as deep agents learn to cooperate Mordatch & Abbeel (2018) ; Kottur et al. (2017) . Both Mordatch & Abbeel (2018) and Kottur et al. (2017) find that limiting the vocabulary size so that there aren't too many more words than there are objects to refer to encourages compositionality, which follows earlier results in evolutionary linguistics Nowak et al. (2000) . Follow up work has continued to investigate the emergence of compositional language among neural agents, mainly focusing on perceptual as opposed to symbolic input and how the structure of the input relates to the tendency for compositional language to emerge Choi et al. (2018) ; Havrylov & Titov (2017) ; Lazaridou et al. (2018) . Other work has shown that Multi Agent interaction leads to better emergent translation Lee et al. (2018) , but it does not measure compositionality.

Cultural Evolution and Neural Nets. Somewhat recently, Bengio (2012) suggested that culturally transmitted ideas may help in escaping from local minima. Experiments in Gülc ¸ehre & Bengio (2016) support this idea by showing that supervision of intermediate representations allows a more complex toy task to be learned. Unlike our work, these experiments use direct supervision provided by the designed environment rather than indirect and implicit supervision provided by other agents.

Two concurrent works examine the role of periodic agent replacement on language emergencealbeit in different environments. In Li & Bowling (2019) replacement is used to encourage languages to be easy to teach, and this in turn causes compositionality. In Dagan et al. (2019) neural language is transmitted through a bottleneck caused by replacement. The resulting language has increased efficiency and effectiveness, with further results showing that co-evolving the agents themselves with the language amplifies the effect. Both of these works support our central observations.

7. CONCLUSION:

In this work we investigated cultural transmission in deep neural dialog agents, applying it to language emergence. The evolutionary linguistics community has long used cultural transmission to explain how compositional languages could have emerged. The deep learning community, having recently become interested in language emergence, has not investigated that link until now. Instead of explicit models of cultural transmission familiar in evolutionary linguistics, we favor an implicit model where language is transmitted from generation to generation only because it helps agents achieve their goals. We show that this does indeed cause cultural transmission and compositionality.

Future work. While our work used an implicit version of cultural transmission, we are interested in investigating the effect of explicit versions of cultural transmission on language structure. In another direction, cultural transmission may also provide an appropriate prior for neural representations of non-language information.

:

There is 1 batch per epoch because there are only 384 instances (64 objects × 6 tasks).

Due to the stochastic nature of our Multi Agent approach.

Figures and Tables:

$ % learn language from other bots

Training with Replacement and Multiple Agents 1 for epoch e = 1, . . . , N epochs do 2

if e mod E = 0 then 10 Sample replacement set B under policy π and re-initialize all agents in B 11 return all Q-bots and A-bots.

Figure4: Each sub-figure summarizes an A-bot's language, as described in Section 5.3. By comparing the baseline of Fig.4ato a similar pair of bots from our approach Section 4b we can see that our approach encourages compositional language to emerge. Furthermore, the similarity between Fig.4band Fig.4csuggests language is indeed transmitted in our approach.

;Cornish et al. (2009);Scott-Phillips & Kirby (2010) experiments. Even though cultural transmission may aid the emergence of compositionality, recent results in evolutionary linguisticsRaviv et al. (2018) and deep learningKottur et al. (2017);Mordatch & Abbeel (2018) also emphasize other factors.

Peer review (iclr2020):
This paper studies whether language composition may emerge by partially re-sampling new agents inside a pool of language agents. They set up a consistent experimental setting for assessing compositionality, assess different agent architectures, e.g., memory vs. memoryless agents, and explore how the language remains close to each other by re-sampling new agents. The paper is well-motivated with substantial background literature on the cognitive science and emergent communication side. The claim is clear, the hypotheses are well-stated, and the experiments look solid (I particularly appreciated the paragraph on shortcoming evaluation). In the end, I enjoy reading the paper despite its density, and I could see that the authors made quite some effort in that direction. Improvement direction, questions: - The authors made were careful not to take ownership of Kottur et al. *s works. Yet, the writing sometimes gives the feeling that their work is solely an extension of Kottur*s work, which is not the case. It also gives the feeling that Kottur et al. is the only valid experimental setting, which is not the case (at the authors pointed out in the related work section). Thus, I would recommend to summarise at some point the similarity/difference between the two papers, or at least stop referring the paper every two lines! - Replacement strategy: the authors use simple replacement strategies, and conclude that it has little impact. Althought It sounds reasonable in the current setting, the conclusion may be a bit premature. I would recommend to discuss further this result with complementary experiments could be the following: see the impact of epsilon, trying tournament strategies, why 8 populations (this sounds a bit arbitrary too). I would also like to put those observations in perspective with the evolutionary literature [1], and even provide a full paragraph in the related work section. - Population dynamics: I am missing a key element in the paper: an analysis of the population dynamics. Although the paper deals with generational transmissions, there are no experiments that analyze the evolution of language generations after generations. Most of the experiments deal with the final convergence state. Again, I would recommend having a look at the evolutionary literature to see which protocol they use to analyze such behavior. - Literature side: The authors did an excellent job on the emergent communication and cognitive science side. I think that it is worth extending the comparison further. For instance: * generational transmission can be studied in the light of game theory [2] where compositionality can be seen as a Nash Equilibrium between agent. * generational transmission is a form of dynamic distillation [3] * and evolutionary algorithms! - I had some difficulties in understanding Fig4, and the final-take away correctly. Would it be possible to give me one or two examples to correctly parse the table? More generally, I would recommend to add a few lines with some concrete and cherry-picked examples from the experiments to help the reader to have more intuition). - In a similar spirit, it is hard to interpret the distance in Figure 3. What would correspond to an increase of 1pt of distance? Having said that, the experiment is sound, and it is insightful. - reproducible: having a final table in the array in the appendix could be very helpful - crazy experiment: even if I am also a DRL addict, I would be curious to train one of the models with evolutionary algorithms (CMA-ES over parameters, for instance) to assess whether RL has an impact on compositionality (or it is solely the experimental protocol that matters). - I may have missed this point, but how many seeds did you use to run your experiments? - I may have also missed this point, what is the average length of the dialogue. Can you upload (non-understandable) dialogue example? Last point... but it does not undermine the soundness of the experimental protocol! - In the end, Is 25 accuracy points really compositionality? What would be the score of simple strategies with overcomplete tokens? What is the score of the minimal vocab if we are only correct with one modality, two modalities? Remarks: - in the introduction, you mention that previous old agents have grounded language, I am not sure whether we can speak of grounded language here, they have a predefined language, but it is not grounded. - Please remove the bold sentence in the introduction :) The claim is clear! - P11: Alg undefined - P12: the legend cannot be read Conclusion I am familiar with this type of experimental protocols, and I am well aware that they are never-ending works. There are always more experiments to do, more parameters to analyze. The final question is the following: is this paper have enough of these never-ending experiments? I think that this paper is just above this threshold by a short margin, and I vouch for weak accept. However, I am missing at least one dynamic figure (to see the impact of the population along time, which is one of the core concepts of the paper), and there are several links with other ML communities that still have to be highlighted (especially evolutionary algorithms). Besides, I somehow feel that the authors pursue two different goals in this paper: they both analyze memory/memoryless complete/overcomplete agents, which is somehow orthogonal to the general transmission hypothesis. Maybe, It would have made more sense to focus on one (or two) of the models and change the experimental setting on them (population size, training time, etc.) In the end, I would favor a weak accept. I am open to discussion regarding this scoring. [1] Bäck, Thomas, and Frank Hoffmeister. *Extended selection mechanisms in genetic algorithms.* (1991). [2] Lanctot, Marc, et al. *A unified game-theoretic approach to multiagent reinforcement learning.* Advances in Neural Information Processing Systems. 2017. [3] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. *Distilling the knowledge in a neural network.* arXiv preprint arXiv:1503.02531 (2015).