Emergent Communication at Scale
Authors: Rahma Chaabouni
Florian Strub
Florent Altché
Eugene Tarassov
Corentin Tallec
Elnaz Davoodi
Kory Wallace Mathewson
Olivier Tieleman
Angeliki Lazaridou
Bilal Piot
Venue: ICLR 2022 Spotlight (2022)
Keywords: emergent communication
multi-agent reinforcement learning
representation learning

Abstract: Emergent communication aims for a better understanding of human language evolution and building more efficient representations. We posit that reaching these goals will require scaling up, in contrast to a significant amount of literature that focuses on setting up small-scale problems to tease out desired properties of the emergent languages. We focus on three independent aspects to scale up, namely the dataset, task complexity, and population size. We provide a first set of results for large populations solving complex tasks on realistic large-scale datasets, as well as an easy-to-use codebase to enable further experimentation. In more complex tasks and datasets, we find that RL training can become unstable, but responds well to established stabilization techniques.
We also identify the need for a different metric than topographic similarity, which does not correlate with the generalization performances when working with natural images. In this context, we probe ease-of-learnability and transfer methods to assess emergent languages. Finally, we observe that larger populations do not induce robust emergent protocols with high generalization performance, leading us to explore different ways to leverage population, through voting and imitation learning. 

Published as a conference paper at ICLR 2022
EMERGENT COMMUNICATION AT SCALE
Rahma Chaabouni', Florian Strub,, Florent Altché. Corentin Tallec. Eugene Trassov Elnaz Davoodi, Kory Mathewson, Olivier Tieleman, Angeliki Lazaridou, Bilal Piot
DeepMind
ABSTRACT
Emergent communication aims for a better understanding of human language evo- lution and building more efficient representations. We posit that reaching these goals will require scaling up, in contrast to a significant amount of literature that focuses setting up small-scale problems tease out desired properties of the emergent languages. We focus on three independent aspects to scale up, namely the dataset, task complexity, and population size. We provide first of re- sults for large populations solving complex tasks on realistic large-scale datasets, well easy-to-use codebase to enable further experimentation[] In more complex tasks and datasets, we find that RL training can become unstable, but re- sponds well established stabilization techniques. We also identify the need for a different metric than topographic similarity, which does not correlate with the generalization performances when working with natural images. In this context, we probe case-of-leammabile Finally, we observe that larger populations do not induce robust emergent proto- cols with high generalization performance, leading to explore different ways leverage population, through voting and imitation learning.
and transfer methods to assess emergent languages.
INTRODUCTION
Language emergence is at the intersection of cognitive science and machine learning. From a cogni- tive science view, researchers have been looking at artificial agents as another expressive species to shed light on the source of linguistic regularities (Wagner et al. 2003 Guo et al., 2019 Chaabouni jet al. 202T). From a machine learning view, language evolution is deemed as promising direction to shape agents' representation and design interactive Al (Steels 1997 Lazaridou et al. 2020).
Most of the literature in this field relies on different variants of the Lewis game (Lewis) 1969)
There, speaker network must describe an object to listener network, which must then retrieve it among a set of other objects. To solve the game, the two agents need to settle on a communication protocol. While deep agents manage to solve the Lewis game, their communication protocols usually degenerate, lacking core properties of human languages (Bouchacourt & Baroni 2018 Chaabouni et al.) 2019). In response to these findings, several works proposed ad-hoc solutions by constraining the game and agents' capacity (Kottur et al. 2017] Resnick et al.) (020).. While reducing problem complexity is tempting, it can lead to unexpected outcomes and may miss general language emergence behaviors (Hayes) 1985). We take a different route and advocate that scaling up communication games is a prerequisite building interactive AI (Baroni al. 2017 Sutton 2019) or modeling language evolution (Barsalou 2008). Indeed, contrary to other machine learning communities, the emergent communication field mostly relies on small-scale games where only one speaker and one listener communicate about disentangled stimuli, which can hinder the generality of its conclusions. this paper, we focus on three scaling dimensions: the dataset, the task complexity. and the population size. That is, argue that making populations of deep agents communicate about larger and more realistic datasets and solve more complex tasks, is necessary if in the end we want these agents to interact with or if we want to model human communication.
We study three properties of emergent languages: generalization, robustness to input noise, and ease of learning over transfer tasks, analyzing different facets of communication protocols. The proposed
Contributed equally.
Corresponding authors:
Source code: thub. http:/twww....aaae..c...cceeecer

---

Published as a conference paper at ICLR 2022
scaling up remains computationally tractable, most of the experiments can be done within a day on a single GPU. The source code is based on the Jaxline pipeline (Babuschkin et al. 2020
Overall, our experiments provide large spectrum of observations, both positive and negative. First, we found that scaling up the Lewis game quickly entails unstable RL optimization. We propose KL regularization (Geist al., 2019) as an effective way to overcome this issue. Second, that complexifying the task has two positive aspects: it better discriminates between methods and improves the generalization of the learned communication protocol. Third, we note no correlation between generalization and the widely used topographic similarity metric, which suggests that the latter is not adequate to assess the compositionality of the languages in our more complex setup. In- stead, we take inspiration from the self-supervised learning literature and explore transfer learning as a new evaluation metric. Fourth, unlike what was observed in human communication (Gary Lupyan 2010 Raviv et al. (b), we find little to no systematic benefit on emergent languages' properties when increasing the population size. Instead, we propose alternative methods to leverage popula- tions, namely voting and imitation among speakers (Hester al. 2018 Vecerik et al. 2017). In particular, we show that such population dynamics lead to more robust, productive, and in some cases easy-to-learn languages even when compared to our best seed without population, opening up new research opportunities. In the end, we expect that these observations, baselines, and good practices would allow the language emergence community to benefit further from deep RL advances and move the field closer to its goals of improving representations and interactive systems.
observe
2 SETUP
2.1 DISCRIMINATION GAME
The discrimination game involves two players, Speaker and Listener, and is decomposed into three sequential stages. First, Speaker receives a target image x and outputs a message m that is sent to Listener. Second, Listener receives and selects an image å among a set C of different images containing the target image x. The set C is called candidates. Finally, the target image x is revealed to Listener. If Listener selects the target image, then both players receive positive reward of 1, and 0 otherwise. Speaker and Listener are parameterized by a set of parameters 0 and e respectively. The message m(z, 8) = (w2 (x, 0) is a sequence of length T of words we we(r, (I., 0). When no confusion and 0 to simplify notations. A word w, is an element of finite vocabulary set W. For the image selected by Listener ă(Ø m, C), we also omit the dependence on Φi m and C. Finally, the reward for Listener and Speaker is denoted by R(x, .) 1
is possible, omit the dependence
Communication in a population of agents. A straightforward extension of the standard discrimi- nation game is to train population of agents. In this case, given a population of N Speakers and N Listeners, we sample S Speakers and S Listeners with replacement at each training step to construct random pairs; where each Speaker is paired with only one Listener ? All pairs observe the same examples and are trained independently to play the discrimination game as described in the 1-pair setting above. our case, S
so that each agent is trained on average once per step, which allows fair comparison with the baseline of 1-pair. At inference time, we take the first P Speak- ers and Listeners and construct all the possible p2 pairs to compute the accuracy, by averaging the rewards of all pairs as in (Modatch & Abbeel 2018).
Exploiting the population. The training dynamic of the population of agents described above does not take advantage of the diversity of the population. Indeed, we treat all agents in the same way. Yet, one motivation for the effectiveness of the population is the variety between Speakers to invent new structures and between Listeners to avoid over-specialization (Brombam al.] 2015). Prior works have looked at the impact of training agents with different dynamics (e.g., Ren et al., 2019] Guo al. 2019 Cogswell et al. 2019 Li & Bowling, 2019). In this work, we introduce two different ways to exploit the population; on Speakers' side, we add imitation, and on Listeners' side, we allow Listeners to vote to get the final prediction. Mathematical details are found in See. 2.3
Imitation training consists of two different steps. First, as described above, Speakers and Listeners are paired randomly for M interaction steps to learn to communicate. Second, Speakers are trained for 1 imitation step as follows: we select the best Speaker (called "teacher") among K randomly sampled Speakers without replacement, and use the remaining K-1 samples as "students". We then train all students in supervised way to imitate the teacher. We alternate between the interaction
In this work, we only consider populations with the same number of Speakers and Listeners.
2

---

Published as a conference paper at ICLR 2022
and imitation steps. Both M and K are hyper-parameters (see Appendix A.5) Finally, Voting is only used at inference time, irrespective of the training mode. Here, instead of averaging P pairs' rewards to compute accuracy,
now allow P Listeners to vote to get unique prediction. This is inspired by ensemble methods to reduce prediction errors (Polikar 2012). More details on imitation and voting are reported in Sec. 2.3 For completeness, we also reproduce ease-of- -feeehhing protocol as another effective way to exploit population in Appendix B.4 Li & Bowling 2019).
2.2 DATASETS AND NEURAL ARCHITECTURES
Datasets. We use the ImageNet (Deng et al., 2009 Russakovsky et al. 2013), and CclebA datasets (Liu et al., 2015), which respectively contain 1400k and 200k labelled images. CelebA contains 40 binary attributes per image, such as the presence of glasses, blond_haid As the official CelebA splits have non-overlapping identities, it is impossible to perform regular identity classifica- tion on the test/valid sets. We thus construct new split, where we shuffle images have overlapping identities across splits. In both datasets, each image is center-cropped before being processed by ResNet-50 encoder f pretrained on ImageNet with the self-supers supervieee method BYOL (Grill al. 2020 to extract representation of size 2048. Full implementation details are in Appendix A.6
Speaker architecture. Speaker neural network that receives an image x and outputs a mes- sage m = (w, ) It is composed of fixed (non-trainable) image encoder f that transforms x into an embedding fx), before being projected by a state adapter co to an initial state of an LSTM (Hochreiter & Schmidhuber 1997). 2-1,9 (x)). The LSTM receives as input a word embedding 5,, = go (0000) (0,-1) and outputs the next state =r,0 ho(21-1,0. cr,0). The first word em- bedding 60,0 = go(sos) is initialized with start of sequence. The state 2r,0 is then fed to two different heads; the value head vo estimates the value of the expected reward knowing 21,0 and the policy head he computes the probability of the chosen word given 2,,0 Then, a sampling function picks the word u; ((TT))12(,,)) Finally w, is fed back to go to produce the next word and so on until the maximum length T is reached. At training, the word wy is sampled according to the policy whereas it is greedily selected at evaluation. Contrary prior works (e.g.. KKotur et al.) 2017 Resnick et al. 2020). we do not restrict the channel capacity of agents to spur the emergence of systematic languages. Instead, we endow Speaker with message space of size |W| 20 (WW = 20, T = 10), significantly larger than the number of available training examples.
Listener architecture. Listener is neural network that receives Speaker's message m and a set of image candidates C, containing the target image x. It outputs the probability over each image i C of being the target image. Listener composed of an LSTM cell ha, which is initialized to the null vector 2-1.0- The message m is decoded by processing the sequence of word embeddings 6, = [ through the LSTM such as Zt. = (2. ct.o). The state 2T- is then fed to the network Ps to output Pm. (PT ). In parallel, each image 7 is projected through the encoder and the network le to obtain the image embedding L3 = (z)(. Both message and image embeddings are then compared through a score function score の) cos( 2lm I[
Pm.e to
The scores over all images are normalized via a softmax to get probability mp(-Im,C) C). Finally, Listener selects an image by taking the best guess according to T i.e. á E arg maxxec To m, C).
Architecture details, hyper- syper-parameters and graphical descriptions are provided in Appendices A.2 and A.4 All experiments are run over 10 seeds.
2.3
OPTIMIZATION
Speaker training and loss. The goal of Speaker is to optimize the message m sent to Listener such that the expected reward of the game is the highest possible. This can be framed sequential decision making problem where the decision is the choice of each word w.. Therefore, following the policy gradient approach with baseline (Sutton al. 2000), train Speaker's 's network by (i) minimizing a value loss (f) to make the value head ool (2,00) fit the expected reward over a hatch
X: Lv = زेे ExEX ΣT- (R(x, 1, .)
vol vo()),))) (ii) maximizing the expected reward through
minimizing the policy loss I (0) | Exex Et- sg (Rxx,)) () vo(21.0)) (og(To(())22..))
where s.()) is the stop gradient function.
In addition, it is common practice in RL and emergent language literature to minimize entropy
loss L (0) encouraging to explore other choices of words by Speaker (Mnih et al. 2016 Williams
& Peng 1991: Espeholt et al. 2018). Finally, several deep RI. (Schulman et al. 2015 2017) and
3

---

Published as a conference paper at ICLR 2022
theoretical RL papers (Geist et al. 2019: Vieillard et al., 2020a b) argued that minimizing KL loss (f) between the online policy Ty and a target policy "y instead of or in addition to entropy regularization could be beneficial for better final performance well as stabilizing the learning. The policy "T is obtained by doing an exponential moving average of the weights 0 over training:
0 + (1 n)0 + .78 where is the exponential moving average parameter. The KL minimization encourages the online policy to change slowly and smoothly.
To sum up, the speaker training loss [()) on a batch of images X' is: L( = (v) + (0) +
(0) + (lgg(() where ar and 3 are paper-.arrmeters
Listener training and loss. Listener is also trained to maximize the reward but acts by predicting the best guess for the game & E arg max (m)m, C). For each image æ in a batch X. a set of image candidates C is sampled randomly (uniform without replacement over X' {x}) chosen in X.
When C = |. we take C = X. Finally, Listener's s goal is to retrieve x among C, i.e. outputting probability Fg (2) mm, C) = 1 if = 2 and To m, C) 0 otherwise. Therefore,
use multi- class classification loss where the correct class is the index of x in the set of candidates C also called InfoNCE loss (van den Oord et al. 2018: L(Ø)=
Exex log (T (x/mm, C)).
Imitation training among a group of Speakers. In a training imitation step, a group of K speakers among the total population of N speakers is sampled without replacement. Among those K speak- ers, one speaker plays the role of the teacher and K - 1 play the role of the students. To choose the teacher, we compute, for each sampled speaker i, the exponential moving average of the accu- racies over each batch on which the speaker has been trained on, o₁, Then the teacher is simply the speaker with the highest o; For convenience,ww we respectively note 0r and 0g the parameters of the teacher and student. A student 0g is trained on a batch of data X' by imitating the messages of the teacher Or with the following loss: Ly(0s) = 지 Σzex Ei=o log Tee (we(2; 0r) |2c, .e).)). where (6g) is cross-entropy loss encourage student to output the same words as the teacher.
I Listener's voting inference time. At inference time, can use all listeners (o)) of the pop- ulation as an ensemble of networks. Together, they vote for a joint guess & over a set of candidates C of images for a message m coming from speaker. One simple way consists in averaging the score probabilities of the listeners and taking the best guess of this average. Formally, for each lis- tener Φi each message m and batch X, we have the score probability T6. (-mm, C). Then the choice amm, C, (o=)) of the population of listeners for the message m and the set is the best guess of the average distribution which is 2 2m, C, (o)) arg maxsec N Zi= Th. (ø/m,C) C).
More details about how we derive the different losses are found in Appendix A.3
2.4 LANGUAGE PROPERTIES
Generalization. It measures the ability of agents communicate about never-seen inputs. To com- pute generalization, we simply report test accuracy. In the simple case where inputs are constructed by disentangled attributes, test inputs are new combinations of previously seen attributes at training.
Robustness. We report agents' drop of accuracy when faced with noisy inputs relatively to clean inputs test time. To construct noisy stimuli, add Gaussian noise for each batch, with mean 0 and a standard deviation that is half of the standard deviation of the batch. Note that we sample different noises for Speakers' and Listeners' inputs. That is, while Listeners are trained find the exact input of Speakers at train time, they are now required to uncover a different input at eval time.
Topographic Similarity (TopSim). TopSim (Brighton& & Kirby 2006) is used as a proxy for com- positionality (e.g.. Li & Bowling 2019; Lazaridou et al. 2018). and it is widely considered crucial for generalization (Fodor & Lepore| 2002 Marcus 2003). TopSim tests whether close objects in the input space are described by close messages by computing the Spearman correlation between the pairwise distances in the input and message spaces. Following prior works, we use the edit-distance and the cosine-distance in the message and input spaces respectively.
Ease and transfer learning (ETL). It captures how fast and well the emergent language is transmit- ted to new Listeners performing distinct task. It is similar to the ease-of-teaching (Li & Bowling. 2019) to the exception that Listeners are trained to solve a different task from the initial one that the emergent language was optimized for. This task transfer is inspired by the self-supervised linear evaluation protocol (Chen et al., 2020). To measure ETL, we take min(N, 5) fixed Speakers from the population after convergence, and feed their deterministic language to 3 newly initialized Lis-
4

---

Published as a conference paper at ICLR 2022
teners. We argmax from Speakers' distribution to construct the deterministic languages. Finally, show how fast and well new Listeners perform in given task when trained on fixed languages by reporting the training curve for 10k steps. We look at the ease of learning when Listeners are trained to solve harder discrimination, classification, and image reconstruction tasks. Hence, E77. not only captures the generality of the language new Listeners but also its ability to transfer to new tasks.
EXPERIMENTS
3.1 TASK SCALING UP: INCREASING THE NUMBER OF CANDIDATES
In the emergent communication literature, agents are typically required to discriminate between less than 20 objects Mu & Goodman 2021 reconstruct hand-crafted attributes (Rita et al.) 2020
Yet, this simple training setting was shown to sometimes lead to degenerate communication proto- cols (Bouchacourt & Baroni, 2018). As highlighted by (Dessi al.) 2021 Guo et al. 2021 the Lewis game is discrete variant of Contrastive Predictive Coding in the self-supervised learning field (van den Oord et al. 2018). There, it was observed that the intermediate representation qual- ity improved when increasing the number of candidates (He et al., 2019). We hence look at the impact of the number of candidates |C] on languages generalization and the technical challenges introduces. In this subsection, we consider the 1-pair setting trained on ImageNet while varying [C].
Scaling up task difficulty requires to carefully tume optimization. We first train agents without KI. regularization (3 () as it is commonly done in the field (see e.g., Strub et al. 2017 Cao (et al., 2018 Lu et al., 2020] among many others). As displayed in Fig. [(a)] the training becomes unstable when increasing the number of candidates from 20 to 1024 candidates. The mean train accuracy drops from 98.9% to 76.0%, and we observe a large variance across seeds. This high variance demonstrates how crucial it is to run multiple seeds to state robust conclusions. A remedy against such instability in RL consists in adding a KL regularisation LgT. loss between the online policy and target policy described in Sec. 2.3 Such solution has yet not been explored in the emergent communication field. We add a KL. regularization with a coefficient ,3 = 0.5 and display the results Fig. [(b) We note that, with this better regularization, 20 and 1024 candidates converge to 99.8% and 98.3 mean accuracy respectively with small variance during training. Adding Lya thus stabilizes the training and leads to better performances. We illustrate further the effect of this regularization across multiple sweeps in Appendix B.3
0.6
Accuracy
6.6
0.6
20
0.0
1524
0.0
1924
50K
100K
150K
200K
2536
300K
505
103K
15
2006
250K
300K
Steps
Steps
(a) KL coefficient& 3 = 0.0
(b) KL coefficient 3 = 0.5
Figure 1: Training curves per task complexity (20 vs. 1024 candidates) w/ and w/o KL regularization on ImageNet. Thin lines are the accuracy of 10 seeds while thick lines represent the mean.
Scaling up task difficulty is necessary generalization. In Fig. 2(a) we train 1 1-pair of agents on different number of candidates ranging from 64 to 1024 and evaluate them on 16 candidates Note that, in the literature, the number of dis- tractors rarely exceeds a dozen both at train and eval times (Mu & Goodman 2021 Li & Bowling, 2019). In such settings, our experiments do not provide any interesting conclusions: all methods are above 99.7% test accuracy and within standard deviation. However, when evaluating the same communication protocols now on harder tasks with 1024 candidates, as shown in Fig. 2(b) we note that these protocols actually different. In particular, agents achieve accuracy of 87.36%, 93.25%. 96.09% when trained on 64, 256, and 1024 candidates respectively, differentiating the var- ious representation's 's generalization capacity. While expected in light of the recent self-supervised results (Chen et al. 2020)- harder training tasks lead to better representation and harder evaluation tasks better discriminate algorithms- these results clearly illustrate how current emergent language
differentiate representations and enhance protocol's
We kept the training batch size 1024, and only varied the number of candidates in the InfoNCE loss.
5

---

Published as a conference paper at ICLR 2022
experiments may be ill-scaled to have robust conclusions. We report further experiments in Ap- pendix B.5 In the following, we use 1024 candidates at train and eval to fully leverage our findings.
1.0
Accuracy
5.6
0.0
Num cand dates (rai))
0.4
Num canchsates trar))
15
16
0.2
64
256
256
1024
0.0
1024
100K. 200K 3O0K 40IK SOOK, 6OOK 700K BOOK SOOK
100K 200K.
303K 4O0K DOOK
BOOK,
700K, DOOK BOOK
Steps
Steps
(a) Num candidates (eval): 16
(b) Num candidates at (eval): 1024
Figure 2:
Test accuracy for different [C] at train (lines) and eval (subplot) times on ImageNet. (a) Easy eval task: no differentiation between methods.
(b) Hard eval task: more candidates at training induces higher test accuracy. Shaded region represents the standard deviation across 10 seeds.
3.2 DATASET SCALING UP: RETHINKING EVALUATION
In most emergent communication works, deep agents are situated in simple disentangled environ- ment where objects are one-hot vectors (Kottur et al. 2017 Li & Bowling 2019). As a result, the state space rarely goes beyond few thousands of discrete samples. Besides, such representations are unambiguous as they are inherently compositional. We argue that such environments may be too simplistic to reach complex or diverse communication behaviors. In contrast, the major success stories in machine learning collectively prove the importance of training neural networks on a rich and large amount of data emulate complex distributions (Brown et al. 2020, Krizhevsky al.) 2012 Sutton 2019). Furthermore, the goal is to reproduce human interaction, it is fundamen- tal to incorporate complex stimuli develop advanced concepts language evolution (Miller & Johnson-Laine 1976, Barsalou 2008). In this subsection, we scale up the input space by using large natural image datasets, namely ImageNet and CelebA. In particular, we investigate further emergent language properties in the same setting of 1-pair of agents while varying |C] at train time.
TopSim
ImageNet Image
CelebA
Training
Image
Attributes
C = 16
15.5020.a1
31.92
14.22+1.56
C = 64
19.01 20.30
28.7 72.71..50
12: 12.23
17.0641.09
30.60_1.o4
12.29
1024
16.5221.16
30.21_1.22
13.4920.80
Figure 3: (left) Som (x100) with different representation of the input space (Image and Attributes) for different training difficulties, (C). We do not observe correlation between TopSim and [C], and thus generalization. + denotes standard error of the mean. (right) Example of predicted images by new Listener trained on reconstruction task given message of 1-pair for [C]=1024. Reconstructions are expected to be blurry, as we are regressing the mean of all faces associated to one message.
TopSim may fall short with natural images. We compute TopSim using the pretrained image log- its on ImageNet and CelebA input representation while varying the task complexity train, |C]. Results are provided in Fig. 3 left). We do not observe a consistent pattern between TopSim and [C]. Furthermore, we
non-significant (p-value>00.1 19) Spearman correlation of -0.09 and 0.08 be- tween TopSim and generalization for ImageNet and CelebA respectively. Although pretrained logits are excellent linear features (Grill et al. 2020). they may be compositional. We thus reiterate the evaluation by using the attributes from CelebA as input representation to compute TopSim. In that case too we do not note any significant correlation (p-value=0.13), Similar findings have been observed in (Andreas) 2019 Chaabouni et al.) 2020). This could be due to different reasons. First, agents could be communicating in compositional way, to support good generalization, by encoding unexpected compositional features that are not labeled, like forehead shape or smile angle. That is, numan-labeled features may differ from the agents' ones. Second, TopSim relies on strong assump- tions, such as the chosen distance, and the use of linear correlation. Such hypotheses may not hold when moving from artificial to realistic data. Finally, due to the large channel capacity, agents could be using synonyms to generalize with low Top values. In sum, results demonstrate that

---

Published as a conference paper at ICLR 2022
Table 1: Evaluation of Speakers on multiple settings on CelebA in %. denotes one standard error of the mean. We report final accuracy for all ETL tasks but Recons. where we report the final loss.
Metrics
Generalization
Ease and Transfer Learning
Training
Accuracy f
Discr. t
Identity t
Attributes 1
Recons.
[ 16
04
56.31+1007
15.52
86.9920.10
2448+16
60.19
66.53+1.00
23.12+1.06
10.11
2419412
C = 256
85.03
76.61+1.14
33.72+1.62
88.81 20.11
2355413
[C] = 1024
89.0040.48
83.57
44.00
90.08
2351:114
TopSim is not a good predictor of communication protocol generalization for large-scale settings. Instead, we investigate the information content via ETL.
ETL is a robust metric to evaluate languages with natural images. ETI, contrary to TopSim, evaluates how useful the emergent language is for target task beyond overfitting. As such it is an important metric, since the goal of emergent communication is to endow agents with commu- nication skills, rather than solving specific game. Results are reported in Table [ over distinct tasks on CelebA. We observe that, even though only looks at the information content and not at compositionality. is better predictor for language generalization performances than Top,im, with a strong and significant correlation >0.90 (p-value~O) for all considered tasks. As a visual sanity check, we generate predicted images of the reconstruction ETL task in Fig. rrgght). There, new Listener is trained to minimize an 2 loss with the target images given the precomputed messages. We observe that gender, makeup, and hair color are fairly encoded, while others such as the smile are not. More samples and training details are available in Appendix [B.]] Interestingly, some ETL tasks, like discrimination or reconstruction, do not depend on (human- jereen-)preeefied input representa- tions, and are all more robust to channel capacity and linearity assumptions, as opposed to TopSim, which makes ETL a more general evaluation metric.
Zero-shot dataset transfer highlights emergent protocols' differences and limits. Scaling up to natural image opens a large diversity of visual distributions that we can leverage by using zero- shot dataset transfer (Yogatama et al. 2019 Lambert et al., 2020h. We hence measure how generic and high-level the emergent protocols are. We take agents trained on ImageNet with |C|=1024, and evaluate them on CelebA in zero-shot fashion. There, the mean accuracy dropped from 95.96 to 36.73, and the standard deviation raised from 0.49 to 12.86. Noticeably, the most different agents obtain a zero-shot accuracy of 27.88 and 68.49, while their initial ImageNet accuracy only differed by 0.23. Firstly, such a standard deviation gap suggests that different emergent protocols have emerged between agent pairs. Secondly, the accuracy drop indicates that the emergent protocols remain specific to the initial data-distribution, and no systematic language has yet emerged.
3.3
POPULATION SCALING UP: EXPLOITING THE POPULATION DYNAMIC
The emergent communication literature often considers single pair of agents (Lzaridouu et al.) 2017 Foorster et al. 2016). However, there is evidence in the multi-agent literature that such a set- ting may lead to extreme co-adaptation and overfitting (Lanctot al. 2017). One
consists of sampling agents within a population (Jaderberg et al. 2018). From a linguistic per- spective, different large-scale corpora analyses and human simulations support the importance of the population in shaping our languages (Gary Lupyan, 2010 Bromham et af.] 2015) Raviv et al.) 2019a b). In this context, Raviv al. (2019a) show that larger populations develop more system- atic languages through human experimentation. Nonetheless, a few artificial simulations tentatively consider population sizes up 32 agents, but with mixed results about their advantage (Mordatch & Abbeel 2018 Tieleman et al. 2018 Graesser et al., 2019; Cogswell et al.) 2019). In this part, we consider up to 100 agents. Specifically, we explore the impact of the population size on languages properties when we deal with more complex tasks ([C|=1024 at train and eval) and realistic inputs (CelebA & ImageNet) while varying the population size N E (1, 10, 50) pairs.
The best single-pair seed should be the baseline against population. A few works in the emergent communication framework looked at the benefit of the population by comparing 2. 2.-agent's 's perfor- mances to the 2-agent baseline (Cogswell et al. 2019). However, the former introduces x more parameters, which could be responsible for its slight observed benefit. Here, we consider instead the "best 1 pair" setting to investigate whether it is computationally advantageous to train a population of size rather than training N independent pairs. That is, for a given computational budget, we ask whether it is beneficial to train agents within a population as opposed to N pairs in parallel, as
7

---

Published as a conference paper at ICLR 2022
Table Different language properties on CelebA and ImageNet datasets, in %. For each setting we report the mean over 10 seeds. + denotes standard error of the mean.
CelebA
ImageNet
Setting
Generalization
Robustness
Generalization f
Robustness +
best pair
90.73
35.82
97.55
27.83
1 pair
89.00.0.44
37.90
96.0940.21
18,90
10 pairs
91.0620.22 23
37.56
95.78+0029
14.58
50 pairs
90.69 90.69#0.61
38.87
95.29+40.34
15.21
10 pairs + imitation
91.84 31
35.47
96.79+0.12
13.46
10 pains + imitation + vote
92. 10.1990.30 300
35.21
96.9510.11
13.27
50 pairs + imitation
92.82 02.62
32.69
96.70+o.16
12.92
50 pairs + imitation vote
93. 13 13.13
32.40
96.85+o.15
12.75
also done in (Tieleman et al. 2018). "Best pair" is constructed by selecting the best seed among the 10 seeds of "I pair", based on validation accuracy, offering fairer baseline specifically for the "10 pairs" setting. Results are shown in Table and Fig. 4 We observe different behavior of "best pair" across datasets. Particularly, if it always achieves better generalization and ETL discrimination compared to "I pair", it shows, for ImageNet only, the worst Robusiness and E77. classification. We hypothesize that in that specific case, the emergent protocol overspecializes the task. Still, "best pair" outperforms the standard baseline "I pair" 6/8 times, which makes it a stronger baseline. In the following, we thus always compare the population setting with "best I pair" for a fair evaluation.
Population size does not lead to a systematic advantage. We look at the language generalization and robustness performances when increasing the population size. Results in Table 2 do not show a clear trend between population size and these metrics. For example, for both datasets, "50 pairs" achieves lower Generalization and Robustness than "10 pairs". Furthermore, "best seed" always achieves better Generalization compared to the largest population setting "50 pairs". In Fig. 4 we look at ETI for a further investigation of the languages' properties. CelebA results exhibit a benefit of the population. However, this benefit does not always correlate with population size where "10 pairs" outperforms "50 pairs" on the discrimination task. This non-systematic benefit is also found when experimenting with ImageNet. Again, our results suggest no benefit of the population size in the current setting. We thus explore new approaches to leverage the population dynamics and improve the representation of the emergent protocol in the following.
better use of the population is needed to see larger benefit. As population size does not improve the emergent language in itself, we took advantage of the richness of the population by consider- ing imitation at training and vote at inference. In Table 2 we observe that both imitation and vote introduce systematic improvement compared the standard training better performances compared to "I pair" in all metrics, Moreover, when considering the stronger baseline "best pair", the latter only outperforms the imitation and vote settings 1/4 times (Gen- eralization for ImageNet). However, as noted previously, this case is an over-specialized protocol. Furthermore, if both imitation and vote introduce a systematic improvement of the protocol, their benefit is even more noticeable for distribution-shifeed settings. For example, the imitation for 50 pairs adds a relative improvement of 12.8% in Robusteness compared to 2.3% for Generalization on CelebA. The same observation holds for the vote but with lower degree. Note that if vote induces only a slight benefit, it does not incur any training cost. When considering ETL on discrimination observe that imitation leads to systematic improvement. In CelebA, if the
population and lead to
in Figs. ([a)) standard population was already beneficial, adding the imitation strengthens the results. In Ima- geNet, "10/50 pairs + imitation" perform similarly at the end of training than the overspecialized "best 1 pair" while having more stable optimization and it is considerably better than the standard training w/ and w/o population. Yet, the results are more nuanced when considering ETL over clas- sification in Fig. [d) Specifically, while CelebA experiments suggest an advantage when training a population agents w/o imitation as opposed w/ imitation or "1 pair", this pattern is ab- sent with ImageNet. There all settings with population are on par. In sum, both considered dynamics outperform in most cases the strong "best pair" baseline (7/8 times), highlighting their viability. 4 DISCUSSION AND CONCLUSION
The emergent communication framework has been extensively studied for decades before the suc- cesses of deep RL. In this context, many works have revisited this framework through deep RL settings (e.g.. Kirby 2002 vs. Ren et al. 2019 Myers-Scotton et al. 2002 vs. Graesser et al. 2019 or
8

---

Published as a conference paper at ICLR 2022
0.52
Accurace
Steps
Steps
(a) CelebA Discrimination
(b) CelebA Classification of Identities
0.25
Accuraa
2.90
Accuracy
2.20
0.13
0.80
0.0
0.88
0.5
1009
2000
3020
4000
3200 6000 7000 8000 Stops
2020
1000
2000 3200
4030 5000 6002 7000 8003 5000 Steps
(c) ImageNet Discrimination
(d) ImageNet Classification
best pair
1 pair
10 pairs
50 pairs
10 pairs imitation
50 pairs Imitation
Figure 4: ETL per datasets and tasks. The results are averaged across min(N, 5) Speakers, new-
born Listeners' over 10 different seeds. The shaded region represents the standard deviation.
Batali 1998 vs. Choi al. 2018). However, the experimental settings have barely evolved for twenty years. For instance, Kirby (2002) and Ren et al. (2019) both used binary input vector of size eight, and only a few papers try to go beyond such artificial input spaces or simple tasks (Lazaridou et al,, 2017 Havrylov & Titov 2017 Dessi al. 2021). While computational constraints may have been limiting factor, we show that our setting is possible with widely available hardware in Appendix B.2 For example, Table [ requires approximately 400 hours of compute, i.e. 16 GPUs for a single day. Compared to other communities, this equivalent to medium-scale studies in vision or NLP. We argue that it finally time for the emergent language community scale up!
In this spirit, we start clearing the way by identifying some scaling up challenges: optimization instabilities, ill-adapted metries, lack of population synergy. We show how face these new difficulties by proposing: alternative optimization (KL regularization), new evaluation protocols (ETL, zero-shot dataset transfer, best seed baseline), and new dynamics to leverage populations (imitation, voting). There are different theories about the necessity of complex tasks to model human communication (e.g., (Barrett & Skyrms) 2017) vs. (Bickerton) 2015] Dupoux [018]). Hence, we endorse here Bickerton's 's view and adopt performance-inspired solutions to scale up such KL- regularization and imitation. An interesting future debate is whether our findings could influence the status quo of similar research in human communication.
Although we only examine three scaling up dimensions, many other directions can be pursued the future: using larger different architectures improve agent abilities (Desai & Johnson] 2021 experimenting with multimodal data like video or sounds for more realistic stimuli (Arandjelovic & Zisserman 2017), building symmetric communication channels have emergent dialogues (Gao et al. 22119) Another frontier would be to incorporate interaction within environments to ground language into actions (Bisk al. 2020). As such embodiment is crucial for human language un- derstanding (Harnad) 1990 Barsalou 2008), we may start exploring small grid world (Kajić et al.
2020) before scaling up to 3D-environments (Abramson al. 2020).
9

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
In this paper, we ensure the reproduciblity of the different findings through several (and voluntary redundant) ways, namely:
The task is detailed in Section 2.1 and provide visual sketch in Appendix A.1
We use open-source datasets (ImageNet and CelebA), Also, dataset processing is explained, with pseudo-code to reproduce our splits, Appendix|A..
The speaker and listener architectures are first explained in Section 2.2 we then detail model size values, and provide visual sketches in Appendix A.2 The specifics for image reconstruction for ETL are A.p a pseudo-code cocuddo-oddetooo for the reconstruction head.
The optimization is first described Section 2.3 Appendix A.3
The equations are then fully detailed in
All training and evaluation hyperparameters are listed in Appendix A.4
Through the paper, we voluntary provide training curves, test curves, and final scores to have global view of the training trends.
We listed computation time and memory footprint for multiple experiments and multiple hardware in Appendix B.2
The source code should be released upon paper acceptance.
ACKNOWLEDGEMENT
We would like to thank Will Dabney, Remi Munos, Karl Tuyls. Nathalie Beauguerlange as well as the rest of the Deep Paris team for their continuous support. We would also like to thank Marco Baroni, Eugene Kharitonov, Olivier Pietquin and Mathieu Rita for their discussions and helpful feedback at the different stages of the project. Finally, we thank Alison Reid and Saran Tunyasuvunakool for their help in open-sourcing the codebase.
REFERENCES
Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.0567.. 2020.
Jacob Andreas. Mcasuring compositionality in representation learning. In Proc. of International Conference on Learning Representations (ICLR), 2019.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proc. of the IEFF International Confer- ence on Computer Vision (ICCV), 2017.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci. Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King. Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider. Eren Sezener, Stephen Spencer. Srivatsan Srinivasan. Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL httt:://gi thub. www/deepmmmeeecccc nd
Marco Baroni, Armand Joulin, Allan Jabri, Germàn Kruszewski, Angeliki Lazaridou, Klemen Simonic, and Tomas Mikolov. CommAl: Evaluating the first steps towards a useful general AI. In Proc. of International Conference on Learning Representations (ICLR) Workshop Track, 2017.
Jeffrey A Barrett and Brian Skyrms. Self-assembling games. The British Journal for the Philosophy of Science, 68(2):329-33 353,,,399333000 2017.
Lawrence Barsalou. Grounded cognition. Annual Review of Psychology. 59:617-641 2008.
John Batali. Computational simulations of the emergence of grammar. In James Hurford, Michael Studdert- Kennedy, and Chris Knight (eds.), Approaches the Evolution of Language: Social and Cognitive Bases, pp. 405-426. Cambridge University Press, Cambridge, UK, 1998.
10

---

Published as a conference paper at ICLR 2022
Derek Bickerton. Roots of language. Language Science Press, 2015.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata. Angeliki Lazaridou, Jonathan May. Aleksandr Nisnevich, et al. Experience grounds language. In Proc. of Empirical Methods in Natural Language Processing (EMNLP). 2020.
Diane Bouchacourt and Marco Baroni. How agents things: On visual representations in an emergent language game. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), 2018.
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topo- graphic mappings. Artificial life, 12(2):229-2) 242. 2006.
Lindell Bromham, Xia Hua, Thomas G Fitzpatrick, and Simon J Greenhill. Rate of language evolution is affected by population size. Proc. of the National Academy of Sciences (PNAS), 112(7)20097 20002 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners, arXiv
preprint arXi::2005. 14165, 2020.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Leibo, Karl Tuyls, and Stephen Clark. Emergent commu- nication through negotiation. In Proc. of International Conference on Learning Reprexentations (ICLR), 2018.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2019.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Composi- tionality and generalization in emergent languages. In Proc. of the Association for Computational Linguistics (ACL), 2020.
Rahma Chaahouni, Fugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proc. the National Academy of Sciences (PNAS), 118(12). 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for con- trastive learning visual representations. In Proc. of International Conference on Machine Learning (ICML), 2020.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas Compositional obverter communication learning from raw visual input. In Proc. of International Conference on Learning Representations (ICLR), 2018.
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of compositional language with deep generational transmission. arXiv preprint (XXX::1904009000 2019.
Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei. ImageNet: A Large-Scale Hierarchical Image
Database. In Proc. of Conference on Computer Vision and Puttern Recognition (CVPR), 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proc. of Conference on Computer Vision and Pultern Recognition (CVPR), 2021.
Roberto Dessi, Eugene Kharitonov, Marco Baroni. Interpretable agent communication from scratch (with a generic visual processor emerging on the side). In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2021.
Emmanuel Dupoux. Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173:43-59, 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proc. of International Conference on Machine Learning (ICML), 2018.
Jerry Fodor and Ernest Lepore. The Compositionality Papers. Oxford University Press, Oxford, UK, 2002.
Jakob Foerster, loannis Alexandros Assacl, Nando de Freitas, and Shimon Whiteson. Learning to communi- cate with deep multi-agent reinforcement learning. In Proc. of Advances in Neural Information Processing Systems (Neur[PS), 2016.
Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational Al: Question answering. tosk-oriented dialogues and social chatbots. Now Foundations and Trends, 2019.
11

---

Published as a conference paper at ICLR 2022
Rick Dale Gary Lupyan. Language structure is partly determined by social structure. PLoS ONE 5, 2010.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In Proc. of International Conference on Machine Learning (ICML), 2019.
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communi- cation games. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemend, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires. Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Proc. of Advunces in Neural Information Processing Systems (NeurIPS), 2020.
Shangmin Guo, Ren, Serhii Havrylov, Stella Frank, Ivan Titov, Kenny Smith. The emergence of compositional languages for numeric concepts through iterated learning in neural agents. arXiv preprint arXiv:1910.05291. 2019.
Shangmin Guo, Yi Ren, Kory Mathewson, Simon Kirby, Stefano V Albrecht, and Kenny Smith. Expressivity of emergent language is trade-off between contextual complexity and unpredictability. arXiv preprint 1xXiv:2006003922 2021.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlineur Phenomena, 42(1-3):335 346, 1990.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2017.
Patrick J. Hayes. The second naive physics manifesto. In R. Shaw & J. Bransford (ed.), Formal Theories of the Commonsense World, 1985.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, al. Deep q-leaming from demonstrations, In Proc. of Conference on Artificial Intelligence (AAAI), 2018.
Sepp Hochreiter Jürgen Schmidhuber. Long short-term memory. Neural Computation, 175:1735-) 1997.
Max Jaderberg, Czarnecki, Dunning. Marris, G Lever, Castaneda, al. Human-level perfor- mance in first-person multiplayer games with population- deep reinforcement learning. arXiv preprint 1807.01 2018.
Ivana Kajić, Eser Aygün, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. arXiv preprint arXiv:2004.01097, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of International Conference on Learning Representations (ICLR), 2015.
Simon Kirby. Natural language from artificial life. Artificial life, 8(2): 2002.
Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution language. Current Opinion in Neurobiology, 28: 18.1001144 2014.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2012.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Mare Lanctot. Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Proc. of Advances Neural Information Processing Systems (NeurIPS), 2017.
12

---

Published as a conference paper at ICLR 2022
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In Proc. of International Conference on Learning Representations (ICLR), 2017.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic commu- nication from referential games with symbolic and pixel input. In Proc. of International Conference on Learning Representations (ICLR). 2018.
Angeliki Lazaridou, Anna Polapenko, and Olivier Tieleman. Multi-agent communication meets natural lan- guage: Synergies between functional and structural language learning. In Proc. of the Association for Com- putational Linguistics (ACL), 2020.
David Lewis. Convention. Harvard University Press, Cambridge, MA, 1969.
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. of the IEEE International Conference on Computer Vision (ICCV), 2015.
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. Countering language drift with seeded iterated learning. In Proc. of International Conference on Machine Learning (ICML), 2020.
Gary Marcus. The Algebraic Mind. MIT Press, Cambridge, MA, 2003.
George Miller Philip N Johnson-Laird. Language and perception. Havard University Press, 1976.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement Icarning. In Proc. of International Conference on Machine Learning (ICML), 2016.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proc. of Conference on Artificial Intelligence (AAAI), 2018.
Jesse Mu and Noah Goodman. Emergent communication of generalizations. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2021.
Carol Myers-Scotton et al. Contact linguistics: Bilingual encounters and grammatical outcomes. Oxford University Press on Demand, 2002.
Robi Polikar. Ensemble learning. In Ensemble machine learning, pp. 1-34. Springer, 2012.
Alce Radford, Luke Metz, and Soumith Chintala. Unsupervised representation Icarnin with deep convolutional generative adversarial networks. In Proc. of International Conference on Learning Representations (ICLR), 2016.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Larger communities create more systematic languages. Pro- ceedings of the Royal Society B, 286( (977) 2019a.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. Compositional structure can emerge without generational trans- mission. Cognition, 182:151-166 164, 2019b.
Yi Ren, Shangmin Guo, Serhii Havrylov, Shay Cohen, and Simon Kirby. Enhance the compositionality of emer- gent language by iterated learning. In Proc. the NeurlPS Emergent Communication Workshop (EmeCom), 2019.
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew Dai, and Kyunghyun Cho. Capacity, handwidth, and compositionality in emergent language learning. Proc. of Autonomous Agents and Multiagent Systems (AAMAS), 2020.
Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "LazImpa": Lazy and Impatient neural agents learn to communicate efficiently. In Proc. of the Association for Computational Linguistics (ACL., 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision. 115(3):211-255) 2015. doi: 10.1007/s1 1
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy opti- mization. In Proc. of International Conference on Machine Learning (ICMI.), 2015.
13

---

Published as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634, 2017.
Luc Steels. The synthetic modeling of language origins. Evolution of communication, 1(1):1-34, 1997.
Florian Strub, Harm de Vries, Jérémie Mary, Bilal Piot, Aaron C Courville, and Olivier Pietquin. End-to- end optimization of goal-driven and visually grounded dialogue systems. In Proc. of International Joint Conference on Artificial Intelligence (ICCII) 2017.
Rich Sutton. The Biller Lesxon, 2019.
Bitterlesscn htm1
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour, Policy gradient methods for reinforcement learning with function approximation. Proc. of Advances in Neural Information Processing Systems (NIPS), 2000.
Olivier Tieleman, Angeliki Lazaridou, Shibl Mourad, Charles Blundell, Doina Precup. Shaping represen- tations through communication. arXiv preprint arXiv: 2018.
Aäron van den Oord, Yazhe Li. Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.0311 2018.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint wtXiv:1707.08877 2017.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Kyle Wagner, James A Reggia, Juan Uriagereka, and Gerald S Wilkinson. Progress in the simulation of emer- gent communication and language. Adaptive Behavior, 11(1):37 69, 2003.
Ronald J Williams and Jing Peng. Function optimization using connectionisl comeetionnsstriiiffrrreeen learning algorithms. Connection Science, 3(3)2241-268, 1991.
Dani Yogatama. Cyprien de Masson d' Autume, Jerome Connor, Tomas Kocisky. Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou. Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating general linguistic intelligence, arXiv preprint arXiv:1901.11377 2019.
14

---

Published as a conference paper at ICLR 2022
A
ADDITIONAL SETTING DETAILS
A.1 DISCRIMINATION GAME
C
Listenor
T
C
m
Speaker
Spaaker
T
2 -= 1
Figure 5: Example of a discrimination game on Imagenet with a set of 4 candidates. In this specific instance, Listener does not select an image & that is identical the target image received by Speaker. Therefore the reward received by both player R(x, 2) la-a will be 0.
A.2 ARCHITECTURE DETAILS
Speaker The speaker's network architecture is composed of several components to transform the target image æ into message m (w:)
The encoder f is a fixed Resnet-50 architecture that has been previously trained Ima- genet with the BYOL algorithm. The resulting embedding fx) is of size 2048.
The RNN he used is an LSTM of hidden size 256. Therefore the core state 2 is of size 512.
The core-state adapter do a linear layer with input size 2048 and an output size of 512 that allows transform the embedding f(z) into an appropriate core state 2-1,9 ca( (x)). We split 2-1,0 into two equal parts to obtain the initial hidden state 2., and the initial cell state 2,,-1.0-
The word embedder go associates to each discrete symbols in W (sos} an embedding of size 10.
The value head vy first selects the hidden part 2h.0,0 of the core state 2,,8 and then applies linear layer of output size 1.
The policy head me first selects the hidden part 2h,1,0 of the state 2,,8 and then applies a linear layer of output size [W] to obtain logits 4e Then softmax layer transforms the logits into the output distribution m=(-120,0)
The listener's network architecture is composed of several components to transform the message m and the input image ž into score score(m, -i, d):
Listener
The encoder f is a fixed Resnet-50 architecture that has been previously trained on Ima- genet with the BYOL algorithm. The resulting embedding fz)) is of size 2048.
The RNN h& used is an LSTM of hidden size 512. Therefore the core state 21,6 is of size 1024 and is composed of an hidden state th,t, of size 512 and cell state Ze,t, of size 512.
The word embedder go associates to each discrete symbols in W an embedding of size 10.
The target projection ls is a linear layer with output size 256.
The core-state projection Ps is linear layer with output size 256.
15

---

Published as a conference paper at ICLR 2022
As explained in 2.2 the score function is defined as score 6) cos( Pm..
olpm. l|tala
The scores over all images are normalized via a softmax to get probability To To C*) such that:
exp(score(m.2, 2,
Vī E C, (T( Exec exp(score(m,
Finally, the listener selects an image by taking the best guess according to Th. ic. E arg max mT)
Wo
W/
W2
W
S
S
8
V8
V8
VA
6
V
TØ
C0
hg
ho
ho
ho
90
90
90
90
SOS
Figure 6: Graphical representation of a speaker's architecture that shows how the words (w:)(=) T-1 are computed from the inputs æ.
score(me m, F, ())
cos(
Ps
to
h
h
he
hB
9
9
gΦ
wo
W1
W2
W3
Figure 7: Graphical representation of listener's architecture that shows how the score (m ā, Φ) is computed given a message (w:)T=o and an input image T.
A.3 OPTIMISATION DETAILS
Speaker training and loss The goal of speaker, parameterized by 0, is optimize the message m sent to a listener, parameterized by é. such that the mean reward of the game is the highest possi- ble. This can be framed as sequential decision making problem where the decision is the choice of each word w.. Thus, each word is sampled from parameterized stochastic policy n Te(1, (wx ) k=o) that depends on the image and past words (we ) where for 0 we have (w+) 0. Then, goal to maximize the expected reward J(0, Φ) by finding the best policy ne:
J(0, Φ) Expp [ [(xx)))]]
16

---

Published as a conference paper at ICLR 2022
where the expectation Ba-p is over the dataset of training images and the expectation Ex,, is over all possible sequences (xr, (w)) ) that be generated by my starting from For given image x, we define the value (x) = Em [R(x, a)] as the expected reward for given image, here we left the dependence over the choice of the listener & for ease of notations the speaker cannot
on it. For a parameterized policy we can use the policy gradient theorem to obtain the gradient:
(g) = BED.. J0 Σ ( V, (2)) Og log(TTe (wy [ar (ww)t=))
where (2) - Ex [R(x, (wx)) is the value conditioned on all the information revealed at time t - 1. For our particular choice of speaker network, we encode the policy ne (-|2, (wx) and the value V,-1 (ar) with the policy head (0(-120,0) and the value head (0(20,) respectively. Doing so is legitimate because by construction of our recurrent speaker network the embedding 2,,0 is
k= We train then our speaker network by minimizing a value loss (#) to make 22(21.8) fit V (x)
function of the image and the past words (wx) 1-11
over a batch X' of images:
(0)
1
(v
1
Σ
Σ
(R(x, vo vo(2),)))
X
X
t=0
and a policy loss function (0) to maximise the expected reward:
T-1
(0) = X
Σ Ln (0,z) ZEX
Σ
sg (R(2, 2) - us (2( (og))))))ll(())) (wo (2)))2)))))
FEX 1=
where s-()) is the stop gradient function. If we assume that v=((00 fits perfectly V, (a)), one can easily verify that Ig (0.2)) x) is an unhiased estimate of -0g V (x) therefore 0g L.g (0) is also an unbiased estimate of J o). In addition to following the gradient 0g / (r)). is common practice in RL and emergent language literature to maximize an entropy term encouraging to explore other choices of words by Speaker:
T-11
H((TT)) = Exzup
Brggar
Σ
Hl (wx) (wm))
T
= Bexx
EsT a
E
(Te (uwl, (ww)k=b) log( (wlz, (ww) (w
1=0
In practice, sampled version of the negative entropy T.p (0) is minimized with the speaker network:
(+)
E (
(o (wl z2,0) (w]
X
t=
Recently, several deep RL (Schulman al.] 2015 2017) and theoretical RL papers (Geist et al., 2019 Vieillard et al. 2220aa b) argued that minimizing the KL between the online policy wa and a target policy Væ instead of or in addition entropy regularization could be beneficial for better final performance as well as stabilizing the learning. Generally, "z is an older policy or an exponen- tial moving average past policies. Therefore, we also consider the following KL regularization KL(To, Tp):
KL(To, no) =Eaup
E
E
KI(Ta(-aa, (wx)) To( xx, (we (ww))h-b))
E Boo (1x)) b To ne(wla ( log( To (wlax, (ww)k=b) my (url lr, (wx)
-Eaop
E
Eso
In practice, the policy Ty is obtained by doing an exponential moving average of the weights A over training. Then, sampled version of the KL is minimized Lxa (0) with our specifie speaker network:
T
To 21,0)
Lgg (0)
E
To (u( Em
log(
X
Tz 2,
ZEN
0=
17

---

Published as a conference paper at ICLR 2022
To sum up. the speaker training loss L(0) batch of images X is: L = (v + (0) + alH((0) + (0)=
where a and are hyper-parametens
Listener loss One important detail in Sec. 2.3 that for each image in batch X, image candidates C(r, X)) is sampled randomly (umformm without replacement over X {r}} chosen in X' We omitted the dependencies of C(x, X) on and X in the main text for ease of reading. In addition, as explained in Sec. [.3] the listener loss is multi-class classification loss where the correct class
the index of the set of candidates C. also called InfoNCE loss (van den Oord et al. 2018): ((d)
श EzEx log (To(m,),, C)). This can be rewritten more explicitly:
I(Ø)
1
Σ
log (TT())|m,
X
#EA
1
Σ FEX
log
exp(score(m, x, x))
X
Enece (xp(ecore(m,m).2mmmm I,, 6))
exp(cos(
Pun.y
Σ .EX
log
Pmn.
2
X
E exp( expp(co((
P a
Imitation training among a group of speakers. In training imitation step, a group of K speak- ers among the total population of N speakers is selected. Among those K speakers, one speaker going play the role of the teacher and K choose the teacher among the K speakers, we use metric the exponential moving average of the accuracies over each batch on which a given speaker has been trained on. be more precise, let 0 be the i-th speaker and X' the present batch which 0; has just been trained on, then a; is updated according to the following rule:
going to play the role of the students. To
μa; + +1 p1)
X
Σ PEX
-=
and is the exponential moving average of coefficient of the accuracies over each batch which speaker has been trained. Then the teacher is simply the speaker with the highest a; among the K speakers. Now for convenience, let note 0y the parameters of the teacher and 0g the parameters of a student. A student fg is going to be trained on a batch of data X' by imitating the messages of the teacher Or with the following loss:
(Os)
log Tos (u,(x,0))e z0g.t)-
#EX
The loss Lr(0s) is a cross-entropy loss that encourages the student to output the same words as the teacher.
Optimizer hyper-parameters Each member of the population (listener or speaker) uses its own optimiser. All optimisers are Adam optimisers (Kingma & (015) with the same set of hyper- parameters:
learning rate: le - 4,
31: 0.9,
S2: 0.999,
1e-8.
For the speakers' loss, 0.0001 for the entropy regularization and 0.5 for the KL regularization. See Sec. B.3 for more details about the impact of these parameters.
18

---

Published as a conference paper at ICLR 2022
A.4 HYPER-PARAMEN PARAMETERS
We use the same apper-parameters across the different settings. When experimenting with ImageNet vs. CelebA, only vary the number of maximum steps. Specifically, use 300k maximum steps for CelebA (we already start observing some overfitting with this value) and 900k maximum steps for ImageNet. In all cases, we select the best checkpoint for evaluation with respect to the listener loss at validation time. For robust evaluation, we average the scores over 5 epochs to have different candidates for given target and get score less dependent on the sampling of the candidates. Unless mentioned otherwise, the remaining training hyper-parae parameters are reported in Table
Table 3: Hyper-parameters values across datasets and settings.
Learning rate
Ir
0.0001
Batch training size
X
1024
Number of Candidates
[C]
1024
Number of agent sampled
min(N, 10)
KL coefficient
B
0.5
KL EMA
0.99
Entropy Coefficient
0.0001
Vocabulary size
W
20
Message Length
T
10
Imitation EMA
0.99
ETL learning rate
ETL, Ir
0.001
ETL training batch size
ETL, |X(
4096
ETL training candidates
ETL discr., [C]
4096
A.5 IMITATION HYPER-PARAMETIO
In all our experiments, we perform grid search over the parameters of M (number of interaction steps) and K (number of sampled speakers at the imitation step). For the case of population of N 10, K is chosen from {1, 4, 9} and M from {10, 50, 100} according to the validation accuracy. For N 50, K is selected from {4, 9, 24) and M from {1, 10, 50). Table 4 shows the selected hyper- parametasssssrss (K, .)) for the different settings.
Table 4: Imitation
values chosen according to the best accuracy at validation.
10 pairs + imitation (K, M)
50 pairs + imitation
Dataset
(K, M)
CelebA
(1, 10)
(9, 10)
ImageNet
(I, 100)
(9, 10)
19

---

Published as a conference paper at ICLR 2022
A.6 DATASETS DETAILS
ImageNet ILSVRC2012, also known as ImageNet (Deng et al. 2009; Russakovsky et al. 2015) among the historical largest natural RGB image dataset. It mostly contains images of animals and objects over 1000 labels, e.g., camel, ostrich, hourglass, or bow tie. In our experiments, we use 99% of the official train set for training. i.e., 1300k images, the last 1% of the train set for validation, i.e., 13k images, and the official validation set as our test set (i.c., 50k images).
CelebA CelebA a large natural RGB dataset (Liu et al. 2013), which contains image of celebrity faces over 10,177 identities. Noticeably, each face also contains 40 binary attributes, e.g., glasses and hair. Such attributes are interesting to compute, for instance, language topography. Although there is official CelebA split, there and test set. We thus reshuffle the split as described in Listing [ In the end, we respectively obtain 169,444 training, 16,669 valid, and 16,48; test images.
overlapping identities between the train, validation,
Image Processing In both datasets, we resize images to 256 pixels along the shorter side us- ing bicubic resampling before applying a 224x224 center crop. We normalize the color channels subtracting the mean color and dividing by ImageNet std. We then use the ResNet-50 encoder pre- trained on ImageNet with the self-supervised method BYOL (Grill al. 2020) to extract the final representation of dimension 2048.
Listing 1: CelebA Splits
SPLIT_RATIO
the ratio train: valid 5:1 def celeba splits (data):
"""Split the data in three sets with overlapping identities"""
Initialize variables
valid, image test = U. U. 11
counter_label collections. Counter()
for data in dataset. values ():
label data labell label encode the face identity count counter_label [label]
if count 0 and count % SPLIT.RATIO = 0:
# We equally split valid and test by Even and Odd labels if label % 2 == 0:
image v append data)
else:
image-test. append ( (data)
else:
Image-train append data)
counter label [label] +=
return image_train, image_valid, image_test
20

---

Published as a conference paper at ICLR 2022
Figure 8: Face reconstructions. Left: Randomly sampled input images from the CelebA dataset; Middle: Reconstructions, using messages from model trained with 16 candidates; Right: Recon- structions, using messages from a model trained with 1024 candidates.
B ADDITIONAL RESULTS AND ANALYSIS
B.1 FACE RECONSTRUCTION
To visualize some of the features encapsulated in the messages used by the agent, we propose face reconstruction procedure. For single speaker, parameterized by 0, and given an initial image Xx we produce message m (w:) (:))==² We feed the messages to listener-like architecture, consisting of an embedding layer of size 10 followed by an LSTM with 512 units. We keep the last output of the LSTM, hr-1 as message representation. We then pass the message through a deconvolutional architecture, similar to the one in Radford et al. (2016), but without batch normal- izations. Pseudocode for the architecture provided in Listing The full network is trained by minimizing the 12 loss between the input image and the reconstructed one. To optimize the loss, we use Adam W (Kingma & Ba 2015) with a batch size of 128, learning rate of 3 10-4, 31 = .9, 32 = 99. 10 and weight decay of 0.01. We clip gradients by norm with clipping threshold of 500, and skip gradient updates with norm superior to 2000. In Table we report reconstruction losses for different number of candidates |C]. The reconstruction loss computed
squared ror, summed over both spatial and channel axes. Figure 8 displays examples of input images, and corresponding reconstructions for two discrimination games with different number of candidates.
Optimally, each reconstruction should converge to the average face, given the corresponding mes- sage. The more features message contains, the better reconstructions should be. Note first that reconstructions are far from perfect, and much worse than reconstructions that an auto-encoder trained end to end on images would provide: first, solving the discrimination game does not require fully reconstructing the input image, but only capturing enough features to identify uniquely the image batch of candidates; second, the discrete nature of the message, its limited size, and the fact that it is learnt using reinforcement learning act as strong bottlenecks, that prevents messages from containing all the information about input images.
Nonetheless, as qualitatively shown in Figure g messages contain semantic information about in- puts images. For instance, hair color, gender, or open mouth are mostly preserved throughout the reconstruction process. Other features, such as face orientation, skin tone, or age mostly ignored in messages. Furthermore, both quantitative and qualitative difference are visible when going from low number of discriminators to high number of candidates. Quantitatively, going from 16 to 1024 candidates improves the reconstruction error from 2448 to 2351+14 (Table []. Qualitatively, messages with 1024 candidates seem contain information about presence of eye-glasses (top left image), as well as presence of head cover (bottom right image), that are absent in messages from the Speaker trained with 16 candidates.
21

---

Published as a conference paper at ICLR 2022
Listing 2: Face reconstruction Head
def upsample-2d (x, factor: int 2): bs. height width, channels = x. shape
image. resize
x, (bs, height * factor, width * factor, channels), method= nearest') return
def img reconstruction (x):
"""Take the LSTM output, and turn it into an image.
# Project the flat embedding into 2d tensor of dim 4x4x128
x = mn. Linear (4 * 4 (x8)(x)
x. reshape shape 4, 4, 128))
x = nn (x)
x = upsample (x) (64,
# 8x8
Vvddin=="AALID")x)
X = nn (x)
x = peeample.2d(x: # 16x16
x nn (32,3x3(32, with bias=False, VVdIin=="AAIID"
x = nn (x)
x = upsample_2d (x) # 32x32
x = nn. Conv3x3 (16, with.bias=False, VVddin==\AAUI")())
= nn (x)
upsample (x) # 64x64
X = nn Conv3x3 with.bias=False. padding="VALID")CX
nn. relu (x)
X = nn Conv3x3 (3, with.bias=False. padding="VALID")
return nn tanh (x)
B.2 COMPUTATIONAL REQUIREMENTS
As mentioned in the article, our approach remains computationally tractable with widely available hardware. Table summarizes these requirements to reproduce our experimental setup.
B.3 IMPACT OF THE KI. REGULAARIZATION ON TRAINING STABILITY
We showed in the main paper how training pair of agents solve complex task (a discrimination game with 1024 candidates) becomes unstable when using common optimization algorithms. In this section, we look the training curves when agents
trained with and without KL regularization,
E {10- 4 10 10-3, 10 10 Results are shown with population of size 1 (Figure 9 and 10 (Figure TOp. In both cases, agents are trained solve the complex discrimination task with 1024 candidates.
while varying the entropy coefficient
First, we observe that the lower is, the more crucial applying KL. regularization is. For example, in Figure (a) we go from chaotic setting that converges to an accuracy of ~ 40% when the KL coefficient 3 = 0 to a significantly more stable optimization with almost perfect accuracy when 0.5. Second,
observe that, if the KL regularization is useful for stable optimization, it comes at the cost the rapidity convergence. This is clearly shown in Figures [c)) and 10(c) where we observe that the larger 3 is, the slower the convergence is. In other words, one should select the best optimizing the stability/rapidity trade-off. Third, in the presence of a KL regularization, training stability depends less on the entropy coefficient Indeed, we observe stable and high training
22

---

Published as a conference paper at ICLR 2022
Table 5: Computational requirements for our base setup. "GPU memory" refers the peak GPU
memory usage.
Dataset
Device
Pop. size, N
GPU memory (GiB)
Step time (ms)
Train. time (hours)
0.29
42
10.5
ImageNet
p100
10
0.71
381
95.2
2.63
1887
471.7
0.29
25
6.3
v100
0.71
223
55.7
50
2.63
1089
272.1
0.33
7.4
p100
10
0.75
457
38.1
CelebA
50
2.60
2248
187.4
0.33
107
8.9
v100
10
0.75
293
24.4
50
2.60
1408
117.3
1.0
1.0
0.8
Accuracy
Accuracy
0.1
1.0
0.2
0.1
0.0
1.0
100K
200K
300K
400K
100K
200K
300K
400K
Steps
Steps
a = 0.0001
(b) a 0.001
0.030
0.025
Accuracy
Accuracy
0.020
0.015
0.0
0.5
100K
200K
300K
400K
100K
200K
300K
400K
Steps
Steps
(c) a 0.01
(d) a 0.1
Figure 9: Training accuracy for communicating pair when varying the KL coefficient B. Each sub-figure represents the results for fixed entropy coefficient Thin lines represent the training accuracy curves of different seeds. Thick lines represent the average across 10 seeds.
However, training without KL regularization (blue curves with 3 0) is very sensitive the value of a and the most stable case is observed for 0.01. Still even in this best case without KL regularization, having 3 > 0 useful as shown in Figure II
curves for a (10-4,10~3,10-1 10-
*It is also the case for O. Not shown here.
23

---

Published as a conference paper at ICLR 2022
0.8
Accuracy
0.6
Accuracy
0.6
0.5
0.4
0.4
0.2
02
0.0
1.0
0.0
0.0
100K
200K
300K
400K
100K
200K
300K
400k
Steps
Steps
0.0001
(h) 0.001
1.0
0.8
0.025
Accuracy
0.6
Accuracy
0.4
0.010
0.2
0.0
0.0
100K
200K
300K
400K
100K
200K
300K
400K
Steps
Steps
(c) o 0.01
(d) a 0.1
Figure 10: Training accuracy for 10 communicating pairs when varying the KL coefficient B. Each sub-figure represents the results for a fixed entropy coefficient a. Thin lines represent the training accuracy curves of different seeds. Thick lines represent the average across 10 seeds.
1.0
1.0
0.8
0.8
Accuracy
0.6
Accuracy
0.6
0.4
0.2
(KL, H)
0.2
0.033
(0. 0.01)
0.0
10.5, 0.0001)
0.0
0.0001)
Steps
Steps
(a) pair
(b) 10 pairs
Figure 11: Training accuracy for 1 and 10 pairs. Each sub-figure compares the best setting with no KI. regularization (with entropy coefficient of 0.01) and the selected setting for our experiments, with a KL. coefficient of 0.5 and entropy coefficient of 0.0001. Thin lines represent the training accuracy curves of different seeds. Thick lines represent the average across 10 seeds. In both cases, the setting with the KL regularization, (0.5, 0.0001), exhibits more stable convergence with larger difference population of 10 pairs.
24

---

Published as a conference paper at ICLR 2022
Table Different language properties on CelebA dataset, in %. For each setting we report the mean over 10 seeds. + denotes standard error of the mean.
Setting
Generalization
Robustness
best pair
90.73
35.82
pair
89.00=0.48
37.90
10 pairs
91.06.1.22 1.223
37.56
50 pairs
90.69 10.61
38.87
10 pairs imitation
91.84=10.31
35.47
10 pairs imitation + vote
92.19
35.21
50 pairs imitation
92.82 10.51
32.69
50 pairs imitation vote
93.13 1..50
32.40
pair reset
92.21
18.34
10 pairs + reset
90.55 10.56
35.32
50 pairs reset
91.52 10.42
34.46
B.4 STUDY OF THE RESETTING ON THE CELEBA DATASET
In this work, we argue that training deep agents to communicate in population benefits from ploiting its richness, as opposed to a standard training (Mordatch & Abbeel 2018). In this context, introduced imitation and voting mechanisms. However, other prior works have stipulated sim- ilar argument by focusing instead on the idea of the expressivity/comprrsssillty trade-off (Kirby al. 2014). The latter attests that our language is shaped by the two competing pressures of expressivity and compressibility. In practice, there were different methods to implement this trade- off deep agents communication such as iterated learning (Ren al.) 2019), cultural transmis- sion (Cogswell et al. 2019). or sene-of-teaching (Li & Bowling 2019). In this section, we compare imitation and voting to the case-of-teaching 5 To encourage languages' case-of-teaching, Li Bowl- ing (211)) trained deep agents to solve discrimination task, while periodically resetting listeners. In this section, we study the impact of this baseline on the different emergent languages' properties consider 3 additional settings: (1) "I pair + reset" that consists of resetting the only listener, (2) "10 pairs reset" where we reset randomly selected listener among the 10 available ones, and (3) "50 pairs reset" where reset randomly selected listener among the 50 available ones, In all cases, resetting takes place every 51k steps. Note that if
introduced in the main paper. Specifically,
(1) is identical Li & Bowling (2019) setting, (2) and (3) present 10 and 50 speakers respectively unlike 1 speaker as it is the case in (Li & Bowling 2019).
We observe in Table that resetting has a noticeable benefit only in the case of "1 pair" on the generalization and robustness of the languages. In particular, we note an average Generalization of 89.00% for "1 pair" vs. 92.21 for "1 pair reset" and an average Robustness of 18.34% vs. 35.82% However, there is no systematic improvement when agents are trained in a population, ("10 pairs" vs. "10 pairs + resetting") and ("50 pairs" vs. "50 pairs + resetting"). This is line with Li & Bowling 2019 's results which show that abrupt change during training leads to better results. Still, "I pair + reset" does not systematically improve the emergent languages' properties compared to the population when imitation and voting
play.
However, as mentioned in the original paper, the purpose of resetting is not to boost agents' general- ization robustness, but instead incentivize agents to develop easy-to- transmit languages. In the casy-to-transmit following, we look at ETL shown in Figure 12 Here, adding resetting does not lead to significant change showing curves almost identical to the with standard training in all cases.
Overall, our results suggest that resetting listeners is only beneficial the one-pair agents in some cases. Furthermore, resetting does not induce faster or better easy-to-leamt communication protocols over transfer tasks compared to the standard training of the Lewis game (with or without population).
'ee also considered the setup of Cogswell et al. (2019). However, preliminary results showed systemati- cally worse results compared to the case-of-teaching settings. Our codebase provides both options.
25

---

Published as a conference paper at ICLR 2022
0.92
0.90
0.88
Accuracy
0.86
Accuracy
0.84
0.3
0.80
0.76
0.0
2000
5000 Steps
Steps
(a) Discrimination
(b) Classification of Identities
best 1 pair
50 pairs
1 pair reset
pair
pairs imitation
10 pairs reset
10 pairs
50 pairs + imitation
50 pairs + reset
Figure 12: ETL for the CelebA dataset of the emergent languages for different tasks. The results averaged across the emergent languages of min(5, .N) different Speakers, newborn listeners' seeds, and across the 10 seeds of each setting. The shaded region represents the standard deviation.
B.5 IMPACT OF THE NUMBER OF CANDIDATES AT TRAINING AND EVALUATION TIME
In this part we look the impact task complexity at train (Figure 14 and evaluation (Figure 13) times, by varying the number of candidates (C). The results confirm our findings in the main paper. That is, for fixed train [C], the harder the evaluation task is, the lower the test accuracy is. Still, the task complex enough at train time (e.g.. [C] 1024), agents reach overall higher accuracies for all evaluation settings, confirming the importance of hard training tasks to see the emergence of communication protocols with good generalization performances. Also, similarly our results in the main paper, we observe in Figure 14 that harder evaluation task is necessary to discriminate between the communication protocols.
26

---

Published as a conference paper at ICLR 2022
1.0
1.0
0.8
0.8
Accuracy
Accuracy
Num candidates leval)
Num candidates (eval)
0.4
0.4
64
1024
1024
4096
100K 200K 300K 400K S00K 6OOK 700K BOOK 900K
100K 200K 300K 400K 500K Steps
600K 700K 80OK 900K
Steps
(a) Num candidates at (train):
(b) Num candidates at (train): 64
1.0
Accuracy
Accuracy
0.6
0.6
Num candidates leval)
Mum candidates (eval)
16
256
256
0.0
4096
0.0
4096
Steps
Steps
(c) Num candidates at (train): 256
(d) Num candidates (train): 1024
Test accuracy for different number of candidates at training time (subplot) and at evalu- ation time (lines) on ImageNet. As would expect, given one model trained with [C] candidates, the more complex the evaluation task, the lower is the accuracy.
Figure 13:
27

---

Published as a conference paper at ICLR 2022
0.8
Accuracy
Accuracy
0.6
candidates (train)
0.4
Num candidates Itrain)
64
64
0.2
1024
1024
Steps
Steps
(a) Num candidates at (eval): 16
(b) Num candidates at (eval): 64
1.0
0.8
Accuracy
Accuracy
0.6
0.4
Num candidates (train)
0.4
Num candidates Itrain)
16
0.2
256
256
100K 200K 300K
400K 500K
6OOK 700K BOOK 900K
100K 200K 300K
400K 500K 600K 700K BOOK 900K Steps
Steps
(c) Num candidates (eval): 256
(d) Num candidates at (eval); 1024
0.8
Accuracy
candidates (train)
64
0.0
1024
900K
Steps
(e) Num candidates at (eval): 4096
Figure 14: Test accuracy for different number of candidates at training time (lines) and at evaluation time ImageNet. The more complexify evaluation, the better we discriminate representations.
28

---

Published as a conference paper at ICLR 2022
B.6 CELEBA ATTRIBUTES:
Table 7: CelebA ETL attribute accuracies when varying the number of candidates at pretraining at 10k training steps.
Atribute
[C 16
[C] 64
CC] 256
[C] 1024
Clock Shadow
89.93
90.57
91.46.10.5
92.7610.54
Arched Eyebrows
77. 77.73
79.49
81.35+1.15
83. 83.22.0..8
Attractive
78.74+1.01
80.08
81.25
83.27
Bags Under Eyes
81.8640.85
83.5310.72
84.6640.65
86.51+0.77
Bald
98.03
98.03
98.20+0.25
98.43
Bangs
85.60
86.29
87. 87.05
88.42
Big Lips
77.57 10.57 10.92
78.7310.81
80.31 10.97
82.09
Big Nose
80.58 10.74
82.56
83.7710.77
85.56 10.75
Black Hair
78.2240.83
79.78+1.32
81.32+1.39
83.52+0.93
Blond Hair
88. 88.2312.31
89.42+1.77
90. 90.62+2.07
91.94+1.00
Blurry
95.11
95.40
95.65
96.31
Brown Hair
80.83
81.82+0.81
83.2641.0
84.80
Bushy Eyebrows
86.5340.5
87.28
87.87
89.4610.5
Chubby
94.53
94.89
95.44
96.06
Double Chin
95.7640.31
90.37
96.37 +0.25
96.8640.26
Eyeglasses
94.50+0.53
95.98
98.30+0.95
98.48
Goatee
94.28
94.6640.36
95.00
95.88+0.41
Gray Hair
96.03
96.4510.3
96.81 10.25
97.41
Heavy Makeup
88.08
88.7510.64
89.36 10.65
90.49 10.95
High Cheekbones
77.94+1.87
19.70+1.07
79.98
82.39
Male
93.1540.44
93.82+0.43
94.06+0.55
94.73+0.57
Mouth Slightly Open
71.25+2.1
73.62+1.2
74.77+1.98
77.86+1.53
Mustache
95.91+0.32
96.15
96.54+0.33
97.06
Narrow Eyes
89.12
89.53
89.98 10.53
90.35 90.35.0.45
No Beard
88.48 10.68
89.05
90.21 10.84
91.82 10.80
Oval Face
74.54
76.21 76.21
78.15
80.82
Pale Skin
95.6940.23
95.94+0.21
96. 96.13
96.50+0.31
Pointy Nose
75.61
1..31
78. 78.78+1.22
80.96.0.79
Receding Hairline
42.49
92.88
+0.39
93.95
Rosy Cheeks
93.79
94.17+0.3
94.70+0.45
95.42+20.5
Sideburns
94.80
95.06
95.62
96.33
Smiling
79.45
81.39 1.22
81.34 12.28
83.56
Straight Hair
80.5540.57
81.6440.6
82.95+0.75
84.46.0.77
Wavy Hair
76. 42
40.99
80.2 80.24+1.35
82.22+0.92
Wearing Earrings
83.72 83.72+0.59
84.65
85. 7
87.0 87.08
Wearing Hat
97. 97.07 11.46
97. 97.03
98.83
98.95
Wearing Lipstick
90.96
91.52+0.58
91.87 10.58
92.57
Wearing Necklace
87.93
88.41
89. 80.2
90.37
Wearing Necktie
95.1440.22
95.5610.33
96. 10.10.0.3
96.44
Young
83.45+1.31
84.93
85.79+0.77
87.85
B.7 OUT-OF-DISTRIBUTIOO GENERALIZATION THE OFFICIAL CELEBA SPLIT
In the main paper, we consider a new CelebA split where train and test sets have overlapping iden- tities. This allows us to test in-distribution performances. In this Subsection, we look at out-of- distribution generalization considering the official CelebA split, where train, validation, and test sets include distinct identities. In other words, we now consider harder form of generalization by test- ing agents on never-seen seen identities at training. Results are shown in Table g We observe a similar pattern to the in-distribution results shown in Table 2 That is, do not observe benefit for the population, where "best pair" achieves better unseen-gent generalization than "10 pairs" and "50 pairs". However, when considering imitation and voting, agents develop languages that generalize better
29

---

Published as a conference paper at ICLR 2022
never-seen identities. The setting "50 pairs imitation + vote" attains the best performance, with 93.75% success when communicating about previously unseen identities at training time.
Table 8: Generalization performance on the official CelebA split, in %. this case, we look at out-of-distribution generalization train and test sets contain different identities. For each setting we report the mean over 10 seeds. denotes standard error of the mean.
Setting
best I pair
pair
10 pains
50 pairs
+imitation
imitation+vvotc
+imitation
imitation+vote
Generalization
92.60
90.08
90.4220.58
92.01 20.40
92.37
=0.91
93.41go.1
93. 20.75
30


Reviews:
[{"review_id": "Mf2zDKJw2kW", "reviewer": "Reviewer_HRb2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well-written. The overall motivation of the work is very clear, and the motivation of each experiments are supported by observations from previous work and theories in cognitive science. The findings are novel and interesting, bringing out the need for conducting large-scale experiments. A minor drawback is that the work largely presents quantitative results and lacks qualitative analyses. At a large scale, are there anything particular about the learned communication that makes it generalize better?\n\n(I didn't aim to write such a short review, but this is a standard exploratory paper that scales up a well-established setting. The experiment setups are well-designed and I don't find any ambiguous point that needs clarification. I am personally not strongly interested in this fully RL communication-learning direction, but researchers that have good reasons for pursuing this topic may find the results very interesting.)", "summary_of_the_paper": "**After rebuttal**: I am keeping my score\n\n**Before rebuttal**: The paper conducts a large-scale study on RL-based emergent communication. The scale is extended in three dimensions: task complexity, data naturalness, and population size. Scaling up gives new findings that were not observed in previous (small-scale) work. Important findings are (1) evaluating with a large number of candidates is essential for observing the advantage of training with a large number of candidates (2) when experimenting with real images, TopSim, a widely used metric for measuring compositional, no longer correlates with generalization (3) larger population does not necessarily enhance generalization; more clever leverage of the population is needed. ", "summary_of_the_review": "Overall, I find this work thorough and well-motivated. The results would have significant impact in this area. I recommend acceptance. However, I do not work extensively this area and may not follow the latest work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DIvQhVXgovU", "reviewer": "Reviewer_YDs6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - Emergent communication has classically focused on toy tasks, so a scaling paper is a much-needed contribution\n - Overall the paper is quite clear and well-written, with clear notation\n - Analysis of KL regularization in the emergent communication domain is interesting and important\n - Analysis of harder training tasks is also compelling; I wonder if you also add \"hard negatives\" to make the task more difficult\n - ETL is a useful proposed metric, albeit partially based on existing work\n - Best single-pair seed is indeed the correct metric for population-based work, which is missed in prior work (Cogswell, et al.)\n - Analysis of population size is interesting and a compelling rebuttal to previous work\n\nWeaknesses:\n - Not clear to me that a lack of correlation between TopSim and |C| implies that TopSim is not a useful metric for generalization; although |C| does correlate with generalization, I think the usual argument is that TopSim correlates with compositional generalization to specific types of out-of-domain examples (i.e., those which have combinations of previously seen attributes of training examples). The finding is still interesting, but it might be worth tempering the claim here a bit.\n - Given that the paper argues for scaling up a field, more information about compute usage and hardware would be appreciated", "summary_of_the_paper": "This paper provides an analysis of emergent communication behavior at scale and discusses training techniques to reduce instability during RL optimization.", "summary_of_the_review": "I strongly recommend accepting this paper, given the over-emphasis of toy tasks in the emergent communication community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7wpv-JJSmq", "reviewer": "Reviewer_MhrQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This is a timely paper for a problem worth solving.\n- I think measuring how well an emergent language transfers to distinct tasks is a very useful metric to determine the utility of an emergent language.\n- I appreciate the \"best 1 pair\" baseline against the population results, it is a very sensible baseline to compare population-based methods against.\n- Leveraging population for language emergence is difficult and the presented methods help scale up here while providing some benefits in generalization and robustness.\n\nWeaknesses:\n- Although ETL is a useful metric to understand if a language can transfer to a new task, it is useful to measure and understand the structure in the emergent language itself. As indicated in the paper, topographic similarity is probably not the best metric; what about something like TRE [1]? In simple Lewis games, a high topo sim gives a strong clue about what the structure of the language is, but in this case, I have no idea.\n- The authors construct test sets that have overlapping identities for CelebA. While this makes sense for the ETL experiment they set up, in general, this is ensuring that the train and test splits are more in-distribution, making the evaluation task much easier. One of the most important questions w.r.t. emergent languages is if they can generalize systematically. To this end, I would prefer a systematic split where one can study how the emergent language performs in a systematically out-of-distribution setup. An example of something like this would be including CelebA samples that contain a larger proportion of certain types of correlations (e.g. bald with beard) in the train set but have uncorrelated attributes for distinct identities in the test set.\n\nQuestions:\n- You observe that scaling up the task difficulty entails unstable RL optimization. Did you try other ways to backpropagate the gradients to the speaker such as Gumbel straight-through estimation?\n- What is the significance of \"ease\" in \"Ease and transfer learning\"? Do you measure ease of learning, e.g. through acquisition speed or something else?\n\nSuggestions:\n- Section 1: Typo in the second to last line of Introduction: \"emergent\" -> \"emergence\".\n- I would suggest adding a period (.) at the end of paragraph titles (between the title and the paragraph content).\n\nReferences:  \n[1] Andreas, J., 2019. Measuring compositionality in representation learning. arXiv preprint arXiv:1902.07181.", "summary_of_the_paper": "This paper aims to scale up the usual Lewis game language emergence setup. Instead of simple synthetic objects, the authors consider scaling up to image datasets such as ImageNet and CelebA. Another aspect they scale up is the task difficulty, by increasing the number of candidates the listener has to choose the correct answer from. Third, they also attempt scaling up the population size. To deal with the instability arising from a large number of candidates, the authors use established stabilization techniques such as minimization of KL divergence between the online policy and an exponentially moving average target policy. The paper also introduces a new metric called Ease and transfer learning (ETL) to measure how well an emergent language transfers to a novel task as an alternative to topographic similarity. Finally, the authors explore some techniques such as imitation learning and inference-time voting to try to leverage a population of agents.", "summary_of_the_review": "The commonly used Lewis game setups with synthetic data for language emergence are indeed quite limited for real-world applicability. Thus, scaling up the game for the emergence of communication protocols is an important problem. I think this paper is a great contribution, but I would appreciate some more understanding of the structure within the emergent languages. Alternatively (or additionally), demonstrating an ability to systematically generalize can also be very valuable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0R81hYXD_U", "reviewer": "Reviewer_zVAw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n1. Thorough and balanced discussion of benefits/drawbacks among the scaling dimensions.\n2. Research is very well-situated in the literature.\n3. Experiments are well-motivated, novel to my knowledge, and support the authors' case to scale up emergent communication experiments.\n4. Reads very clearly, especially the model and experiment implementation. The technical implementation details in Section 2 are precise.\n\n**Cons**\n\n1. Questions about statistical rigor, especially concerning few datapoints (1, 3 in additional comments)\n\n**Questions and Additional Comments**\n\n(1) The main paper considers |C| = 16, 64, 256, 1024 (the appendix considers also |C| = 4096) and N = 1, 10, 50. It is hard to justify conclusions such as \"increased task complexity systematically improves generalization and ETL\" and \"there is no consistent pattern between TopSim and |C| with a non-significant Spearman correlation\" with 4 datapoints. Were intermediate datapoints tested but not reported?\n\n(2) In Sections 3.1 and 3.2 (Fig. 2, Table 1), several metrics such as accuracy, generalization, and ETL are shown as functions of |C| (task complexity). However, |C| grows exponentially while the performance improvement in these metrics grows roughly linearly. In Table 2, going from 10 to 50 pairs requires roughly 5x more training hours (max is 20 days), whereas their mean performance may not be significantly different. Do the authors think the cost of scaling up is worth the diminishing returns?\n\n(3) In Section 3.2 (TopSim may fall short with natural images): \"As the latter correlates... we expect to find a correlation between TopSim and |C| by transitivity.\" The authors imply that |C| ~ generalization and TopSim !~ |C| (Fig 3) ---> TopSim !~ generalization. In general, transitivity of correlation is not guaranteed. (For example, take X and Y are independent and X + Y = Z.) I would avoid this argument and replace it by directly computing the correlation between TopSim and generalization accuracy. \n\nIn addition, \"...TopSim and |C| with a ... between TopSim and generalization for ImageNet and CelebA respectively\". I found this sentence hard to parse-- are the reported correlations between TopSim and |C| or TopSim and generalization?\n\n(4) It would improve the focus of the paper to state the work's relationship to the cognitive science and the RL sides of emergent communication more explicitly. After reading the paper, it is clear that the authors focus on the robotics/modelling aspect of emergent communication rather than the evolutionary origins of human language. However, the authors begin by discussing both [cognitive science/evolutionary linguistics] and RL as reasons to scale up. It's not clear that starting with complex tasks is the best model of human language emergence. Work in cognitive science theorizes that from both an evolutionary and a first language acquisition perspective, complex language develops from simple communication such as pointing and pantomiming in simple settings rather than from immediately solving difficult tasks (see Barrett and Skyrms 2017, Deacon 1998, Stern 1974). Given the authors partially motivated their work with researching the origins of human language, it would be nice to see more discussion on how their experiments mirror (or contradict) hypothesized settings of language development. For example, in Section 3.3's voting, this is a technique that improves model performance but is likely not how language originated in humans. Alternatively, this can be avoided if the authors state that their work just focuses on the RL side.", "summary_of_the_paper": "The paper argues for scaling up experiments in emergent communication to better understand the evolution of human language and to construct more efficient representations. To achieve this, the authors demonstrate scaling up along dataset, task complexity, and population size in the context of a multi-agent Lewis game, evaluating the benefits and drawbacks of each using generalization, robustness to input noise, and ETL. They find that (1) KL regularization mitigates unstable optimization when scaling up; (2) increasing task complexity improves generalization and better separates performance of different algorithms; (3) topsim and generalization are uncorrelated, necessitating alternative metrics of compositionality at scale; (4) while scaling up population in itself is not obviously beneficial, imitation learning and voting in larger populations produce more \"robust, productive, and easy-to-learn\" languages. Overall, the paper makes a strong case to scale up emergent communication experiments supported by empirical contributions.", "summary_of_the_review": "**Recommendation**\n\nI recommend to accept this paper based on its valuable experimental results, which demonstrate that scaling up emergent communication can find more efficient and generalizable representations. Supported by empirical contributions, the paper steps beyond toy experiments that are common in emergent communication literature and towards increasingly complex and realistic settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript expands the range of recent work in reinforcement learning for language games to much larger and more realistic datasets. A timely and relevant contribution and one that is well evaluated. Further work in stabilizing RL approaches for such large-scale problems is likely to have other far-reaching consequences. Reviewers were unanimous that this is a strong submission after the author discussion period.
Decision: Accept (Spotlight)