Year: 2022

Abstract:
Emergent communication aims for a better understanding of human language evolution and building more efficient representations. We posit that reaching these goals will require scaling up, in contrast to a significant amount of literature that focuses on setting up small-scale problems to tease out desired properties of the emergent languages. We focus on three independent aspects to scale up, namely the dataset, task complexity, and population size. We provide a first set of results for large populations solving complex tasks on realistic large-scale datasets, as well as an easy-to-use codebase to enable further experimentation. In more complex tasks and datasets, we find that RL training can become unstable, but responds well to established stabilization techniques. We also identify the need for a different metric than topographic similarity, which does not correlate with the generalization performances when working with natural images. In this context, we probe ease-of-learnability and transfer methods to assess emergent languages. Finally, we observe that larger populations do not induce robust emergent protocols with high generalization performance, leading us to explore different ways to leverage population, through voting and imitation learning. source_code

text: Emergent communication aims for a better understanding of human language evolution and building more efficient representations. We posit that reaching these goals will require scaling up, in contrast to a significant amount of literature that focuses on setting up small-scale problems to tease out desired properties of the emergent languages. We focus on three independent aspects to scale up, namely the dataset, task complexity, and population size. We provide a first set of results for large populations solving complex tasks on realistic large-scale datasets, as well as an easy-to-use codebase to enable further experimentation. In more complex tasks and datasets, we find that RL training can become unstable, but responds well to established stabilization techniques. We also identify the need for a different metric than topographic similarity, which does not correlate with the generalization performances when working with natural images. In this context, we probe ease-of-learnability and transfer methods to assess emergent languages. Finally, we observe that larger populations do not induce robust emergent protocols with high generalization performance, leading us to explore different ways to leverage population, through voting and imitation learning. source_code

heading: 1 INTRODUCTION
text: Language emergence is at the intersection of cognitive science and machine learning. From a cognitive science view, researchers have been looking at artificial agents as another expressive species to shed light on the source of linguistic regularities (Wagner et al., 2003; Guo et al., 2019; Chaabouni et al., 2021). From a machine learning view, language evolution is deemed as a promising direction to shape agents’ representation and design interactive AI (Steels, 1997; Lazaridou et al., 2020).
Most of the literature in this field relies on different variants of the Lewis game (Lewis, 1969). There, a speaker network must describe an object to a listener network, which must then retrieve it among a set of other objects. To solve the game, the two agents need to settle on a communication protocol. While deep agents manage to solve the Lewis game, their communication protocols are usually degenerate, lacking core properties of human languages (Bouchacourt & Baroni, 2018; Chaabouni et al., 2019). In response to these findings, several works proposed ad-hoc solutions by constraining the game and agents’ capacity (Kottur et al., 2017; Resnick et al., 2020). While reducing problem complexity is tempting, it can lead to unexpected outcomes and may miss general language emergence behaviors (Hayes, 1985). We take a different route and advocate that scaling up communication games is a prerequisite to building interactive AI (Baroni et al., 2017; Sutton, 2019) or modeling language evolution (Barsalou, 2008). Indeed, contrary to other machine learning communities, the emergent communication field mostly relies on small-scale games where only one speaker and one listener communicate about disentangled stimuli, which can hinder the generality of its conclusions. In this paper, we focus on three scaling dimensions: the dataset, the task complexity, and the population size. That is, we argue that making populations of deep agents communicate about larger and more realistic datasets and solve more complex tasks, is necessary if in the end we want these agents to interact with us or if we want to model human communication.
∗ Contributed equally. Corresponding authors: {rahmac,fstrub,piot}@deepmind.com
We study three properties of emergent languages: generalization, robustness to input noise, and ease of learning over transfer tasks, analyzing different facets of communication protocols. The proposed scaling up remains computationally tractable, as most of the experiments can be done within a day on a single GPU. The source code is based on the Jaxline pipeline (Babuschkin et al., 2020).
Overall, our experiments provide a large spectrum of observations, both positive and negative. First, we found that scaling up the Lewis game quickly entails unstable RL optimization. We propose KL regularization (Geist et al., 2019) as an effective way to overcome this issue. Second, we observe that complexifying the task has two positive aspects: it better discriminates between methods and improves the generalization of the learned communication protocol. Third, we note no correlation between generalization and the widely used topographic similarity metric, which suggests that the latter is not adequate to assess the compositionality of the languages in our more complex setup. Instead, we take inspiration from the self-supervised learning literature and explore transfer learning as a new evaluation metric. Fourth, unlike what was observed in human communication (Gary Lupyan, 2010; Raviv et al., 2019a;b), we find little to no systematic benefit on emergent languages’ properties when increasing the population size. Instead, we propose alternative methods to leverage populations, namely voting and imitation among speakers (Hester et al., 2018; Vecerik et al., 2017). In particular, we show that such population dynamics lead to more robust, productive, and in some cases easy-to-learn languages even when compared to our best seed without population, opening up new research opportunities. In the end, we expect that these observations, baselines, and good practices would allow the language emergence community to benefit further from deep RL advances and move the field closer to its goals of improving representations and interactive systems.

heading: 2 SETUP

heading: 2.1 DISCRIMINATION GAME
text: The discrimination game involves two players, Speaker and Listener, and is decomposed into three sequential stages. First, Speaker receives a target image x and outputs a message m that is sent to Listener. Second, Listener receives m and selects an image x̂ among a set C of different images containing the target image x. The set C is called candidates. Finally, the target image x is revealed to Listener. If Listener selects the target image, then both players receive a positive reward of 1, and 0 otherwise. Speaker and Listener are parameterized by a set of parameters θ and φ respectively. The message m(x, θ) = (wt(x, θ))T−1t=0 is a sequence of length T of words wt(x, θ). When no confusion is possible, we omit the dependence on x and θ to simplify notations. A word wt is an element of a finite vocabulary setW . For the image selected by Listener x̂(φ,m, C), we also omit the dependence on φ, m and C. Finally, the reward for Listener and Speaker is denoted by R(x, x̂) = 1x=x̂. Communication in a population of agents. A straightforward extension of the standard discrimination game is to train a population of agents. In this case, given a population of N Speakers and N Listeners, we sample S Speakers and S Listeners with replacement at each training step to construct S random pairs; where each Speaker is paired with only one Listener.1 All pairs observe the same examples and are trained independently to play the discrimination game as described in the 1-pair setting above. In our case, S = N so that each agent is trained on average once per step, which allows a fair comparison with the baseline of 1-pair. At inference time, we take the first P Speakers and Listeners and construct all the possible P 2 pairs to compute the accuracy, by averaging the rewards of all pairs as in (Mordatch & Abbeel, 2018).
Exploiting the population. The training dynamic of the population of agents described above does not take advantage of the diversity of the population. Indeed, we treat all agents in the same way. Yet, one motivation for the effectiveness of the population is the variety between Speakers to invent new structures and between Listeners to avoid over-specialization (Bromham et al., 2015). Prior works have looked at the impact of training agents with different dynamics (e.g., Ren et al., 2019; Guo et al., 2019; Cogswell et al., 2019; Li & Bowling, 2019). In this work, we introduce two different ways to exploit the population; on Speakers’ side, we add imitation, and on Listeners’ side, we allow Listeners to vote to get the final prediction. Mathematical details are found in Sec. 2.3. Imitation training consists of two different steps. First, as described above, Speakers and Listeners are paired randomly for M interaction steps to learn to communicate. Second, Speakers are trained for 1 imitation step as follows: we select the best Speaker (called “teacher”) among K randomly
1In this work, we only consider populations with the same number of Speakers and Listeners.
sampled Speakers without replacement, and use the remaining K-1 samples as “students”. We then train all students in a supervised way to imitate the teacher. We alternate between the interaction and imitation steps. Both M and K are hyper-parameters (see Appendix A.5). Finally, Voting is only used at inference time, irrespective of the training mode. Here, instead of averaging P 2 pairs’ rewards to compute accuracy, we now allow P Listeners to vote to get a unique prediction. This is inspired by ensemble methods to reduce prediction errors (Polikar, 2012). More details on imitation and voting are reported in Sec. 2.3. For completeness, we also reproduce ease-of-teaching protocol as another effective way to exploit population in Appendix B.4 (Li & Bowling, 2019).

heading: 2.2 DATASETS AND NEURAL ARCHITECTURES
text: Datasets. We use the ImageNet (Deng et al., 2009; Russakovsky et al., 2015), and CelebA datasets (Liu et al., 2015), which respectively contain 1400k and 200k labelled images. CelebA contains 40 binary attributes per image, such as the presence of glasses, blond hair. As the official CelebA splits have non-overlapping identities, it is impossible to perform regular identity classification on the test/valid sets. We thus construct a new split, where we shuffle images to have overlapping identities across splits. In both datasets, each image is center-cropped before being processed by a ResNet-50 encoder f pretrained on ImageNet with the self-supervised method BYOL (Grill et al., 2020) to extract a representation of size 2048. Full implementation details are in Appendix A.6.
Speaker architecture. Speaker is a neural network that receives an image x and outputs a message m = (wt)T−1t=0 . It is composed of a fixed (non-trainable) image encoder f that transforms x into an embedding f(x), before being projected by a state adapter cθ to an initial state of an LSTM (Hochreiter & Schmidhuber, 1997), z−1,θ = cθ(f(x)). The LSTM receives as input a word embedding et,θ = gθ(wt−1) and outputs the next state zt,θ = hθ(zt−1,θ, et,θ). The first word embedding e0,θ = gθ(sos) is initialized with a start of sequence. The state zt,θ is then fed to two different heads; the value head vθ estimates the value of the expected reward knowing zt,θ and the policy head πθ computes the probability of the chosen word given zt,θ. Then, a sampling function s picks the word wt = s(πθ(.|zt,θ)). Finally wt is fed back to gθ to produce the next word and so on until the maximum length T is reached. At training, the word wt is sampled according to the policy whereas it is greedily selected at evaluation. Contrary to prior works (e.g., Kottur et al., 2017; Resnick et al., 2020), we do not restrict the channel capacity of agents to spur the emergence of systematic languages. Instead, we endow Speaker with a message space of size |W |T = 2010 (|W | = 20, T = 10), significantly larger than the number of available training examples. Listener architecture. Listener is a neural network that receives Speaker’s message m and a set of image candidates C, containing the target image x. It outputs the probability over each image x̃ ∈ C of being the target image. Listener is composed of an LSTM cell hφ, which is initialized to the null vector z−1,φ. The message m is decoded by processing the sequence of word embeddings et,φ = gφ(wt) through the LSTM such as zt,φ = hφ(zt−1,φ, et,φ). The state zT−1,φ is then fed to the network pφ to output pm,φ = pφ(zT−1,φ). In parallel, each image x̃ is projected through the encoder f and the network tφ to obtain the image embedding tx̃ = tφ(f(x̃)). Both message and image embeddings are then compared through a score function score(m, x̃, φ) = cos( pm,φ‖pm,φ‖2 , tx̃ ‖tx̃‖2 ). The scores over all images are normalized via a softmax to get a probability πφ(.|m, C). Finally, Listener selects an image by taking the best guess according to πφ, i.e. x̂ ∈ argmaxx̃∈C πφ(x̃|m, C). Architecture details, hyper-parameters and graphical descriptions are provided in Appendices A.2 and A.4. All experiments are run over 10 seeds.

heading: 2.3 OPTIMIZATION
text: Speaker training and loss. The goal of Speaker is to optimize the message m sent to Listener such that the expected reward of the game is the highest possible. This can be framed as a sequential decision making problem where the decision is the choice of each word wt. Therefore, following the policy gradient approach with a baseline (Sutton et al., 2000), we train Speaker’s network by (i) minimizing a value loss LV (θ) to make the value head vθ(zt,θ) fit the expected reward over a batch X : LV (θ) = 1|X | ∑ x∈X ∑T−1 t=0 (R(x, x̂)− vθ(zt,θ)) 2 , (ii) maximizing the expected reward through
minimizing the policy loss Lπ(θ) = 1|X | ∑ x∈X ∑T−1 t=0 sg (R(x, x̂)− vθ(zt,θ)) log(πθ(wt|zt,θ)), where sg(.) is the stop gradient function.
In addition, it is common practice in RL and emergent language literature to minimize an entropy loss LH(θ) encouraging to explore other choices of words by Speaker (Mnih et al., 2016; Williams & Peng, 1991; Espeholt et al., 2018). Finally, several deep RL (Schulman et al., 2015; 2017) and theoretical RL papers (Geist et al., 2019; Vieillard et al., 2020a;b) argued that minimizing a KL loss LKL(θ) between the online policy πθ and a target policy πθ instead of or in addition to entropy regularization could be beneficial for better final performance as well as stabilizing the learning. The policy πθ is obtained by doing an exponential moving average of the weights θ over training: θ ← (1 − η)θ + ηθ where η is the exponential moving average parameter. The KL minimization encourages the online policy to change slowly and smoothly. To sum up, the speaker training loss L(θ) on a batch of images X is: L(θ) = LV (θ) + Lπ(θ) + αLH(θ) + βLKL(θ), where α and β are hyper-parameters.
Listener training and loss. Listener is also trained to maximize the reward but acts by predicting the best guess for the game x̂ ∈ argmaxx̃∈C πφ(x̃|m, C). For each image x in a batch X , a set of image candidates C is sampled randomly (uniform without replacement over X \ {x}) chosen in X . When |C| = |X |, we take C = X . Finally, Listener’s goal is to retrieve x among C, i.e. outputting a probability πφ(x̃|m, C) = 1 if x = x̃ and πφ(x̃|m, C) = 0 otherwise. Therefore, we use a multiclass classification loss where the correct class is the index of x in the set of candidates C also called InfoNCE loss (van den Oord et al., 2018): L(φ) = − 1|X | ∑ x∈X log (πφ(x|m, C)).
Imitation training among a group of Speakers. In a training imitation step, a group ofK speakers among the total population of N speakers is sampled without replacement. Among those K speakers, one speaker plays the role of the teacher and K − 1 play the role of the students. To choose the teacher, we compute, for each sampled speaker i, the exponential moving average of the accuracies over each batch on which the speaker i has been trained on, σi. Then the teacher is simply the speaker with the highest σi. For convenience,we respectively note θT and θS the parameters of the teacher and student. A student θS is trained on a batch of data X by imitating the messages of the teacher θT with the following loss: LI(θS) = − 1|X | ∑ x∈X ∑T−1 t=0 log πθS (wt(x, θT )|x, zθS ,t), where LI(θS) is a cross-entropy loss to encourage a student to output the same words as the teacher.
Listener’s voting at inference time. At inference time, we can use all listeners (φi)N−1i=0 of the population as an ensemble of networks. Together, they vote for a joint guess x̂ over a set of candidates C of images for a message m coming from a speaker. One simple way consists in averaging the score probabilities of the listeners and taking the best guess of this average. Formally, for each listener φi, each message m and batch X , we have the score probability πφi(.|m, C). Then the choice x̂(m, C, (φi)N−1i=0 ) of the population of listeners for the message m and the set C is the best guess of the average distribution which is x̂(m, C, (φi)N−1i=0 ) = argmaxx̃∈C 1N ∑N−1 i=0 πφi(x̃|m, C).
More details about how we derive the different losses are found in Appendix A.3.

heading: 2.4 LANGUAGE PROPERTIES
text: Generalization. It measures the ability of agents to communicate about never-seen inputs. To compute generalization, we simply report test accuracy. In the simple case where inputs are constructed by disentangled attributes, test inputs are new combinations of previously seen attributes at training.
Robustness. We report agents’ drop of accuracy when faced with noisy inputs relatively to clean inputs at test time. To construct noisy stimuli, we add a Gaussian noise for each batch, with mean 0 and a standard deviation that is half of the standard deviation of the batch. Note that we sample different noises for Speakers’ and Listeners’ inputs. That is, while Listeners are trained to find the exact input of Speakers at train time, they are now required to uncover a different input at eval time.
Topographic Similarity (TopSim). TopSim (Brighton & Kirby, 2006) is used as a proxy for compositionality (e.g., Li & Bowling, 2019; Lazaridou et al., 2018), and it is widely considered crucial for generalization (Fodor & Lepore, 2002; Marcus, 2003). TopSim tests whether close objects in the input space are described by close messages by computing the Spearman correlation between the pairwise distances in the input and message spaces. Following prior works, we use the edit-distance and the cosine-distance in the message and input spaces respectively.
Ease and transfer learning (ETL). It captures how fast and well the emergent language is transmitted to new Listeners performing a distinct task. It is similar to the ease-of-teaching (Li & Bowling, 2019) to the exception that Listeners are trained to solve a different task from the initial one that
the emergent language was optimized for. This task transfer is inspired by the self-supervised linear evaluation protocol (Chen et al., 2020). To measure ETL, we take min(N, 5) fixed Speakers from the population after convergence, and feed their deterministic language to 3 newly initialized Listeners. We argmax from Speakers’ distribution to construct the deterministic languages. Finally, we show how fast and well new Listeners perform in a given task when trained on fixed languages by reporting the training curve for 10k steps. We look at the ease of learning when Listeners are trained to solve harder discrimination, classification, and image reconstruction tasks. Hence, ETL not only captures the generality of the language to new Listeners but also its ability to transfer to new tasks.

heading: 3 EXPERIMENTS

heading: 3.1 TASK SCALING UP: INCREASING THE NUMBER OF CANDIDATES
text: In the emergent communication literature, agents are typically required to discriminate between less than 20 objects (Mu & Goodman, 2021) or reconstruct hand-crafted attributes (Rita et al., 2020). Yet, this simple training setting was shown to sometimes lead to degenerate communication protocols (Bouchacourt & Baroni, 2018). As highlighted by (Dessı̀ et al., 2021; Guo et al., 2021), the Lewis game is a discrete variant of Contrastive Predictive Coding in the self-supervised learning field (van den Oord et al., 2018). There, it was observed that the intermediate representation quality improved when increasing the number of candidates (He et al., 2019). We hence look at the impact of the number of candidates |C| on languages generalization and the technical challenges it introduces. In this subsection, we consider the 1-pair setting trained on ImageNet while varying |C|. Scaling up task difficulty requires to carefully tune optimization. We first train agents without KL regularization (β = 0) as it is commonly done in the field (see e.g., Strub et al., 2017; Cao et al., 2018; Lu et al., 2020, among many others). As displayed in Fig. 1(a), the training becomes unstable when increasing the number of candidates from 20 to 1024 candidates. The mean train accuracy drops from 98.9% to 76.0%, and we observe a large variance across seeds. This high variance demonstrates how crucial it is to run multiple seeds to state robust conclusions. A remedy against such instability in RL consists in adding a KL regularisation LKL loss between the online policy and a target policy as described in Sec. 2.3. Such a solution has yet not been explored in the emergent communication field. We add a KL regularization with a coefficient β = 0.5 and display the results in Fig. 1(b). We note that, with this better regularization, 20 and 1024 candidates converge to 99.8% and 98.3% mean accuracy respectively with small variance during training. Adding LKL thus stabilizes the training and leads to better performances. We illustrate further the effect of this regularization across multiple sweeps in Appendix B.3.
Scaling up task difficulty is necessary to differentiate representations and enhance protocol’s generalization. In Fig. 2(a), we train 1-pair of agents on different number of candidates ranging from 64 to 1024 and evaluate them on 16 candidates.2 Note that, in the literature, the number of distractors rarely exceeds a dozen both at train and eval times (Mu & Goodman, 2021; Li & Bowling, 2019). In such settings, our experiments do not provide any interesting conclusions: all methods are above 99.7% test accuracy and within standard deviation. However, when evaluating the same communication protocols now on harder tasks with 1024 candidates, as shown in Fig. 2(b), we note that these protocols are actually different. In particular, agents achieve an accuracy of 87.36%,
2We kept the training batch size to 1024, and only varied the number of candidates in the InfoNCE loss.
93.25%, 96.09% when trained on 64, 256, and 1024 candidates respectively, differentiating the various representation’s generalization capacity. While expected in light of the recent self-supervised results (Chen et al., 2020) - harder training tasks lead to better representation and harder evaluation tasks better discriminate algorithms - these results clearly illustrate how current emergent language experiments may be ill-scaled to have robust conclusions. We report further experiments in Appendix B.5. In the following, we use 1024 candidates at train and eval to fully leverage our findings.

heading: 3.2 DATASET SCALING UP: RETHINKING EVALUATION
text: In most emergent communication works, deep agents are situated in a simple disentangled environment where objects are one-hot vectors (Kottur et al., 2017; Li & Bowling, 2019). As a result, the state space rarely goes beyond a few thousands of discrete samples. Besides, such representations are unambiguous as they are inherently compositional. We argue that such environments may be too simplistic to reach complex or diverse communication behaviors. In contrast, the major success stories in machine learning collectively prove the importance of training neural networks on a rich and large amount of data to emulate complex distributions (Brown et al., 2020; Krizhevsky et al., 2012; Sutton, 2019). Furthermore, if the goal is to reproduce human interaction, it is fundamental to incorporate complex stimuli to develop advanced concepts in language evolution (Miller & Johnson-Laird, 1976; Barsalou, 2008). In this subsection, we scale up the input space by using large natural image datasets, namely ImageNet and CelebA. In particular, we investigate further emergent language properties in the same setting of 1-pair of agents while varying |C| at train time.
TopSim may fall short with natural images. We compute TopSim using the pretrained image logits on ImageNet and CelebA as input representation while varying the task complexity at train, |C|. Results are provided in Fig. 3(left). We do not observe a consistent pattern between TopSim and |C|. Furthermore, we note a non-significant (p-value>0.19) Spearman correlation of−0.09 and 0.08 between TopSim and generalization for ImageNet and CelebA respectively. Although pretrained logits are excellent linear features (Grill et al., 2020), they may not be compositional. We thus reiterate the evaluation by using the attributes from CelebA as input representation to compute TopSim. In that case too we do not note any significant correlation (p-value=0.13). Similar findings have been observed in (Andreas, 2019; Chaabouni et al., 2020). This could be due to different reasons. First, agents could be communicating in a compositional way, to support good generalization, by encoding unexpected compositional features that are not labeled, like forehead shape or smile angle. That is,
human-labeled features may differ from the agents’ ones. Second, TopSim relies on strong assumptions, such as the chosen distance, and the use of a linear correlation. Such hypotheses may not hold when moving from artificial to realistic data. Finally, due to the large channel capacity, agents could be using synonyms to generalize with low TopSim values. In sum, our results demonstrate that TopSim is not a good predictor of communication protocol generalization for large-scale settings. Instead, we investigate the information content via ETL.
ETL is a robust metric to evaluate languages with natural images. ETL, contrary to TopSim, evaluates how useful the emergent language is for a target task beyond overfitting. As such it is an important metric, since the goal of emergent communication is to endow agents with communication skills, rather than solving a specific game. Results are reported in Table 1 over distinct tasks on CelebA. We observe that, even though ETL only looks at the information content and not at compositionality, it is a better predictor for language generalization performances than TopSim, with a strong and significant correlation >0.90 (p-value∼0) for all considered tasks. As a visual sanity check, we generate predicted images of the reconstruction ETL task in Fig. 3(right). There, a new Listener is trained to minimize an l2 loss with the target images given the precomputed messages. We observe that gender, makeup, and hair color are fairly encoded, while others such as the smile are not. More samples and training details are available in Appendix B.1. Interestingly, some ETL tasks, like discrimination or reconstruction, do not depend on (human-)predefined input representations, and are all more robust to channel capacity and linearity assumptions, as opposed to TopSim, which makes ETL a more general evaluation metric.
Zero-shot dataset transfer highlights emergent protocols’ differences and limits. Scaling up to natural image opens a large diversity of visual distributions that we can leverage by using zeroshot dataset transfer (Yogatama et al., 2019; Lambert et al., 2020). We hence measure how generic and high-level the emergent protocols are. We take agents trained on ImageNet with |C|=1024, and evaluate them on CelebA in a zero-shot fashion. There, the mean accuracy dropped from 95.96 to 36.73, and the standard deviation raised from 0.49 to 12.86. Noticeably, the most different agents obtain a zero-shot accuracy of 27.88 and 68.49, while their initial ImageNet accuracy only differed by 0.23. Firstly, such a standard deviation gap suggests that different emergent protocols have emerged between agent pairs. Secondly, the accuracy drop indicates that the emergent protocols remain specific to the initial data-distribution, and no systematic language has yet emerged.

heading: 3.3 POPULATION SCALING UP: EXPLOITING THE POPULATION DYNAMIC
text: The emergent communication literature often considers a single pair of agents (Lazaridou et al., 2017; Foerster et al., 2016). However, there is evidence in the multi-agent literature that such a setting may lead to extreme co-adaptation and overfitting (Lanctot et al., 2017). One counter-measure consists of sampling agents within a population (Jaderberg et al., 2018). From a linguistic perspective, different large-scale corpora analyses and human simulations support the importance of the population in shaping our languages (Gary Lupyan, 2010; Bromham et al., 2015; Raviv et al., 2019a;b). In this context, Raviv et al. (2019a) show that larger populations develop more systematic languages through human experimentation. Nonetheless, a few artificial simulations tentatively consider population sizes up to 32 agents, but with mixed results about their advantage (Mordatch & Abbeel, 2018; Tieleman et al., 2018; Graesser et al., 2019; Cogswell et al., 2019). In this part, we consider up to 100 agents. Specifically, we explore the impact of the population size on languages properties when we deal with more complex tasks (|C|=1024 at train and eval) and realistic inputs (CelebA & ImageNet) while varying the population size N ∈ {1, 10, 50} pairs. The best single-pair seed should be the baseline against population. A few works in the emergent communication framework looked at the benefit of the population by comparing 2N -agent’s performances to the 2-agent baseline (Cogswell et al., 2019). However, the former introduces ×N more
parameters, which could be responsible for its slight observed benefit. Here, we consider instead the “best 1 pair” setting to investigate whether it is computationally advantageous to train a population of size N rather than training N independent pairs. That is, for a given computational budget, we ask whether it is beneficial to train agents within a population as opposed to N pairs in parallel, as also done in (Tieleman et al., 2018). “Best 1 pair” is constructed by selecting the best seed among the 10 seeds of “1 pair”, based on validation accuracy, offering a fairer baseline specifically for the “10 pairs” setting. Results are shown in Table 2 and Fig. 4. We observe different behavior of “best 1 pair” across datasets. Particularly, if it always achieves better generalization and ETL discrimination compared to “1 pair”, it shows, for ImageNet only, the worst Robustness and ETL classification. We hypothesize that in that specific case, the emergent protocol overspecializes to the task. Still, “best 1 pair” outperforms the standard baseline “1 pair” 6/8 times, which makes it a stronger baseline. In the following, we thus always compare the population setting with “best 1 pair” for a fair evaluation.
Population size does not lead to a systematic advantage. We look at the language generalization and robustness performances when increasing the population size. Results in Table 2 do not show a clear trend between population size and these metrics. For example, for both datasets, “50 pairs” achieves lower Generalization and Robustness than “10 pairs”. Furthermore, “best 1 seed” always achieves better Generalization compared to the largest population setting “50 pairs”. In Fig. 4, we look at ETL for a further investigation of the languages’ properties. CelebA results exhibit a benefit of the population. However, this benefit does not always correlate with population size where “10 pairs” outperforms “50 pairs” on the discrimination task. This non-systematic benefit is also found when experimenting with ImageNet. Again, our results suggest no benefit of the population size in the current setting. We thus explore new approaches to leverage the population dynamics and improve the representation of the emergent protocol in the following.
A better use of the population is needed to see larger benefit. As population size does not improve the emergent language in itself, we took advantage of the richness of the population by considering imitation at training and vote at inference. In Table 2, we observe that both imitation and vote introduce a systematic improvement compared to the standard training in a population and lead to better performances compared to “1 pair” in all metrics. Moreover, when considering the stronger baseline “best 1 pair”, the latter only outperforms the imitation and vote settings 1/4 times (Generalization for ImageNet). However, as noted previously, this case is an over-specialized protocol. Furthermore, if both imitation and vote introduce a systematic improvement of the protocol, their benefit is even more noticeable for distribution-shifted settings. For example, the imitation for 50 pairs adds a relative improvement of 12.8% in Robusteness compared to 2.3% for Generalization on CelebA. The same observation holds for the vote but with a lower degree. Note that if vote induces only a slight benefit, it does not incur any training cost. When considering ETL on discrimination in Figs. 4(a)&4(c), we observe that imitation leads to a systematic improvement. In CelebA, if the standard population was already beneficial, adding the imitation strengthens the results. In ImageNet, “10/50 pairs + imitation” perform similarly at the end of training than the overspecialized “best 1 pair” while having more stable optimization and it is considerably better than the standard training w/ and w/o population. Yet, the results are more nuanced when considering ETL over classification in Fig. 4(b)&4(d). Specifically, while CelebA experiments suggest an advantage when training a population of agents w/o imitation as opposed w/ imitation or “1 pair”, this pattern is absent with ImageNet. There all settings with population are on par. In sum, both considered dynamics outperform in most cases the strong “best 1 pair” baseline (7/8 times), highlighting their viability.

heading: 4 DISCUSSION AND CONCLUSION
text: The emergent communication framework has been extensively studied for decades before the successes of deep RL. In this context, many works have revisited this framework through deep RL settings (e.g., Kirby 2002 vs. Ren et al. 2019, Myers-Scotton et al. 2002 vs. Graesser et al. 2019, or Batali 1998 vs. Choi et al. 2018). However, the experimental settings have barely evolved for twenty years. For instance, Kirby (2002) and Ren et al. (2019) both used a binary input vector of size eight, and only a few papers try to go beyond such artificial input spaces or simple tasks (Lazaridou et al., 2017; Havrylov & Titov, 2017; Dessı̀ et al., 2021). While computational constraints may have been a limiting factor, we show that our setting is possible with widely available hardware in Appendix B.2. For example, Table 1 requires approximately 400 hours of compute, i.e. 16 GPUs for a single day. Compared to other communities, this is equivalent to medium-scale studies in vision or NLP. We argue that it is finally time for the emergent language community to scale up!
In this spirit, we start clearing the way by identifying some scaling up challenges: optimization instabilities, ill-adapted metrics, or lack of population synergy. We show how to face these new difficulties by proposing: alternative optimization (KL regularization), new evaluation protocols (ETL, zero-shot dataset transfer, best seed baseline), and new dynamics to leverage populations (imitation, voting). There are different theories about the necessity of complex tasks to model human communication (e.g., (Barrett & Skyrms, 2017) vs. (Bickerton, 2015; Dupoux, 2018)). Hence, we endorse here Bickerton’s view and adopt performance-inspired solutions to scale up such as KLregularization and imitation. An interesting future debate is whether our findings could influence the status quo of similar research in human communication.
Although we only examine three scaling up dimensions, many other directions can be pursued in the future: using larger or different architectures to improve agent abilities (Desai & Johnson, 2021), experimenting with multimodal data like video or sounds for more realistic stimuli (Arandjelovic & Zisserman, 2017), or building symmetric communication channels to have emergent dialogues (Gao et al., 2019). Another frontier would be to incorporate interaction within environments to ground language into actions (Bisk et al., 2020). As such embodiment is crucial for human language understanding (Harnad, 1990; Barsalou, 2008), we may start exploring small grid world (Kajić et al., 2020) before scaling up to 3D-environments (Abramson et al., 2020).

heading: REPRODUCIBILITY STATEMENT
text: In this paper, we ensure the reproduciblity of the different findings through several (and voluntary redundant) ways, namely:
• The task is detailed in Section 2.1 and we provide a visual sketch in Appendix A.1. • We use open-source datasets (ImageNet and CelebA). Also, dataset processing is explained,
with pseudo-code to reproduce our splits, in Appendix A.6. • The speaker and listener architectures are first explained in Section 2.2, we then detail
model size values, and provide visual sketches in Appendix A.2. The specifics for image reconstruction for ETL are in Appendix B.1 with a pseudo-code for the reconstruction head.
• The optimization is first described in Section 2.3. The equations are then fully detailed in Appendix A.3.
• All training and evaluation hyperparameters are listed in Appendix A.4. • Through the paper, we voluntary provide training curves, test curves, and final scores to
have a global view of the training trends. • We listed computation time and memory footprint for multiple experiments and multiple
hardware in Appendix B.2 • The source code should be released upon paper acceptance.

heading: ACKNOWLEDGEMENT
text: We would like to thank Will Dabney, Remi Munos, Karl Tuyls, Nathalie Beauguerlange as well as the rest of the DeepMind Paris team for their continuous support. We would also like to thank Marco Baroni, Eugene Kharitonov, Olivier Pietquin and Mathieu Rita for their discussions and helpful feedback at the different stages of the project.

heading: A ADDITIONAL SETTING DETAILS

heading: A.1 DISCRIMINATION GAME

heading: A.2 ARCHITECTURE DETAILS
text: Speaker The speaker’s network architecture is composed of several components to transform the target image x into a message m = (wt)T−1t=0 :
• The encoder f is a fixed Resnet-50 architecture that has been previously trained on Imagenet with the BYOL algorithm. The resulting embedding f(x) is of size 2048.
• The RNN hθ used is an LSTM of hidden size 256. Therefore the core state zt,θ is of size 512.
• The core-state adapter cθ is a linear layer with input size 2048 and an output size of 512 that allows to transform the embedding f(x) into an appropriate core state z−1,θ = cθ(f(x)). We split z−1,θ into two equal parts to obtain the initial hidden state zh,−1,θ and the initial cell state zc,−1,θ.
• The word embedder gθ associates to each discrete symbols inW ∪ {sos} an embedding of size 10.
• The value head vθ first selects the hidden part zh,t,θ of the core state zt,θ and then applies a linear layer of output size 1.
• The policy head πθ first selects the hidden part zh,t,θ of the core state zt,θ and then applies a linear layer of output size |W| to obtain logits lt,θ. Then a softmax layer transforms the logits into the output distribution πθ(.|zt,θ).
Listener The listener’s network architecture is composed of several components to transform the message m and the input image x̃ into a score score(m, x̃, φ):
• The encoder f is a fixed Resnet-50 architecture that has been previously trained on Imagenet with the BYOL algorithm. The resulting embedding f(x) is of size 2048.
• The RNN hφ used is an LSTM of hidden size 512. Therefore the core state zt,φ is of size 1024 and is composed of an hidden state zh,t,φ of size 512 and a cell state zc,t,φ of size 512.
• The word embedder gφ associates to each discrete symbols inW an embedding of size 10. • The target projection tφ is a linear layer with output size 256. • The core-state projection pφ is a linear layer with output size 256.
As explained in Sec.2.2, the score function is defined as score(m, x̃, φ) = cos( pm,φ‖pm,φ‖2 , tx̃ ‖tx̃‖2 ). The scores over all images are normalized via a softmax to get a probability πφ(.|m, C) such that:
∀x̃ ∈ C, πφ(x̃|m, C) = exp(score(m, x̃, φ))∑ x∈C exp(score(m,x, φ)) .
Finally, the listener selects an image by taking the best guess according to πφ, i.e. x̂ ∈ argmaxx̃∈C πφ(x̃|m, C).

heading: A.3 OPTIMISATION DETAILS
text: Speaker training and loss The goal of a speaker, parameterized by θ, is to optimize the message m sent to a listener, parameterized by φ, such that the mean reward of the game is the highest possible. This can be framed as a sequential decision making problem where the decision is the choice of each word wt. Thus, each word is sampled from a parameterized stochastic policy πθ(.|x, (wk)t−1k=0) that depends on the image x and past words (wk)t−1k=0, where for t = 0 we have (wk) −1 k=0 = ∅. Then, the goal is to maximize the expected reward J(θ, φ) by finding the best policy πθ:
J(θ, φ) = Ex∼ρ [Eπθ,x[R(x, x̂)]] ,
where the expectation Ex∼ρ is over the dataset of training images and the expectation Eπθ,x is over all possible sequences (x, (wt)T−1t=0 ) that can be generated by πθ starting from x. For a given image x, we define the value V πθ (x) = Eπθ,x[R(x, x̂)] as the expected reward for a given image, here we left the dependence over the choice of the listener x̂ for ease of notations as the speaker cannot act on it. For a parameterized policy πθ, we can use the policy gradient theorem to obtain the gradient:
∂θV πθ (x) = Eπθ,x [ T−1∑ t=0 ( R(x, x̂)− V πθt−1(x) ) ∂θ log(πθ(wt|x, (wk)t−1k=0)) ] ,
where V πθt−1(x) = Eπθ,x [ R(x, x̂)|(wk)t−1k=0 ] is the value conditioned on all the information revealed at time t− 1. For our particular choice of speaker network, we encode the policy πθ(.|x, (wk)t−1k=0) and the value V πθt−1(x) with the policy head πθ(.|zt,θ) and the value head vθ(zt,θ) respectively. Doing so is legitimate because by construction of our recurrent speaker network the embedding zt,θ is a function of the image x and the past words (wk)t−1k=0. We train then our speaker network by minimizing a value loss LV (θ) to make vθ(zt,θ) fit V πθt−1(x) over a batch X of images:
LV (θ) = 1 |X | ∑ x∈X LV (θ, x) = 1 |X | ∑ x∈X T−1∑ t=0 (R(x, x̂)− vθ(zt,θ))2 ,
and a policy loss function Lπ(θ) to maximise the expected reward:
Lπ(θ) = 1 |X | ∑ x∈X Lπ(θ, x) = 1 |X | ∑ x∈X T−1∑ t=0 sg (R(x, x̂)− vθ(zt,θ)) log(πθ(wt|zt,θ)),
where sg(.) is the stop gradient function. If we assume that vθ(zt,θ) fits perfectly V πθt−1(x), one can easily verify that ∂θLπ(θ, x) is an unbiased estimate of −∂θV πθ (x) therefore ∂θLπ(θ) is also an unbiased estimate of −∂θJ(θ, φ). In addition to following the gradient ∂θV πθ (x), it is common practice in RL and emergent language literature to maximize an entropy term encouraging to explore other choices of words by Speaker:
H(πθ) =Ex∼ρ [ Eπθ,x [ T−1∑ t=0 H(πθ(.|x, (wk)t−1k=0)) ]] ,
= Ex∼ρ [ Eπθ,x [ T−1∑ t=0 Ew∼πθ(.|x,(wk)t−1k=0) [ πθ(w|x, (wk)t−1k=0) log(πθ(w|x, (wk) t−1 k=0)) ]]] .
In practice, a sampled version of the negative entropy LH(θ) is minimized with the speaker network:
LH(θ) = − 1 |X | ∑ x∈X T−1∑ t=0 Ew∼πθ(.|zt,θ) [πθ(w|zt,θ) log(πθ(w|zt,θ))] .
Recently, several deep RL (Schulman et al., 2015; 2017) and theoretical RL papers (Geist et al., 2019; Vieillard et al., 2020a;b) argued that minimizing the KL between the online policy πθ and a target policy πθ instead of or in addition to entropy regularization could be beneficial for better final performance as well as stabilizing the learning. Generally, πθ is an older policy or an exponential moving average of past policies. Therefore, we also consider the following KL regularization KL(πθ, πθ):
KL(πθ, πθ) =Ex∼ρ [ Eπθ,x [ T−1∑ t=0 KL(πθ(.|x, (wk)t−1k=0), πθ(.|x, (wk) t−1 k=0)) ]] ,
=Ex∼ρ [ Eπθ,x [ T−1∑ t=0 Ew∼πθ(.|x,(wk)t−1k=0) [ πθ(w|x, (wk)t−1k=0) log( πθ(w|x, (wk)t−1k=0) πθ(w|x, (wk) t−1 k=0) ) ]]] .
In practice, the policy πθ is obtained by doing an exponential moving average of the weights θ over training. Then, a sampled version of the KL is minimized LKL(θ) with our specific speaker network:
LKL(θ) = 1 |X | ∑ x∈X T−1∑ t=0 Ew∼πθ(.|zt,θ)
[ πθ(w|zt,θ) log(
πθ(w|zt,θ) πθ(w|zt,θ) )
] .
To sum up, the speaker training loss L(θ) on a batch of images X is:
L(θ) = LV (θ) + Lπ(θ) + αLH(θ) + βLKL(θ),
where α and β are hyper-parameters.
Listener loss One important detail in Sec. 2.3 is that for each image x in a batch X , a set of image candidates C(x,X ) is sampled randomly (uniform without replacement over X \ {x}) chosen in X . We omitted the dependencies of C(x,X ) on x andX in the main text for ease of reading. In addition, as explained in Sec. 2.3, the listener loss is a multi-class classification loss where the correct class is the index of x in the set of candidates C also called InfoNCE loss (van den Oord et al., 2018): L(φ) = − 1|X | ∑ x∈X log (πφ(x|m, C)). This can be rewritten more explicitly:
L(φ) = − 1 |X | ∑ x∈X log (πφ(x|m, C)) ,
= − 1 |X | ∑ x∈X log ( exp(score(m,x, φ))∑ x∈C exp(score(m,x, φ)) ) ,
= − 1 |X | ∑ x∈X log
( exp(cos(
pm,φ ‖pm,φ‖2 , tx ‖tx‖2 ))∑
x∈C exp(cos( pm,φ ‖pm,φ‖2 , tx ‖tx‖2 ))
) .
Imitation training among a group of speakers. In a training imitation step, a group of K speakers among the total population of N speakers is selected. Among those K speakers, one speaker is going to play the role of the teacher and K − 1 are going to play the role of the students. To choose the teacher among the K speakers, we use as metric the exponential moving average of the accuracies over each batch on which a given speaker has been trained on. To be more precise, let θi be the i-th speaker and X the present batch on which θi has just been trained on, then σi is updated according to the following rule:
σi ← µσi + (1− µ) 1 |X | ∑ x∈X 1x=x̂
and is the exponential moving average of coefficient µ of the accuracies over each batch on which speaker i has been trained. Then the teacher is simply the speaker with the highest σi among the K speakers. Now for convenience, let us note θT the parameters of the teacher and θS the parameters of a student. A student θS is going to be trained on a batch of data X by imitating the messages of the teacher θT with the following loss:
LI(θS) = − 1 |X | ∑ x∈X T−1∑ t=0 log πθS (wt(x, θT )|x, zθS ,t).
The loss LI(θS) is a cross-entropy loss that encourages the student to output the same words as the teacher.
Optimizer hyper-parameters Each member of the population (listener or speaker) uses its own optimiser. All optimisers are Adam optimisers (Kingma & Ba, 2015) with the same set of hyperparameters:
• learning rate: 1e− 4, • β1: 0.9, • β2: 0.999, • : 1e−8.
For the speakers’ loss, we use α = 0.0001 for the entropy regularization and β = 0.5 for the KL regularization. See Sec. B.3 for more details about the impact of these parameters.

heading: A.4 HYPER-PARAMETERS
text: We use the same hyper-parameters across the different settings. When experimenting with ImageNet vs. CelebA, we only vary the number of maximum steps. Specifically, we use 300k maximum steps for CelebA (we already start observing some overfitting with this value) and 900k maximum steps for ImageNet. In all cases, we select the best checkpoint for evaluation with respect to the listener loss at validation time. For robust evaluation, we average the scores over 5 epochs to have different candidates for a given target and get a score less dependent on the sampling of the candidates. Unless mentioned otherwise, the remaining training hyper-parameters are reported in Table 3.
A.5 IMITATION HYPER-PARAMETERS
In all our experiments, we perform a grid search over the parameters of M (number of interaction steps) and K (number of sampled speakers at the imitation step). For the case of a population of N = 10,K is chosen from {1, 4, 9} andM from {10, 50, 100} according to the validation accuracy. For N = 50, K is selected from {4, 9, 24} and M from {1, 10, 50}. Table 4 shows the selected hyper-parameters (K,M) for the different settings.

heading: A.6 DATASETS DETAILS
text: ImageNet ILSVRC2012, also known as ImageNet (Deng et al., 2009; Russakovsky et al., 2015) is among the historical largest natural RGB image dataset. It mostly contains images of animals and objects over 1000 labels, e.g., camel, ostrich, hourglass, or bow tie. In our experiments, we use 99% of the official train set for training, i.e., 1300k images, the last 1% of the train set for validation, i.e., 13k images, and the official validation set as our test set (i.e., 50k images).
CelebA CelebA is a large natural RGB dataset (Liu et al., 2015), which contains image of celebrity faces over 10,177 identities. Noticeably, each face also contains 40 binary attributes, e.g., glasses and hair. Such attributes are interesting to compute, for instance, language topography. Although there is an official CelebA split, there are no overlapping identities between the train, validation, and test set. We thus reshuffle the split as described in Listing 1. In the end, we respectively obtain 169,444 training, 16,669 valid, and 16,486 test images.
Image Processing In both datasets, we resize images to 256 pixels along the shorter side using bicubic resampling before applying a 224x224 center crop. We normalize the color channels subtracting the mean color and dividing by ImageNet std. We then use the ResNet-50 encoder pretrained on ImageNet with the self-supervised method BYOL (Grill et al., 2020) to extract the final representation of dimension 2048.
Listing 1: CelebA Splits

heading: SPLIT RATIO = 5 # t h e r a t i o t r a i n : v a l i d i s 5 :1
text: def c e l e b a s p l i t s ( d a t a ) : ””” S p l i t t h e da ta i n t h r e e s e t s w i t h o v e r l a p p i n g i d e n t i t i e s ”””
# I n i t i a l i z e v a r i a b l e s i m a g e t r a i n , i m a g e v a l i d , i m a g e t e s t = [ ] , [ ] , [ ] c o u n t e r l a b e l = c o l l e c t i o n s . Coun te r ( )
f o r d a t a in d a t a s e t . v a l u e s ( ) :
l a b e l = d a t a [ ’ l a b e l ’ ] # l a b e l encode t h e f a c e i d e n t i t y c o u n t = c o u n t e r l a b e l [ l a b e l ]
i f c o u n t > 0 and c o u n t % SPLIT RATIO == 0 : # We e q u a l l y s p l i t v a l i d and t e s t by Even and Odd l a b e l s i f l a b e l % 2 == 0 :
i m a g e v a l i d . append ( d a t a ) e l s e :
i m a g e t e s t . append ( d a t a ) e l s e :
i m a g e t r a i n . append ( d a t a )
c o u n t e r l a b e l [ l a b e l ] += 1
re turn i m a g e t r a i n , i m a g e v a l i d , i m a g e t e s t

heading: B ADDITIONAL RESULTS AND ANALYSIS

heading: B.1 FACE RECONSTRUCTION
text: To visualize some of the features encapsulated in the messages used by the agent, we propose a face reconstruction procedure. For a single speaker, parameterized by θ, and given an initial image x ∈ X , we produce a message m = (wt)T−1t=0 . We feed the messages to a listener-like architecture, consisting of an embedding layer of size 10 followed by an LSTM with 512 units. We keep the last output of the LSTM, hT−1 as a message representation. We then pass the message through a deconvolutional architecture, similar to the one in Radford et al. (2016), but without batch normalizations. Pseudocode for the architecture is provided in Listing 2. The full network is trained by minimizing the `2 loss between the input image and the reconstructed one. To optimize the loss, we use AdamW (Kingma & Ba, 2015) with a batch size of 128, a learning rate of 3 · 10−4, β1 = .9, β2 = .9, ε = 10−8 and a weight decay of 0.01. We clip gradients by norm with a clipping threshold of 500, and skip gradient updates with norm superior to 2000. In Table 1, we report reconstruction losses for different number of candidates |C|. The reconstruction loss is computed as a squared error, summed over both spatial and channel axes. Figure 8 displays examples of input images, and corresponding reconstructions for two discrimination games with different number of candidates.
Optimally, each reconstruction should converge to the average face, given the corresponding message. The more features a message contains, the better reconstructions should be. Note first that reconstructions are far from perfect, and much worse than reconstructions that an auto-encoder trained end to end on images would provide: first, solving the discrimination game does not require fully reconstructing the input image, but only capturing enough features to identify uniquely the image in a batch of candidates; second, the discrete nature of the message, its limited size, and the fact that it is learnt using reinforcement learning act as strong bottlenecks, that prevents messages from containing all the information about input images.
Nonetheless, as qualitatively shown in Figure 8, messages contain semantic information about inputs images. For instance, hair color, gender, or open mouth are mostly preserved throughout the reconstruction process. Other features, such as face orientation, skin tone, or age are mostly ignored in messages. Furthermore, both quantitative and qualitative difference are visible when going from low number of discriminators to high number of candidates. Quantitatively, going from 16 to 1024 candidates improves the reconstruction error from 2448±16 to 2351±14 (Table 1). Qualitatively, messages with 1024 candidates seem to contain information about presence of eye-glasses (top left image), as well as presence of head cover (bottom right image), that are absent in messages from the Speaker trained with 16 candidates.
Listing 2: Face reconstruction Head
def upsample 2d ( x , f a c t o r : i n t = 2 ) : bs , h e i g h t , width , c h a n n e l s = x . shape x = image . r e s i z e (
x , ( bs , h e i g h t ∗ f a c t o r , w id th ∗ f a c t o r , c h a n n e l s ) , method= ’ n e a r e s t ’ ) re turn x
def i m g r e c o n s t r u c t i o n ( x ) : ”””Take t h e LSTM o u t p u t , and t u r n i t i n t o an image . ”””
# P r o j e c t t h e f l a t embedding i n t o a 2d t e n s o r o f dim 4 x4x128 x = nn . L i n e a r (4 ∗ 4 ∗ 1 2 8 ) ( x ) x = x . r e s h a p e ( ( x . shape [ 0 ] , 4 , 4 , 1 2 8 ) ) x = nn . r e l u ( x )
x = upsample 2d ( x ) # 8 x8 x = nn . Conv3x3 ( 6 4 , w i t h b i a s = F a l s e , padd ing =”VALID” ) ( x ) x = nn . r e l u ( x )
x = upsample 2d ( x ) # 16 x16 x = nn . Conv3x3 ( 3 2 , w i t h b i a s = F a l s e , padd ing =”VALID” ) ( x ) x = nn . r e l u ( x )
x = upsample 2d ( x ) # 32 x32 x = nn . Conv3x3 ( 1 6 , w i t h b i a s = F a l s e , padd ing =”VALID” ) ( x ) x = nn . r e l u ( x )
x = upsample 2d ( x ) # 64 x64 x = nn . Conv3x3 ( 1 6 , w i t h b i a s = F a l s e , padd ing =”VALID” ) ( x ) x = nn . r e l u ( x ) x = nn . Conv3x3 ( 3 , w i t h b i a s = F a l s e , padd ing =”VALID” ) ( x )
re turn nn . t a n h ( x )

heading: B.2 COMPUTATIONAL REQUIREMENTS
text: As mentioned in the article, our approach remains computationally tractable with widely available hardware. Table 5 summarizes these requirements to reproduce our experimental setup.
B.3 IMPACT OF THE KL REGULARIZATION ON TRAINING STABILITY
We showed in the main paper how training a pair of agents to solve a complex task (a discrimination game with 1024 candidates) becomes unstable when using common optimization algorithms. In this section, we look at the training curves when agents are trained with and without KL regularization, while varying the entropy coefficient α ∈ {10−4, 10−3, 10−2, 10−1}. Results are shown with a population of size 1 (Figure 9) and 10 (Figure 10). In both cases, agents are trained to solve the complex discrimination task with 1024 candidates.
First, we observe that the lower α is, the more crucial applying a KL regularization is. For example, in Figure 9(a), we go from a chaotic setting that converges to an accuracy of ∼ 40% when the KL coefficient β = 0 to a significantly more stable optimization with almost perfect accuracy when β ≥ 0.5. Second, we observe that, if the KL regularization is useful for stable optimization, it comes at the cost of the rapidity of convergence. This is clearly shown in Figures 9(c) and 10(c), where we observe that the larger β is, the slower the convergence is. In other words, one should select the best β optimizing the stability/rapidity trade-off. Third, in the presence of a KL regularization, training stability depends less on the entropy coefficient α. Indeed, we observe a stable and high training
curves for α ∈ {10−4, 10−3, 10−2}.3 However, training without KL regularization (blue curves with β = 0) is very sensitive to the value of α and the most stable case is observed for α = 0.01. Still even in this best case without KL regularization, having a β > 0 is useful as shown in Figure 11.
3It is also the case for α = 0. Not shown here.

heading: B.4 STUDY OF THE RESETTING ON THE CELEBA DATASET
text: In this work, we argue that training deep agents to communicate in a population benefits from exploiting its richness, as opposed to a standard training (Mordatch & Abbeel, 2018). In this context, we introduced imitation and voting mechanisms. However, other prior works have stipulated a similar argument by focusing instead on the idea of the expressivity/compressibility trade-off (Kirby et al., 2014). The latter attests that our language is shaped by the two competing pressures of expressivity and compressibility. In practice, there were different methods to implement this tradeoff in deep agents communication such as iterated learning (Ren et al., 2019), cultural transmission (Cogswell et al., 2019), or ease-of-teaching (Li & Bowling, 2019). In this section, we compare imitation and voting to the ease-of-teaching.4 To encourage languages’ ease-of-teaching, Li & Bowling (2019) trained deep agents to solve a discrimination task, while periodically resetting listeners. In this section, we study the impact of this baseline on the different emergent languages’ properties introduced in the main paper. Specifically, we consider 3 additional settings: (1) “1 pair + reset” that consists of resetting the only listener, (2) “10 pairs + reset” where we reset a randomly selected listener among the 10 available ones, and (3) “50 pairs + reset” where we reset a randomly selected listener among the 50 available ones. In all cases, resetting takes place every 51k steps. Note that if (1) is identical to Li & Bowling (2019) setting, (2) and (3) present 10 and 50 speakers respectively unlike 1 speaker as it is the case in (Li & Bowling, 2019).
We observe in Table 6 that resetting has a noticeable benefit only in the case of “1 pair” on the generalization and robustness of the languages. In particular, we note an average Generalization of 89.00% for “1 pair” vs. 92.21% for “1 pair + reset” and an average Robustness of 18.34% vs. 35.82%. However, there is no systematic improvement when agents are trained in a population, (“10 pairs” vs. “10 pairs + resetting”) and (“50 pairs” vs. “50 pairs + resetting”). This is in line with Li & Bowling (2019)’s results which show that an abrupt change during training leads to better results. Still, “1 pair + reset” does not systematically improve the emergent languages’ properties compared to the population when imitation and voting are at play.
However, as mentioned in the original paper, the purpose of resetting is not to boost agents’ generalization or robustness, but instead to incentivize agents to develop easy-to-transmit languages. In the following, we look at ETL shown in Figure 12. Here, adding resetting does not lead to a significant change showing curves almost identical to the ones with standard training in all cases.
Overall, our results suggest that resetting listeners is only beneficial in the one-pair agents in some cases. Furthermore, resetting does not induce faster or better easy-to-learn communication protocols over transfer tasks compared to the standard training of the Lewis game (with or without population).
4We also considered the setup of Cogswell et al. (2019). However, preliminary results showed systematically worse results compared to the ease-of-teaching settings. Our codebase provides both options.
B.5 IMPACT OF THE NUMBER OF CANDIDATES AT TRAINING AND EVALUATION TIME
In this part we look the impact of task complexity at train (Figure 14) and evaluation (Figure 13) times, by varying the number of candidates |C|. The results confirm our findings in the main paper. That is, for a fixed train |C|, the harder the evaluation task is, the lower the test accuracy is. Still, if the task is complex enough at train time (e.g., |C| = 1024), agents reach overall higher accuracies for all evaluation settings, confirming the importance of hard training tasks to see the emergence of communication protocols with good generalization performances. Also, similarly to our results in the main paper, we observe in Figure 14 that harder evaluation task is necessary to discriminate between the communication protocols.

heading: B.6 CELEBA ATTRIBUTES:

heading: B.7 OUT-OF-DISTRIBUTION GENERALIZATION: THE OFFICIAL CELEBA SPLIT
text: In the main paper, we consider a new CelebA split where train and test sets have overlapping identities. This allows us to test in-distribution performances. In this Subsection, we look at out-ofdistribution generalization considering the official CelebA split, where train, validation, and test sets include distinct identities. In other words, we now consider a harder form of generalization by testing agents on never-seen identities at training. Results are shown in Table 8. We observe a similar pattern to the in-distribution results shown in Table 2. That is, we do not observe a benefit for the population, where “best 1 pair” achieves better unseen-generalization than “10 pairs” and “50 pairs”. However, when considering imitation and voting, agents develop languages that generalize better to
never-seen identities. The setting “50 pairs + imitation + vote” attains the best performance, with a 93.75% success when communicating about previously unseen identities at training time.

Review summaries:
**After rebuttal**: I am keeping my score  **Before rebuttal**: The paper conducts a large-scale study on RL-based emergent communication. The scale is extended in three dimensions: task complexity, data naturalness, and population size. Scaling up gives new findings that were not observed in previous (small-scale) work. Important findings are (1) evaluating with a large number of candidates is essential for observing the advantage of training with a large number of candidates (2) when experimenting with real images, TopSim, a widely used metric for measuring compositional, no longer correlates with generalization (3) larger population does not necessarily enhance generalization; more clever leverage of the population is needed. 

This paper aims to scale up the usual Lewis game language emergence setup. Instead of simple synthetic objects, the authors consider scaling up to image datasets such as ImageNet and CelebA. Another aspect they scale up is the task difficulty, by increasing the number of candidates the listener has to choose the correct answer from. Third, they also attempt scaling up the population size. To deal with the instability arising from a large number of candidates, the authors use established stabilization techniques such as minimization of KL divergence between the online policy and an exponentially moving average target policy. The paper also introduces a new metric called Ease and transfer learning (ETL) to measure how well an emergent language transfers to a novel task as an alternative to topographic similarity. Finally, the authors explore some techniques such as imitation learning and inference-time voting to try to leverage a population of agents.

The paper argues for scaling up experiments in emergent communication to better understand the evolution of human language and to construct more efficient representations. To achieve this, the authors demonstrate scaling up along dataset, task complexity, and population size in the context of a multi-agent Lewis game, evaluating the benefits and drawbacks of each using generalization, robustness to input noise, and ETL. They find that (1) KL regularization mitigates unstable optimization when scaling up; (2) increasing task complexity improves generalization and better separates performance of different algorithms; (3) topsim and generalization are uncorrelated, necessitating alternative metrics of compositionality at scale; (4) while scaling up population in itself is not obviously beneficial, imitation learning and voting in larger populations produce more "robust, productive, and easy-to-learn" languages. Overall, the paper makes a strong case to scale up emergent communication experiments supported by empirical contributions.