Emergent Communication at Scale

Emergent communication aims for a better understanding of human language evolution and building more efficient representations . We posit that reaching these goals will require scaling up , in contrast to a significant amount of literature that focuses on setting up small-scale problems to tease out desired properties of the emergent languages . We focus on three independent aspects to scale up , namely the dataset , task complexity , and population size . We provide a first set of results for large populations solving complex tasks on realistic large-scale datasets , as well as an easy-to-use codebase to enable further experimentation . In more complex tasks and datasets , we find that RL training can become unstable , but responds well to established stabilization techniques . We also identify the need for a different metric than topographic similarity , which does not correlate with the generalization performances when working with natural images . In this context , we probe ease-of-learnability and transfer methods to assess emergent languages . Finally , we observe that larger populations do not induce robust emergent protocols with high generalization performance , leading us to explore different ways to leverage population , through voting and imitation learning . source_code 1 INTRODUCTION . Language emergence is at the intersection of cognitive science and machine learning . From a cognitive science view , researchers have been looking at artificial agents as another expressive species to shed light on the source of linguistic regularities ( Wagner et al. , 2003 ; Guo et al. , 2019 ; Chaabouni et al. , 2021 ) . From a machine learning view , language evolution is deemed as a promising direction to shape agents ’ representation and design interactive AI ( Steels , 1997 ; Lazaridou et al. , 2020 ) . Most of the literature in this field relies on different variants of the Lewis game ( Lewis , 1969 ) . There , a speaker network must describe an object to a listener network , which must then retrieve it among a set of other objects . To solve the game , the two agents need to settle on a communication protocol . While deep agents manage to solve the Lewis game , their communication protocols are usually degenerate , lacking core properties of human languages ( Bouchacourt & Baroni , 2018 ; Chaabouni et al. , 2019 ) . In response to these findings , several works proposed ad-hoc solutions by constraining the game and agents ’ capacity ( Kottur et al. , 2017 ; Resnick et al. , 2020 ) . While reducing problem complexity is tempting , it can lead to unexpected outcomes and may miss general language emergence behaviors ( Hayes , 1985 ) . We take a different route and advocate that scaling up communication games is a prerequisite to building interactive AI ( Baroni et al. , 2017 ; Sutton , 2019 ) or modeling language evolution ( Barsalou , 2008 ) . Indeed , contrary to other machine learning communities , the emergent communication field mostly relies on small-scale games where only one speaker and one listener communicate about disentangled stimuli , which can hinder the generality of its conclusions . In this paper , we focus on three scaling dimensions : the dataset , the task complexity , and the population size . That is , we argue that making populations of deep agents communicate about larger and more realistic datasets and solve more complex tasks , is necessary if in the end we want these agents to interact with us or if we want to model human communication . ∗ Contributed equally . Corresponding authors : { rahmac , fstrub , piot } @ deepmind.com We study three properties of emergent languages : generalization , robustness to input noise , and ease of learning over transfer tasks , analyzing different facets of communication protocols . The proposed scaling up remains computationally tractable , as most of the experiments can be done within a day on a single GPU . The source code is based on the Jaxline pipeline ( Babuschkin et al. , 2020 ) . Overall , our experiments provide a large spectrum of observations , both positive and negative . First , we found that scaling up the Lewis game quickly entails unstable RL optimization . We propose KL regularization ( Geist et al. , 2019 ) as an effective way to overcome this issue . Second , we observe that complexifying the task has two positive aspects : it better discriminates between methods and improves the generalization of the learned communication protocol . Third , we note no correlation between generalization and the widely used topographic similarity metric , which suggests that the latter is not adequate to assess the compositionality of the languages in our more complex setup . Instead , we take inspiration from the self-supervised learning literature and explore transfer learning as a new evaluation metric . Fourth , unlike what was observed in human communication ( Gary Lupyan , 2010 ; Raviv et al. , 2019a ; b ) , we find little to no systematic benefit on emergent languages ’ properties when increasing the population size . Instead , we propose alternative methods to leverage populations , namely voting and imitation among speakers ( Hester et al. , 2018 ; Vecerik et al. , 2017 ) . In particular , we show that such population dynamics lead to more robust , productive , and in some cases easy-to-learn languages even when compared to our best seed without population , opening up new research opportunities . In the end , we expect that these observations , baselines , and good practices would allow the language emergence community to benefit further from deep RL advances and move the field closer to its goals of improving representations and interactive systems . 2 SETUP . 2.1 DISCRIMINATION GAME . The discrimination game involves two players , Speaker and Listener , and is decomposed into three sequential stages . First , Speaker receives a target image x and outputs a message m that is sent to Listener . Second , Listener receives m and selects an image x̂ among a set C of different images containing the target image x . The set C is called candidates . Finally , the target image x is revealed to Listener . If Listener selects the target image , then both players receive a positive reward of 1 , and 0 otherwise . Speaker and Listener are parameterized by a set of parameters θ and φ respectively . The message m ( x , θ ) = ( wt ( x , θ ) ) T−1t=0 is a sequence of length T of words wt ( x , θ ) . When no confusion is possible , we omit the dependence on x and θ to simplify notations . A word wt is an element of a finite vocabulary setW . For the image selected by Listener x̂ ( φ , m , C ) , we also omit the dependence on φ , m and C. Finally , the reward for Listener and Speaker is denoted by R ( x , x̂ ) = 1x=x̂ . Communication in a population of agents . A straightforward extension of the standard discrimination game is to train a population of agents . In this case , given a population of N Speakers and N Listeners , we sample S Speakers and S Listeners with replacement at each training step to construct S random pairs ; where each Speaker is paired with only one Listener.1 All pairs observe the same examples and are trained independently to play the discrimination game as described in the 1-pair setting above . In our case , S = N so that each agent is trained on average once per step , which allows a fair comparison with the baseline of 1-pair . At inference time , we take the first P Speakers and Listeners and construct all the possible P 2 pairs to compute the accuracy , by averaging the rewards of all pairs as in ( Mordatch & Abbeel , 2018 ) . Exploiting the population . The training dynamic of the population of agents described above does not take advantage of the diversity of the population . Indeed , we treat all agents in the same way . Yet , one motivation for the effectiveness of the population is the variety between Speakers to invent new structures and between Listeners to avoid over-specialization ( Bromham et al. , 2015 ) . Prior works have looked at the impact of training agents with different dynamics ( e.g. , Ren et al. , 2019 ; Guo et al. , 2019 ; Cogswell et al. , 2019 ; Li & Bowling , 2019 ) . In this work , we introduce two different ways to exploit the population ; on Speakers ’ side , we add imitation , and on Listeners ’ side , we allow Listeners to vote to get the final prediction . Mathematical details are found in Sec . 2.3 . Imitation training consists of two different steps . First , as described above , Speakers and Listeners are paired randomly for M interaction steps to learn to communicate . Second , Speakers are trained for 1 imitation step as follows : we select the best Speaker ( called “ teacher ” ) among K randomly 1In this work , we only consider populations with the same number of Speakers and Listeners . sampled Speakers without replacement , and use the remaining K-1 samples as “ students ” . We then train all students in a supervised way to imitate the teacher . We alternate between the interaction and imitation steps . Both M and K are hyper-parameters ( see Appendix A.5 ) . Finally , Voting is only used at inference time , irrespective of the training mode . Here , instead of averaging P 2 pairs ’ rewards to compute accuracy , we now allow P Listeners to vote to get a unique prediction . This is inspired by ensemble methods to reduce prediction errors ( Polikar , 2012 ) . More details on imitation and voting are reported in Sec . 2.3 . For completeness , we also reproduce ease-of-teaching protocol as another effective way to exploit population in Appendix B.4 ( Li & Bowling , 2019 ) . 2.2 DATASETS AND NEURAL ARCHITECTURES . Datasets . We use the ImageNet ( Deng et al. , 2009 ; Russakovsky et al. , 2015 ) , and CelebA datasets ( Liu et al. , 2015 ) , which respectively contain 1400k and 200k labelled images . CelebA contains 40 binary attributes per image , such as the presence of glasses , blond hair . As the official CelebA splits have non-overlapping identities , it is impossible to perform regular identity classification on the test/valid sets . We thus construct a new split , where we shuffle images to have overlapping identities across splits . In both datasets , each image is center-cropped before being processed by a ResNet-50 encoder f pretrained on ImageNet with the self-supervised method BYOL ( Grill et al. , 2020 ) to extract a representation of size 2048 . Full implementation details are in Appendix A.6 . Speaker architecture . Speaker is a neural network that receives an image x and outputs a message m = ( wt ) T−1t=0 . It is composed of a fixed ( non-trainable ) image encoder f that transforms x into an embedding f ( x ) , before being projected by a state adapter cθ to an initial state of an LSTM ( Hochreiter & Schmidhuber , 1997 ) , z−1 , θ = cθ ( f ( x ) ) . The LSTM receives as input a word embedding et , θ = gθ ( wt−1 ) and outputs the next state zt , θ = hθ ( zt−1 , θ , et , θ ) . The first word embedding e0 , θ = gθ ( sos ) is initialized with a start of sequence . The state zt , θ is then fed to two different heads ; the value head vθ estimates the value of the expected reward knowing zt , θ and the policy head πθ computes the probability of the chosen word given zt , θ . Then , a sampling function s picks the word wt = s ( πθ ( .|zt , θ ) ) . Finally wt is fed back to gθ to produce the next word and so on until the maximum length T is reached . At training , the word wt is sampled according to the policy whereas it is greedily selected at evaluation . Contrary to prior works ( e.g. , Kottur et al. , 2017 ; Resnick et al. , 2020 ) , we do not restrict the channel capacity of agents to spur the emergence of systematic languages . Instead , we endow Speaker with a message space of size |W |T = 2010 ( |W | = 20 , T = 10 ) , significantly larger than the number of available training examples . Listener architecture . Listener is a neural network that receives Speaker ’ s message m and a set of image candidates C , containing the target image x . It outputs the probability over each image x̃ ∈ C of being the target image . Listener is composed of an LSTM cell hφ , which is initialized to the null vector z−1 , φ . The message m is decoded by processing the sequence of word embeddings et , φ = gφ ( wt ) through the LSTM such as zt , φ = hφ ( zt−1 , φ , et , φ ) . The state zT−1 , φ is then fed to the network pφ to output pm , φ = pφ ( zT−1 , φ ) . In parallel , each image x̃ is projected through the encoder f and the network tφ to obtain the image embedding tx̃ = tφ ( f ( x̃ ) ) . Both message and image embeddings are then compared through a score function score ( m , x̃ , φ ) = cos ( pm , φ‖pm , φ‖2 , tx̃ ‖tx̃‖2 ) . The scores over all images are normalized via a softmax to get a probability πφ ( .|m , C ) . Finally , Listener selects an image by taking the best guess according to πφ , i.e . x̂ ∈ argmaxx̃∈C πφ ( x̃|m , C ) . Architecture details , hyper-parameters and graphical descriptions are provided in Appendices A.2 and A.4 . All experiments are run over 10 seeds .

Summary: **After rebuttal**: I am keeping my score  **Before rebuttal**: The paper conducts a large-scale study on RL-based emergent communication. The scale is extended in three dimensions: task complexity, data naturalness, and population size. Scaling up gives new findings that were not observed in previous (small-scale) work. Important findings are (1) evaluating with a large number of candidates is essential for observing the advantage of training with a large number of candidates (2) when experimenting with real images, TopSim, a widely used metric for measuring compositional, no longer correlates with generalization (3) larger population does not necessarily enhance generalization; more clever leverage of the population is needed.