Emergent Communication at Scale

Emergent communication aims for a better understanding of human language evolution and building more efficient representations . We posit that reaching these goals will require scaling up , in contrast to a significant amount of literature that focuses on setting up small-scale problems to tease out desired properties of the emergent languages . We focus on three independent aspects to scale up , namely the dataset , task complexity , and population size . We provide a first set of results for large populations solving complex tasks on realistic large-scale datasets , as well as an easy-to-use codebase to enable further experimentation . In more complex tasks and datasets , we find that RL training can become unstable , but responds well to established stabilization techniques . We also identify the need for a different metric than topographic similarity , which does not correlate with the generalization performances when working with natural images . In this context , we probe ease-of-learnability and transfer methods to assess emergent languages . Finally , we observe that larger populations do not induce robust emergent protocols with high generalization performance , leading us to explore different ways to leverage population , through voting and imitation learning . source_code 1 INTRODUCTION . Language emergence is at the intersection of cognitive science and machine learning . From a cognitive science view , researchers have been looking at artificial agents as another expressive species to shed light on the source of linguistic regularities ( Wagner et al. , 2003 ; Guo et al. , 2019 ; Chaabouni et al. , 2021 ) . From a machine learning view , language evolution is deemed as a promising direction to shape agents ’ representation and design interactive AI ( Steels , 1997 ; Lazaridou et al. , 2020 ) . Most of the literature in this field relies on different variants of the Lewis game ( Lewis , 1969 ) . There , a speaker network must describe an object to a listener network , which must then retrieve it among a set of other objects . To solve the game , the two agents need to settle on a communication protocol . While deep agents manage to solve the Lewis game , their communication protocols are usually degenerate , lacking core properties of human languages ( Bouchacourt & Baroni , 2018 ; Chaabouni et al. , 2019 ) . In response to these findings , several works proposed ad-hoc solutions by constraining the game and agents ’ capacity ( Kottur et al. , 2017 ; Resnick et al. , 2020 ) . While reducing problem complexity is tempting , it can lead to unexpected outcomes and may miss general language emergence behaviors ( Hayes , 1985 ) . We take a different route and advocate that scaling up communication games is a prerequisite to building interactive AI ( Baroni et al. , 2017 ; Sutton , 2019 ) or modeling language evolution ( Barsalou , 2008 ) . Indeed , contrary to other machine learning communities , the emergent communication field mostly relies on small-scale games where only one speaker and one listener communicate about disentangled stimuli , which can hinder the generality of its conclusions . In this paper , we focus on three scaling dimensions : the dataset , the task complexity , and the population size . That is , we argue that making populations of deep agents communicate about larger and more realistic datasets and solve more complex tasks , is necessary if in the end we want these agents to interact with us or if we want to model human communication . ∗ Contributed equally . Corresponding authors : { rahmac , fstrub , piot } @ deepmind.com We study three properties of emergent languages : generalization , robustness to input noise , and ease of learning over transfer tasks , analyzing different facets of communication protocols . The proposed scaling up remains computationally tractable , as most of the experiments can be done within a day on a single GPU . The source code is based on the Jaxline pipeline ( Babuschkin et al. , 2020 ) . Overall , our experiments provide a large spectrum of observations , both positive and negative . First , we found that scaling up the Lewis game quickly entails unstable RL optimization . We propose KL regularization ( Geist et al. , 2019 ) as an effective way to overcome this issue . Second , we observe that complexifying the task has two positive aspects : it better discriminates between methods and improves the generalization of the learned communication protocol . Third , we note no correlation between generalization and the widely used topographic similarity metric , which suggests that the latter is not adequate to assess the compositionality of the languages in our more complex setup . Instead , we take inspiration from the self-supervised learning literature and explore transfer learning as a new evaluation metric . Fourth , unlike what was observed in human communication ( Gary Lupyan , 2010 ; Raviv et al. , 2019a ; b ) , we find little to no systematic benefit on emergent languages ’ properties when increasing the population size . Instead , we propose alternative methods to leverage populations , namely voting and imitation among speakers ( Hester et al. , 2018 ; Vecerik et al. , 2017 ) . In particular , we show that such population dynamics lead to more robust , productive , and in some cases easy-to-learn languages even when compared to our best seed without population , opening up new research opportunities . In the end , we expect that these observations , baselines , and good practices would allow the language emergence community to benefit further from deep RL advances and move the field closer to its goals of improving representations and interactive systems . 2 SETUP . 2.1 DISCRIMINATION GAME . The discrimination game involves two players , Speaker and Listener , and is decomposed into three sequential stages . First , Speaker receives a target image x and outputs a message m that is sent to Listener . Second , Listener receives m and selects an image x̂ among a set C of different images containing the target image x . The set C is called candidates . Finally , the target image x is revealed to Listener . If Listener selects the target image , then both players receive a positive reward of 1 , and 0 otherwise . Speaker and Listener are parameterized by a set of parameters θ and φ respectively . The message m ( x , θ ) = ( wt ( x , θ ) ) T−1t=0 is a sequence of length T of words wt ( x , θ ) . When no confusion is possible , we omit the dependence on x and θ to simplify notations . A word wt is an element of a finite vocabulary setW . For the image selected by Listener x̂ ( φ , m , C ) , we also omit the dependence on φ , m and C. Finally , the reward for Listener and Speaker is denoted by R ( x , x̂ ) = 1x=x̂ . Communication in a population of agents . A straightforward extension of the standard discrimination game is to train a population of agents . In this case , given a population of N Speakers and N Listeners , we sample S Speakers and S Listeners with replacement at each training step to construct S random pairs ; where each Speaker is paired with only one Listener.1 All pairs observe the same examples and are trained independently to play the discrimination game as described in the 1-pair setting above . In our case , S = N so that each agent is trained on average once per step , which allows a fair comparison with the baseline of 1-pair . At inference time , we take the first P Speakers and Listeners and construct all the possible P 2 pairs to compute the accuracy , by averaging the rewards of all pairs as in ( Mordatch & Abbeel , 2018 ) . Exploiting the population . The training dynamic of the population of agents described above does not take advantage of the diversity of the population . Indeed , we treat all agents in the same way . Yet , one motivation for the effectiveness of the population is the variety between Speakers to invent new structures and between Listeners to avoid over-specialization ( Bromham et al. , 2015 ) . Prior works have looked at the impact of training agents with different dynamics ( e.g. , Ren et al. , 2019 ; Guo et al. , 2019 ; Cogswell et al. , 2019 ; Li & Bowling , 2019 ) . In this work , we introduce two different ways to exploit the population ; on Speakers ’ side , we add imitation , and on Listeners ’ side , we allow Listeners to vote to get the final prediction . Mathematical details are found in Sec . 2.3 . Imitation training consists of two different steps . First , as described above , Speakers and Listeners are paired randomly for M interaction steps to learn to communicate . Second , Speakers are trained for 1 imitation step as follows : we select the best Speaker ( called “ teacher ” ) among K randomly 1In this work , we only consider populations with the same number of Speakers and Listeners . sampled Speakers without replacement , and use the remaining K-1 samples as “ students ” . We then train all students in a supervised way to imitate the teacher . We alternate between the interaction and imitation steps . Both M and K are hyper-parameters ( see Appendix A.5 ) . Finally , Voting is only used at inference time , irrespective of the training mode . Here , instead of averaging P 2 pairs ’ rewards to compute accuracy , we now allow P Listeners to vote to get a unique prediction . This is inspired by ensemble methods to reduce prediction errors ( Polikar , 2012 ) . More details on imitation and voting are reported in Sec . 2.3 . For completeness , we also reproduce ease-of-teaching protocol as another effective way to exploit population in Appendix B.4 ( Li & Bowling , 2019 ) . 2.2 DATASETS AND NEURAL ARCHITECTURES . Datasets . We use the ImageNet ( Deng et al. , 2009 ; Russakovsky et al. , 2015 ) , and CelebA datasets ( Liu et al. , 2015 ) , which respectively contain 1400k and 200k labelled images . CelebA contains 40 binary attributes per image , such as the presence of glasses , blond hair . As the official CelebA splits have non-overlapping identities , it is impossible to perform regular identity classification on the test/valid sets . We thus construct a new split , where we shuffle images to have overlapping identities across splits . In both datasets , each image is center-cropped before being processed by a ResNet-50 encoder f pretrained on ImageNet with the self-supervised method BYOL ( Grill et al. , 2020 ) to extract a representation of size 2048 . Full implementation details are in Appendix A.6 . Speaker architecture . Speaker is a neural network that receives an image x and outputs a message m = ( wt ) T−1t=0 . It is composed of a fixed ( non-trainable ) image encoder f that transforms x into an embedding f ( x ) , before being projected by a state adapter cθ to an initial state of an LSTM ( Hochreiter & Schmidhuber , 1997 ) , z−1 , θ = cθ ( f ( x ) ) . The LSTM receives as input a word embedding et , θ = gθ ( wt−1 ) and outputs the next state zt , θ = hθ ( zt−1 , θ , et , θ ) . The first word embedding e0 , θ = gθ ( sos ) is initialized with a start of sequence . The state zt , θ is then fed to two different heads ; the value head vθ estimates the value of the expected reward knowing zt , θ and the policy head πθ computes the probability of the chosen word given zt , θ . Then , a sampling function s picks the word wt = s ( πθ ( .|zt , θ ) ) . Finally wt is fed back to gθ to produce the next word and so on until the maximum length T is reached . At training , the word wt is sampled according to the policy whereas it is greedily selected at evaluation . Contrary to prior works ( e.g. , Kottur et al. , 2017 ; Resnick et al. , 2020 ) , we do not restrict the channel capacity of agents to spur the emergence of systematic languages . Instead , we endow Speaker with a message space of size |W |T = 2010 ( |W | = 20 , T = 10 ) , significantly larger than the number of available training examples . Listener architecture . Listener is a neural network that receives Speaker ’ s message m and a set of image candidates C , containing the target image x . It outputs the probability over each image x̃ ∈ C of being the target image . Listener is composed of an LSTM cell hφ , which is initialized to the null vector z−1 , φ . The message m is decoded by processing the sequence of word embeddings et , φ = gφ ( wt ) through the LSTM such as zt , φ = hφ ( zt−1 , φ , et , φ ) . The state zT−1 , φ is then fed to the network pφ to output pm , φ = pφ ( zT−1 , φ ) . In parallel , each image x̃ is projected through the encoder f and the network tφ to obtain the image embedding tx̃ = tφ ( f ( x̃ ) ) . Both message and image embeddings are then compared through a score function score ( m , x̃ , φ ) = cos ( pm , φ‖pm , φ‖2 , tx̃ ‖tx̃‖2 ) . The scores over all images are normalized via a softmax to get a probability πφ ( .|m , C ) . Finally , Listener selects an image by taking the best guess according to πφ , i.e . x̂ ∈ argmaxx̃∈C πφ ( x̃|m , C ) . Architecture details , hyper-parameters and graphical descriptions are provided in Appendices A.2 and A.4 . All experiments are run over 10 seeds .

Summary: This paper aims to scale up the usual Lewis game language emergence setup. Instead of simple synthetic objects, the authors consider scaling up to image datasets such as ImageNet and CelebA. Another aspect they scale up is the task difficulty, by increasing the number of candidates the listener has to choose the correct answer from. Third, they also attempt scaling up the population size. To deal with the instability arising from a large number of candidates, the authors use established stabilization techniques such as minimization of KL divergence between the online policy and an exponentially moving average target policy. The paper also introduces a new metric called Ease and transfer learning (ETL) to measure how well an emergent language transfers to a novel task as an alternative to topographic similarity. Finally, the authors explore some techniques such as imitation learning and inference-time voting to try to leverage a population of agents.