INTERNAL-CONSISTENCY CONSTRAINTS FOR EMERGENT COMMUNICATION

1 INTRODUCTION Emergent communication is the study of how linguistic protocols evolve when agents are tasked to cooperate . For example , agents engaged in a simple object retrieval task learn to communicate with one another in order to get the items they want ( Lazaridou et al. , 2018 ) . To date , work of this type has each agent assume a conversational role . Thus , agents are often trained only to speak or only to listen ( Lazaridou et al. , 2018 ) , or similarily trained to speak using a vocabulary disjoint from the vocabulary it is understands as a listener–e.g . speaking only to ask questions ( “ what color ? ” ) and listening only to comprehend the answer ( “ blue ” ) ( Kottur et al. , 2017 ; Das et al. , 2017 ) . These assumptions are misaligned with how we think about human communication , and with the way we ’ d like computational models to work in practice . As humans , not only can we easily shift between roles , we also know that there is inherent symmetry between these roles : we expect others to speak ( or listen ) similarly to the way we do , and we know that others expect the same of us . We test if dialog agents that incorporate the symmetry between themselves and their communicative partners learn more generalizable representations than those which do not . We introduce three modifications to the agents to encourage that they abide by the “ golden rule ” : speak/listen as you would want to be spoken/listened to . Specifically , these modifications include self-play training objectives , shared embedding spaces , and symmetric decoding and encoding mechanisms that share parameters . We test two hypotheses about the effect of the proposed modifications on emergent communication : 1 . Internal-consistency constraints improve agents ’ ability to generalize to unseen items–e.g . training on “ red square ” and “ blue circle ” and then testing on “ blue square ” . 2 . Internal-consistency constraints improve agents ’ ability to generalize across communicative roles–e.g . training on “ blue ” as a listener , and using “ blue ” as a speaker when testing . We evaluate the effect of each of the proposed modifications with two reference game datasets and two model architectures , an RNN model used by Lazaridou et al . ( 2018 ) and a Transformer model . We find no evidence to support that internal-consistency improves generalization to unseen items ( Hypothesis 1 ) , but significant evidence that these proposed constraints enable models to generalize learned representations across communicative roles ( Hypothesis 2 ) , even in the case of where the agent receives no direct training in the target ( test ) role . All of our code and data are available at bit.ly/internal-consistency-emergent-communication . 2 EMERGENT COMMUNICATION VIA REFERENCE GAMES Following past work on emergent communication ( Lazaridou et al. , 2018 ) , we evaluate our models in a cooperative reference game ( depicted in Fig . 1 ) ( Lewis , 2008 ) . A basic reference game consists of two agents and a set of items ( each a vector of attribute : value pairs ) . One agent ( the “ speaker ” ) has a target item in mind , and must describe it to the other agent ( the “ listener ” ) . After training , we are primarily interested in the properties and the generalizability of the resulting lexicon : the mapping from symbols in the vocabulary onto items and/or their attributes . Implemented naively , agents converge to a trivial lexicon which assigns a unique symbolic name to each item in the training set ( Kottur et al. , 2017 ) . Thus , a challenge in emergent communication is to build agents which learn a lexicon that maps symbols to attributes or other primitive concepts , so the agents can generalize to novel combinations of attributes at test time . For example , if trained on items trshape : circle , color : reds , rshape : square , color : bluesu , an ideal lexicon would map each attribute-value pair to a unique word ( e.g . word1=blue , word2=circle ) so the agents could refer to an unseen combination ( rshape : circle , color : blues ) at test time . Like past work , this paper focuses on evaluating inductive biases that result in lexicons which generalize to novel items . In addition , we focus on a different type of generalization : generalization across communicative roles . Prior work has tended to treat these roles as distinct , such that an agent may learn to comprehend “ blue ” and “ circle ” perfectly as a listener , and yet be unable to refer to rshape : circle , color : blues as a speaker . Thus , we test whether models that are trained in both roles ( either with direct supervision , or only via self-play ) learn lexicons that not only generalize to novel items but also which can be transferred across roles . Notation . The space of possible references is parameterized by the number of attributes nf that describe each item ( e.g . color ) and the number of values nv each attribute can take ( e.g . { red , blue } ) . Each item o is a bag-of-features vector o P t0 , 1uN where N “ nf ¨ nv . Each index oi is 1 if o expresses the ith feature value . The speaker produces a message with symbols from a vocabulary V with length L. For comparison , we use the best-performing setting |V| “ 100 and L “ 10 from previous work ( Lazaridou et al. , 2018 ) . Symbols in V are represented as 1-hot vectors . In each round of the reference game , we construct xC , r , ry where C is the context ( set of item column vectors stacked into a matrix ) , r is a vector representing the referent , and r is the index of the referent in C. We uniformly sample k´1 items as distractors to form C “ to1 , . . .ok´1uYtru . The distractors are is sampled randomly each round ( in every epoch ) . 3 MODELS We begin with a general architecture and training objective to underly all of our models ( Sections 3.1 and 3.2 ) . We then introduce three modifications which can be used to encourage internallyconsistent representations : a self-play training objective , a shared embedding space , and a symmetric decoding and encoding mechanism with shared parameters ( Section 3.3 ) 1 . 1For implementation details , refer to Appendices A.1 , A.2 , A.3 , or source code bit.ly/ internal-consistency-emergent-communication . 3.1 ARCHITECTURE Agents contain four modules . Embedding modules 1 ) Eitem P RNˆD , Emessage P R|V|ˆD . E˚pxq embed items and messages . When speaking , the decoder module 2 ) consumes the embedded referent Eitemprq and produces a discrete message M P VL . Next , when listening , the encoder module 3 ) consumes embedded messages EmessagepMq P RWˆD and then produces a representation of the referent r̂ P RD . Finally , a non-parametric pointing module 4 ) produces a distribution P pCq over the context by matrix multiplying r̂ with the embedded context EitempCq . The decoders emit one symbol at a time , auto-regressively producing fixed-length messages . The messages are discretized with the straight-through Gumbel Softmax ( Jang et al. , 2016 ) as in Mordatch & Abbeel ( 2018 ) . This converts a distribution to a one-hot vector while permitting gradients to flow through the operation and enables the agents to be optimized without reinforcement methods . The Recurrent model uses a LSTM ( Hochreiter & Schmidhuber , 1997 ) decoder when speaking and a LSTM encoder when listening , as in ( Lazaridou et al. , 2018 ) . The Transformer model ( Vaswani et al. , 2017 ) uses a Transformer Decoder when speaking and a Transformer Encoder to encode when listening . See Appendix A for implementation details and hyperparameters . 3.2 TRAINING OBJECTIVES Speaking S : r , θ Ñ M and listening L : C , M , θ Ñ P pCq are both functions where M P VL Ă t0 , 1uLˆ|V| is a discrete-valued message with length L , P pCq is a distribution over the items in context , and θ are optimizable parameters . We optimize the parameters θA , θB of agents A , B over a dataset D to select each referent r from among the distractors in its context C by minimizing the negative log likelihood of selecting the correct referent in context ( Eq 1 ) . In our experiments , the speaker modules and the listener modules instantiate the function S and L respectively . LAÑB “ ÿ xC , r , ryPD ´ logL pC , S pr ; θAq ; θBqr , ( 1 ) 3.3 IMPOSING INTERNAL-CONSISTENCY CONSTRAINTS We investigate three internal-consistency constraints , that encourage internally-consistent representations . Baseline agents consist of two separate sets of parameters , one for listening and one for speaking . For example , the baseline recurrent model consists of two recurrent models . This corresponds to the scenario where agents either speak or listen , but not both ( Lazaridou et al. , 2018 ) . We introduce a 1 ) self-play loss for both agents of the same form as Eq . 1 , except the given agent fulfills both roles , encouraging it to speak/listen to others the same way it speaks/listens to itself . When we use the self-play training objective , we use it for both agents . Next , 2 ) shared embedding agents use the same item embeddings and the same message embeddings when listening and speaking . Finally , 3 ) symmetric encoding and decoding agents use the same parameters ( but different mechanisms ) to decode a message when speaking as it does to encode a message when listening . Parameters are only ever shared between roles within an agent , and never between agents . 4 DATASETS Our evaluation is based on the simple reference game2 as described in Section 2 , played across two datasets . The datasets , summarized in Table 1 , target different aspects of lexical reference . The first , Visual Attributes for Concepts ( CONCEPTS ) Silberer et al . ( 2013 ) , is derived from annotated images of actual items ( animals , tools , etc ) . Thus , the data contains realistic co-occurance patterns : groups of attributes like has-head , has-mouth , and has-snout appears together several times , whereas has-seeds , has-mouth , made-of-metal never co-occur . The intuition is that a good lexicon will exploit the structure by developing words which refer to groups of frequently co-occurring attributes ( e.g . “ mammal ” ) and will describe unseen referents in terms of 2We use a “ context-hidden ” game : the context is visible to the listener but not the speaker . This is to ensure that the speaker does not have access to additional attributes ( e.g . position of item within the context ) that could enable the agents to find degenerate solutions to the game . these primitive concepts . The second dataset , SHAPES , is one we create ourselves in contrast with the CONCEPTS data . In SHAPES , items correspond to abstract shapes which are exactly describable by their attributes ( e.g . blue , shaded , hexagon ) . All the attributes are independent and there is no co-occurence structure to exploit , so a good lexicon should ideally provide a means for uniquely specifying each attribute ’ s value independently of the others . 5 RESULTS We present experimental results aimed at testing the hypotheses stated in Section 1 . To provide intuition , we frame experiments in terms of two agents , “ Alice ” ( Agent A ) and “ Bob ” ( Agent B ) , who are taking turns asking for and fetching toys . 5.1 HYPOTHESIS 1 : GENERALIZATION TO NEW ITEMS 5.1.1 SETUP We first test whether any of the proposed internal-consistency constraints improve the agents ’ ability to generalize to novel items–i.e . items which consist of unseen combinations of features . Here , we focus on the performance when models are trained and tested within the same communicative role . This corresponds to the setting that has typically been used in prior work on emergent communication : Alice always speaks in order to ask for toys , Bob always responds by fetching them , and the pair ’ s success is evaluated in terms of Alice ’ s ability to describe new toys such that Bob correctly gets them . For evaluation , we hold out a subset of the value combinations from each dataset to use for testing . For example , the agents might be trained to refer to trshape : circle , color : reds , rshape : square , color : bluesu and then tested on its ability to refer to rshape : circle , color : blues . We compute validation and test accuracies.3 5.1.2 COMPARISON OF MODEL ARCHITECTURES Before introducing our internal-consistency constraints , we first evaluate our various architectures in this vanilla setting . Thus , these results serve chiefly to calibrate differences between model architectures . If we see differences between the test conditions , these should be attributed to general architectural advantages associated with implementing these internal-consistency constraints , rather than interpreted as evidence for/against internal-consistency per se . Table 2 shows the results . Overall , the transformer architecture outperforms the recurrent architecture , and the transformer outperforms the previously best-reported result for this task ( which is only available on the CONCEPTS data ) . We observe no clear trend associated with the shared embedding module ( sometimes it helps , sometimes it hurts ) , but do see a slight positive effect associated with the symmetric transformer . 5.1.3 EFFECTS OF INTERNAL-CONSISTENCY CONSTRAINTS . We next look at whether there is an advantage to using the the internal-consistency constraints , even when agents remain in fixed conversational roles . Intuitively , this corresponds to a scenario in which both Alice and Bob are capable of performing either role ( speaking or listening ) , but nonetheless , they only ever interact within the same routine : Alice only every speaks and Bob only ever listens and fetches items in response . However , both Alice and Bob can imagine how they would behave if they were to assume the opposite role , and thus , via self-play , each can enforce internal-consistency between their own actions as speaker ( listener ) and the way they imagine they would respond as listener ( speaker ) . In this manner , although Alice and Bob only ever receive direct feedback in 3We additionally tried computing Tree Reconstruction Error as introduced by Andreas ( 2019 ) . See Appendix A.5 for implementation details . However , we did not find an interesting differences across conditions . Therefore , for compactness , we omit these results . one role , they can still impose internally consistent behavior across both roles . It is conceivable that doing so might improve performance even though each agent remains in a fixed role , either by providing the model with additional information about the task structure , or simply by acting as a regularizer . Thus , for completeness , we assess whether the internal-consistency constraints provide any advantage to the models in the vanilla emergent communication setting . Table 3 shows the effect of adding the self-play objective in the fixed-role setting , across architectures and datasets . The trends are mixed : it appears the additional signal only noises the baseline and symmetric models , whereas the shared embeddings models are able to leverage it effectively . Thus , the effect is not clear enough to establish conclusively that the internal-consistency constraints help the agents generalize in this fixed-role setting , and in fact it may hurt . 5.2 HYPOTHESIS 2 : GENERALIZATION TO NEW ROLES 5.2.1 SETUP We now look at whether internal-consistency improves the agents ’ ability to generalize linguistic knowledge across roles . For example , we can picture the following scenario : Alice is speaking to Bob , and asks for the “ truck ” . Bob hands her the doll , and Alice replies negatively , indicating that what she actually wanted was the truck . Now , without additional direct supervision , when Bob wants the truck , will he know do use the word “ truck ” ? Such a setting is particularly relevant in practical settings , for example when robotic agents must reach high accuracy despite only limited access to human interaction for training . We consider two versions of this setting , involving different levels of direct supervision ( i.e . interaction with the other agent ) as described below . Training in one role . Our first experimental setting assumes that Alice and Bob each only receive direct training in one role , e.g . Alice only ever speaks to Bob , so Alice only receives feedback on how she is performing as a speaker , and Bob on how he is performing as a listener . However , both Alice and Bob are able to practice in the opposite role via self-play . This setup is analogous to the experiment just discussed in Section 5.1.3 . However , unlike before , Alice and Bob will be tested in the roles opposite of those in which they were trained . That is , if Alice was trained as a speaker , then she will be tested as a listener ( on her ability to correctly identify items to which Bob refers ) . Training in both roles . In our second experimental setting , we assume Alice and Bob enjoy a healthy friendship , in which both take turns speaking and listening to each other , and thus both receive direct supervision in both roles . However , they do not necessarily receive equal training on every vocabulary item . Rather , there are some contexts in which Alice only speaks and other contexts in which she only listens . Intuitively , this corresponds to a scenario in which Alice speaks exclusively about food ( while Bob listens ) , while Bob speaks exclusively about toys ( while Alice listens ) . We are interested in testing how well Alice is able to speak about toys at test time . We use the SHAPES dataset4 to create two training splits , each having the same attributes but covering disjoint sets of values . For example , the first training split ( train-1 ) might have colorP { blue , red , yellow } whereas the second training split ( train-2 ) has colorP { green , orange , purple } . We use train-1 to train Alice as speaker and Bob as listener and train-2 to train them in the reverse roles . We then report performance with Alice as listener and Bob as speaker using a test set that uses the same attribute values as train-1 . 5.2.2 RESULTS Our results for both training conditions are shown in Table 4 . The baseline model ( which includes no internal-consistency constraints ) performs , unsurprisingly , at chance . Adding the self-play objective gives improvements across the board . Again , while seemingly straight forward , this result has promising practical interpretations in settings in which a model has access to only a small amount of interaction . For example , a human may be willing to train a robot via speaking ( pointing and naming items ) , but not patient enough to train it via listening ( responding to the robot ’ s noisy commands ) . In such a setting , the ability to massively augment performance via self-play is significant . In addition to the self-play objective , we see that enforcing shared-embedding spaces yields further significant performance gains ( in the range of 30 percentage points in some cases ) . The symmetric constraints on top of self-play and shared embeddings seem to hurt performance in general.5 5.3 TAKEAWAYS Overall , when agents can be trained directly in the role in which they are tested , there is no clear evidence that adding internal-consistency constraints improves the ability of agents to generalize 4We can not construct analogous splits with CONCEPTS since the train set is small . 5We found that , in absence of self-play , neither symmetric encoding/decoding nor shared embeddings performed competitively . Therefore , for compactness , we only report the results of the constraints in combination . to new items . However , internal-consistency constraints improve performance significantly when agents have limited ability to train in a given role . Specifically , models which are equipped with selfplay training objectives and shared embedding spaces show superior ability to generalize learned representations across roles , and perform about as well as if they had been trained on the target role . 6 ANALYSIS In this section we provide additional analyses to highlight the effects of internal-consistency ( in particular , self-play ) on training efficiency and on the emerged protocol . Here , we use a smaller SHAPES dataset ( see B.1 ) , and reduce the vocabulary size and message length ( |V| “ 10 , L “ 3 ) . 6.2 DOES SELF-PLAY ENCOURAGE BETTER PROTOCOLS TO EMERGE ? We measure whether self-play leads to better communication protocols in general . First , we measure the agents ’ speaking and listening capacities separately , using measures proposed by Lowe et al . ( 2019 ) ; Eccles et al . ( 2019 ) . Specifically , positive signaling ( S ` ) measures if the speaker ’ s messages depend on the features of the referent , and positive listening ( L ` ) measures if the listener ’ s actions depend on the message6 . Table 5 shows that self-play improves the agents ’ communication as measured by accuracy as well as these orthogonal metrics . We also find that the model architectures and self-play impact the agents ’ lexicons . The recurrent models produces fewer unique messages than the transformer models ( on average 110 versus 300 ) , and often neglect to use all the vocabulary . Fig . 3 shows that self-play helps the recurrent model use more of the vocabulary , and leads to both the recurrent and transformer models to develop sparser mappings from symbols onto features . 6See Appendix B.2 for details . 7 RELATED WORK Work in emergent communication ( Das et al. , 2017 ; Lazaridou et al. , 2018 ) analyzes agents that develop a shared protocol by playing reference games ( Lewis , 2008 ) . Kottur et al . ( 2017 ) presented results showing that computational models do not learn compositional protocols by default . Instead , the agents tend to develop brittle protocols that have trouble generalizing to novel items . Several approaches have been proposed which could encourage models to learn more generalizable representations of language , including compression ( Kirby et al. , 2015 ) , efficiency ( Gibson et al. , 2019 ) , memory constraints ( Kottur et al. , 2017 ) , pragmatic constraints ( Tomlin & Pavlick , 2018 ) , and positive biases to respond to other agents ( Jaques et al. , 2018 ; Eccles et al. , 2019 ) . Some work , like ours , assumes access to symbolic representations of referents and their attributes , whereas others ’ are set in pixel-based multi-agent games ( Jaques et al. , 2018 ; Eccles et al. , 2019 ; Das et al. , 2018 ) or multi-agent grid-worlds ( Sukhbaatar et al. , 2016 ; Lowe et al. , 2019 ) . Our work also relates to a broader body of work on speaker-listener models , specifically pragmatically-informed models in which speakers reason recursively about listeners ’ beliefs ( and vice-versa ) ( Frank & Goodman , 2012 ; Goodman & Frank , 2016 ) . Such models have been used in applications such image captioning ( Andreas & Klein , 2016 ; Yu et al. , 2017 ; Monroe & Potts , 2015 ) , and robotics ( Vogel & Jurafsky , 2010 ; Vogel et al. , 2013 ; Fried et al. , 2018 ) , as well as in linguistics and psychology in order to explain complex linguistic inferences ( Tessler & Goodman , 2016 ; Monroe et al. , 2017 ) . Conceptually , our proposed internal-consistency constraints share something in common with these neural speaker-listener models developed outside of emergent communication . However , again , past work has tended to assume that a speaker ’ s mental model of their listener is not necessarily consistent–in fact , it is often assumed explicitly to be inconsistent ( Frank & Goodman , 2012 ) –with the way the speaker themself would behave as a listener . We note , however , that our proposed model architecture ( because it lacks the recursion typical in other pragmatics models ) is likely unable to handle the types of higher-level inferences ( e.g . implicatures ) targeted by the mentioned prior work on computational pragmatics , though this is an interesting avenue to explore . 8 CONCLUSION We propose three methods for encouraging dialog agents to follow “ the golden rule ” : speak/listen to others as you would expect to be spoken/listened to . In the emergent communication setting , we find that the internal-consistency constraints do not systematically improve models ’ generalization to novel items , but both the self-play objective and shared embeddings significantly improve performance when agents are tested on roles they were not directly trained for . In fact , when trained in one role and tested on another , these internal-consistency constraints allow the agents to perform about as well as if they had been trained in the target role . REFERENCES Jacob Andreas . Measuring compositionality in representation learning . arXiv preprint arXiv:1902.07181 , 2019 . Jacob Andreas and Dan Klein . Reasoning about pragmatics with neural listeners and speakers . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pp . 1173–1182 , Austin , Texas , November 2016 . Association for Computational Linguistics . URL https : //aclweb.org/anthology/D16-1125 . Abhishek Das , Satwik Kottur , José MF Moura , Stefan Lee , and Dhruv Batra . Learning cooperative visual dialog agents with deep reinforcement learning . In Proceedings of the IEEE International Conference on Computer Vision , pp . 2951–2960 , 2017 . Abhishek Das , Théophile Gervet , Joshua Romoff , Dhruv Batra , Devi Parikh , Michael Rabbat , and Joelle Pineau . Tarmac : Targeted multi-agent communication . arXiv preprint arXiv:1810.11187 , 2018 . Tom Eccles , Yoram Bachrach , Guy Lever , Angeliki Lazaridou , and Thore Graepel . Biases for emergent communication in multi-agent reinforcement learning . 2019 . Michael C Frank and Noah D Goodman . Predicting pragmatic reasoning in language games . Science , 336 ( 6084 ) :998–998 , 2012 . Daniel Fried , Ronghang Hu , Volkan Cirik , Anna Rohrbach , Jacob Andreas , Louis-Philippe Morency , Taylor Berg-Kirkpatrick , Kate Saenko , Dan Klein , and Trevor Darrell . Speaker-follower models for vision-and-language navigation . arXiv preprint arXiv:1806.02724 , 2018 . Edward Gibson , Richard Futrell , Steven T Piandadosi , Isabelle Dautriche , Kyle Mahowald , Leon Bergen , and Roger Levy . How efficiency shapes human language . Trends in cognitive sciences , 2019 . Noah D Goodman and Michael C Frank . Pragmatic language interpretation as probabilistic inference . Trends in Cognitive Sciences , 20 ( 11 ) :818–829 , 2016 . Sepp Hochreiter and Jürgen Schmidhuber . Long short-term memory . Neural computation , 9 ( 8 ) : 1735–1780 , 1997 . Eric Jang , Shixiang Gu , and Ben Poole . Categorical reparameterization with gumbel-softmax . arXiv preprint arXiv:1611.01144 , 2016 . Natasha Jaques , Angeliki Lazaridou , Edward Hughes , Caglar Gulcehre , Pedro A Ortega , DJ Strouse , Joel Z Leibo , and Nando de Freitas . Intrinsic social motivation via causal influence in multi-agent rl . arXiv preprint arXiv:1810.08647 , 2018 . Simon Kirby , Monica Tamariz , Hannah Cornish , and Kenny Smith . Compression and communication in the cultural evolution of linguistic structure . Cognition , 141:87–102 , 2015 . Satwik Kottur , José Moura , Stefan Lee , and Dhruv Batra . Natural language does not emerge ‘ naturally ’ in multi-agent dialog . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pp . 2962–2967 , Copenhagen , Denmark , September 2017 . Association for Computational Linguistics . doi : 10.18653/v1/D17-1321 . URL https : //www . aclweb.org/anthology/D17-1321 . Angeliki Lazaridou , Karl Moritz Hermann , Karl Tuyls , and Stephen Clark . Emergence of linguistic communication from referential games with symbolic and pixel input . arXiv preprint arXiv:1804.03984 , 2018 . David Lewis . Convention : A philosophical study . John Wiley & Sons , 2008 . Ryan Lowe , Jakob Foerster , Y-Lan Boureau , Joelle Pineau , and Yann Dauphin . On the pitfalls of measuring emergent communication . In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pp . 693–701 . International Foundation for Autonomous Agents and Multiagent Systems , 2019 . Will Monroe and Christopher Potts . Learning in the rational speech acts model . arXiv preprint arXiv:1510.06807 , 2015 . Will Monroe , Robert X. D. Hawkins , N. D. Goodman , and Christopher Potts . Colors in context : A pragmatic neural model for grounded language understanding . Transactions of the Association for Computational Linguistics , 2017 . Igor Mordatch and Pieter Abbeel . Emergence of grounded compositional language in multi-agent populations . In Thirty-Second AAAI Conference on Artificial Intelligence , 2018 . Carina Silberer , Vittorio Ferrari , and Mirella Lapata . Models of semantic representation with visual attributes . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics ( Volume 1 : Long Papers ) , pp . 572–582 , 2013 . Sainbayar Sukhbaatar , Rob Fergus , et al . Learning multiagent communication with backpropagation . In Advances in Neural Information Processing Systems , pp . 2244–2252 , 2016 . Michael Henry Tessler and Noah D Goodman . A pragmatic theory of generic language . arXiv preprint arXiv:1608.02926 , 2016 . Nicholas Tomlin and Ellie Pavlick . Incremental pragmatics and emergent communication . In Emergent Communication Workshop at NeurIPS , 2018 . Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , Łukasz Kaiser , and Illia Polosukhin . Attention is all you need . In Advances in neural information processing systems , pp . 5998–6008 , 2017 . Adam Vogel and Dan Jurafsky . Learning to follow navigational directions . In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pp . 806–814 . Association for Computational Linguistics , 2010 . Adam Vogel , Max Bodoia , Christopher Potts , and Daniel Jurafsky . Emergence of gricean maxims from multi-agent decision theory . In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies , pp . 1072–1081 , Atlanta , Georgia , June 2013 . Association for Computational Linguistics . URL http : //www.aclweb.org/anthology/N13-1127 . Licheng Yu , Hao Tan , Mohit Bansal , and Tamara L Berg . A joint speakerlistener-reinforcer model for referring expressions . In Computer Vision and Pattern Recognition ( CVPR ) , volume 2 , 2017 . A IMPLEMENTATION DETAILS We use the deep learning framework Pytorch7 ( v1.2.0 ) to implement our models ( and Python 3.7.3 ) . For reproducibility , all random seeds ( random , numpy , torch ) are arbitrarily set to 42 . The general architecture , the four modules that comprise each agent are shown in Figure 4 . A.1 RECURRENT IMPLEMENTATION The recurrent model decodes and encodes message as follows : to generate a message , the first input is the embedding of a SOS start-of-sentence symbol and the initial hidden state is set as the embedded referent ( and the cell memory is all zeroes ) . From here , at each step , the outputted hidden state ( P RD ) is projected by the transposed word embeddings ( E⊺message P RDˆ|V | ) , and the next word is sampled from this resulting distribution across the vocabulary . Moving forward , the next input is the embedding of the sampled word , and the hidden state and cell memory are set those emitted at the previous step . We produce words until the maximum length is reached . When encoding a message , a learned embedding is set to the first hidden state , and the input at each time step is the corresponding embedded word . The last hidden state is set as the encoding . In the symmetric variant of this model , the LSTM cell used for encoding and decoding is the same . 7https : //pytorch.org/docs/master/torch.html A.2 TRANSFORMER IMPLEMENTATION The transformer model decodes and encodes message as follows : to generate a message describing the referent auto-regressively , all the embeddings of the words produced so far M and the embedding of a NEXT symbol are concatenated together into a matrix X ( Eq . 2 ) . Next , the transformer decoder consumes this matrix and the referent embedding and produces a contextualized representation of the input matrix ( Eq . 3 ) . The last column vector X̃ : ,W , which corresponds to the NEXT embedding , is the internal representation of the next word . The next word m is sampled from the projection of this representation with the transposed word embedding . More words are produced in this way until the maximum length message is formed . Producing the i ` 1th word ( so M consists of the first i words ) , works as follows : X “ rEmessagepMq ; ENEXTs P Rpi ` 1qˆD , ( 2 ) X̃ “ TransformerDecoderpX , Eitemprqq P RpW ` 1qˆD . ( 3 ) Mi ` 1 “ GumbelSoftmaxpX̃ : ,i ˆ E ⊺ messageq ( 4 ) To encode incoming messages when listening , all the embeddings of the words in the message plus the embedding of a ITEM symbol are concatenated together into a matrix X ( Eq . 5 ) . Then , the transformer encoder is used to produce a contextualized embedding X̃ ( Eq . 6 ) . The last column vector X̃ : ,W , which corresponds to the ITEM embedding , is set as the message encoding . Note , W is the number of words in each message ( and the length of M ) . X “ rEmessagepMq ; EITEMs P RpW ` 1qˆD , ( 5 ) X̃ “ TransformerEncoderpXq P RpW ` 1qˆD . ( 6 ) r̂ “ X̃ : ,W ( 7 ) A.3 SYMMETRIC AGENTS For the Symmetric Recurrent Model , a LSTM cell is shared between the encoder and decoder . Otherwise , the recurrent model is unchanged . For the Symmetric Transformer Model , Transformer Encoders and Transformer Decoders have different structures , so to share parameters between them , we have change either how the transformer agent speaks or how it listens . We opt to replace the Transformer Decoder with Transformer Encoder , and use it to decode messages in-place when speaking . A.3.1 SYMMETRIC TRANSFORMER IMPLEMENTATION The Symmetric Transformer uses the same mechanism for encoding messages when listening as the default transformer model ( described directly above ) . However , it uses a Transformer Encoder when speaking instead of a Transformer Decode . When speaking , to produce the next symbol , the embeddings of the i words produced so far , the referent embedding , and the embedding of a NEXT symbol are concatenated together into a matrix X ( Eq . 8 ) . The Transformer Encoder then maps X to a contextualized representation X̃ ( Eq . 9 ) . Finally , the column vector in X that corresponds to NEXT , is used to sample the next symbol : X “ rEitemprq ; EmessagepMq ; EITEMs , X P Rpi ` 2qˆD , ( 8 ) X̃ “ TransformerDecoderpXq , X̃ P Rpi ` 2qˆD , ( 9 ) Mi ` 1 “ GumbelSoftmaxpX̃ : ,i ` 1 ˆ E ⊺ messageq , Mi ` 1 P V Ă t0 , 1u|V | . ( 10 ) A.4 HYPER PARAMETERS We uniformly sampled 25 hyperparameter configurations for each model architecture , experiment , and dataset split . In every case , we fixed the hidden size dimensionality and embedding dimensionality to be the same . We searched over three different learning rate schedulers : ( 1 ) None ( or no scheduler , ( 2 ) ReduceLROnPlateau with a patience of 25 , reduction factor of 0.1 , and uses validation accuracy as its measure of progress , and ( 3 ) CyclicLR rising from 0.00001 to the given learning rate over 500 batches and then declines towards 0.00001 for the rest of training ( 10,000 batches ) . This is similar to the Noam Update in Vaswani et al . ( 2017 ) . To save space , we relegate the hyper-parameter selections in our code bit.ly/internal-consistency-emergent-communication – see /lib/hyperparamters.py . A.5 TREE RECONSTRUCTION ERROR TRE takes two important hyperparameters , an error function and a composition function . We select the same choices as the author , for what amounts to the same task ( producing a discrete message ) as detailed in the original paper . This method requires structured feature representations , so we assume that the features in each item are entirely right-branching . The composition function is learned , and set the number of update steps to 1000 . The original implementation is at https : //github.com/jacobandreas/tre , and our modification is at bit.ly/internal-consistency-emergent-communication in the file ./lib/compositionality.py . B FURTHER ANALYSIS B.1 SHAPES SMALL We simplify the SHAPES dataset in order to be able to empirically compute the positive listening and signaling scores , which requires iterating overall possible messages . In the original version of SHAPES this is impractical as there would be 5010 possible messages . The details of this smaller version of SHAPES detailed in Table 9 . We fix the settings for the recurrent and transformer models as we found that a majority of models across experiments used the same parameters . See Tables 10 , 11 . Furthermore , all results in Sec . 6 are averaged over 5 arbitrary random seeds ( both trained and tested ) ( 43 , 44 , 45 , 46 , 46 ) . B.2 POSITIVE SIGNALING AND LISTENING Positive listening was introduced by Lowe et al . ( 2019 ) as Causal Influence of Communication , and the precursor to positive signaling was introduced by Jaques et al . ( 2018 ) . We use the definitions Eccles et al . ( 2019 ) modified for a one step game for both metrics : Positive Listening ( L ` ) . “ DKLpPrpa|mq Prlpaqq , Positive Speaking ( S ` ) . “ Ipm , xq “ Hpmq ´ Hpm|xq , where m is a given message , a are the actions the agent can take , x is state , DKL is the Kuller-bach divergence , I is the mutual information , H is the entropy . We can compute these quantities without sampling messages because the number of messages is tractable . In our setting , the game is a single step , and the average policy over actions independent of the message converges to the uniform distribution over actions , as the order of referents is randomized . Thus we have : L ` “ DKLpPrpa|mq Uq . The positive speaking metric is computed over a sampling of the dataset ( the distractors are random ) as follows as in ( Eccles et al. , 2019 ) : S ` “ Hpmq ´ Hpm|xq , “ ´ ÿ m πpmq log πpmq ` Exr ÿ m πpm|xq log πpm|xqs , where the summations are over all possible messages , x is the given referent , and πpmq is empirically likelihood of the message being produced irrespective of x , and πpm|xq is the likelihood of the given message being produced given the referent x . We report empirical averages of L ` and S ` over all items in the dataset , also averaged over 5 arbitrary random seeds . B.3 ANALYSIS DETAILS Figs . 5 , 6 , 7 , 8 show lexicons across random seeds . See Table 8 for additional results in the vanilla setting for SHAPES-small that were elided for space .

Summary: This paper investigates the question of internal consistency in emergent communication. In other words, the paper aims to answer the question ‘how is emergent communication improved if we enforce the constraint that an agent must speak in the same way that it listens?’ The paper explores three methods of enforcing internal consistency: self-play, shared embeddings, and symmetric encoding/decoding. They find that, while internal consistency does not help generalization to unseen objects, it does allow agents to generalize over conversational roles (i.e. to perform well as a speaker despite only being trained as a listener).