Compositional languages emerge in a neural iterated learning model

1 INTRODUCTION . Natural language understanding ( NLU ) , which is exemplified by challenging problems such as machine reading comprehension , question answering and machine translation , plays a crucial role in artificial intelligence systems . So far , most of the existing methods focus on building statistical associations between textual inputs and semantic representations , e.g . using first-order logic ( Manning et al. , 1999 ) or other types of representations such as abstract meaning representation ( Banarescu et al. , 2013 ) . Recently , grounded language learning has gradually attracted attention in various domains , inspired by the hypothesis that early language learning was focused on problemsolving ( Kirby & Hurford , 2002 ) . While related to NLU , it focuses on the pragmatics ( Clark , 1996 ) of learning natural language , as it implies learning language from scratch , grounded in experience . This research is often practiced through the development of neural agents which are made to communicate with each other to accomplish specific tasks ( for example , playing a game ) . During this process , the agents build mappings between the concepts they wish to communicate about and the symbols used to represent them . These mappings are usually referred to as ‘ emergent language ’ . So far , an array of recent work ( Havrylov & Titov , 2017 ; Mordatch & Abbeel , 2018 ; Kottur et al. , 2017 ; Foerster et al. , 2016 ) has shown that in many game settings , the neural agents can use their emergent language to exchange useful coordinating information . While the best way to design games to favour language emergence is still open to debate , there is a consensus on the fact that we should gear these emergent languages towards sharing similarities with natural language . Among the properties of natural language , compositionality is considered to be critical , because it enables representation of complex concepts through the combinination of several simple ones . While work on incorporating compositionality into emergent languages is still in its early stage , several experiments have already demonstrated that by properly choosing the maximum message length and vocabulary size , the agents can be brought together to develop a compositional language that shares similarities with natural language ( Li & Bowling , 2019 ; Lazaridou et al. , 2018 ; Cogswell et al. , 2019 ) . In a different body of language research literature , evolutionary linguists have already studied the origins of compositionality for decades ( Kirby & Hurford , 2002 ; Kirby et al. , 2014 ; 2015 ) . They proposed a cultural evolutionary account of the origins of compositionality and designed a framework called iterated learning to simulate the language evolution process , based on the idea that the simulated language must be learned by new speakers at each generation , while also being used for communication . Their experiments show that highly compositional languages may indeed emerge through iterated learning . However , the models they introduced were mainly studied by means of experiments with human participants , in which the compositional languages is favored by the participants because human brain favors structure . Hence , directly applying this framework to ground language learning is not straightforward : we should first verify the existence of the preference of compositional language at the neural agent , and then design an effective training procedure for the neural agent to amplify such an advantage . In this project , we analyze whether and how the learning speed advantage of the highly compositional languages exists in the context of communication between neural agents playing a game . Then we propose a three-phase neural iterated learning algorithm ( NIL ) and a probabilistic explanation of it . The experimental results demonstrate that our algorithm can significantly enhance the topological similarity ( Brighton & Kirby , 2006 ) between the emergent language and the original meaning space in a simple referential game ( Lewis , 1969 ) . Such highly compositional languages also generalize better , because they perform well on a held-out validation set . We highlight our contribution as : • We discover the learning speed advantages of languages with high topological similarity for neural agents communicating in order to play a referential game . • We propose the NIL based on those advantages , which is quite robust compared to most of the related works . • We propose a probabilistic framework to explain the mechanisms of NIL . 2 BACKGROUND . 2.1 REFERENTIAL GAME . We analyze a typical and straightforward object selection game , in which a speaking agent ( Alice , or speaker ) and a listening agent ( Bob , or listener ) must cooperate to accomplish a task . In each round of the game , we show Alice a target object x selected from an object space X and let her send a discrete-sequence message m to Bob . We then show Bob c different objects ( x must be one of them ) and use c1 , ... , cc ∈ X to represent these candidates . Bob must use the message received from Alice to select the object that Alice refers among the c candidates . If Bob ’ s selection c̄ is correct , both Alice and Bob are rewarded . The objects are shuffled and candidates are randomly selected in each round to avoid the agents recognizing the objects using their order of presentation . In our game , each object in X has Na attributes ( color and shape are often used in the literature ) , and each attribute has Nv possible values . To represent objects , similarly to the settings chosen in ( Kottur et al. , 2017 ) , we encode each attribute as a one-hot vector and concatenate the Na one-hot vectors to represent one object . The message delivered by Alice is a fixed-length discrete sequence m = ( m1 , ... , mNL ) , in which each mi is selected from a fixed size meaningless vocabulary V . 2.2 NEURAL AGENT STRUCTURES . Neural agents usually have separate modules for speaking and listening , which we name Alice and Bob . Their architectures , shown in Figure 1 , are similar to those studied in ( Havrylov & Titov , 2017 ) and ( Lazaridou et al. , 2018 ) . Alice first applies a multi-layer perceptron ( MLP ) to encode x into an embedding , then feeds it to an encoding LSTM ( Hochreiter & Schmidhuber , 1997 ) . Its output will go through a softmax layer , which we use to generate the message m1 , m2 , · · · . Bob uses a decoding LSTM to read the message and uses a MLP to encode c1 , ... , cc into embeddings . Bob then takes the dot product between the hidden states of the decoding LSTM and the embeddings to generate scores sc for each object . These scores are then used to calculate the cross-entropy loss when training Bob . When Alice and Bob are trained using reinforcement learning , we can use pA ( m|x ; θA ) and pB ( c̄|m , c1 , ... , cc ; θB ) to represent their respective policies , where θA and θB contain the parameters of each of the neural agents . When the agents are trained to play the game together , we use the REINFORCE algorithm ( Williams , 1992 ) to maximize the expected reward under their policies , and add the entropy regularization term to encourage exploration during training , as explained in ( Mnih et al. , 2016 ) . The gradients of the objective function J ( θA , θB ) are : ∇θAJ = E [ R ( c̄ , x ) ∇ log pA ( m|x ) ] + λA∇H [ pA ( m|x ) ] ( 1 ) ∇θBJ = E [ R ( c̄ , x ) ∇ log pB ( c̄|m , c1 , ... , cc ) ] + λB∇H [ pB ( c̄|m , c1 , ... , cc ) ] , ( 2 ) where R ( c̄ , x ) = 1 ( c̄ , x ) is the reward function , H is the standard entropy function , and λA , λB > 0 are hyperparameters . A formal definition of the agents can be found in Appendix C . 2.3 MEASURING COMPOSITIONALITY . Compositionality is a crucial feature of natural languages , allowing us to use small building blocks ( e.g. , words , phrases ) to generate more complex structures ( e.g. , sentences ) , with the meaning of the larger structure being determined by the meaning of its parts ( Clark , 1996 ) . However , there is no consensus on how to quantitatively assess it . Besides a subjective human evaluation , topological similarity has been proposed as a possible quantitative measure ( Brighton & Kirby , 2006 ) . To define topological similarity , we first define the language studied in this work as L ( · ) : X 7→ M. Then we need to measure the distances between pairs of objects : ∆ijX = dX ( xi , xj ) , where dX ( · ) is a distance in X . Similarly , we compute the corresponding quantity for the associated messages mi = L ( xi ) in the message space M with ∆ijM = dM ( mi , mj ) , where dM ( · ) is a distance in M. Then the topological similarity ( i.e. , ρ ) is defined as the correlation between these quantities across X . Following the setup of ( Lazaridou et al. , 2018 ) and ( Li & Bowling , 2019 ) , we use the negative cosine similarity in the object space and Levenshtein distances ( Levenshtein , 1966 ) in the message space . We provide an example in Appendix B to give a better intuition about this metric . 3 NEURAL ITERATED LEARNING MODEL . The idea of iterated learning requires the agent in current generation be partially exposed to the language used in the previous generation . Even this idea is proven to be effective when experimenting with human participants , directly applying it to games played by neural agents is not trivial : for example , we are not sure where to find the preference for high-ρ languages for the neural agents . Besides , we must carefully design an algorithm that can simulate the “ partially exposed ” procedure , which is essential for the success of iterated learning . 3.1 LEARNING SPEED ADVANTAGE FOR THE NEURAL AGENTS . As mentioned before , the preference of high-ρ language by the learning agents is essential for the success of iterated learning . In language evolution , highly compositional languages are favored because they are structurally simple and hence are easier to learn ( Carr et al. , 2017 ) . We believe that a similar phenomenon applies to communication between neural agents : Hypothesis 1 : High topological similarity improves the learning speed of the speaking neural agent . We speculate that high-ρ languages are easier to emulate for a neural agent than low-ρ languages . Concretely , that means that Alice , when pre-trained with object-message pairs describing a high- ρ language at a given generation , will be faster to successfully output the right message for each object . Intuitively , this is because the structured mapping described by a language with high ρ is smoother , and hence has a lower sample complexity , which makes resulting examples easier to learn for the speaker agent ( Vapnik , 2013 ) . Hypothesis 2 : High topological similarity allows the listening agent to successfully recognize more concepts , using less samples . We speculate that high-ρ languages are easier for a neural agent to recognize . That means that Bob , when pre-trained with message-object pairs corresponding to a high-ρ language , will be faster to successfully choose the right object . Intuitively , the lower topological similarity is , the more difficult it will be to infer unseen object-message pairs from seen examples . The more complex mapping of a low-ρ language implies that more object-message pairs need to be provided to describe it . This translates as an inability for the listening agent to generalize the information it obtained from one object-message associated to a low-ρ language to other examples . Thus , the general performance of Bob on any example will improve much faster when trained with pairs corresponding to a high-ρ language than with a low-ρ language . We provide experimental results in section 4.1 to verify our hypotheses . We also provide a detailed example in Appendix D to illustrate our reasoning .

Summary: This paper studies the emergence of compositional language in neural agents. They propose an iterated learning method that consists of three phases: a supervised learning phase for a randomly-initialized speaker and listener, a self-play phase (where both agents are updated together), and a phase where a new dataset is created based on the current speaker’s language. This dataset is then passed on to the next ‘generation’ of speaker and listener. The paper finds that this procedure, with the right hyperparameters, leads to the emergence of more compositional languages in a simple symbolic referential game.