Compositional languages emerge in a neural iterated learning model

1 INTRODUCTION . Natural language understanding ( NLU ) , which is exemplified by challenging problems such as machine reading comprehension , question answering and machine translation , plays a crucial role in artificial intelligence systems . So far , most of the existing methods focus on building statistical associations between textual inputs and semantic representations , e.g . using first-order logic ( Manning et al. , 1999 ) or other types of representations such as abstract meaning representation ( Banarescu et al. , 2013 ) . Recently , grounded language learning has gradually attracted attention in various domains , inspired by the hypothesis that early language learning was focused on problemsolving ( Kirby & Hurford , 2002 ) . While related to NLU , it focuses on the pragmatics ( Clark , 1996 ) of learning natural language , as it implies learning language from scratch , grounded in experience . This research is often practiced through the development of neural agents which are made to communicate with each other to accomplish specific tasks ( for example , playing a game ) . During this process , the agents build mappings between the concepts they wish to communicate about and the symbols used to represent them . These mappings are usually referred to as ‘ emergent language ’ . So far , an array of recent work ( Havrylov & Titov , 2017 ; Mordatch & Abbeel , 2018 ; Kottur et al. , 2017 ; Foerster et al. , 2016 ) has shown that in many game settings , the neural agents can use their emergent language to exchange useful coordinating information . While the best way to design games to favour language emergence is still open to debate , there is a consensus on the fact that we should gear these emergent languages towards sharing similarities with natural language . Among the properties of natural language , compositionality is considered to be critical , because it enables representation of complex concepts through the combinination of several simple ones . While work on incorporating compositionality into emergent languages is still in its early stage , several experiments have already demonstrated that by properly choosing the maximum message length and vocabulary size , the agents can be brought together to develop a compositional language that shares similarities with natural language ( Li & Bowling , 2019 ; Lazaridou et al. , 2018 ; Cogswell et al. , 2019 ) . In a different body of language research literature , evolutionary linguists have already studied the origins of compositionality for decades ( Kirby & Hurford , 2002 ; Kirby et al. , 2014 ; 2015 ) . They proposed a cultural evolutionary account of the origins of compositionality and designed a framework called iterated learning to simulate the language evolution process , based on the idea that the simulated language must be learned by new speakers at each generation , while also being used for communication . Their experiments show that highly compositional languages may indeed emerge through iterated learning . However , the models they introduced were mainly studied by means of experiments with human participants , in which the compositional languages is favored by the participants because human brain favors structure . Hence , directly applying this framework to ground language learning is not straightforward : we should first verify the existence of the preference of compositional language at the neural agent , and then design an effective training procedure for the neural agent to amplify such an advantage . In this project , we analyze whether and how the learning speed advantage of the highly compositional languages exists in the context of communication between neural agents playing a game . Then we propose a three-phase neural iterated learning algorithm ( NIL ) and a probabilistic explanation of it . The experimental results demonstrate that our algorithm can significantly enhance the topological similarity ( Brighton & Kirby , 2006 ) between the emergent language and the original meaning space in a simple referential game ( Lewis , 1969 ) . Such highly compositional languages also generalize better , because they perform well on a held-out validation set . We highlight our contribution as : • We discover the learning speed advantages of languages with high topological similarity for neural agents communicating in order to play a referential game . • We propose the NIL based on those advantages , which is quite robust compared to most of the related works . • We propose a probabilistic framework to explain the mechanisms of NIL . 2 BACKGROUND . 2.1 REFERENTIAL GAME . We analyze a typical and straightforward object selection game , in which a speaking agent ( Alice , or speaker ) and a listening agent ( Bob , or listener ) must cooperate to accomplish a task . In each round of the game , we show Alice a target object x selected from an object space X and let her send a discrete-sequence message m to Bob . We then show Bob c different objects ( x must be one of them ) and use c1 , ... , cc ∈ X to represent these candidates . Bob must use the message received from Alice to select the object that Alice refers among the c candidates . If Bob ’ s selection c̄ is correct , both Alice and Bob are rewarded . The objects are shuffled and candidates are randomly selected in each round to avoid the agents recognizing the objects using their order of presentation . In our game , each object in X has Na attributes ( color and shape are often used in the literature ) , and each attribute has Nv possible values . To represent objects , similarly to the settings chosen in ( Kottur et al. , 2017 ) , we encode each attribute as a one-hot vector and concatenate the Na one-hot vectors to represent one object . The message delivered by Alice is a fixed-length discrete sequence m = ( m1 , ... , mNL ) , in which each mi is selected from a fixed size meaningless vocabulary V . 2.2 NEURAL AGENT STRUCTURES . Neural agents usually have separate modules for speaking and listening , which we name Alice and Bob . Their architectures , shown in Figure 1 , are similar to those studied in ( Havrylov & Titov , 2017 ) and ( Lazaridou et al. , 2018 ) . Alice first applies a multi-layer perceptron ( MLP ) to encode x into an embedding , then feeds it to an encoding LSTM ( Hochreiter & Schmidhuber , 1997 ) . Its output will go through a softmax layer , which we use to generate the message m1 , m2 , · · · . Bob uses a decoding LSTM to read the message and uses a MLP to encode c1 , ... , cc into embeddings . Bob then takes the dot product between the hidden states of the decoding LSTM and the embeddings to generate scores sc for each object . These scores are then used to calculate the cross-entropy loss when training Bob . When Alice and Bob are trained using reinforcement learning , we can use pA ( m|x ; θA ) and pB ( c̄|m , c1 , ... , cc ; θB ) to represent their respective policies , where θA and θB contain the parameters of each of the neural agents . When the agents are trained to play the game together , we use the REINFORCE algorithm ( Williams , 1992 ) to maximize the expected reward under their policies , and add the entropy regularization term to encourage exploration during training , as explained in ( Mnih et al. , 2016 ) . The gradients of the objective function J ( θA , θB ) are : ∇θAJ = E [ R ( c̄ , x ) ∇ log pA ( m|x ) ] + λA∇H [ pA ( m|x ) ] ( 1 ) ∇θBJ = E [ R ( c̄ , x ) ∇ log pB ( c̄|m , c1 , ... , cc ) ] + λB∇H [ pB ( c̄|m , c1 , ... , cc ) ] , ( 2 ) where R ( c̄ , x ) = 1 ( c̄ , x ) is the reward function , H is the standard entropy function , and λA , λB > 0 are hyperparameters . A formal definition of the agents can be found in Appendix C . 2.3 MEASURING COMPOSITIONALITY . Compositionality is a crucial feature of natural languages , allowing us to use small building blocks ( e.g. , words , phrases ) to generate more complex structures ( e.g. , sentences ) , with the meaning of the larger structure being determined by the meaning of its parts ( Clark , 1996 ) . However , there is no consensus on how to quantitatively assess it . Besides a subjective human evaluation , topological similarity has been proposed as a possible quantitative measure ( Brighton & Kirby , 2006 ) . To define topological similarity , we first define the language studied in this work as L ( · ) : X 7→ M. Then we need to measure the distances between pairs of objects : ∆ijX = dX ( xi , xj ) , where dX ( · ) is a distance in X . Similarly , we compute the corresponding quantity for the associated messages mi = L ( xi ) in the message space M with ∆ijM = dM ( mi , mj ) , where dM ( · ) is a distance in M. Then the topological similarity ( i.e. , ρ ) is defined as the correlation between these quantities across X . Following the setup of ( Lazaridou et al. , 2018 ) and ( Li & Bowling , 2019 ) , we use the negative cosine similarity in the object space and Levenshtein distances ( Levenshtein , 1966 ) in the message space . We provide an example in Appendix B to give a better intuition about this metric . 3 NEURAL ITERATED LEARNING MODEL . The idea of iterated learning requires the agent in current generation be partially exposed to the language used in the previous generation . Even this idea is proven to be effective when experimenting with human participants , directly applying it to games played by neural agents is not trivial : for example , we are not sure where to find the preference for high-ρ languages for the neural agents . Besides , we must carefully design an algorithm that can simulate the “ partially exposed ” procedure , which is essential for the success of iterated learning . 3.1 LEARNING SPEED ADVANTAGE FOR THE NEURAL AGENTS . As mentioned before , the preference of high-ρ language by the learning agents is essential for the success of iterated learning . In language evolution , highly compositional languages are favored because they are structurally simple and hence are easier to learn ( Carr et al. , 2017 ) . We believe that a similar phenomenon applies to communication between neural agents : Hypothesis 1 : High topological similarity improves the learning speed of the speaking neural agent . We speculate that high-ρ languages are easier to emulate for a neural agent than low-ρ languages . Concretely , that means that Alice , when pre-trained with object-message pairs describing a high- ρ language at a given generation , will be faster to successfully output the right message for each object . Intuitively , this is because the structured mapping described by a language with high ρ is smoother , and hence has a lower sample complexity , which makes resulting examples easier to learn for the speaker agent ( Vapnik , 2013 ) . Hypothesis 2 : High topological similarity allows the listening agent to successfully recognize more concepts , using less samples . We speculate that high-ρ languages are easier for a neural agent to recognize . That means that Bob , when pre-trained with message-object pairs corresponding to a high-ρ language , will be faster to successfully choose the right object . Intuitively , the lower topological similarity is , the more difficult it will be to infer unseen object-message pairs from seen examples . The more complex mapping of a low-ρ language implies that more object-message pairs need to be provided to describe it . This translates as an inability for the listening agent to generalize the information it obtained from one object-message associated to a low-ρ language to other examples . Thus , the general performance of Bob on any example will improve much faster when trained with pairs corresponding to a high-ρ language than with a low-ρ language . We provide experimental results in section 4.1 to verify our hypotheses . We also provide a detailed example in Appendix D to illustrate our reasoning .

Summary: This paper proposed a neural iterated learning algorithm to encourage the dominance of high compositional language in the multi-agent communication game. The author shows that the iterative training of two agents playing a referential game can incrementally increase the agent to use the language with high topological similarity. The authors also demonstrated that topological similarity is correlated with zero-shot performance. And Experiment results show the authors could propose alternative pre-training strategies for the neural agent can prefer high compositional language and achieve high task performance.