Dynamic population-based meta-learning for multi-agent communication with natural language

1 Introduction . Humans excel at large-group coordination , with natural language playing a key role in their problem solving ability [ 49 , 8 ] . Inspired by these findings , in this work our goal is to endow artificial agents with communication skills in natural language , which can in turn allow them to coordinate effectively in three key situations when playing referential games ( i ) with agents they have been paired with during their training ( ii ) with new agents they have not been paired with before and ( iii ) with humans . Our starting point is recent work in the emergent communication literature , which proposes to learn protocols “ from scratch ” , allowing like these agents that have been trained together ( and for a large number of iterations ) to form conventions [ 47 , 24 , 12 , 19 ] . While such communication approaches result in agents achieving higher rewards at test time when paired with their training partners [ 21 , 14 , 23 ] , there is growing evidence that these agents do not acquire communication skills with general properties [ 31 , 1 , 3 , 18 ] but rather learn to form communication protocols that are based solely on idiosyncratic conventions . Interestingly , these type of communication pathologies are also observed in natural languages , where they are linked to the size of a linguistic community [ 42 , 54 , 50 , 35 ] , i.e . the smaller the ⇤Work partially done during an internship at DeepMind . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . community the more complex the languages get . This has recently inspired the use of populationbased approaches to multi-agent communication in which agents within a population interact with one another [ 48 , 5 , 11 , 13 ] . One such approach is L2C [ 32 ] which uses meta-learning approaches like MAML [ 10 ] combined with a static population of agents . Indeed , L2C results in communication protocols with some primitive forms of compositionality , allowing agents to successfully communicate with unseen partners . Nevertheless , even in this case , the induced protocols fail to adapt to natural language , leaving open questions with regard to coordination with humans . We hypothesize that a potential bottleneck of this and related approaches lies in the use of static populations . Intuitively , the initial diversity induced by different random initializations of the population gradually vanishes when given enough capacity and training iterations , thus allowing all agents in the population to co-adapt to each other and converge to the same ad-hoc conventions . Simply put , it is challenging to maintain diversity in static populations , which is what drives the learning of generalizable protocols . With this in mind , we propose to address this issue by dynamically building the population ( §3 ) , to gradually induce constructive diversity within the population , so as to enable population-based algorithms ( like meta-learning ) to train agents that can then generalize to seen and unseen partners as well as humans.Since communication with humans is one of our goals , grounding to natural language is of paramount importance . Consequently , we want the agents to coordinate with humans using limited amount of human data while not drifting away from human behavior . In order to maximize efficiency and minimize the use task-specific language data , we combine two learning signals [ 25 , 33 ] , each bootstrapping the other ( §2 ) . First , maximizing task rewards allows the agents to learn to communicate with one another in a purely self-play manner , but using emergent protocols not grounding in natural language . Hence , combining this signal with supervision from a limited dataset of task-specific language data , allows grounding of the emergent protocols in natural language . All in all , our proposed approaches consists of three different phases ( i.e. , interactive reward-based learning , supervised learning , and meta-learning ) , each imposing an inductive bias . We present extensive experiments in two referential game environments ( an image-based and a textbased defined in §4 ) using natural language communication and compare our model against previous approaches , strong baselines and ablations of our method . Moreover , we follow a holistic approach to evaluating multi-agent communication , reporting results on : ( a ) task performance measured by referential accuracy in ( §4.1 ) ( b ) the agents ’ natural language generation skills by computing BLEU score with language data ( §4.2 ) ( c ) human evaluation of both agents in ( §4.3 ) ( d ) cross-play evaluation with unseen partners in ( §4.5 ) , and ( e ) robustness against implict bias in the dataset ( §4.6 ) . 2 Learning in multi-agent games with natural language : The case of referential games Referential Games Referential games are a type of Lewis Signaling games [ 28 ] that have been used in human as well as artificial language experiments [ 15 , 23 , 9 , 27 , 29 , 13 ] . This fully cooperative game consists of two players , a speaker S and a listener L who interact with each other to solve a common task . In our setup , both agents are parameterized using deep neural networks , where the speaker ’ s and listener ’ s parameters are denoted by ✓ and respectively . The speaker gets as input a target object t , encodes it in its own embedding space and sends a discrete message m = S ( t ) to the listener . The listener receives two inputs , the message m and a set of distractor objects D ( |D| = K ) and the target object t. The listener embeds both of these into a shared vector representation to compute a similarity score between the message and the objects and outputs a prediction t0 about the target object . Both agents are given a reward if the listener is able to correctly predict the target object . We denote the maximum length of the message m with l and the vocabulary set with V . In our work , we use two different types of objects : images and sentences as described in §4 . Interactive Learning The most common approach to train agents to solve the main communicative task in the multi-agent communication framework is by using interactive ( reward-driven ) learning [ 9 , 37 , 15 , 29 ] . The reward function r for both agents is the same and given by : r = 1 if t = t 0 and 0.1 otherwise . We optimize the agents ’ parameters using reinforcement learning , and specifically , using policy gradients ( REINFORCE [ 53 ] ) , similar to [ 9 , 25 ] . The listener is additionally optimized using a supervised learning ( cross-entropy ) loss since we know the ground truth label ( which is the target object ) . The corresponding interactive loss functions for the speaker ( J intS ) and the listener ( J intL ) are given by : J int S ( t ; ✓ ) = r l lX j=1 log p ( mj |m < j , t ; ✓ ) + hsHS ( ✓ ) ( 1 ) J int L ( m , t , D ; ) = r log p ( t 0 |m , t , D ; ) + s log p ( t 0 = t|m , t , D ; ) + hlHL ( ) ( 2 ) where HS and HL denote entropy regularization for the speaker and listener policies respectively . hs and hl are non-negative regularization coefficients and s 0 is a scalar quantity . Behavior Grounding If we wish to enable agents to communicate with humans , it is essential to ground their behavior into the corresponding human distribution . Recent work [ 25 , 33 , 34 ] achieves this by performing supervised learning with the use of a natural language dataset . Consequently , the speaker needs to send messages that are human-interpretable while the listener should be able to understand natural language . To achieve this , we let humans provide descriptions of the objects and collect such ( object , description ) pairs to train our agents using supervised learning . Let us denote this dataset by D with m ⇤ being the description for the target object t present in D. Then the corresponding cross-entropy losses for the speaker ( J supS ) and the listener ( J sup L ) are : J sup S ( m ⇤ , t ; ✓ ) = 1 l lX j=1 |V |X c=1 m ⇤ j , c log p ( mj , c|m < j , t ; ✓ ) ( 3 ) J sup L ( m ⇤ , t , D ; ) = K+1X j=1 1 ( tj=t ) p ( tj |m ⇤ , t , D ; ) ( 4 ) 3 Dynamic Population-based Meta-Learning . The previous two approaches help in learning protocols that are either closer to human prior or are ad-hoc conventions that get emerged during training . A common pitfall of training a single set of speaker/listener is that both agents tend to overfit to their partners resulting in loss of generalization across multiple partners including humans [ 11 , 5 , 29 ] . Since our objective is for the agents to learn robust protocols that match the human distribution and are high rewarding , we adopt a populationbased approach to train agents that generalize across diverse agents in a population . Previous approaches have used population as means to regularize the behavior of the agents using a community of fixed number of agents [ 48 ] or by meta-learning an agent on this fixed population of agents . In a given population , each agent learns a best-response to the other agent with the underlying hypothesis being the meta-agent learning the overall best-response to all agents in the population . Algorithm 1 : Algorithm Input : Dataset D , collection of objects O , randomly initialized speaker parameters ✓0 , listener parameters 0 , meta-speaker parameters # 0 , and meta-listener parameters ' 0 , empty buffers B0S and B0L ✓1 SupervisedLearning ( { m , t } ⇠ D ; ✓0 ) . Eq ( 3 ) 1 SupervisedLearning ( { m , t , D } ⇠ D ; 0 ) . Eq ( 4 ) i 1 repeat B i S B i 1 S S ✓i B i L B i 1 L S i # 00 # i 1 ; ' 0 0 ' i 1 for j 2 { 1 , 2 , . . . , nmeta } do # 0j MetaLearning ( O ; # 0j 1 , { 8 2 B i L } ) . Eq ( 5 ) '0j MetaLearning ( O ; '0j 1 , { 8✓ 2 B i S } ) . Eq ( 6 ) end ✓00 ✓i ; 0 0 0 i ; # i # 0 j ; ' i ' 0 j for l 2 { 1 , 2 , . . . , nint } do ✓0l InteractiveLearning ( { t , D } ⇠ O ; ✓0l 1 , ' i ) . Eq ( 1 ) 0l InteractiveLearning ( { t , D } ⇠ O ; 0l 1 , # i ) . Eq ( 2 ) end ✓000 ✓ 0 l ; 00 0 0 l for m 2 { 1 , 2 , . . . , nsup } do ✓00m SupervisedLearning ( { m , t } ⇠ D ; ✓00m 1 ) . Eq ( 3 ) 00m SupervisedLearning ( { m , t , D } ⇠ D ; 00m 1 ) . Eq ( 4 ) end ✓i+1 ✓ 00 m ; i+1 00 m i i+ 1 until performance of # and ' converge The main aim of meta-learning techniques [ 51 , 45 , 41 , 10 ] is to learn a meta-agent that is able to achieve good generalization across multiple tasks . In the reinforcement learning setting , these tasks can be different sets of goals in the same or different environments . In a multiagent interactive setting , we can reformulate these goals as generalizing to different agents in a population where the agent ’ s partner is its environment . Consequently , meta-learning can only aim to learn generic protocols if there is a rich signal emerging from a diverse set of populations . A previous approach [ 32 ] ( L2C ) investigating learning diverse agents using different seeds for random initialization of the agent parameters . However , this simplistic way of inducing diversity is limited and only works with more toy ( artificial ) languages , thus leaving a big performance gap when agents communicate in natural language [ 33 ] . We hypothesize that the desired useful diversity ( for meta-learning ) is absent in these static populations , hence resulting in lower performance of the meta-agent . We aim to tackle this issue by building a dynamic population in an iterative manner , similar to Policy-Space Response Oracles ( PSRO ) [ 22 ] , which uses empirical game-theoretic analysis in a non-cooperative game to learn a meta-strategy for policy selection . An agent learns a bestresponse to an Oracle which is then added to a buffer that stores this growing population of agents . This buffer is used to obtain an improved Oracle which is used in subsequent iteration . We split the training into two phases as depicted in Fig 2b . We will describe the method for training a meta-listener ( an equivalent method is used for the meta-speaker ) : Learning a meta-listener We train a metalistener that plays with all speakers present in the current population buffer . At each training iteration , the meta-listener aims to distill the behavior of all past listeners that interacted with the current speaker population . Building the speaker population After obtaining the newly trained meta-listener , we expand the speaker population by playing the meta-listener with the speaker that was added to the buffer in the previous iteration . This encourages new behavior to emerge as a result of interacting with the distilled meta-listener . Our hypothesis is that by adopting this iterative mechanism of distillation and expansion , we are able to obtain a diverse population where the behavior of each consecutive agent in the buffer changes gradually . In this work , we use techniques that use gradient descent for optimizing the meta-agent . In particular , we use the popular Model Agnostic Meta Learning ( MAML ) [ 10 ] to train the meta-agents . MAML based approaches provide a good initialization over the parameters to be able to quickly adapt to new tasks . An important point to note here is that in our setup , the task distribution is changing over time and is determined as a result of multi-agent learning . We also show some results using other algorithms that are derived from MAML in the Appendix . We denote the parameters of meta-speaker S m by # and meta-listener Lm by ' . We assume a buffer of speakers denoted by BS and listeners by BL . We split the collection of objects O into two sets Oi and Oo in order to compute the inner and outer loop losses in MAML respectively . Now , we can define the objective functions for the meta-speaker ( J metaSm ) and the meta-listener ( J meta Lm ) as follows : J meta Sm ( O ; # ) = X L2BL J int Sm ⇣ t o ; # ↵r # J int Sm t i ; # ⌘ ( 5 ) J meta Lm ( O ; ' ) = X S2BS J int Lm ⇣ m o , t o , D o ; ' ↵r ' J int Lm m i , t i , D i ; ' ⌘ ( 6 ) where ti 2 Oi , to 2 Oo , mo = S ( to ) , mi = S ( ti ) , and Di and Do are sets of distractor objects sampled from Oi and Oo respectively . Finally , the trained meta-agents are fine-tuned using the dataset D before the testing phase . The fine-tuning losses are the same as the supervised losses Eq ( 3 ) ( 4 ) described in the previous section . Since the number of training iterations is unknown and could be potentially much larger than the size of the buffer the memory can hold , we use reservoir sampling to keep a uniform sample of past agents in the buffer . The detailed algorithm can be found in Algorithm 1 .

Summary: The paper is motivated by group coordination through natural language and creates a goal of developing agents that can communicate effectively with other agents and humans. They build off of work on emergent communication and their method is most related to L2C [Lowe19], which uses a meta-learning approach to learn communication protocols.  Their training method is as follows: ``` Train with some human data while not_converged:     for n_meta steps:         Learn a meta-listener on a population of speakers     for n_selfplay steps:         Learn new speakers through self-play with the meta-listener     for n_supervised:         Fine-tune on human data train  ```  And their evaluations are: 1. Image based referential games 2. A text based referential game that they design where agents are trained to communicate in German about English sentences  Their experiments show that: 1. They outperform baselines on referential accuracy in both modalities 2. They have a high sentence quality (BLEU, caption score) wrt baselines 3. The meta-speaker in the text task works reasonably effectively as an English to German translation system (BLEU) 4. Their speakers and listeners perform better on human evaluation in the image task  Learning to learn to communicate. Lowe et. al., 2019.