Dynamic population-based meta-learning for multi-agent communication with natural language

1 Introduction . Humans excel at large-group coordination , with natural language playing a key role in their problem solving ability [ 49 , 8 ] . Inspired by these findings , in this work our goal is to endow artificial agents with communication skills in natural language , which can in turn allow them to coordinate effectively in three key situations when playing referential games ( i ) with agents they have been paired with during their training ( ii ) with new agents they have not been paired with before and ( iii ) with humans . Our starting point is recent work in the emergent communication literature , which proposes to learn protocols “ from scratch ” , allowing like these agents that have been trained together ( and for a large number of iterations ) to form conventions [ 47 , 24 , 12 , 19 ] . While such communication approaches result in agents achieving higher rewards at test time when paired with their training partners [ 21 , 14 , 23 ] , there is growing evidence that these agents do not acquire communication skills with general properties [ 31 , 1 , 3 , 18 ] but rather learn to form communication protocols that are based solely on idiosyncratic conventions . Interestingly , these type of communication pathologies are also observed in natural languages , where they are linked to the size of a linguistic community [ 42 , 54 , 50 , 35 ] , i.e . the smaller the ⇤Work partially done during an internship at DeepMind . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . community the more complex the languages get . This has recently inspired the use of populationbased approaches to multi-agent communication in which agents within a population interact with one another [ 48 , 5 , 11 , 13 ] . One such approach is L2C [ 32 ] which uses meta-learning approaches like MAML [ 10 ] combined with a static population of agents . Indeed , L2C results in communication protocols with some primitive forms of compositionality , allowing agents to successfully communicate with unseen partners . Nevertheless , even in this case , the induced protocols fail to adapt to natural language , leaving open questions with regard to coordination with humans . We hypothesize that a potential bottleneck of this and related approaches lies in the use of static populations . Intuitively , the initial diversity induced by different random initializations of the population gradually vanishes when given enough capacity and training iterations , thus allowing all agents in the population to co-adapt to each other and converge to the same ad-hoc conventions . Simply put , it is challenging to maintain diversity in static populations , which is what drives the learning of generalizable protocols . With this in mind , we propose to address this issue by dynamically building the population ( §3 ) , to gradually induce constructive diversity within the population , so as to enable population-based algorithms ( like meta-learning ) to train agents that can then generalize to seen and unseen partners as well as humans.Since communication with humans is one of our goals , grounding to natural language is of paramount importance . Consequently , we want the agents to coordinate with humans using limited amount of human data while not drifting away from human behavior . In order to maximize efficiency and minimize the use task-specific language data , we combine two learning signals [ 25 , 33 ] , each bootstrapping the other ( §2 ) . First , maximizing task rewards allows the agents to learn to communicate with one another in a purely self-play manner , but using emergent protocols not grounding in natural language . Hence , combining this signal with supervision from a limited dataset of task-specific language data , allows grounding of the emergent protocols in natural language . All in all , our proposed approaches consists of three different phases ( i.e. , interactive reward-based learning , supervised learning , and meta-learning ) , each imposing an inductive bias . We present extensive experiments in two referential game environments ( an image-based and a textbased defined in §4 ) using natural language communication and compare our model against previous approaches , strong baselines and ablations of our method . Moreover , we follow a holistic approach to evaluating multi-agent communication , reporting results on : ( a ) task performance measured by referential accuracy in ( §4.1 ) ( b ) the agents ’ natural language generation skills by computing BLEU score with language data ( §4.2 ) ( c ) human evaluation of both agents in ( §4.3 ) ( d ) cross-play evaluation with unseen partners in ( §4.5 ) , and ( e ) robustness against implict bias in the dataset ( §4.6 ) . 2 Learning in multi-agent games with natural language : The case of referential games Referential Games Referential games are a type of Lewis Signaling games [ 28 ] that have been used in human as well as artificial language experiments [ 15 , 23 , 9 , 27 , 29 , 13 ] . This fully cooperative game consists of two players , a speaker S and a listener L who interact with each other to solve a common task . In our setup , both agents are parameterized using deep neural networks , where the speaker ’ s and listener ’ s parameters are denoted by ✓ and respectively . The speaker gets as input a target object t , encodes it in its own embedding space and sends a discrete message m = S ( t ) to the listener . The listener receives two inputs , the message m and a set of distractor objects D ( |D| = K ) and the target object t. The listener embeds both of these into a shared vector representation to compute a similarity score between the message and the objects and outputs a prediction t0 about the target object . Both agents are given a reward if the listener is able to correctly predict the target object . We denote the maximum length of the message m with l and the vocabulary set with V . In our work , we use two different types of objects : images and sentences as described in §4 . Interactive Learning The most common approach to train agents to solve the main communicative task in the multi-agent communication framework is by using interactive ( reward-driven ) learning [ 9 , 37 , 15 , 29 ] . The reward function r for both agents is the same and given by : r = 1 if t = t 0 and 0.1 otherwise . We optimize the agents ’ parameters using reinforcement learning , and specifically , using policy gradients ( REINFORCE [ 53 ] ) , similar to [ 9 , 25 ] . The listener is additionally optimized using a supervised learning ( cross-entropy ) loss since we know the ground truth label ( which is the target object ) . The corresponding interactive loss functions for the speaker ( J intS ) and the listener ( J intL ) are given by : J int S ( t ; ✓ ) = r l lX j=1 log p ( mj |m < j , t ; ✓ ) + hsHS ( ✓ ) ( 1 ) J int L ( m , t , D ; ) = r log p ( t 0 |m , t , D ; ) + s log p ( t 0 = t|m , t , D ; ) + hlHL ( ) ( 2 ) where HS and HL denote entropy regularization for the speaker and listener policies respectively . hs and hl are non-negative regularization coefficients and s 0 is a scalar quantity . Behavior Grounding If we wish to enable agents to communicate with humans , it is essential to ground their behavior into the corresponding human distribution . Recent work [ 25 , 33 , 34 ] achieves this by performing supervised learning with the use of a natural language dataset . Consequently , the speaker needs to send messages that are human-interpretable while the listener should be able to understand natural language . To achieve this , we let humans provide descriptions of the objects and collect such ( object , description ) pairs to train our agents using supervised learning . Let us denote this dataset by D with m ⇤ being the description for the target object t present in D. Then the corresponding cross-entropy losses for the speaker ( J supS ) and the listener ( J sup L ) are : J sup S ( m ⇤ , t ; ✓ ) = 1 l lX j=1 |V |X c=1 m ⇤ j , c log p ( mj , c|m < j , t ; ✓ ) ( 3 ) J sup L ( m ⇤ , t , D ; ) = K+1X j=1 1 ( tj=t ) p ( tj |m ⇤ , t , D ; ) ( 4 ) 3 Dynamic Population-based Meta-Learning . The previous two approaches help in learning protocols that are either closer to human prior or are ad-hoc conventions that get emerged during training . A common pitfall of training a single set of speaker/listener is that both agents tend to overfit to their partners resulting in loss of generalization across multiple partners including humans [ 11 , 5 , 29 ] . Since our objective is for the agents to learn robust protocols that match the human distribution and are high rewarding , we adopt a populationbased approach to train agents that generalize across diverse agents in a population . Previous approaches have used population as means to regularize the behavior of the agents using a community of fixed number of agents [ 48 ] or by meta-learning an agent on this fixed population of agents . In a given population , each agent learns a best-response to the other agent with the underlying hypothesis being the meta-agent learning the overall best-response to all agents in the population . Algorithm 1 : Algorithm Input : Dataset D , collection of objects O , randomly initialized speaker parameters ✓0 , listener parameters 0 , meta-speaker parameters # 0 , and meta-listener parameters ' 0 , empty buffers B0S and B0L ✓1 SupervisedLearning ( { m , t } ⇠ D ; ✓0 ) . Eq ( 3 ) 1 SupervisedLearning ( { m , t , D } ⇠ D ; 0 ) . Eq ( 4 ) i 1 repeat B i S B i 1 S S ✓i B i L B i 1 L S i # 00 # i 1 ; ' 0 0 ' i 1 for j 2 { 1 , 2 , . . . , nmeta } do # 0j MetaLearning ( O ; # 0j 1 , { 8 2 B i L } ) . Eq ( 5 ) '0j MetaLearning ( O ; '0j 1 , { 8✓ 2 B i S } ) . Eq ( 6 ) end ✓00 ✓i ; 0 0 0 i ; # i # 0 j ; ' i ' 0 j for l 2 { 1 , 2 , . . . , nint } do ✓0l InteractiveLearning ( { t , D } ⇠ O ; ✓0l 1 , ' i ) . Eq ( 1 ) 0l InteractiveLearning ( { t , D } ⇠ O ; 0l 1 , # i ) . Eq ( 2 ) end ✓000 ✓ 0 l ; 00 0 0 l for m 2 { 1 , 2 , . . . , nsup } do ✓00m SupervisedLearning ( { m , t } ⇠ D ; ✓00m 1 ) . Eq ( 3 ) 00m SupervisedLearning ( { m , t , D } ⇠ D ; 00m 1 ) . Eq ( 4 ) end ✓i+1 ✓ 00 m ; i+1 00 m i i+ 1 until performance of # and ' converge The main aim of meta-learning techniques [ 51 , 45 , 41 , 10 ] is to learn a meta-agent that is able to achieve good generalization across multiple tasks . In the reinforcement learning setting , these tasks can be different sets of goals in the same or different environments . In a multiagent interactive setting , we can reformulate these goals as generalizing to different agents in a population where the agent ’ s partner is its environment . Consequently , meta-learning can only aim to learn generic protocols if there is a rich signal emerging from a diverse set of populations . A previous approach [ 32 ] ( L2C ) investigating learning diverse agents using different seeds for random initialization of the agent parameters . However , this simplistic way of inducing diversity is limited and only works with more toy ( artificial ) languages , thus leaving a big performance gap when agents communicate in natural language [ 33 ] . We hypothesize that the desired useful diversity ( for meta-learning ) is absent in these static populations , hence resulting in lower performance of the meta-agent . We aim to tackle this issue by building a dynamic population in an iterative manner , similar to Policy-Space Response Oracles ( PSRO ) [ 22 ] , which uses empirical game-theoretic analysis in a non-cooperative game to learn a meta-strategy for policy selection . An agent learns a bestresponse to an Oracle which is then added to a buffer that stores this growing population of agents . This buffer is used to obtain an improved Oracle which is used in subsequent iteration . We split the training into two phases as depicted in Fig 2b . We will describe the method for training a meta-listener ( an equivalent method is used for the meta-speaker ) : Learning a meta-listener We train a metalistener that plays with all speakers present in the current population buffer . At each training iteration , the meta-listener aims to distill the behavior of all past listeners that interacted with the current speaker population . Building the speaker population After obtaining the newly trained meta-listener , we expand the speaker population by playing the meta-listener with the speaker that was added to the buffer in the previous iteration . This encourages new behavior to emerge as a result of interacting with the distilled meta-listener . Our hypothesis is that by adopting this iterative mechanism of distillation and expansion , we are able to obtain a diverse population where the behavior of each consecutive agent in the buffer changes gradually . In this work , we use techniques that use gradient descent for optimizing the meta-agent . In particular , we use the popular Model Agnostic Meta Learning ( MAML ) [ 10 ] to train the meta-agents . MAML based approaches provide a good initialization over the parameters to be able to quickly adapt to new tasks . An important point to note here is that in our setup , the task distribution is changing over time and is determined as a result of multi-agent learning . We also show some results using other algorithms that are derived from MAML in the Appendix . We denote the parameters of meta-speaker S m by # and meta-listener Lm by ' . We assume a buffer of speakers denoted by BS and listeners by BL . We split the collection of objects O into two sets Oi and Oo in order to compute the inner and outer loop losses in MAML respectively . Now , we can define the objective functions for the meta-speaker ( J metaSm ) and the meta-listener ( J meta Lm ) as follows : J meta Sm ( O ; # ) = X L2BL J int Sm ⇣ t o ; # ↵r # J int Sm t i ; # ⌘ ( 5 ) J meta Lm ( O ; ' ) = X S2BS J int Lm ⇣ m o , t o , D o ; ' ↵r ' J int Lm m i , t i , D i ; ' ⌘ ( 6 ) where ti 2 Oi , to 2 Oo , mo = S ( to ) , mi = S ( ti ) , and Di and Do are sets of distractor objects sampled from Oi and Oo respectively . Finally , the trained meta-agents are fine-tuned using the dataset D before the testing phase . The fine-tuning losses are the same as the supervised losses Eq ( 3 ) ( 4 ) described in the previous section . Since the number of training iterations is unknown and could be potentially much larger than the size of the buffer the memory can hold , we use reservoir sampling to keep a uniform sample of past agents in the buffer . The detailed algorithm can be found in Algorithm 1 .

Summary: In this paper, the authors aim at training agents that can coordinate with seen, unseen/human partners in a multi-agent communication environment. Instead of using static populations, the authors propose a dynamic population-based meta-learning approach that dynamically builds the population. Such an approach enables the trained agents to generalize to seen and unseen/humans partners. On the other hand, the authors try to use a limited amount of human data to fine-tune the agents to coordinate with humans using natural language. They also compare the proposed method and baselines on two referential games.