Countering Language Drift via Grounding

1 INTRODUCTION . In the summer of 2017 , the internet was briefly abuzz with the mistaken viral message that a leading AI research lab had to “ unplug its AI ” because it “ had gone rogue ” . What had in fact happened was that two chatbots , under certain conditions , had , rather unsurprisingly , started diverging from their English training data and had instead reverted to their own ungrammatical communication protocol for solving a negotiation task ( Lewis et al. , 2017 ) . Instead of saying something like “ hats have no value for me ” the system would starting saying things like “ hat have zero to me to me to me to me ” . As was soon made clear by the parties involved , this sort of language drift is to be expected if we are optimizing for an external reward , for example one based on whether or not two agents successfully accomplish a negotiation . While language drift is to be expected under external reward , it is natural to ask what we can do to avoid it . Consider policy gradient methods , for example , and suppose we sample an output sequence from an English language decoder for a given task : sampling a non-grammatical sequence might still be rewarded if we manage to solve the task ( e.g. , due to some correct words ; or because an interlocutor understood us anyway , or guessed correctly ) , which would quickly move the decoder away ( drifting ) from English . If we were able to keep drift in check , we could maximize reward while retaining the “ Englishness ” of the decoder , with obvious benefits for interpretability and interaction with humans . That is , while search space size prohibits the direct usage of policy gradient methods for training natural language decoders from scratch , we could prevent pre-trained models from drifting while we optimize for the desired reward using policy gradients . Thus , the ability to stop policy gradient methods from diverging from natural language enables interesting long-term possibilities for exploration : imagine e.g . fine-tuning a pre-trained language model trained on large amounts of data , call it a “ language module ” , for a given generation task with limited data . When training chit-chat dialogue agents , for example , we often want to optimize for some very high-level reward , such as engagingness or consistency , with hardly enough data to learn simple English grammar . Or consider what might happen when we train agents using self-play to actively use natural language to change the other agent ’ s ( mental ) state , rather than having a model passively observe language usage in some corpus or dataset , as usually happens . In this work , we study the question of language drift . Drawing inspiration from Lee et al . ( 2018 ) , we re-cast translation as a communication game . Two machine translation ( MT ) agents—i.e. , encoderdecoder models with attention—are tasked with successfully translating source language sequences Reward to the target language using a third pivot language as an intermediary . The communication channel ( the output of the first agent ’ s decoder , which is fed to the second agent ’ s encoder as input ) is updated via policy gradient methods to optimize for translating into the target language , effectively fine-tuning two separate pre-trained MT models via a pivot language . While the subject of communication need not be language ( e.g. , for Lee et al . ( 2018 ) , agents learn to translate by communicating about images ) , three-way translation via pivot is an excellent way for studying the current problem : we can check exactly to what extent the communicated sequence corresponds to both the intended meaning , as well as to the gold standard sequence . We can think of this setup as an agent aiming to communicate its state to another agent via some protocol—yet in this case , the mental states and intermediary communication protocol are completely interpretable . In what follows , we show that language drift happens , and quite dramatically so , when fine-tuning using policy gradients . We then show that the most intuitive way of solving this problem—adding an “ Englishness ” constraint , such as the log-probability assigned by a language model , to the reward function—does not in fact lead to the desired consequences . Indeed , there is nothing preventing such models from learning to translate “ Two giraffes standing next to a white truck in the savanna ” from French to German via “ Democracy is a political system ” as the English intermediary . Hence , we contend that what is missing is grounding : while language model constraints impose syntactic conformity , they do not lead to semantic correspondence . Humans don ’ t invent unique idiolects for every individual interlocutor , exactly because they are grounded : we share strong priors , social and behavioral norms , and a common sensorimotor experience of our physical environment . Thus , “ not going rogue ” means not only sticking to the prescribed language , but more importantly preserving meaning , which means staying grounded . Our experiments show that fine-tuning the communication channel with visual grounding leads to the highest communication performance ( Fr→En→De ) as well as the best retention of original syntax and intended semantics . Our token frequency analysis corroborates our hypothesis , and shows that grounding is key for preserving the token frequency distribution of the pivot language ( English ) . 2 PRIOR WORK . Our work is inspired by recent work in protocols or languages that emerge from multi-agent interaction ( Lazaridou et al. , 2017 ; Lee et al. , 2018 ; Andreas et al. , 2017 ; Evtimova et al. , 2018 ; Kottur et al. , 2017 ; Havrylov & Titov , 2017 ; Mordatch & Abbeel , 2017 ) . Work on the emergence of language in multi-agent settings goes back a long way ( Steels , 1997 ; Nowak & Krakauer , 1999 ; Kirby , 2001 ; Briscoe , 2002 ; Skyrms , 2010 ) . In our case , we are specifically interested in tabula inscripta agents that are already pre-trained to generate natural language , and we are primarily concerned with keeping their language natural during further training . Reinforcement Learning ( RL ) has been applied to fine-tuning models for various natural language generation tasks , including summarization ( Ranzato et al. , 2015 ; Paulus et al. , 2017 ) , information retrieval ( Nogueira & Cho , 2017 ) , MT ( Gu et al. , 2017 ; Bahdanau et al. , 2016 ) and dialogue ( Li et al. , 2017 ) . Our work can be viewed as fine-tuning MT systems using an intermediary pivot language . In MT , there is a long line of work of pivot-based approaches , most notably Muraki ( 1986 ) and more recently with neural approaches ( Wang et al. , 2017 ; Cheng et al. , 2017 ; Chen et al. , 2018 ) . There has also been work on using visual pivots directly ( Hitschler et al. , 2016 ; Nakayama & Nishida , 2017 ; Lee et al. , 2018 ) . Grounded language learning in general has been shown to give significant practical improvements in various natural language understanding tasks ( Gella et al. , 2017 ; Elliott & Kádár , 2017 ; Chrupała et al. , 2015 ; Kiela et al. , 2017 ; Kádár et al. , 2018 ) . Meanwhile , Bowman et al . ( 2016 ) found a powerful decoder to ignore the latent representation in VAEs for language . 3 TASK AND MODELS . We recast translation as a communication game involving two MT agents : Fr→En and En→De ( see Figure 1 ) . Our dataset consists of N triples of aligned sentences { Fri , Eni , Dei } Ni=1 , where Eni is only used for evaluation . We first feed the French sentence Fri to Agent A , which generates an English message Eni as output . Agent B is then trained to maximize the log likelihood of the ground truth German sentence given the English message , i.e . log p ( Dei|Eni ) . Agent A is trained using REINFORCE ( Williams , 1992 ) with reward R = log pB ( Dei|Eni ) .1 This encourages Agent A to develop helpful communication policies for Agent B , and allows Agent B to adapt to Agent A ’ s new policies . In other words : communication via the pivot language ( English ) is a success if we are able to translate the intended source sequence ( French ) into the desired target sequence ( German ) . Both agents are pre-trained individually before communication , meaning that we start off with English as an intermediate language in the early stages of the game . This work examines what happens to the intermediate language as we fine-tune the system jointly : will the agents keep communicating in English , or diverge ? And if so , what can we do to prevent that from happening ? 3.1 AUXILIARY TASKS . To help reduce the search space of intermediate languages , we use two auxiliary tasks : language modelling ( LM ) and image-caption retrieval ( henceforth called the grounding model ) . Language Model Given a language model pre-trained on a standard English corpus , the log likelihood of the English message informs its general “ Englishness ” . We incorporate this into the reward for Agent A , so that it learns to send messages that are plausible English.2 Reward for Agent A is : RLM = log pB ( Dei|Eni ) + βLM log pLM ( Eni ) . Grounding Model Let us assume we have access to a set of images { Imgi } associated with each triple { Fri , Eni , Dei } . Given a pre-trained image-caption retrieval model , such as VSE++ ( Faghri et al. , 2018 ) , the log likelihood of the image given the English message ( and vice versa ) informs how much the English message is grounded in the original semantic content ( Kiela et al. , 2017 ) . We incorporate the ranking loss into Agent A ’ s reward . RG = log pB ( Dei|Eni ) + βG log pG ( Imgi|Eni ) . Note that βLM , βG are hyperparameters . 3.2 TRAINING OBJECTIVE . For brevity the t-th token in the i-th English sentence Eni ; t is abbreviated to Ent , and Eni to En . Policy Gradient Training At decoding timestep t , Agent A takes an action ( outputs token Ent ) given an environment ( previous hidden states and previous token Ent−1 ) . It receives reward R at the end of the sequence , from which we subtract a state-dependent baseline Rt to reduce variance . Therefore , we maximize ( R −Rt ) log p ( Ent|En < t , Fr ) . In addition , we employ entropy regularization on Agent A ’ s decoder to encourage exploration . Hence , Agent A ’ s overall objective function is given as : LA = αpg ( R−Rt ) log p ( Ent|En < t , Fr ) + αentrH ( p ( Ent|En < t , Fr ) ) − αbMSE ( R , Rt ) , where H and MSE denote entropy and mean squared error losses . Cross Entropy Training Agent B is trained using standard cross entropy loss , i.e . LB = log p ( Det|De < t , En ) . We jointly train both agents by maximizing L = LA + LB . 1We use subscript B to denote that the probability is computed with Agent B . 2We also experimented with a dense LM reward on the word-level , but found this to lead to worse performance . We hypothesize that the model might be focusing too much on the dense LM reward , ignoring the sparse reward for the communication task and leading to poor performance .

Summary: This paper poses and addresses the problem of language drift in multi-agent communication paradigms. When two pretrained natural-language agents are jointly optimized to communicate and solve some external non-linguistic objective, their internal communication often diverges to a code-like, unnatural communication system. This paper solves this “language drift” problem by requiring that the messages between agents be usable as inputs to an image caption retrieval system. They demonstrate that the jointly optimized agents perform best when regularized in this manner to prevent language drift.