Countering Language Drift via Grounding

1 INTRODUCTION . In the summer of 2017 , the internet was briefly abuzz with the mistaken viral message that a leading AI research lab had to “ unplug its AI ” because it “ had gone rogue ” . What had in fact happened was that two chatbots , under certain conditions , had , rather unsurprisingly , started diverging from their English training data and had instead reverted to their own ungrammatical communication protocol for solving a negotiation task ( Lewis et al. , 2017 ) . Instead of saying something like “ hats have no value for me ” the system would starting saying things like “ hat have zero to me to me to me to me ” . As was soon made clear by the parties involved , this sort of language drift is to be expected if we are optimizing for an external reward , for example one based on whether or not two agents successfully accomplish a negotiation . While language drift is to be expected under external reward , it is natural to ask what we can do to avoid it . Consider policy gradient methods , for example , and suppose we sample an output sequence from an English language decoder for a given task : sampling a non-grammatical sequence might still be rewarded if we manage to solve the task ( e.g. , due to some correct words ; or because an interlocutor understood us anyway , or guessed correctly ) , which would quickly move the decoder away ( drifting ) from English . If we were able to keep drift in check , we could maximize reward while retaining the “ Englishness ” of the decoder , with obvious benefits for interpretability and interaction with humans . That is , while search space size prohibits the direct usage of policy gradient methods for training natural language decoders from scratch , we could prevent pre-trained models from drifting while we optimize for the desired reward using policy gradients . Thus , the ability to stop policy gradient methods from diverging from natural language enables interesting long-term possibilities for exploration : imagine e.g . fine-tuning a pre-trained language model trained on large amounts of data , call it a “ language module ” , for a given generation task with limited data . When training chit-chat dialogue agents , for example , we often want to optimize for some very high-level reward , such as engagingness or consistency , with hardly enough data to learn simple English grammar . Or consider what might happen when we train agents using self-play to actively use natural language to change the other agent ’ s ( mental ) state , rather than having a model passively observe language usage in some corpus or dataset , as usually happens . In this work , we study the question of language drift . Drawing inspiration from Lee et al . ( 2018 ) , we re-cast translation as a communication game . Two machine translation ( MT ) agents—i.e. , encoderdecoder models with attention—are tasked with successfully translating source language sequences Reward to the target language using a third pivot language as an intermediary . The communication channel ( the output of the first agent ’ s decoder , which is fed to the second agent ’ s encoder as input ) is updated via policy gradient methods to optimize for translating into the target language , effectively fine-tuning two separate pre-trained MT models via a pivot language . While the subject of communication need not be language ( e.g. , for Lee et al . ( 2018 ) , agents learn to translate by communicating about images ) , three-way translation via pivot is an excellent way for studying the current problem : we can check exactly to what extent the communicated sequence corresponds to both the intended meaning , as well as to the gold standard sequence . We can think of this setup as an agent aiming to communicate its state to another agent via some protocol—yet in this case , the mental states and intermediary communication protocol are completely interpretable . In what follows , we show that language drift happens , and quite dramatically so , when fine-tuning using policy gradients . We then show that the most intuitive way of solving this problem—adding an “ Englishness ” constraint , such as the log-probability assigned by a language model , to the reward function—does not in fact lead to the desired consequences . Indeed , there is nothing preventing such models from learning to translate “ Two giraffes standing next to a white truck in the savanna ” from French to German via “ Democracy is a political system ” as the English intermediary . Hence , we contend that what is missing is grounding : while language model constraints impose syntactic conformity , they do not lead to semantic correspondence . Humans don ’ t invent unique idiolects for every individual interlocutor , exactly because they are grounded : we share strong priors , social and behavioral norms , and a common sensorimotor experience of our physical environment . Thus , “ not going rogue ” means not only sticking to the prescribed language , but more importantly preserving meaning , which means staying grounded . Our experiments show that fine-tuning the communication channel with visual grounding leads to the highest communication performance ( Fr→En→De ) as well as the best retention of original syntax and intended semantics . Our token frequency analysis corroborates our hypothesis , and shows that grounding is key for preserving the token frequency distribution of the pivot language ( English ) . 2 PRIOR WORK . Our work is inspired by recent work in protocols or languages that emerge from multi-agent interaction ( Lazaridou et al. , 2017 ; Lee et al. , 2018 ; Andreas et al. , 2017 ; Evtimova et al. , 2018 ; Kottur et al. , 2017 ; Havrylov & Titov , 2017 ; Mordatch & Abbeel , 2017 ) . Work on the emergence of language in multi-agent settings goes back a long way ( Steels , 1997 ; Nowak & Krakauer , 1999 ; Kirby , 2001 ; Briscoe , 2002 ; Skyrms , 2010 ) . In our case , we are specifically interested in tabula inscripta agents that are already pre-trained to generate natural language , and we are primarily concerned with keeping their language natural during further training . Reinforcement Learning ( RL ) has been applied to fine-tuning models for various natural language generation tasks , including summarization ( Ranzato et al. , 2015 ; Paulus et al. , 2017 ) , information retrieval ( Nogueira & Cho , 2017 ) , MT ( Gu et al. , 2017 ; Bahdanau et al. , 2016 ) and dialogue ( Li et al. , 2017 ) . Our work can be viewed as fine-tuning MT systems using an intermediary pivot language . In MT , there is a long line of work of pivot-based approaches , most notably Muraki ( 1986 ) and more recently with neural approaches ( Wang et al. , 2017 ; Cheng et al. , 2017 ; Chen et al. , 2018 ) . There has also been work on using visual pivots directly ( Hitschler et al. , 2016 ; Nakayama & Nishida , 2017 ; Lee et al. , 2018 ) . Grounded language learning in general has been shown to give significant practical improvements in various natural language understanding tasks ( Gella et al. , 2017 ; Elliott & Kádár , 2017 ; Chrupała et al. , 2015 ; Kiela et al. , 2017 ; Kádár et al. , 2018 ) . Meanwhile , Bowman et al . ( 2016 ) found a powerful decoder to ignore the latent representation in VAEs for language . 3 TASK AND MODELS . We recast translation as a communication game involving two MT agents : Fr→En and En→De ( see Figure 1 ) . Our dataset consists of N triples of aligned sentences { Fri , Eni , Dei } Ni=1 , where Eni is only used for evaluation . We first feed the French sentence Fri to Agent A , which generates an English message Eni as output . Agent B is then trained to maximize the log likelihood of the ground truth German sentence given the English message , i.e . log p ( Dei|Eni ) . Agent A is trained using REINFORCE ( Williams , 1992 ) with reward R = log pB ( Dei|Eni ) .1 This encourages Agent A to develop helpful communication policies for Agent B , and allows Agent B to adapt to Agent A ’ s new policies . In other words : communication via the pivot language ( English ) is a success if we are able to translate the intended source sequence ( French ) into the desired target sequence ( German ) . Both agents are pre-trained individually before communication , meaning that we start off with English as an intermediate language in the early stages of the game . This work examines what happens to the intermediate language as we fine-tune the system jointly : will the agents keep communicating in English , or diverge ? And if so , what can we do to prevent that from happening ? 3.1 AUXILIARY TASKS . To help reduce the search space of intermediate languages , we use two auxiliary tasks : language modelling ( LM ) and image-caption retrieval ( henceforth called the grounding model ) . Language Model Given a language model pre-trained on a standard English corpus , the log likelihood of the English message informs its general “ Englishness ” . We incorporate this into the reward for Agent A , so that it learns to send messages that are plausible English.2 Reward for Agent A is : RLM = log pB ( Dei|Eni ) + βLM log pLM ( Eni ) . Grounding Model Let us assume we have access to a set of images { Imgi } associated with each triple { Fri , Eni , Dei } . Given a pre-trained image-caption retrieval model , such as VSE++ ( Faghri et al. , 2018 ) , the log likelihood of the image given the English message ( and vice versa ) informs how much the English message is grounded in the original semantic content ( Kiela et al. , 2017 ) . We incorporate the ranking loss into Agent A ’ s reward . RG = log pB ( Dei|Eni ) + βG log pG ( Imgi|Eni ) . Note that βLM , βG are hyperparameters . 3.2 TRAINING OBJECTIVE . For brevity the t-th token in the i-th English sentence Eni ; t is abbreviated to Ent , and Eni to En . Policy Gradient Training At decoding timestep t , Agent A takes an action ( outputs token Ent ) given an environment ( previous hidden states and previous token Ent−1 ) . It receives reward R at the end of the sequence , from which we subtract a state-dependent baseline Rt to reduce variance . Therefore , we maximize ( R −Rt ) log p ( Ent|En < t , Fr ) . In addition , we employ entropy regularization on Agent A ’ s decoder to encourage exploration . Hence , Agent A ’ s overall objective function is given as : LA = αpg ( R−Rt ) log p ( Ent|En < t , Fr ) + αentrH ( p ( Ent|En < t , Fr ) ) − αbMSE ( R , Rt ) , where H and MSE denote entropy and mean squared error losses . Cross Entropy Training Agent B is trained using standard cross entropy loss , i.e . LB = log p ( Det|De < t , En ) . We jointly train both agents by maximizing L = LA + LB . 1We use subscript B to denote that the probability is computed with Agent B . 2We also experimented with a dense LM reward on the word-level , but found this to lead to worse performance . We hypothesize that the model might be focusing too much on the dense LM reward , ignoring the sparse reward for the communication task and leading to poor performance .

Summary: This paper tries to verify a hypothesis that language grounding DO help to overcome language drift when two agents creating their own protocol in order to communicate with each other. There are several constraints to enforce: 1) naturalness, say "Englishness", 2) grounded in visual semantics. The experiments prove that both constraints help the most (say, BLUE score). 1) w/o 2) restricts the vocabulary into a small set with the most frequent words, while 1) with 2) can resemble the original distribution.