Year: 2018

Abstract:
While reinforcement learning (RL) shows a lot of promise for natural language processing—e.g. when fine-tuning natural language systems for optimizing a certain objective—there has been little investigation into potential language drift: when an external reward is used to train a system, the agents’ communication protocol may easily and radically diverge from natural language. By re-casting translation as a communication game, we show that language drift indeed happens when pre-trained agents are fine-tuned with policy gradient methods. We contend that simply adding a “naturalness” constraint to the reward, e.g. by using language model log likelihood, does not fully address the issue, and argue that (perceptual) grounding is required. That is, while language model constraints impose syntactic conformity, they do not lead to semantic correspondence. Our experiments show that grounded models give the best communication performance, while retaining English syntax along with the ability to convey the intended semantics.

heading: 1 INTRODUCTION
text: In the summer of 2017, the internet was briefly abuzz with the mistaken viral message that a leading AI research lab had to “unplug its AI” because it “had gone rogue”. What had in fact happened was that two chatbots, under certain conditions, had, rather unsurprisingly, started diverging from their English training data and had instead reverted to their own ungrammatical communication protocol for solving a negotiation task (Lewis et al., 2017). Instead of saying something like “hats have no value for me” the system would starting saying things like “hat have zero to me to me to me to me”. As was soon made clear by the parties involved, this sort of language drift is to be expected if we are optimizing for an external reward, for example one based on whether or not two agents successfully accomplish a negotiation.
While language drift is to be expected under external reward, it is natural to ask what we can do to avoid it. Consider policy gradient methods, for example, and suppose we sample an output sequence from an English language decoder for a given task: sampling a non-grammatical sequence might still be rewarded if we manage to solve the task (e.g., due to some correct words; or because an interlocutor understood us anyway, or guessed correctly), which would quickly move the decoder away (drifting) from English. If we were able to keep drift in check, we could maximize reward while retaining the “Englishness” of the decoder, with obvious benefits for interpretability and interaction with humans. That is, while search space size prohibits the direct usage of policy gradient methods for training natural language decoders from scratch, we could prevent pre-trained models from drifting while we optimize for the desired reward using policy gradients.
Thus, the ability to stop policy gradient methods from diverging from natural language enables interesting long-term possibilities for exploration: imagine e.g. fine-tuning a pre-trained language model trained on large amounts of data, call it a “language module”, for a given generation task with limited data. When training chit-chat dialogue agents, for example, we often want to optimize for some very high-level reward, such as engagingness or consistency, with hardly enough data to learn simple English grammar. Or consider what might happen when we train agents using self-play to actively use natural language to change the other agent’s (mental) state, rather than having a model passively observe language usage in some corpus or dataset, as usually happens.
In this work, we study the question of language drift. Drawing inspiration from Lee et al. (2018), we re-cast translation as a communication game. Two machine translation (MT) agents—i.e., encoderdecoder models with attention—are tasked with successfully translating source language sequences
Reward
to the target language using a third pivot language as an intermediary. The communication channel (the output of the first agent’s decoder, which is fed to the second agent’s encoder as input) is updated via policy gradient methods to optimize for translating into the target language, effectively fine-tuning two separate pre-trained MT models via a pivot language.
While the subject of communication need not be language (e.g., for Lee et al. (2018), agents learn to translate by communicating about images), three-way translation via pivot is an excellent way for studying the current problem: we can check exactly to what extent the communicated sequence corresponds to both the intended meaning, as well as to the gold standard sequence. We can think of this setup as an agent aiming to communicate its state to another agent via some protocol—yet in this case, the mental states and intermediary communication protocol are completely interpretable.
In what follows, we show that language drift happens, and quite dramatically so, when fine-tuning using policy gradients. We then show that the most intuitive way of solving this problem—adding an “Englishness” constraint, such as the log-probability assigned by a language model, to the reward function—does not in fact lead to the desired consequences. Indeed, there is nothing preventing such models from learning to translate “Two giraffes standing next to a white truck in the savanna” from French to German via “Democracy is a political system” as the English intermediary.
Hence, we contend that what is missing is grounding: while language model constraints impose syntactic conformity, they do not lead to semantic correspondence. Humans don’t invent unique idiolects for every individual interlocutor, exactly because they are grounded: we share strong priors, social and behavioral norms, and a common sensorimotor experience of our physical environment. Thus, “not going rogue” means not only sticking to the prescribed language, but more importantly preserving meaning, which means staying grounded.
Our experiments show that fine-tuning the communication channel with visual grounding leads to the highest communication performance (Fr→En→De) as well as the best retention of original syntax and intended semantics. Our token frequency analysis corroborates our hypothesis, and shows that grounding is key for preserving the token frequency distribution of the pivot language (English).

heading: 2 PRIOR WORK
text: Our work is inspired by recent work in protocols or languages that emerge from multi-agent interaction (Lazaridou et al., 2017; Lee et al., 2018; Andreas et al., 2017; Evtimova et al., 2018; Kottur et al., 2017; Havrylov & Titov, 2017; Mordatch & Abbeel, 2017). Work on the emergence of language in multi-agent settings goes back a long way (Steels, 1997; Nowak & Krakauer, 1999; Kirby, 2001; Briscoe, 2002; Skyrms, 2010). In our case, we are specifically interested in tabula inscripta agents that are already pre-trained to generate natural language, and we are primarily concerned with keeping their language natural during further training.
Reinforcement Learning (RL) has been applied to fine-tuning models for various natural language generation tasks, including summarization (Ranzato et al., 2015; Paulus et al., 2017), information retrieval (Nogueira & Cho, 2017), MT (Gu et al., 2017; Bahdanau et al., 2016) and dialogue (Li et al., 2017). Our work can be viewed as fine-tuning MT systems using an intermediary pivot language. In MT, there is a long line of work of pivot-based approaches, most notably Muraki (1986) and more recently with neural approaches (Wang et al., 2017; Cheng et al., 2017; Chen et al., 2018). There has also been work on using visual pivots directly (Hitschler et al., 2016; Nakayama & Nishida, 2017; Lee et al., 2018). Grounded language learning in general has been shown to give significant practical improvements in various natural language understanding tasks (Gella et al., 2017; Elliott & Kádár, 2017; Chrupała et al., 2015; Kiela et al., 2017; Kádár et al., 2018). Meanwhile, Bowman et al. (2016) found a powerful decoder to ignore the latent representation in VAEs for language.

heading: 3 TASK AND MODELS
text: We recast translation as a communication game involving two MT agents: Fr→En and En→De (see Figure 1). Our dataset consists of N triples of aligned sentences {Fri,Eni,Dei}Ni=1, where Eni is only used for evaluation. We first feed the French sentence Fri to Agent A, which generates an English message Eni as output. Agent B is then trained to maximize the log likelihood of the ground truth German sentence given the English message, i.e. log p(Dei|Eni). Agent A is trained using REINFORCE (Williams, 1992) with reward R = log pB(Dei|Eni).1 This encourages Agent A to develop helpful communication policies for Agent B, and allows Agent B to adapt to Agent A’s new policies. In other words: communication via the pivot language (English) is a success if we are able to translate the intended source sequence (French) into the desired target sequence (German).
Both agents are pre-trained individually before communication, meaning that we start off with English as an intermediate language in the early stages of the game. This work examines what happens to the intermediate language as we fine-tune the system jointly: will the agents keep communicating in English, or diverge? And if so, what can we do to prevent that from happening?

heading: 3.1 AUXILIARY TASKS
text: To help reduce the search space of intermediate languages, we use two auxiliary tasks: language modelling (LM) and image-caption retrieval (henceforth called the grounding model).
Language Model Given a language model pre-trained on a standard English corpus, the log likelihood of the English message informs its general “Englishness”. We incorporate this into the reward for Agent A, so that it learns to send messages that are plausible English.2 Reward for Agent A is:
RLM = log pB(Dei|Eni) + βLM log pLM (Eni).
Grounding Model Let us assume we have access to a set of images {Imgi} associated with each triple {Fri,Eni,Dei}. Given a pre-trained image-caption retrieval model, such as VSE++ (Faghri et al., 2018), the log likelihood of the image given the English message (and vice versa) informs how much the English message is grounded in the original semantic content (Kiela et al., 2017). We incorporate the ranking loss into Agent A’s reward.
RG = log pB(Dei|Eni) + βG log pG(Imgi|Eni).
Note that βLM , βG are hyperparameters.

heading: 3.2 TRAINING OBJECTIVE
text: For brevity the t-th token in the i-th English sentence Eni;t is abbreviated to Ent, and Eni to En.
Policy Gradient Training At decoding timestep t, Agent A takes an action (outputs token Ent) given an environment (previous hidden states and previous token Ent−1). It receives reward R at the end of the sequence, from which we subtract a state-dependent baseline Rt to reduce variance. Therefore, we maximize (R −Rt) log p(Ent|En<t,Fr). In addition, we employ entropy regularization on Agent A’s decoder to encourage exploration. Hence, Agent A’s overall objective function is given as:
LA = αpg(R−Rt) log p(Ent|En<t,Fr) + αentrH(p(Ent|En<t,Fr))− αbMSE(R,Rt), where H and MSE denote entropy and mean squared error losses.
Cross Entropy Training Agent B is trained using standard cross entropy loss, i.e. LB = log p(Det|De<t,En).
We jointly train both agents by maximizing L = LA + LB . 1We use subscript B to denote that the probability is computed with Agent B. 2We also experimented with a dense LM reward on the word-level, but found this to lead to worse performance. We hypothesize that the model might be focusing too much on the dense LM reward, ignoring the sparse reward for the communication task and leading to poor performance.

heading: 4 EXPERIMENTAL SETTINGS
text: In this section we provide the details of our experimental setup: a Fr→X→De translation task where the intermediate language X is initialized as English, and subsequently fine-tuned with policy gradient. On a trilingual corpus consisting of three languages (Fr, En and De), we can measure communication success with Fr→De BLEU (Papineni et al., 2002), while Fr→En BLEU informs how closely the intermediate language resembles English, at any given point during fine-tuning.
Datasets Agents are initially pre-trained on IWSLT Fr→En and En→De. Fine-tuning is performed on Multi30k Task 1(Elliott et al., 2016). That is, importantly, there is no overlap in the pre-training data and the fine-tuning data. Multi30k Task 1 consists of 30k images and one caption per image in English, French, German and Czech (of which we only use the first three). For the English language model, we compare four different datasets: WikiText103, MS COCO and Flickr30k. The image-caption retrieval model is trained on Flickr30k: the same set of 30k images as Multi30k but containing 5 English captions per image. Following Faghri et al. (2018), we randomly crop training images at every epoch. We use 2048-dimensional final-layer features from a pretrained and fixed ResNet-152 (He et al., 2016).
Preprocessing The same tokenization and vocabulary are used across different tasks and datasets. We lowercase and tokenize our corpora with Moses (Koehn et al., 2007) and use subword tokenization with Byte Pair Encoding (BPE) (Sennrich et al., 2016) with 10k merge operations. This allows us to use the same vocabulary across different models seamlessly (translation, language model, image-caption ranker model).
Controlling the English message length When fine-tuning the agents, we observe that the length of English messages becomes excessively long. As Agent A has no explicit incentive to output the 〈EOS〉 symbol, it tends to keep transmitting the same token repeatedly. Excessively long messages obscure evaluation of the communication protocol. For instance, BLEU score quickly deteriorates as the message length becomes longer, as it is a precision metric. When the message length is fixed, a drop in BLEU score will by necessity mean that the intermediate language has drifted away more. For this reason, we constrain the length of English messages to be no longer than the length of their French source sentence, or shorter if the model outputs the 〈EOS〉 symbol early. Recall that Agent B is supervised to predict the 〈EOS〉 symbol, so does not suffer from this issue.
Model Architecture and Pretraining Our MT agents are standard sequence-to-sequence models with attention (Bahdanau et al., 2015) with unidirectional, 1-layer GRU with 256 hidden units and 256-dimensional embeddings. During initial pre-training on IWSLT, we early-stop based on BLEU score on the development set (tst2013). The best checkpoints give 34.05 BLEU and 21.94 BLEU on IWSLT Fr→En and En→De development sets with greedy decoding. For our value function, we use a 2-layer MLP with a ReLU nonlinearity.
The language model is a 1-layer recurrent language model with 512 LSTM hidden units. The imagecaption retrieval model is a recently proposed VSE++ model (Faghri et al., 2018), with unidirectional 1-layer GRU with 512 hidden units and a single fully connected layer from 2048-dimensional ResNet features to 512-dimensional GRU hidden states. We report the performance of the pretrained models used in our experiments in Tables 1, 2 and 3.
IWSLT Multi30k
Fr→En 34.05 26.80 En→De 21.94 18.56
Table 1: Translation performance of our pre-trained agents (BLEU)
WikiText103 3.51 MS COCO 2.66
Flickr30k 2.85
Table 2: Development NLL of pretrained language models
Training Details When fine-tuning our agents, we perform learning rate annealing and early stopping based on Fr→De BLEU (communication performance) on the Multi30k development
set. We use Adam (Kingma & Ba, 2014) with initial learning rate of 0.001 and dropout (Srivastava et al., 2014) rate of 0.1. We grid search over learning rate schedule and reward coefficients (αpg, αentr, αb, βLM , βG).
For our joint systems with policy gradient fine-tuning, we run every model three times with different random seeds and report averaged results (see Table 4).
Baseline and Upper Bound Our main quantitative experiment has three baselines:
• Pretrained checkpoints (on IWSLT). • Ensembling : Given Fr, we let Agent A generate K English hypotheses with beam search, {Enj}Kj=1. Then, we let Agent B generate the German translation De using an ensemble of K source sentences (Firat et al., 2016; Zoph & Knight, 2016).
• Fr→En fixed : We fix Agent A and only fine-tune Agent B using LB .
Meanwhile, we also train an NMT model of the same architecture and size directly on the Fr→De task in Multi30k Task 1. This serves as an upper bound on the Fr→De performance achievable with available data.

heading: 5 QUANTITATIVE RESULTS
text: In Table 4, the top three rows are our baselines. The pretrained model performs relatively poorly on Fr→De, conceivably because it was pretrained on a different corpus, and Agent B was was given Agent A’s output as source. Ensembling multiple English hypotheses for Agent B (row 2) gives negligible increase in Fr→De performance. When only Agent B is fine-tuned, we observe 6 BLEU score increase in Fr→De. When the joint system is fine-tuned on German log likelihood with policy gradients (PG), we observe a large, 8 BLEU increase increase in Fr→De at the cost of a substanstial, 15 BLEU score drop in Fr→En. This clearly shows that optimizing on some external reward causes a drastic language drift.
When the agent is trained with the “Englishness” constraint (PG+LM), we notice a significant improvement in Fr→En BLEU. When the LM is trained on WikiText103, a widely used language modelling dataset, we observe improvement of 9 BLEU scores. When the training corpus is closer to the target domain, such as MS COCO or Flickr30k, we see more than 10 BLEU score increase. Fr→De translation also improves by 2–3 BLEU scores. However, we see the biggest improvements in performance when agents are trained using visual grounding feedback. This is particularly pronounced with the LM trained on WikiText103: introducing visual grounding leads to more than 2 BLEU score improvement in Fr→En, and 1 BLEU score improvement in Fr→De. We hypothesize that the “Englishness” constraint forces agents to communicate with correct syntax and fluency, while the image-caption retrieval model restricts the search space of languages to ones that are grounded by visual semantics. To see if grounding is really necessary, we train a stronger LM on all three datasets combined, but find this still leads to more language drift than using visual grounding: the PG+LM+G model with the LM trained on MS COCO outperforms this by 3 BLEU scores on Fr→En. In Figure 2, we observe that vanilla PG fine-tuning quickly leads to highly “un-English” communication, as can be seen from a distinct increase in LM NLL. It is also worth noting that while PG+LM achieves better LM NLL than PG+LM+G, it gives much lower Fr→En BLEU score than the grounded model (PG+LM+G). This is another indication that simply encouraging naturalness is not enough; grounding is key.
A close investigation into the token statistics of each communication strategy reveals that PG finetuning causes the word frequency distribution to be flatter. The PG model has negative frequency difference values for the most frequent tokens, indicating that PG downweighs frequent words severely. On the other hand, PG+LM gives highly positive frequency differences, meaning that language modelling alone disproportionately emphasizes frequent tokens. Visual grounding keeps the token frequency distribution close to the original pretrained regimes. Analyzing the top-k most frequent
words shows that PG+LM disproportionately favors quotation marks, which are very common tokens in many language modelling datasets but occur rarely in Multi30k (see also Appendix A).
Table 5 reinforces the finding that vanilla PG fine-tuning leads to flatter token frequency distributions, as the number of unique tokens used by PG is greater than that of the pretrained model. Meanwhile, PG+LM uses fewer tokens overall, signifying that it uses a relatively small set of tokens frequently.
Also note that PG, despite using a more diverse set of tokens, uses the smallest number of unique symbols per sentence (/sent) and overall (/all). This implies that PG communication is often repetitive. Introducing extra tasks seems to mitigate this, and the grounded model (PG+LM+G) learns a frequency distribution that most closely resembles the original distribution.
To gain further insight into the agents’ communication protocols, we compare the degree of drift by part-of-speech. Table 6 shows that PG tends to ignore function words, such as periods and infinitives. Models trained with LM and grounding losses retain function words with much higher accuracy. PG fares relatively better with content words (nouns and verbs), but adding LM and grounding losses still outperform PG. Grounding leads to overall improvements in recall, particularly with content words.
Conceivably, when optimizing Agent A’s policy on the communication task alone, it is more crucial to relay content information to Agent B, and this might cause agents to ignore syntactic conformity in the original intermediate language. We argue that LM and grounding reduces the space of intermediate languages to a much reasonable language space, facilitating learning.

heading: 6 QUALITATIVE RESULTS
text: In the first example of Table 7, it is clear that PG’s English message has significantly diverged from English: it is highly repetitive (“table table table table table”) and is missing some key content words such as “man” and “jacket”. However, Agent B still generates the German word for ‘man’. The grounded model’s message (PG+LM+G) is distinctly the most fluent and semantically correct.
In the second example, observe that the PG Agent B misinterprets “talking talking a coach a coach” into “spricht mit einem spieler” (talking to a player). The PG+LM+G model again generates a flawless English sentence. Also note that it communicates both colors (red and white) successfully from French to German, while the other two models fail to do so.
We observe some instances of token flipping with the PG model. For example, one particular PG model uses “punk” to describe “child” (see Table 8). As no occurrence of “punk” in any training data is associated with “child”, the agents must have acquired this new meaning assignment during fine-tuning. Among 35 examples in Multi30k development set where the English reference contains “child”, the model uses “punk” 15 times, indicating this is no random phenomenon. We show similar examples from the PG+LM model in Appendix B. We did not observe such examples with the PG+LM+G model.

heading: 7 CONCLUSION
text: In this paper, we show that language drift happens when fine-tuning natural language agents with some external (non-linguistic) reward using policy gradients, and propose a few approaches to avoid this. Most importantly, we find that simply encouraging “naturalness”, e.g. via adding a language model log likelihood to the reward, does not lead to the desired consequences. Instead, we contend that grounding is what we need to avoid language drift. Our empirical results show that grounding leads to best communication performance (highest Fr→De BLEU), while also showing least signs of language drift (highest Fr→En BLEU). Analyzing token frequencies in exchanged messages reveals that pure PG finetuning tends to learn flatter token distributions, and encouraging naturalness disproportionately emphasizes frequent tokens, while the grounded model best retains the original token frequencies.

heading: ACKNOWLEDGMENTS

heading: A FREQUENCY ANALYSIS
text: In Figure 4, where the x-axis is fixed to the token indices sorted with respect to the English reference, we observe that the PG+LM model does indeed favor one word particularly strongly. From investigating top-k most frequent tokens in each model, we find that quotation mark is the most common token for PG+LM in both datasets we experimented with. It is plausible that quotation marks occur with high frequency in language modelling datasets, causing them to be disproportionately overweighed during fine-tuning.
IWSLT
In Figure 5, we show the token frequency curves before subtracting the reference frequencies. Similarly to Figure 4, we observe that the PG model discourages frequent (mostly functional) words, while the PG+LM model excessively prefers frequent words.

heading: B EVIDENCE OF TOKEN FLIPPING IN THE PG+LM MODEL
text: Similar to Table 8, we find evidence of token flipping for the PG+LM model, where the agents use “can@@” (@@ is a subword BPE token marker) to mean “poodle”. This shows that language drift still happens even when a language model is used.

Review summaries:
This paper tries to verify a hypothesis that language grounding DO help to overcome language drift when two agents creating their own protocol in order to communicate with each other. There are several constraints to enforce: 1) naturalness, say "Englishness", 2) grounded in visual semantics. The experiments prove that both constraints help the most (say, BLUE score). 1) w/o 2) restricts the vocabulary into a small set with the most frequent words, while 1) with 2) can resemble the original distribution. 

This paper poses and addresses the problem of language drift in multi-agent communication paradigms. When two pretrained natural-language agents are jointly optimized to communicate and solve some external non-linguistic objective, their internal communication often diverges to a code-like, unnatural communication system. This paper solves this “language drift” problem by requiring that the messages between agents be usable as inputs to an image caption retrieval system. They demonstrate that the jointly optimized agents perform best when regularized in this manner to prevent language drift.