Title: COUNTERING LANGUAGE DRIFT VIA GROUNDING

Abstract: While reinforcement learning (RL) shows a lot of promise for natural language processing-e.g. when fine-tuning natural language systems for optimizing a certain objective-there has been little investigation into potential language drift: when an external reward is used to train a system, the agents' communication protocol may easily and radically diverge from natural language. By re-casting translation as a communication game, we show that language drift indeed happens when pre-trained agents are fine-tuned with policy gradient methods. We contend that simply adding a "naturalness" constraint to the reward, e.g. by using language model log likelihood, does not fully address the issue, and argue that (perceptual) grounding is required. That is, while language model constraints impose syntactic conformity, they do not lead to semantic correspondence. Our experiments show that grounded models give the best communication performance, while retaining English syntax along with the ability to convey the intended semantics.

1. INTRODUCTION:

In the summer of 2017, the internet was briefly abuzz with the mistaken viral message that a leading AI research lab had to "unplug its AI" because it "had gone rogue". What had in fact happened was that two chatbots, under certain conditions, had, rather unsurprisingly, started diverging from their English training data and had instead reverted to their own ungrammatical communication protocol for solving a negotiation task (Lewis et al., 2017) . Instead of saying something like "hats have no value for me" the system would starting saying things like "hat have zero to me to me to me to me". As was soon made clear by the parties involved, this sort of language drift is to be expected if we are optimizing for an external reward, for example one based on whether or not two agents successfully accomplish a negotiation.

While language drift is to be expected under external reward, it is natural to ask what we can do to avoid it. Consider policy gradient methods, for example, and suppose we sample an output sequence from an English language decoder for a given task: sampling a non-grammatical sequence might still be rewarded if we manage to solve the task (e.g., due to some correct words; or because an interlocutor understood us anyway, or guessed correctly), which would quickly move the decoder away (drifting) from English. If we were able to keep drift in check, we could maximize reward while retaining the "Englishness" of the decoder, with obvious benefits for interpretability and interaction with humans. That is, while search space size prohibits the direct usage of policy gradient methods for training natural language decoders from scratch, we could prevent pre-trained models from drifting while we optimize for the desired reward using policy gradients.

Thus, the ability to stop policy gradient methods from diverging from natural language enables interesting long-term possibilities for exploration: imagine e.g. fine-tuning a pre-trained language model trained on large amounts of data, call it a "language module", for a given generation task with limited data. When training chit-chat dialogue agents, for example, we often want to optimize for some very high-level reward, such as engagingness or consistency, with hardly enough data to learn simple English grammar. Or consider what might happen when we train agents using self-play to actively use natural language to change the other agent's (mental) state, rather than having a model passively observe language usage in some corpus or dataset, as usually happens.

In this work, we study the question of language drift. Drawing inspiration from Lee et al. (2018) , we re-cast translation as a communication game. Two machine translation (MT) agents-i.e., encoderdecoder models with attention-are tasked with successfully translating source language sequences to the target language using a third pivot language as an intermediary. The communication channel (the output of the first agent's decoder, which is fed to the second agent's encoder as input) is updated via policy gradient methods to optimize for translating into the target language, effectively fine-tuning two separate pre-trained MT models via a pivot language.

While the subject of communication need not be language (e.g., for Lee et al. (2018) , agents learn to translate by communicating about images), three-way translation via pivot is an excellent way for studying the current problem: we can check exactly to what extent the communicated sequence corresponds to both the intended meaning, as well as to the gold standard sequence. We can think of this setup as an agent aiming to communicate its state to another agent via some protocol-yet in this case, the mental states and intermediary communication protocol are completely interpretable.

In what follows, we show that language drift happens, and quite dramatically so, when fine-tuning using policy gradients. We then show that the most intuitive way of solving this problem-adding an "Englishness" constraint, such as the log-probability assigned by a language model, to the reward function-does not in fact lead to the desired consequences. Indeed, there is nothing preventing such models from learning to translate "Two giraffes standing next to a white truck in the savanna" from French to German via "Democracy is a political system" as the English intermediary.

Hence, we contend that what is missing is grounding: while language model constraints impose syntactic conformity, they do not lead to semantic correspondence. Humans don't invent unique idiolects for every individual interlocutor, exactly because they are grounded: we share strong priors, social and behavioral norms, and a common sensorimotor experience of our physical environment. Thus, "not going rogue" means not only sticking to the prescribed language, but more importantly preserving meaning, which means staying grounded.

Our experiments show that fine-tuning the communication channel with visual grounding leads to the highest communication performance (Fr→En→De) as well as the best retention of original syntax and intended semantics. Our token frequency analysis corroborates our hypothesis, and shows that grounding is key for preserving the token frequency distribution of the pivot language (English).

2. PRIOR WORK:

Our work is inspired by recent work in protocols or languages that emerge from multi-agent interaction (Lazaridou et al., 2017; Lee et al., 2018; Andreas et al., 2017; Evtimova et al., 2018; Kottur et al., 2017; Havrylov & Titov, 2017; Mordatch & Abbeel, 2017) . Work on the emergence of language in multi-agent settings goes back a long way (Steels, 1997; Nowak & Krakauer, 1999; Kirby, 2001; Briscoe, 2002; Skyrms, 2010) . In our case, we are specifically interested in tabula inscripta agents that are already pre-trained to generate natural language, and we are primarily concerned with keeping their language natural during further training.

Reinforcement Learning (RL) has been applied to fine-tuning models for various natural language generation tasks, including summarization (Ranzato et al., 2015; Paulus et al., 2017) , information retrieval (Nogueira & Cho, 2017) , MT (Gu et al., 2017; Bahdanau et al., 2016) and dialogue (Li et al., 2017) . Our work can be viewed as fine-tuning MT systems using an intermediary pivot language. In MT, there is a long line of work of pivot-based approaches, most notably Muraki (1986) and more recently with neural approaches (Wang et al., 2017; Cheng et al., 2017; Chen et al., 2018) . There has also been work on using visual pivots directly (Hitschler et al., 2016; Nakayama & Nishida, 2017; Lee et al., 2018) . Grounded language learning in general has been shown to give significant practical improvements in various natural language understanding tasks (Gella et al., 2017; Elliott & Kádár, 2017; Chrupała et al., 2015; Kiela et al., 2017; Kádár et al., 2018) . Meanwhile, Bowman et al. (2016) found a powerful decoder to ignore the latent representation in VAEs for language.

3. TASK AND MODELS:

We recast translation as a communication game involving two MT agents: Fr→En and En→De (see Figure 1 ). Our dataset consists of N triples of aligned sentences {Fr i , En i , De i } N i=1 , where En i is only used for evaluation. We first feed the French sentence Fr i to Agent A, which generates an English message En i as output. Agent B is then trained to maximize the log likelihood of the ground truth German sentence given the English message, i.e. log p(De i |En i ). Agent A is trained using REINFORCE (Williams, 1992) with reward R = log p B (De i |En i ).1 This encourages Agent A to develop helpful communication policies for Agent B, and allows Agent B to adapt to Agent A's new policies. In other words: communication via the pivot language (English) is a success if we are able to translate the intended source sequence (French) into the desired target sequence (German).

Both agents are pre-trained individually before communication, meaning that we start off with English as an intermediate language in the early stages of the game. This work examines what happens to the intermediate language as we fine-tune the system jointly: will the agents keep communicating in English, or diverge? And if so, what can we do to prevent that from happening?

3.1. AUXILIARY TASKS:

To help reduce the search space of intermediate languages, we use two auxiliary tasks: language modelling (LM) and image-caption retrieval (henceforth called the grounding model).

Language Model Given a language model pre-trained on a standard English corpus, the log likelihood of the English message informs its general "Englishness". We incorporate this into the reward for Agent A, so that it learns to send messages that are plausible English.2 Reward for Agent A is:

R LM = log p B (De i |En i ) + β LM log p LM (En i ).

Grounding Model Let us assume we have access to a set of images {Img i } associated with each triple {Fr i , En i , De i }. Given a pre-trained image-caption retrieval model, such as VSE++ (Faghri et al., 2018) , the log likelihood of the image given the English message (and vice versa) informs how much the English message is grounded in the original semantic content (Kiela et al., 2017) . We incorporate the ranking loss into Agent A's reward.

R G = log p B (De i |En i ) + β G log p G (Img i |En i ).

Note that β LM , β G are hyperparameters.

3.2. TRAINING OBJECTIVE:

For brevity the t-th token in the i-th English sentence En i;t is abbreviated to En t , and En i to En.

Policy Gradient Training At decoding timestep t, Agent A takes an action (outputs token En t ) given an environment (previous hidden states and previous token En t-1 ). It receives reward R at the end of the sequence, from which we subtract a state-dependent baseline R t to reduce variance. Therefore, we maximize (R -R t ) log p(En t |En <t , Fr). In addition, we employ entropy regularization on Agent A's decoder to encourage exploration. Hence, Agent A's overall objective function is given as:

L A = α pg (R -R t ) log p(En t |En <t , Fr) + α entr H(p(En t |En <t , Fr)) -α b MSE(R, R t )

, where H and MSE denote entropy and mean squared error losses.

Cross Entropy Training Agent B is trained using standard cross entropy loss, i.e. L B = log p(De t |De <t , En).

We jointly train both agents by maximizing L:

= L A + L B .

4. EXPERIMENTAL SETTINGS:

In this section we provide the details of our experimental setup: a Fr→X→De translation task where the intermediate language X is initialized as English, and subsequently fine-tuned with policy gradient. On a trilingual corpus consisting of three languages (Fr, En and De), we can measure communication success with Fr→De BLEU (Papineni et al., 2002) , while Fr→En BLEU informs how closely the intermediate language resembles English, at any given point during fine-tuning.

Datasets Agents are initially pre-trained on IWSLT Fr→En and En→De. Fine-tuning is performed on Multi30k Task 1 (Elliott et al., 2016) . That is, importantly, there is no overlap in the pre-training data and the fine-tuning data. Multi30k Task 1 consists of 30k images and one caption per image in English, French, German and Czech (of which we only use the first three). For the English language model, we compare four different datasets: WikiText103, MS COCO and Flickr30k. The image-caption retrieval model is trained on Flickr30k: the same set of 30k images as Multi30k but containing 5 English captions per image. Following Faghri et al. (2018) , we randomly crop training images at every epoch. We use 2048-dimensional final-layer features from a pretrained and fixed ResNet-152 (He et al., 2016) .

Preprocessing The same tokenization and vocabulary are used across different tasks and datasets. We lowercase and tokenize our corpora with Moses (Koehn et al., 2007) and use subword tokenization with Byte Pair Encoding (BPE) (Sennrich et al., 2016) with 10k merge operations. This allows us to use the same vocabulary across different models seamlessly (translation, language model, image-caption ranker model).

Controlling the English message length When fine-tuning the agents, we observe that the length of English messages becomes excessively long. As Agent A has no explicit incentive to output the EOS symbol, it tends to keep transmitting the same token repeatedly. Excessively long messages obscure evaluation of the communication protocol. For instance, BLEU score quickly deteriorates as the message length becomes longer, as it is a precision metric. When the message length is fixed, a drop in BLEU score will by necessity mean that the intermediate language has drifted away more. For this reason, we constrain the length of English messages to be no longer than the length of their French source sentence, or shorter if the model outputs the EOS symbol early. Recall that Agent B is supervised to predict the EOS symbol, so does not suffer from this issue.

Model Architecture and Pretraining Our MT agents are standard sequence-to-sequence models with attention (Bahdanau et al., 2015) with unidirectional, 1-layer GRU with 256 hidden units and 256-dimensional embeddings. During initial pre-training on IWSLT, we early-stop based on BLEU score on the development set (tst2013). The best checkpoints give 34.05 BLEU and 21.94 BLEU on IWSLT Fr→En and En→De development sets with greedy decoding. For our value function, we use a 2-layer MLP with a ReLU nonlinearity.

The language model is a 1-layer recurrent language model with 512 LSTM hidden units. The imagecaption retrieval model is a recently proposed VSE++ model (Faghri et al., 2018) , with unidirectional 1-layer GRU with 512 hidden units and a single fully connected layer from 2048-dimensional ResNet features to 512-dimensional GRU hidden states. We report the performance of the pretrained models used in our experiments in Tables 1, 2 and 3 For our joint systems with policy gradient fine-tuning, we run every model three times with different random seeds and report averaged results (see Table 4 ).

Baseline and Upper Bound Our main quantitative experiment has three baselines:

• Pretrained checkpoints (on IWSLT).

• Ensembling : Given Fr, we let Agent A generate K English hypotheses with beam search, {En j } K j=1 . Then, we let Agent B generate the German translation De using an ensemble of K source sentences (Firat et al., 2016; Zoph & Knight, 2016) .

• Fr→En fixed : We fix Agent A and only fine-tune Agent B using L B .

Meanwhile, we also train an NMT model of the same architecture and size directly on the Fr→De task in Multi30k Task 1. This serves as an upper bound on the Fr→De performance achievable with available data. 4 : Results in BLEU score on Multi30k Task 1. For our models using policy gradient fine-tuning, we report results averaged over three runs and provide standard deviations in brackets. PG: trained with vanilla policy gradient fine-tuning. PG+LM: trained with the "Englishness" constraint in reward. For MS COCO and Flickr30k, the LM was trained directly on image captions. PG+LM+G: trained with grounding loss as well as the LM loss. Fr→En: degree of intermediate language drift from English; lower indicates more drift. Fr→En→De: metric for communication accuracy; higher is better. All: LM was trained on all three datasets combined. Improvements of PG+LM+G over PG+LM were found to be significant in all cases, using the approximate randomization test for significance testing (Riezler & Maxwell III, 2005) .

5. QUANTITATIVE RESULTS:



LM:

In Table 4 , the top three rows are our baselines. The pretrained model performs relatively poorly on Fr→De, conceivably because it was pretrained on a different corpus, and Agent B was was given Agent A's output as source. Ensembling multiple English hypotheses for Agent B (row 2) gives negligible increase in Fr→De performance. When only Agent B is fine-tuned, we observe 6 BLEU score increase in Fr→De.

When the joint system is fine-tuned on German log likelihood with policy gradients (PG), we observe a large, 8 BLEU increase increase in Fr→De at the cost of a substanstial, 15 BLEU score drop in Fr→En. This clearly shows that optimizing on some external reward causes a drastic language drift. When the agent is trained with the "Englishness" constraint (PG+LM), we notice a significant improvement in Fr→En BLEU. When the LM is trained on WikiText103, a widely used language modelling dataset, we observe improvement of 9 BLEU scores. When the training corpus is closer to the target domain, such as MS COCO or Flickr30k, we see more than 10 BLEU score increase.

Fr→De translation also improves by 2-3 BLEU scores.

However, we see the biggest improvements in performance when agents are trained using visual grounding feedback. This is particularly pronounced with the LM trained on WikiText103: introducing visual grounding leads to more than 2 BLEU score improvement in Fr→En, and 1 BLEU score improvement in Fr→De. We hypothesize that the "Englishness" constraint forces agents to communicate with correct syntax and fluency, while the image-caption retrieval model restricts the search space of languages to ones that are grounded by visual semantics. To see if grounding is really necessary, we train a stronger LM on all three datasets combined, but find this still leads to more language drift than using visual grounding: the PG+LM+G model with the LM trained on MS COCO outperforms this by 3 BLEU scores on Fr→En.

In Figure 2 , we observe that vanilla PG fine-tuning quickly leads to highly "un-English" communication, as can be seen from a distinct increase in LM NLL. It is also worth noting that while PG+LM achieves better LM NLL than PG+LM+G, it gives much lower Fr→En BLEU score than the grounded model (PG+LM+G). This is another indication that simply encouraging naturalness is not enough; grounding is key.

A close investigation into the token statistics of each communication strategy reveals that PG finetuning causes the word frequency distribution to be flatter. The PG model has negative frequency difference values for the most frequent tokens, indicating that PG downweighs frequent words severely.

On the other hand, PG+LM gives highly positive frequency differences, meaning that language modelling alone disproportionately emphasizes frequent tokens. Visual grounding keeps the token frequency distribution close to the original pretrained regimes. Analyzing the top-k most frequent words shows that PG+LM disproportionately favors quotation marks, which are very common tokens in many language modelling datasets but occur rarely in Multi30k (see also Appendix A).

IWSLT Multi30k unique /sent /all unique /sent /all

Table 5 reinforces the finding that vanilla PG fine-tuning leads to flatter token frequency distributions, as the number of unique tokens used by PG is greater than that of the pretrained model. Meanwhile, PG+LM uses fewer tokens overall, signifying that it uses a relatively small set of tokens frequently.

Also note that PG, despite using a more diverse set of tokens, uses the smallest number of unique symbols per sentence (/sent) and overall (/all). This implies that PG communication is often repetitive. Introducing extra tasks seems to mitigate this, and the grounded model (PG+LM+G) learns a frequency distribution that most closely resembles the original distribution.

To gain further insight into the agents' communication protocols, we compare the degree of drift by part-of-speech. Table 6 shows that PG tends to ignore function words, such as periods and infinitives. Models trained with LM and grounding losses retain function words with much higher accuracy. PG fares relatively better with content words (nouns and verbs), but adding LM and grounding losses still outperform PG. Grounding leads to overall improvements in recall, particularly with content words.

Conceivably, when optimizing Agent A's policy on the communication task alone, it is more crucial to relay content information to Agent B, and this might cause agents to ignore syntactic conformity in the original intermediate language. We argue that LM and grounding reduces the space of intermediate languages to a much reasonable language space, facilitating learning.

6. QUALITATIVE RESULTS:

In the first example of Table 7 , it is clear that PG's English message has significantly diverged from English: it is highly repetitive ("table table table table table" ) and is missing some key content words such as "man" and "jacket". However, Agent B still generates the German word for 'man'.

The grounded model's message (PG+LM+G) is distinctly the most fluent and semantically correct.

In the second example, observe that the PG Agent B misinterprets "talking talking a coach a coach" into "spricht mit einem spieler" (talking to a player). The PG+LM+G model again generates a flawless English sentence. Also note that it communicates both colors (red and white) successfully from French to German, while the other two models fail to do so. We observe some instances of token flipping with the PG model. For example, one particular PG model uses "punk" to describe "child" (see Table 8 ). As no occurrence of "punk" in any training data is associated with "child", the agents must have acquired this new meaning assignment during fine-tuning. Among 35 examples in Multi30k development set where the English reference contains "child", the model uses "punk" 15 times, indicating this is no random phenomenon. We show similar examples from the PG+LM model in Appendix B. We did not observe such examples with the PG+LM+G model.

7. CONCLUSION:

In this paper, we show that language drift happens when fine-tuning natural language agents with some external (non-linguistic) reward using policy gradients, and propose a few approaches to avoid this. Most importantly, we find that simply encouraging "naturalness", e.g. via adding a language model log likelihood to the reward, does not lead to the desired consequences. Instead, we contend that grounding is what we need to avoid language drift. Our empirical results show that grounding leads to best communication performance (highest Fr→De BLEU), while also showing least signs of language drift (highest Fr→En BLEU). Analyzing token frequencies in exchanged messages reveals that pure PG finetuning tends to learn flatter token distributions, and encouraging naturalness disproportionately emphasizes frequent tokens, while the grounded model best retains the original token frequencies.

A FREQUENCY ANALYSIS:

Figure 4 : Token frequency analysis similar to Figure 3 , but with the x-axis fixed to the token indices sorted with respect to English reference, in decreasing order.

In Figure 4 , where the x-axis is fixed to the token indices sorted with respect to the English reference, we observe that the PG+LM model does indeed favor one word particularly strongly. From investigating top-k most frequent tokens in each model, we find that quotation mark is the most common token for PG+LM in both datasets we experimented with. It is plausible that quotation marks occur with high frequency in language modelling datasets, causing them to be disproportionately overweighed during fine-tuning.

IWSLT:



Reference:

, . the and to of a that i in is it you we &apos;s this &quot; Pretrained , the . to of and a i that in it we you &apos;s is this &quot; was PG a the and , . in i &quot; this of to is we you ? that not for PG+LM &quot; the , of . and in a to this is i es you for we that with PG+LM+G the , . of a and to in is i this es we for that you at what Multi30k Reference a . in the on of with and is man two woman to are people at an Pretrained a . the in of on with and , man @-@ is to woman two people at white PG a in the on and with . man water red two woman street blue city its people white PG+LM &quot; a the . in of on with and man , to two woman his people es young PG+LM+G a the . in of on with and man at es to , woman two his people is Table 9 : Top 20 most frequent tokens in English reference (Reference) or the output from Fr→En models. In Figure 5 , we show the token frequency curves before subtracting the reference frequencies. Similarly to Figure 4 , we observe that the PG model discourages frequent (mostly functional) words, while the PG+LM model excessively prefers frequent words.

:

We use subscript B to denote that the probability is computed with Agent B.

We also experimented with a dense LM reward on the word-level, but found this to lead to worse performance. We hypothesize that the model might be focusing too much on the dense LM reward, ignoring the sparse reward for the communication task and leading to poor performance.

Figures and Tables:

Figure 1: Diagram of our communication game.

Figure 2: Learning curves for PG, PG+LM and PG+LM+G. En LM NLL curves show the NLL of English messages, computed by a language model trained on WikiText103. Lower En BLEU indicates more language drift, and higher En LM NLL indicates more language drift.

Figure3: Token frequency analysis on three different models (PG, PG+LM, PG+LM+G) as well as the pretrained model before any fine-tuning (Pretrained). We show the word frequency curves (sorted in decreasing order) for each model, after subtracting the reference English frequency statistics (also sorted). Positive y values indicate higher frequency values than the English reference, and negative y values indicate lower frequency values than English. Note that y-axis is the frequency difference in thousands, and x-axis shows the vocabulary index (sorted with frequency) in log scale.

in einem schwarzen hemd schaut auf den tisch . +LM ein alter mann in einer jacke beobachtet einen tisch . +G ein älterer mann in einer schwarzen jacke schaut auf den tisch . Ref Fr un joueur de football américain en blanc et rouge parle à un entraîneur . De ein rot-weiß gekleideter footballspieler spricht mit einem trainer . En a football player in red and white is talking to a coach . En PG a player football american football american and red talking talking a coach +LM a player of white and red talking to a coach . " " " +G a football player in white and red talking to a coach . De PG ein footballspieler spricht mit einem spieler in einem roten trikot . +LM ein weiß gekleideter fußballspieler spricht zu einem trainer . +G ein fußballspieler in einem rot-weißen trikot spricht mit einem trainer . Table 7: Two random examples from Multi30k development set with different models (PG, PG+LM, PG+LM+G). The top three rows list the ground truth sentences, the middle three rows are the English messages sent by the Fr→En agent, and the bottom three rows show the German output from the En→De agent. We also show the corresponding images, which were only used to train the image-caption retrieval modal. Fr src un enfant assis sur un rocher. En ref a child sitting on a rock formation. En hyp a punk sitting sitting on on a broken De ref ein kind sitzt auf einem felsen . De hyp ein kind sitzt auf einem felsen . Fr src un petit enfant est assis à une

Figure 5: Token frequency curves (before subtracting the reference frequencies). Both x (vocabulary index) and y (frequency) axes are in log scale.

. Retrieval results for our VSE++ model on Flickr30k test set. We use Adam(Kingma & Ba, 2014) with initial learning rate of 0.001 and dropout(Srivastava et al., 2014) rate of 0.1. We grid search over learning rate schedule and reward coefficients (α pg , α entr , α b , β LM , β G ).

Table content: <table><tr><td/><td colspan="2">IWSLT Multi30k</td><td>WikiText103 3.51</td><td/><td colspan="3">R@1 R@5 R@10</td></tr><tr><td>Fr→En</td><td>34.05</td><td>26.80</td><td>MS COCO 2.66</td><td>Caption</td><td>50.1</td><td>76.3</td><td>84.6</td></tr><tr><td>En→De</td><td>21.94</td><td>18.56</td><td>Flickr30k 2.85</td><td>Image</td><td>35.7</td><td>65.3</td><td>75.9</td></tr><tr><td colspan="3">Table 1: Translation performance</td><td>Table 2: Development NLL of</td><td/><td/><td/><td/></tr><tr><td colspan="3">of our pre-trained agents (BLEU)</td><td>pretrained language models</td><td/><td/><td/><td/></tr></table>

Exact-match word recall by POS-tag on IWSLT development set: when the English reference contains a word of a certain POS tag, how often does the agent correctly produces that word. TO: infinitive to, (.): period, DT: determiner, Noun: (NN, NNS, NNP, NNPS), Verb: (VB, VBD, VBG, VBN, VBP, VBZ), Adj: adjective (JJ, JJR, JJS), Adv: adverb (RB, RBR, RBS) Ref Fr un vieil homme vêtu d'une veste noire regarde sur la table De ein alter mann in einer schwarzen jacke blickt auf den tisch En an old man wearing a black jacket is looking on the table En PG a old teaching black watching on the table table table table table table +LM a old man in a jacket looking on the table . " " +G an old man in a black jacket looking on the table .

Table content: <table><tr><td/><td colspan="3">Function words</td><td>Content words</td></tr><tr><td/><td>TO</td><td>.</td><td colspan="2">DT Noun Verb</td><td>Adj Adv</td></tr><tr><td>PG</td><td colspan="3">0.22 0.36 0.57</td><td>0.38 0.17 0.32 0.26</td></tr><tr><td>PG+LM</td><td colspan="3">0.55 0.84 0.72</td><td>0.39 0.18 0.21 0.25</td></tr><tr><td colspan="4">PG+LM+G 0.62 0.88 0.74</td><td>0.43 0.26 0.33 0.29</td></tr></table>

Evidence of token flipping in the PG model.

Table content: <table/>

Peer review (iclr2019):
 This paper poses and addresses the problem of language drift in multi-agent communication paradigms. When two pretrained natural-language agents are jointly optimized to communicate and solve some external non-linguistic objective, their internal communication often diverges to a code-like, unnatural communication system. This paper solves this “language drift” problem by requiring that the messages between agents be usable as inputs to an image caption retrieval system. They demonstrate that the jointly optimized agents perform best when regularized in this manner to prevent language drift. 1. Framing: I’m uncertain about the framing of this paper. The authors pose the problem of “language drift,” which is indeed a frequent problem in multi-agent communication tasks where the principle supervision involves non-linguistic inputs and outputs. They then design a three-language MT task as a test case, where the inputs and outputs are both linguistic. Why attack this particular task and grounding solution? I can imagine some potential goals of the paper, but also see more direct ways to address each of the potential goals than what the authors have chosen: 1a. Study how to counter language drift in general — why not choose a more intuitive two-agent communication task, e.g. navigation, game playing, etc.? 1b. Study how to counter language drift in the MT task — aren’t there simpler solutions to prevent language drift in this particular task? e.g. require “cycle-consistency” – that it be possible to reconstruct the French input using the French output? Why pick multimodal grounding, given that it imposes substantial additional data requirements? 1c. Build a better/more data-efficient machine translation system — this could be an interesting goal and suitable for the paper, but this doesn’t seem to be the framing that the authors intend. 2. Interpretation of first results: 2a. Thanks for including standard deviation estimates! I think it’s also important that you do some sort of significance testing on the comparison between PG+LM+G and PG+LM performance for Fr->En->De — these numbers look pretty close to me. You could run e.g. a simple sign test on examples within each corpus between the two conditions. 2b. It would also be good to know how robust your results are to hyperparameter settings (especially the entropy regularization hyperparameter). 3. Token frequency results: These are intriguing but quite confusing to me! 3a. How sensitive are these results to your entropy regularization setup? How does PG behave without entropy regularization? 3b. Table 6 shows that the PG model has very different drift for different POS categories. Does this explain away the change in the token frequency distribution? What do the token frequency effects look like for PG within the open-class / content word categories (i.e., controlling for the huge difference in closed-class behavior)? 4. Minor comments: 4a. There’s a related problem in unsupervised representation learning for language. Work on VAEs for language, for example, has shown that the encoder often collapses meaning differences in the latent representation, and leans on an overly powerful decoder in order to pick up all of the lost information. It would be good to reference this work in your framing (see e.g. Bowman et al. (2015)). 4b. In sec. 3.1 you overload notation for R. Can you subscript these so that it’s especially clear in your results which systems are following which reward function? 4c. Great to show some qualitative examples in Table 7 — can you explicitly state where these are from (dev set vs. test set?) and whether they are randomly sampled? References: Bowman et al. (2015). Generating sentences from a continuous space. https://arxiv.org/abs/1511.06349