SlowGuess/ACL_2020_OCR huggingface dataset:
Noisy channel reranker. Following Bayes rule, we factorize the speaker's policy as follows:  $\pi_{\theta_S}(s|u,t) \propto p(t|s,u)p(s|u)$ , where  $u = [u_t; u_d]$ . We omit the distractor vector  $u_d$  in the conditioning of the prior, arriving to  $p(s|u_t)$  from the PoE reranker above. The crucial difference is that the first term now represents the speaker's approximation of the listener's behaviour. As before, we represent samples with the transformed bag-of-words, but then compute their dot-product similarities with each image separately and normalize with softmax across the images to obtain the probability of the target  $p(t|s,u)$ . This reranker model is closely related to pragmatic speakers in Rational Speech Act (RSA) framework (Andreas and Klein, 2016; Monroe and Potts, 2015; Cohn-Gordon et al., 2018; Fried et al., 2018). However, while the RSA model assumes a given and fixed listener model, here we are learning the model of the listener that the speaker is using by optimizing end-to-end the listener's reward. Thus, when doing multi-agent communication using the noisy channel model, there exist two components that produce probability distributions of the same type  $p(t|s,u)$ ; one belongs to the listener, thus the speaker has no access to it (e.g., this listener in the future could be a human sending rewards), while the other belongs to the speaker corresponding to their model of the listener.
5 Speakers trained jointly with listeners
Table 2 presents referential success when speakers are trained with rewards from a joint listener, i.e., a listener being learned jointly with the speaker.
We conduct three different evaluations: at test time we play against the fixed listener, human listeners and the joint listener the speaker was trained with. While fixed listener is the same for all speakers, the joint listener is speaker-specific. We report results on two splits: for the easy and difficult split we report referential success of the joint listener, and for the latter split, we also report results of the fixed and human listener.
To compute referential success using human listeners, we collect 400 annotations for each speaker model. To avoid annotators adapting to model-specific strategies, we group predictions of similar models and collect annotations in three sessions (one for each group), during which we present annotators with predictions from a model sampled from that group.[5]
Easy split joint	Difficult split
joint	fixed	human
Functional-only learning
emergent (§4.1)	0.99	0.98	-	0.5
Structural-only learning
image captioning (§4.2)
sample	0.92	0.78	0.77	0.77
greedy	0.91	0.77	0.73	0.78
Structural & functional learning
Gradients from reward affect base captioning model
reward finetuning (§4.3.1)
no KL	0.95	0.82	0.63	0.62
with KL	0.93	0.79	0.77	0.69
multi-task learning (§4.3.2)
λs=0.1	0.98	0.94	0.71	0.71
λs=1	0.96	0.90	0.69	0.69
Reranking (§4.3.3), base captioning model unchanged
PoE, λs=0	0.99	0.92	0.81	0.81
PoE, λs=1	0.98	0.91	0.83	0.78
noisy channel	0.96	0.83	0.84	0.86*
Utilizing ground-truth captions from the dataset
Oracle speakers, no weights learned (§3.1)
random	0.87	0.74	0.72	0.81
discriminative	0.87	0.73	0.82	0.87*
Reranking (§4.3.3) ground-truth captions
PoE (§4.3.3)	0.95	0.88	0.85	0.93*
noisy channel (§4.3.3)	0.95	0.78	0.83	0.88*
Table 2: Referential success of speakers (by rows) trained with joint listener and then tested with joint, fixed and human listener (by columns).* indicates significance over the image captioning (greedy) when tested with humans  $(p <   0.005$  , bootstrapping test).
5.1 Referential success of joint listeners
All models perform quite similarly in the easy split, whereas we observe larger gaps in the difficult split. In terms of joint accuracy results in the difficult split, reward finetuning has the lowest performance among models that are optimizing rewards, perhaps due to its large action space (i.e., the vocabulary size  $|V| = 2685$ ), making it a hard RL exploration problem. multi-task, despite having the same action space performs better, probably due to the captioning objective being optimized concurrently facilitating the learning dynamics. Finally, the best results in both splits are obtained by the emergent communication model, that achieves near perfect performance. We believe this is the case since this speaker is the least constrained of all, since we can think of all other speakers (i.e., the ones that combine both types of learning) as being regularized towards producing natural language.
nel, PoE). Group 2: multi-task, reward finetuning. Group 3: random, discriminative, PoE and noisy channel with ground-truth captions.
5Group 1: image captioning (greedy/sample), noisy chan
7668
5.2 Referential success of human listeners
Somewhat alarmingly, we observe the joint performance is not predictive of the human's one across the board, hinting to issues regarding pragmatic drift (we will further discuss this in Section 6). In the most extreme case, while the emergent communication speaker achieved the highest results when playing against a listener jointly learned with the speaker, this comes in the expense of human performance: functional learning alone results in maximally uninterpretable protocols, and as such humans are at random when playing against such a model.
Speakers that combine both types of learning achieve good human performance, with reward-learned reranker models, i.e., noisy channel and PoE being the best. In their case, they outperform the image captioning baselines, even approaching the discriminative oracle speaker based on ground-truth captions. This indicates their effectiveness in extending the conditioning of the underlying image captioning to the distractor image with the reward coming from the listener, turning like this the base image-captioning model into a task-specific referential captioning model. Moreover, when giving the rerankers a perfect captioning model in the form of ground-truth captions of target images, performance of noisy channel and PoE surpass the oracles' (see last two columns of Table 2); as the community improves the base language models, we should expect this to also result in net improvement in the reranker models.
Finally, we also observe that the fixed grounded listener is significantly predictive of the human performance  $(p < 0.005$ , t-test). This is encouraging, since as we will show in Section 7, we can use this listener as a fixed model that provides rewards to the speaker model.
6 Language drift and how to detect it
We show that the multi-agent communication framework is prone to language drift (Lee et al., 2019), i.e., when protocols diverge from human language. We present a taxonomy of different types that occur in this framework, alongside a set of automatic measures to detect it.
6.1 Structural drift: Definition and measures
The most basic type of drift that manifests in the emergent communication setup relates to the core
Distractor Image
Structural-only learning
image captioning (§4.2)
jenny is wearing a hat
mike is wearing a hat
Structural and functional learning
Gradients from reward affect base captioning model reward finetuning (§4.3.1)
no KL
it is camping camping [...] camping
with KL
mike is sitting on the tent
multi-task learning (§4.3.2)
$\lambda_{s} = 0.1$
mike is jenny on the the tent
$\lambda_{s} = 1$
Reranking  $(\S 4.3.3)$ , base captioning model unchanged
PoE,  $\lambda_{s} = 0$
the tent is in the tree
PoE,  $\lambda_{s} = 1$
mike and jenny are sitting on the ground
noisy channel
jenny is wearing a funny hat
Table 3: Examples of generated messages. We underline wrong and bold correct cases of language usage.
structural properties of the generated language, i.e., its fluency and grammaticality with respect to natural language (this is also referred to by Lee et al. (2019) as "syntactic"). Looking at Table 3, a clear example of this type of drift happens when models update the base captioning model. reward finetuning (no KL) does not produce at all grammatical sentences, while multi-task  $(\lambda_{s} = 0.1)$  appears to suffer less, only occasionally producing slightly ungrammatical sentences by repeating consecutive words. We term this structural drift and we quantify it as the log probability of the generated message under a pre-trained unconditional language model (column  $\log p(m)$  in Table 4).
6.2 Semantic drift: Definition and measures
The second type of drift is the semantic drift. This relates to whether the generated message is grounded with regards to the target object, i.e., its adequacy with respect to the literal semantics of the target (this is also referenced by Lee et al. (2019) as "semantic"). We have qualitatively observed instances of this type of drift in the PoE, which occasionally shifts the semantics of words, e.g., using the word tree to refer to ground as seen in Table 3. To measure it, we use a pre-trained image-conditional language model and compute the target-image conditional log probability of the generated
6All t-tests are conducted between two distributions of scores dichotomized on human performance.
7669
log p(m)	log p(m|i)	1-gram	3-gram
Structural-only learning
image captioning (§4.2)
sample	-8.71	-7.77	0.81	0.37
greedy	-8.63	-7.72	0.73*	0.30*
Structural & functional learning
Gradients from reward affect base captioning model
reward finetuning (§4.3.1)
no KL	-442.00	-279.55	0.33	0.00
with KL	-11.75	-10.78	0.70*	0.22*
multi-task learning (§4.3.2)
λs=0.1	-18.08*	-19.67	0.78*	0.18*
λs=1	-10.68*	-10.64	0.63*	0.18
Reranking (§4.3.3), base captioning model unchanged
PoE, λs=0	-10.18*	-8.79	0.71*	0.24*
PoE, λs=1	-8.95	-7.94	0.78*	0.30*
noisy channel	-10.02	-8.59	0.76	0.30
Table 4: Language drift measures, lower scores mean higher drift. *indicates that the measure was significantly predictive of the human listener performance on the referential task (p<0.005, t-test).
message (column  $\log p(m|i)$  in Table 4).
These two log probability-based measures do not assume access to language data for the target objects, and as such can be computed from general unconditional and domain-specific conditional language models. In this particular case though, since we also have access to language data for the target images (i.e., captions in English), and assuming that these data describe everything that is true about the target, we can use simple n-gram statistics as proxies of semantic drift (i.e., in this case 1-gram word overlap ignoring stop word and 3-gram word overlap between the ground-truth captions and the speaker-generated message). Moreover, all these measures do not take into account the specific communication task the speaker has to perform, i.e., our measures do not consider any information about the distractor object, making them easily adaptable to other tasks.
6.3 Structural and semantic drift results
In Table 4 we report performance of different models under these automatic measures. The structural score  $\log p(m)$  reflects the qualitative observations made from Table 3, i.e., multi-task and reward finetuning, have the highest structural drift, with the latter performing significantly worse than all the models. In contrast, the reranker models that do not update the base captioning model, i.e., PoE and noisy channel, perform the best on the semantic score by construction; both models directly incorporate in their
models a component associated with the semantic score (i.e., the samples taken from the image-conditional model alongside the associated probabilities). Moreover, they also perform well on all other measures, indicating their robustness against language drift. Finally, all the model-specific language regularizers (KL for reward finetining,  $\lambda_{s} = 1$  for multi-task and  $\lambda_{s} = 1$  for PoE) we introduced were effective in limiting both types of language drift (as also seen in Table 3).
6.4 Pragmatic drift
Finally, we identify a novel type of drift, i.e., pragmatic drift, which relates to the divergence between a human's interpretation of the message from the interpretation a speaker will assume. Unfortunately, this type of drift is perhaps the most difficult to capture in an automatic way as it is task specific and requires access to the exact interpretation that the human would ascribe to the message. As a proxy of pragmatic drift, we use the difference between the agent- and human-listener referential success; if the joint referential success is higher than the human's one, then the speaker assumes an interpretation of the message that is different from the human's one, resulting in lower human performance. An extreme example of this drift manifests when the joint listener achieves almost perfect referential success whereas a human listener is at random, as in the case of emergent communication. However, in this case the messages are maximally uninterpretable with the lowest possible performance in both structural and semantic scores.
Hence, a natural question to ask is to what degree (if at all possible) pragmatic drift can manifest in the absence of the other two types of language drift. Or, put differently, does the emergent communication for learning language use hide any other pathological behaviour for models that do not suffer a lot from structural and semantic drift, as in the case of PoE and noisy channel? To study this, we create a setup where PoE is guaranteed to have a perfect knowledge of (grounded) language. Namely, it uses the reward to rerank ground-truth captions associated with the target image (note, our dataset provides up to five captions per image). Moreover, we perform several ablations where we allow the updating of different parameters in the speaker's and listener's model by unfreezing components.
Table 5 presents the results of the joint and human referential success. The main finding is that
7670
Weights learned with RL	joint	human	Δ
reranker	0.88	0.92	-0.04
reranker + speaker ResNet	0.92	0.90	+0.02
reranker + both agent ResNets	0.96	0.88	+0.08
by increasing the number of components that get updated using the joint reward, the margin between the referential success of the two types of listeners increases. Despite the fact that the speaker is using human language that is perfectly fluent and accurate with respect to the target image (since the reranker operates on captions associated with the target image), while the joint listener is able to communicate with the agent speaker, the human listener achieves significantly lower performance.
In one test example, the speaker said Mike has a hat, which was equally true for both images making the human pick at random. So, how could the listener pick correctly? The speaker had reached a pact with the listener that the interpretation of this message will be something beyond what the phrase means (e.g., Mike has a yellow hat or the intensity of the pixels in the target image is lower). Since speaker and listener learn together, they co-adapt, forming conventions (or conceptual pacts (Brennan and Clark, 1996)) that differ from humans', even in the presence of fluent and grounded language.
7 Speakers trained using fixed listener
In the previous section we showed that learning a speaker using a learned reward module as a scaffolding (i.e., the joint listener) can lead to pragmatic drift. In this section, we use a grounded reward as scaffolding. In the absence of a human listener to provide rewards for learning, we use the oracle fixed listener, which was found in Section 5 to be predictive of human referential success. It is pre-trained, stays fixed and just provides rewards for training the speaker. As speakers, we use the models that scored the highest in Table 2 and retrain them against fixed. Table 6 presents the results of referential success against fixed and human listeners. Using a grounded reward results in better performance for the weaker models. The small gap between the rerankers in the two experimental setups points that using a learned reward module (joint) holds promise, despite the different types of language drift. Moreover, we show that our mod
Table 5: Referential success for PoE with gold captions when updating different components during training with joint listener.
Model	fixed	human
reward finetuning, with KL	0.81	0.75
multi-task learning, λs=0.1	0.80	0.68
PoE, λs=0	0.93	0.86
noisy channel	0.88	0.87
Table 6: Referential success when training speakers with the fixed listener.
els for learning language can be used against fixed reward models, potentially learning directly from human rewards (Ziegler et al., 2019).
We presented a method for teaching agents to communicate with humans in natural language, by combining two learning signals coming from multi-agent communication and traditional data-driven natural language learning techniques, which adds on recent efforts of blending emergent communication with natural language (Lowe et al., 2020; Lu et al., 2020).
Self-play between speakers and listeners can result in language drift, the most severe of which being pragmatic drift. Since speakers and listeners are learning concurrently, they can co-adapt to pair-specific policies that deviate from the policies that humans learn. This pathological behaviour of self-play is not specific to language and extends to other policies (Carroll et al., 2019).
Finally, we introduced the reward-learned reranker approach which alleviates language drift and achieves the highest human performance, by constraining the functional learning to happen on the level of utterances generated by a pre-trained language model. However, since the functional signal is not currently influencing the sampling from the language model, this will lead to poor performance when using more general language models with weaker conditioning (e.g. GPT2 (Radford et al., 2019)) whose samples potentially do not fit the functional context. Moving towards integrating our findings into more realistic applications of self-play, e.g., user simulation in dialogue (Schatzmann et al., 2006; Shah et al., 2008), these shortcomings need to be addressed.
We thank Kris Cao, Laura Rimell, Chris Dyer, Phil Blunsom, Aida Nematzadeh and the DeepMind Language Team for useful feedback, Susie Young and Adam Liska for help with annotations.
7671
Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182, Austin, Texas. Association for Computational Linguistics.
John Langshaw Austin. 1975. How to do things with words. Oxford university press.
Diane Bouchacourt and Marco Baroni. 2018. How agents see things: On visual representations in an emergent language game. arXiv preprint arXiv:1808.10696.
Susan E Brennan and Herbert H Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6):1482.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, pages 5175-5186.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. 2019. Anti-efficient encoding in emergent communication. arXiv preprint arXiv:1905.12561.
Herbert H Clark. 1996. Using language. Cambridge university press.
Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, volume 2, pages 439-443.
Bo Dai and Dahua Lin. 2017. Contrastive learning for image captioning. In Advances in Neural Information Processing Systems, pages 898-907.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. 2017. Emergent communication in a multi-modal, multi-step referential game. arXiv preprint arXiv:1705.10369.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. 2016. Learning to communicate with deep multi-agent reinforcement learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2137-2145. Curran Associates, Inc.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In Advances in Neural Information Processing Systems, pages 3314-3325.
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. 2019. Emergent linguistic phenomena in multiagent communication games. arXiv preprint arXiv:1901.08706.
Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in neural information processing systems, pages 2149-2159.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
R. Kneser and H. Ney. 1995. Improved backing-off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 181-184 vol.1.
Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge'naturally'in multi-agent dialog. arXiv preprint arXiv:1706.08502.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. 2018. Emergence of linguistic communication from referential games with symbolic and pixel input. ICLR.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017. Multi-agent cooperation and the emergence of (natural) language. ICLR.
Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019. Countering language drift via visual grounding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4376-4386.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Ryan Lowe, Abhinav Gupta, Jakob N. Foerster, Douwe Kiela, and Joelle Pineau. 2020. On the interaction between supervision and self-play in emergent communication. ArXiv, abs/2002.01093.
Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron C. Courville. 2020. Countering language drift with seeded iterated learning. *ArXiv*, abs/2003.12694.
7672