SlowGuess/ACL_2020_OCR huggingface dataset:
Angeliki Lazaridou*, Anna Potapenko*, Olivier Tieleman*
DeepMind, London, UK
{angeliki,apotapenko,tieleman}@google.com
We present a method for combining multiagent communication and traditional data-driven approaches to natural language learning, with an end goal of teaching agents to communicate with humans in natural language. Our starting point is a language model that has been trained on generic, not task-specific language data. We then place this model in a multi-agent self-play environment that generates task-specific rewards used to adapt or modulate the model, turning it into a task-conditional language model. We introduce a new way for combining the two types of learning based on the idea of reranking language model samples, and show that this method outperforms others in communicating with humans in a visual referential communication task. Finally, we present a taxonomy of different types of language drift that can occur alongside a set of measures to detect them.
In this work, we aim at making agents communicate with humans in natural language. Our starting point is a language model that has been trained on generic, not task-specific language data. We then place this model in a multi-agent communication environment that generates task-specific rewards, which are used to adapt or modulate the model, making it task-conditional. We thus propose to decompose the problem of learning language use into two components: learning "what" to say based on a given situation, and learning "how" to say it. The "what" is the essence of communication that underlies our intentions and is chosen by maximizing any given utility, making it a functional, utility-driven process. On the other hand, the "how" is a surface realization of our intentions, i.e., the words we use
to communicate this "what" successfully. This factorization into content planning (here, "what") and surface realization (here, "how") moves us away from end-to-end neural generation systems and is in line with traditional methods of natural language generation (Reiter and Dale, 1997). More importantly, it enables us to bring together two different strands of research: traditional data-driven natural language learning and multi-agent communication.
Traditional approaches to natural language learning (Kneser and Ney, 1995; Mikolov et al., 2010; Sutskever et al., 2014; Vinyals and Le, 2015; Radford et al., 2019) are based on inferring structural properties of language from text corpora, often in a passive regime, dissociated from communication. While this type of learning is great for learning general statistical associations between symbols (e.g., adjectives come before nouns) and even inferring semantic relations, it ignores the functional aspects of communication, i.e., the fact that people use words to coordinate with others and make things happen in the world (Wittgenstein, 1953; Austin, 1975; Clark, 1996).
On the other hand, multi-agent communication research (Foerster et al., 2016; Lazaridou et al., 2017; Havrylov and Titov, 2017; Evtimova et al., 2017; Lee et al., 2019) puts communication at the heart of agents' (language) learning. Implemented within a multi-agent reinforcement learning setup, agents start tabula rasa and form communication protocols that maximize task rewards. While this purely utilitarian framework results in agents that successfully learn to solve the task by creating a communication protocol, these emergent communication protocols do not bear core properties of natural language. Chaabouni et al. (2019) show that protocols found through emergent communication, unlike natural language, do not conform to Zipf's Law of Abbreviation; Kottur et al. (2017) find that communication protocols do not follow composi
7663
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tionality patterns of natural language, and Lazari-dou et al. (2018) find emerged protocols to be sensitive to different experimental conditions. This growing set of alarming results on emergent communication raises doubts about the use of this type of functional learning as a viable alternative to language learning.
Concluding that neither approach on its own is adequate for learning language use, we propose a method for combining the best of both worlds. Generic language data can be used effectively as a good prior model of language, encapsulating its intrinsic structural properties, i.e., are only used for the "how" in the form of generic language models. Conversely, multi-agent interactions, that provide rewards specific to the task of interest, now only need to be used for the functional learning of language use, i.e., learning the "what".<sup>1</sup>
The contributions of this paper are as follows. First, we propose a general research program of language learning that combines two learning signals coming from multi-agent communication and traditional data-driven natural language learning techniques. We present a concrete study in the context of a referential communication game (see Section 2) between a speaker and a listener, where the traditional data-driven language learning takes the form of image captioning, and the functional learning takes the form of agent self-play (see Section 3). We then present a new approach for combining the two learning signals, i.e., reward-learned rerankers (see Section 4), and compare this to existing approaches using a human study (see Section 5). We discuss shortcomings of this program with respect to different types of language drift that can occur, and introduce a number of automatic measures to detect them (see Section 6). Finally, we show how such a program under oracle rewards can be a viable approach moving towards learning language use from human rewards (see Section 7).
2 Research framing
Our research can be framed in the following scenario. An agent needs to perform a functional communication task in a natural language (in this work, English). However, examples of linguistic communication about this functional task are not available - the only natural language data that can be used consist of examples of generic natural language, which are not grounded in the functional task. Recasting the task as a multi-agent language game provides a way to obtain a reward that judges whether an utterance elicited the correct behaviour by a listener.
In this work, we instantiate the research in the following way: the functional task is a visual referential communication game for a target image in the context of a distractor, the reward is based on success in referential communication where a listener is tasked to pick the correct image within distractors guided by the speaker's description, and the generic natural language data are captioning data.
Visual referential communication game.
There are two players, the speaker and the listener. The speaker sees a target object and needs to communicate an utterance about it in the context of distractors; both target and distractors are represented as images. The listener is presented with the same set of images, but without the knowledge of which is the target, and needs to identify the target image relying on the utterance being communicated by the speaker. The utterance takes the form of sequences of word-like units. If the listener's choice is correct they both receive a positive reward, else they receive the same negative reward.[2]
Dataset and referential splits. For playing the visual referential communication game, we use a multi-modal dataset, the Abstract Scenes (Zitnick and Parikh, 2013) which contains 10k synthetic images accompanied with descriptive captions (on average 6 per image) (see Figure 1). The cap
2The task we consider is essentially discriminative image captioning (Vedantam et al., 2017; Dai and Lin, 2017; Andreas and Klein, 2016). Here we are using it as a placeholder of a communication task to illustrate our general framework. Thus, we are not incorporating any explicit bias in the model about this particular task. The only task-specific information we use is communicated via the reward.
3Other multi-modal datasets like MSCOCO (Lin et al., 2014) or Flickr (Thomee et al., 2016), while providing com
<sup>1</sup>About the terminology: by 'traditional data-driven natural language learning', we mean language modelling of the next-word-prediction variety. This type of learning does not involve any use of the language or other context, and as such only focuses on word statistics. Since the structure of the language is a large part of those statistics, and the role of the generic language models in our proposed combined systems is to provide structural knowledge of language, we also use the term 'structural learning'. We contrast this with the purely usage-driven, reward-based learning of the type seen in emergent communication research. Since the function, rather than the structure or statistics, is the only thing that matters for such a learner, we also use the term 'functional learning'.
7664
Figure 1: Example image and ground-truth captions from the Abstract Scenes dataset used in this study.
speaker (human)	easy	difficult
random	0.92	0.81
discriminative	1.0	0.97
Table 1: Accuracy performance of a human listener with a human speaker producing either random or discriminative caption on the easy and difficult splits.
tions typically refer to diverse aspects of the scene (characters and actions), providing a rich and challenging environment for an agent to evolve the captioning skills for successful communication. In our experiments, we split the dataset into 80/10/10 for train/validation/test sets. We use the test images to create two referential splits, i.e., easy and difficult, as a function of the similarity between the target and distractor images. Each split contains 1000 pairs of a target and a distractor.
Human performance and setup validation. In order to assess the difficulty of the task in the presence of the particular data (images and captions) we perform a human study in the reference game with a human speaker and a human listener, where the human speaker can only communicate one of the existing captions of the target image. We perform the human study under two conditions. In the first condition, the human speaker has only access to the ground-truth captions and does not have access to the distractor image, thus has to pick a random caption. This corresponds to the perfect structural knowledge of English but no knowledge of the functional task and it is the human upper-bound of a captioning system performance on this task. In the second condition, the speaker has access to both the ground-truth captions and the distractor image, thus is able to pick a discriminative caption to communicate. For each condition, we collect 50 rounds
of games and present results in Table 1. We see that the task-specific condition outperforms the first condition, indicating that in our current setup there is enough space to improve upon models based on structural-only learning (i.e., captioning models). Moreover, the good performance of discriminative caption speaker demonstrates that (in principle) the captioning data can be used in a successful communication with a human for this task.
3 Multi-agent communication setup
3.1 Speaker
The speaker is the primary learner in this research, aiming at creating a model that is able to use natural language in a communicative scenario, and consists of standard visual and language modules. To convert images to embeddings  $u$ , we use a pretrained ResNet (He et al., 2016) (parametrized by  $\theta^{resnet}$ ) and feed its last layer output into a one-layer MLP (parametrized by  $\theta_S^{MLP}$ ). To generate a message  $m$ , we use a one-layer LSTM (Hochreiter and Schmidhuber, 1997) (parametrized by  $\theta_S^{LSTM}$ ), adding embeddings  $u$  at each time step as additional context. Section 4 presents different speaker models consisting of these modules.
We also design two oracle speakers (with no weights) that have direct access to ground-truth captions of images at test time. The random caption speaker outputs one of the ground-truth captions for the target image at random. Since this speaker is not aware of the functional goal, their performance will indicate whether having only good grounded language skills is enough for communication success in our setup. We also build an oracle speaker that is task-aware; discriminative caption speaker uses a simple word-overlap heuristic to pick the target's caption that has the least word overlap with any of the distractor's captions (the score is normalized by the captions' length excluding stop-words).
3.2 Listener
Throughout the experiments, we need a way to estimate performance on the functional communication task, either for evaluation or to provide rewards during training acting as a scaffolding to learn the speaker model. Ideally, this performance signal should be provided by a human who is interacting online with the speaker agent. While we do so for evaluation reasons, for training we approximate this quantity with a learned component, an agent listener.
Jenny is scared of the bear  
Mike is scared of the bear  
Jenny and Mike sit by a fire  
Jenny and Mike are sitting  
A bear is scaring mike and jenny
plex naturalistic images, often have a repetitive set of captions, highlighting one particular aspect of the scene and suffer from a human reporting bias (Misra et al., 2016). By using Abstract Scenes, we have left certain visual challenges out of the scope of the work, obtained cleaner multi-modal associations between words and objects, and focused on the language use for referential communication.
To convert images to embeddings  $u$ , we use the same pre-trained ResNet as for the speaker and feed its last layer output into a one-layer MLP (parametrized by  $\theta_L^{MLP}$ ). Following that, the listener uses an LSTM (parametrized by  $\theta_L^{LSTM}$ ) to embed the utterance  $m$  received by speaker, creating embedding  $v$ . Finally, the listener picks the image with the highest dot-product similarity between the embedded message  $v$  and the embeddings  $u_t$  and  $u_d$  for target and distractor. Since we know which image candidate is the intended referent, we cast this problem as supervised learning and update the listener's weights  $\theta_L = \{\theta_L^{MLP}, \theta_L^{LSTM}\}$  optimizing cross-entropy. Finally, the listener assigns reward 1 to the speaker if they identified the correct image, else reward -1.
We consider two different setups: a joint listener, which is trained together with the speaker, as commonly done in the emergent communication literature, and a fixed listener that is pre-trained to perform best-response to the oracle discriminative caption speaker and stays fixed throughout the learning of the speakers with the sole use of providing them rewards. We expect the latter setup to be less prone to language drift issues due to the grounding of the discriminative caption speaker to language data. thus potentially resulting in better communication with human listeners. We also use the fixed listener for evaluation of all speakers.
4 Methods for learning language use
We describe ways to estimate the speaker's generative model  $p_{\theta_S}(m|u,t)$  for message  $m$ , conditioned on target and distractor embeddings  $u = [u_t,u_d]$  and target image index  $t\in \{0,1\}$ .
4.1 Functional-only learning
This type of learning language use is identical to experiments commonly conducted in the literature of emergent communication (Lazaridou et al., 2017; Havrylov and Titov, 2017; Bouchacourt and Baroni, 2018; Evtimova et al., 2017; Graesser et al., 2019), i.e., the speaker learns to emit communication utterances  $m$  in order to maximize the communication task reward (see Section 3.2 for a discussion on how this reward is computed). Concretely, the weights  $\theta_{S} = \{\theta_{S}^{MLP}, \theta_{S}^{LSTM}\}$  of the speaker policy  $\pi_{\theta_S}(m|u,t)$  are updated via the REINFORCE update rule (Williams, 1992) using rewards  $r^L$  provided by the listener, i.e., we optimize  $L^{functional} =$
$-r^{L}(m,u,t)\sum_{i = 1}^{I}\log p_{\theta_{S}^{LSTM}}(m^{i}|m^{< i},u),$  where  $u = [u_t;u_d],m^i\in V$  , vocabulary size  $|V| = 100$  and message length  $I = 10$  Note, that while this type of learning results in a language that is maximally functionally correct for the given task reward, this language is not natural language, i.e., the symbols are not grounded to natural language.
4.2 Structural-only learning
This type of learning ignores the functional aspect of communication and communicates utterances that reflect intrinsic structural properties of language, i.e., that are fluent, grammatical and related to the target. Here, we used paired data in the form  $\langle u, c \rangle$ , where  $u$  is a visual embedding and  $c$  is the associated caption, and learn an image captioning model. The speaker's parameters  $\theta_S = \{\theta_S^{MLP}, \theta_S^{LSTM}\}$  are optimized using cross-entropy, i.e.,  $L^{structural} = -\sum_{i=1}^{I} \log p_{\theta_S^{LSTM}}(c^i | c^{<i}, u)$ , where  $u = u_t$ ,  $c^i \in V$ ,  $|V| = 2685$  (the vocabulary size) and  $I = 25$ , i.e., the longest caption in the dataset. We approximate the speaker model  $p_{\theta_S}(m | u, t)$  with the captioning one, which ignores distractor, thus the communication task. We construct two speakers with different decoding schemes: greedy uses greedy decoding, while sample picks the highest probability message among  $k = 20$  stochastic samples (temperature  $\tau = 2.0$ ).
4.3 Structural and functional learning
We now describe several ways in which both types of learning are used to learn language use. In all cases, we equip the speaker with a base image captioning model similar to the one presented in Section 4.2, which is used to calculate  $p_{\theta_S^{LSTM}}(c^i | c^{<i}, u_t)$ . The functional part is learned via the REINFORCE update rule optimizing the task reward (i.e., listener's accuracy in the referential task). However, speakers differ in how they parametrize  $p_{\theta_S}(m|u, t)$  and whether the task reward is used to update the weights  $\{\theta_S^{MLP}, \theta_S^{LSTM}\}$  of the base captioning model.
4.3.1 Reward finetuning
The simplest approach is to first use existing pretrained components for which we have available corpora in order to learn the statistical properties
4In all experiments using REINFORCE we add an entropy regularization term to the loss.
of language, and then steer the language use to be functionally appropriate using reward finetuning for the given task. We use paired data in the form  $\langle u, c \rangle$  to learn the weights  $\theta_{S} = \{\theta_{S}^{MLP}, \theta_{S}^{LSTM}\}$  of a base image captioning model following Section 4.2, and then we perform functional learning by using the listener's reward to optimize the weights  $\theta_{S}$  as in Section 4.1. While this method is conceptually simple, it becomes challenging when the task requires extending the conditioning part of the base model. Here, we need to change the conditioning of the base captioning model from  $u = u_{t}$  to  $u = [u_{t}; u_{d}]$ , to allow conditioning on the distractor. Since this is not trivial (the base image captioning model has been learned by conditioning only on one image embedding), we keep the conditioning  $u = u_{t}$  also during finetuning with REINFORCE. Thus, similar to the image captioning model, we approximate  $p_{\theta_{S}}(m|u, t)$  with  $p_{\theta_{S}^{LSTM}}(m|u_{t})$ . However, unlike image captioning, the information about distractors flows into the policy, since the weights  $\theta_{S}$  are optimized using the listener's reward which considers distractors.
Since the gradients from optimizing the functional task are sent all the way into the base captioning model, this causes catastrophic forgetting of the core knowledge of language, leading to language drift. Thus, we use a language regularizer term in the form of Kullback-Leibler divergence between pre-trained and fine-tuned language modeling distributions (Havrylov and Titov, 2017).
4.3.2 Multi-task learning
An alternative is to conduct both types of learning (i.e., image captioning and functional learning) at the same time (Lazaridou et al., 2017; Lee et al., 2019). This takes the form of multi-task learning optimizing  $\lambda_{f}L^{functional} + \lambda_{s}L^{structural}$ , where  $\lambda_{f} = 1$ . Like in reward finetuning, the gradients of the reward learning flow into the weights of a base captioning model, leaving us with questions about a trade-off between task success and quality of language. Therefore, we introduce two variants of this model depending on the importance of the language component, i.e., one variant with  $\lambda_{s} = 0.1$  and a language-regularized one with  $\lambda_{s} = 1$ .
4.3.3 Reward-learned rerankers
Finally, we introduce a new way of learning language use in the multi-agent communication setup. As before, we train the core language capabilities of a speaker using the image captioning task objec
tive, but after this pre-training phase, the weights of this model are frozen. The functional part is then viewed as learning to use this general knowledge of language grounded in images. This is operationalized as learning to rerank samples obtained from the captioning model optimizing the listener's reward. The action space of this speaker are sentences, as opposed to words used commonly in the literature of emergent communication. We emphasize that by leveraging the idea of reranking, we are able to take a task-unconditional model, i.e., a captioning model that only conditions on the target, and extend its conditioning turning it into a task-conditional model, i.e., a discriminative captioning model that conditions also on the distractor.
Below we consider two concrete reranker models. In both cases, the message generation happens in two steps. First, we sample  $|S| = 20$  candidates from the pre-trained and fixed image captioning model  $p_{\theta_S^{LSTM}}(m|u_t)$ . Then, we pick the best sample  $s$  using a task-conditional reranking score  $p(s|u,t)$ . The reranking score can be viewed as a new policy  $\pi_{\theta_S}(s|u,t)$  that operates in the space of samples  $S$  drawn from the task-unconditional model. This policy introduces an additional set of trainable parameters  $\theta_S^{rerank}$  that are learned with REINFORCE. Thus, the full set of weights for this speaker is  $\theta_S = \{\theta_S^{MLP}, \theta_S^{LSTM}, \theta_S^{rerank}\}$ . Crucially, the two learning signals, i.e., structural and functional, affect different set of weights, i.e.,  $\{\theta_S^{MLP}, \theta_S^{LSTM}\}$  and  $\theta_S^{rerank}$  respectively, allowing submodules to specialize.
Product of experts reranker. In this model we parametrize the policy as a product of experts (PoE):  $\pi_{\theta_S}(s|u,t) \propto p(s|u,t)^{\lambda_f}p(s|u_t)^{\lambda_s}$ , where  $u = [u_t;u_d]$  and  $\lambda_f = 1$ . The second term is the image captioning message probability, renormalized over the samples space, thus bringing general language knowledge grounded in images. The first term adjusts for the task specifics. To model that, we re-embed the samples using transformed bag-of-words, thus the trainable parameters of the reranker  $\theta_S^{rerank}$  are word embeddings and additional MLP weights. We combine target and distractor embeddings into a single vector and compute the dot-product similarity between this vector and each of the bag-of-words representations of samples. Finally, these scores are passed through a softmax layer to obtain  $p(s|u,t)$ . We introduce two variants of the model, one with  $\lambda_s = 0$  and a language-regularized one with  $\lambda_s = 1$ .