Venue: EMNLP 2017

# Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog

 Satwik Kottur\({}^{1}\)

Jose M.F. Moura\({}^{1}\)

Stefan Lee\({}^{2,3}\)

Dhruv Batra\({}^{3,4}\)

Carnegie Mellon University, \({}^{2}\)Virginia Tech, \({}^{3}\)Georgia Tech, \({}^{4}\)Facebook AI Research

###### Abstract

A number of recent works have proposed techniques for end-to-end learning of communication protocols among cooperative multi-agent populations, and have simultaneously found the _emergence of grounded human-interpretable language_ in the protocols developed by the agents, learned without any human supervision! In this paper, using a _Task & Talk_ reference game between two agents as a testbed, we present a sequence of 'negative' results culminating in a 'positive' one - showing that while most agent-invented languages are effective (achieve achieve near-perfect task rewards), they are decidedly _not_ interpretable or compositional. In essence, we find that natural language does not emerge 'naturally', despite the semblance of ease of natural-language-emergence that one may gather from recent literature. We discuss how it is possible to coax the invented languages to become more and more human-like and compositional by increasing restrictions on how two agents may communicate.

## 1 Introduction

One fundamental goal of artificial intelligence (AI) is the development of goal-driven dialog agents - specifically, agents that can perceive their environment (through vision, audition, or other sensors), and communicate with humans or other agents in natural language towards a goal.

While historically such agents have been based on _slot filling_[9], the dominant paradigm today is neural dialog models [5, 6, 7] trained on large quantities of data. Perhaps somewhat counterintuitively, this current paradigm treats dialog as a static supervised learning problem, rather than as the interactive agent learning problem that it naturally is. Specifically, a typical pipeline is to collect a large dataset of human-human dialog [8, 10, 11, 12], inject a machine in the middle of a dialog from the dataset, and supervisite it to mimic the human response. While this teaches the agent correlations between symbols, it does not convey the functional meaning of language, grounding (mapping physical concepts to words), compositionality (combining knowledge of simpler concepts to describe richer concepts), or aspects of planning (why are we having this conversation?).

An alternative paradigm that has a long history [13, 14] and is witnessing a recent resurgence [15, 16, 17, 18, 19, 20] - is situated language learning. A number of recent works have proposed reinforcement learning techniques for learning the communication protocols of agents situated in virtual environments in a completely end-to-end manner - from perceptual input (pixels) to communication (discrete symbols without any pre-specified meanings) to action (signaling in reference games or navigating in an environment) - and have simultaneously found the _emergence of grounded human-interpretable (often compositional) language_ among agents, without any human supervision or pretraining, simply to succeed at the task.

In this short paper, we study the following question - what are the conditions that lead to the emergence of human-interpretable or compositional grounded language? Our key finding is that

natural language _does not emerge 'naturally'_ in multi-agent dialog, despite independently reported successful demonstrations in recent literature.

Specifically, in a sequence of 'negative' results culminating in a 'positive' one, we find that while agents always successfully invent communication protocols and languages to achieve their goals with near-perfect accuracies, the invented languages are decidedly _not_ compositional, interpretable, or 'natural'; and that it is possible to coax the invented languages to become more and more human-like and compositional by increasing restrictions on how two agents may communicate.

* tabular Q-Learning (which does not scale) REINFORCE (which does), and generalization to novel environments (not studied in prior work). In the spirit of Abbeel et al. (2017), we hope our findings shed more light into the interpretability of languages invented in cooperative multi-agent settings, place recent work in appropriate context, and inform fruitful directions for future work.

## 2 The _Task & Talk_ Game

Our testbed is a reference game (_Task & Talk_) between two agents, Q-bot and A-bot. The game is grounded in a synthetic world of objects comprised of three attributes - _color_, _style_, and _shape_ - each with four possible values for a total of \(4\times 4\times 4=64\) objects. Fig. 0(a) shows some example instances from this set.

_Task & Talk_ plays out over multiple rounds of dialog. At the start, A-bot is given an instance (_green, dotted, square)_ unseen by Q-bot, and Q-bot is assigned a task \(G\) (unknown to A-bot) consisting of two attributes for Q-bot to discover from A-bot (_color, style)_. For two rounds, Q-bot and A-bot exchange utterances from finite vocabularies \(V_{Q}\) and \(V_{A}\), with Q-bot speaking first. The game culminates in Q-bot guessing a pair of attribute values (_green, dotted)_ and both agents are rewarded identically based on the accuracy of this prediction.

Note that the Task & Talk game setting involves an informational asymmetry between the agents - A-bot sees the object while Q-bot does not; similarly Q-bot knows the task while A-bot does not. Thus, a two-way communication is necessary for success. Without this asymmetry, A-bot could simply convey the target attributes from the task without Q-bot having to speak. Such a setting has been widely studies in economics and game theory as the classic Lewis Signaling (LS) game [8]. By necessitating _dialog_ between agents, we are able ground both \(V_{A}\) and \(V_{Q}\) in our final setting (Sec. 4.3).

## 3 Modeling Q-bot and A-bot

We formalize Q-bot and A-bot as agents operating in a partially observable world and optimize their policies using deep reinforcement learning.

**States and Actions.** Each agent observes its own input (task \(G\) for Q-bot and object instance \(I\) for A-bot) and the output of the other agent as a stochastic environment. At the beginning of round \(t\), Q-bot observes state \(s^{t}_{Q}{=}[G,q_{1},a_{1},\ldots,q_{t-1},a_{t-1}]\) and acts by uttering some token \(q_{t}\) from its vocabulary \(V_{Q}\). Similarly, A-bot observes the history and this new utterance as state \(s^{t}_{A}{=}[I,q_{1},a_{1},\ldots,q_{t-1},a_{t-1},q_{t}]\) and emits a response \(a_{t}\) from \(V_{A}\). At the last round, Q-bot takes a final action by predicting a pair of attribute values \(\hat{w}^{G}=(\hat{w}^{G}_{1},\hat{w}^{G}_{2})\) to solve the task.

**Cooperative Reward.** Both Q-bot and A-bot are rewarded identically based on the accuracy of

Figure 1: _Task & Talk_: The testbed for our study is cooperative 2-player game, _Task & Talk_, grounded in a synthetic world of objects with 4 _shapes_\(\times\) 4 _colors_\(\times\) 4 _styles_. The two agents, Q-bot and A-bot, are modeled as neural networks, and their policies are learned via REINFORCE. We find that the languages invented by the two agents are typically _not_ ‘natural’.

 

Q-bot's prediction \({\hat{w}^{G}}\), receiving a positive reward of \(R{=}1\) if the prediction matches ground truth and a negative reward of \(R{=}{-}10\) otherwise. We arrive at these values empirically.

**Policy Networks.** We model Q-bot and A-bot as operating under stochastic policies \(\pi_{Q}(q_{t}|s_{t}^{Q};\theta_{Q})\) and \(\pi_{A}(a_{t}|s_{t}^{A};\theta_{A})\) respectively, which we instantiate as LSTM-based models. We use lower case characters (_e.g._\(s_{t}^{Q}\)) to denote the strings (_e.g._ Q-bot's token at round \(t\)), and upper case \(S_{t}^{Q}\) to denote the corresponding vector as encoded by the model.

As shown in Fig. 1, Q-bot is modeled with three modules - speaking, listening, and prediction. The task \(G\) is received as a 6-dimensional one-hot encoding over the space of possible tasks and embedded via the listener LSTM. At each round \(t\), the _speaker network_ models the probability of output utterances \(q_{t}\in V_{Q}\) based on the state \(S_{t-1}^{Q}\). This is modeled as a fully-connected layer followed by a softmax that transforms \(S_{t-1}^{Q}\) to a distribution over \(V_{Q}\). After receiving the reply \(a_{t}\) from A-bot, the _listener LSTM_ updates the state by processing both tokens of the dialog exchange. In the final round, the _prediction LSTM_ is unrolled twice to produce Q-bot's prediction based on the final state \(S_{T}^{Q}\) and the task \(G\). As before, task \(G\) is fed in one-hot to the prediction LSTM for two time steps, resulting in a pair of outputs used as the prediction \({\hat{w}_{G}}\).

Analogously, A-bot is modeled as a combination of a _speaker network_, a _listener LSTM_, and an _instance encoder_. Like in Q-bot, the speaker network models the probability of utterances \(a_{t}\in V_{A}\) given the state \(S_{t}^{A}\) and the listener LSTM updates the state \(S_{t}^{A}\) based on dialog exchanges. The instance encoder embeds each one-hot attribute vector via a linear layer and concatenates all three encodings to obtain a unified instance representation.

**Learning Policies with REINFORCE.** We train these models using the popular REINFORCE Williams (1992) policy gradient algorithm. Note that while the game is fully-cooperative, we do not assume full observability of one agent by another, opting instead to treat one agent as part of the unknown stochastic environment when updating the other. During training, we sample 1000 two round dialog episodes per batch and update policy parameters with Adam Kingma and Ba (2015) based on these approximate gradients. Our code is publicly available1.

Footnote 1: github.com/batra-mlp-lab/lang-emerge

## 4 The Road to Compositionality

This section details our key observation - that while the agents always successfully invent a language to solve the game with near-perfect accuracies, the invented languages are decidedly _not_ compositional, interpretable, or 'natural' (_e.g._ A-bot ignoring Q-bot's utterances and simply encoding every object with a unique symbol if the vocabulary is sufficiently large). In our setting, the language being compositional simply amounts to the ability of the agents to communicate the compositional atoms of a task (_e.g._shape_ or _color_) and an instance (_e.g. square_ or _blue_) independently.

Through this section, we present a series of settings that get progressively more restrictive to coax the agents towards adopting a compositional language, providing analysis of the learned languages developed along the way. Table 1 summarizes results for all settings. In all experiments, optimal policies (achieving near-perfect rewards) were found. For each setting, we provide qualitative analysis of the learned languages and report their ability to generalize to unseen instances. We use 80% of the object-instances for training and the remaining 20% to evaluate these learned policies. Further, greedy argmax policies are used at evaluation time.

### Overcomplete Vocabularies

We begin with the simplest setting where both A-bot and Q-bot are given arbitrarily large vocabularies. We find that when \(|V_{A}|\) is greater than the number of instances (64), the learned policy simply has A-bot ignore what Q-bot asks and instead convey the instance using a single symbol, _e.g._ token \(1{\equiv}\)(_red, square, filled_). Notice that this means no 'dialog' is necessary and amounts to each agent having a codebook that maps symbols to object instances.

Perhaps as expected, the generalization of this language to unseen instances is quite poor (success rate: 25.6%). The adopted strategy of mapping instances to token pairs fails for test instances containing novel combinations of attributes for which the agents lack an agreed-upon code from training. It seems clear that like in human communication Nowak et al. (2000), a limited vocabulary that cannot possibly encode the richness of the world seems to be necessary for non-trivial dialog to emerge. We explore such a setting next.

 



### Attribute & Value Vocabulary

Since our world has 3 attributes (_shape/color/ style_) and \(4+4+4=12\) possible settings of their states, one may believe that the intuitive choice of \(|V_{Q}|=3\) and \(|V_{A}|=12\) will be enough to circumvent the 'cheating' enumeration strategy from the previous experiment. Surprisingly, we find that the new language learned in this setting is not only decidedly non-compositional but also very difficult to interpret! We present two salient observations.

We observe that Q-bot uses only the first round to convey the task to A-bot by encoding tasks in an order-independent fashion _e.g_. the _(style,color)_ and _(color,style)_ tasks are both expressed as the utterance \(Z\) in the first round. Consequentially, multiple rounds of dialog are rendered unnecessary and the second round is inconsistent across instances even for the same task.

Given the task from Q-bot in the first round, A-bot only needs to identify one of the \(4\times 4=16\) attribute pairs for a given task. Rather than ground its symbols into individual states, A-bot follows a 'set partitioning' strategy, _i.e_. A-bot identifies a pair of attributes with a unique combinations of round 1 and 2 utterances (_i.e_. the round 2 utterance has no meaning independent from round 1). Thus, symbols are reused across tasks to describe different attributes (_i.e_. symbols do not have individual consistent groundings). This 'set partitioning' strategy is consistent with known results from game theory on Nash equilibria in 'cheap talk' games Crawford and Sobel (1982).

This strategy has improved generalization to unseen instances because it is able to communicate the task; however, it fails on unseen attribute value combinations because it is not compositional.

### Memoryless A-bot, Minimal Vocabulary

The key problem with the previous setting is that A-bot's utterances _mean different things_ based on the round of dialog (\(a_{1}=1\) is different from \(a_{2}=1\)). Essentially, the communication protocol is overparameterized and we must limit it further. First, we limit A-bot's vocabulary to \(|V_{A}|\)=\(4\) to reduce the number of 'synonyms' the agents learn. Second, we _remove A-bot's memory_ by reseting the state vector \(S^{A}\) at each time step, which eliminates its ability to enumerate all attribute pairs.

These restrictions result in a learned language that grounds _individual symbols_ into attributes and their states. For example, Q-bot learns that \(Y\rightarrow\text{shape}\), \(X\rightarrow\text{color}\), and \(Z\rightarrow\text{style}\). Q-bot does not however learn to always utter these symbols in the same order as the task, _e.g_. asking for _shape_ first for both \((color,shape)\) and \((shape,color)\). Notice that this is perfectly valid as Q-bot can later re-arrange the attributes in the task desired order. Similarly, A-bot learns mappings to attribute values for each attribute query that remain consistent regardless of round (_i.e_. when asked for \(color\), \(1\) always means \(blue\)).

This is similar to learned languages reported in recent works and is most closely related to Das et al. (2017b), who solve this problem by taking away Q-bot's state rather than A-bot's memory. Their approach can be interpreted as Q-bot 'forgetting' the task after interacting with A-bot. However, this behavior of Q-bot to remember the task only during dialog but not while predicting is somewhat unnatural compared to our setting.

Tab. 2 enumerates the learnt groundings for both the agents. Given this mapping, we can predict a plausible dialog between the agents for any unseen instance and task combination. Notice that this is possible only due to the compositionality in the emergent language between the two agents. For example, consider solving _(shape, color)_ for an instance _(red, square, filled)_ from Fig. 2(b). Q-bot queries _Y (shape)_ and _X (color)_ across two rounds, and receives _2 (square)_ and _4 (red)_ as answers.

Intuitively, this consistently grounded and compositional language has the greatest ability to generalize among the settings we have explored, correctly answering 74.4% of the held out instances. We note that errors in this setting seem to largely be due to A-bot giving an incorrect answers despite Q-bot asking the correct questions to accomplish the task. A plausible reason could be the model approximation error stemming from the instance encoder as test instances are unseen and have novel attribute combinations.

Fig. 2(b) shows the dialog for the instance _(red, square, filled)_ and task _(shape, color)_. Q-bot queries _Y (shape)_ and _(color)_ across two rounds,

\begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{Setting} & \multicolumn{2}{c}{Vocab.} & Memory & Gen.(\%) \\  & \(V_{Q}\) & \(V_{A}\) & A & Q & Both & One \\ \hline Overcomplete (§4.1) & 64 & 64 & ✓ & ✓ & 25.6 & 79.5 \\ Attr-Value (§4.2) & 3 & 12 & ✓ & ✓ & 38.5 & 88.4 \\ NoMem-Min (§4.3) & 3 & 4 & ✗ & ✓ & **74.4** & **94.9** \\ \hline \end{tabular}
\end{table}
Table 1: Overview of settings we explore to analyze the language learnt by two agents in a cooperative game, Task & Talk. Last two columns measure generalization in terms of prediction accuracy of **both** or at least **one** of the attribute pair, on a held-out test set containing unseen instances.



and receives _2 (square)_ and _4 (red)_ as answers.

### Evolution of Language Timeline

To gain further insight into the languages learned, we create a _language evolution_ plot in Fig. 2. Specifically, at regular intervals during policy learning, we construct 'dialog trees'. A dialog tree enumerates all plausible dialogs between the two agents \((q_{1},a_{1},q_{2},a_{2})\), as a tree. The root node is \(q_{1}\), and at any node, we go deeper by choosing a branch based on the next utterance in the dialog. Since Task & Talk runs for two rounds, our dialog trees are \(4\) layers deep with \(|V_{A}|^{2}|V_{Q}|^{2}\) leaves. Notice that a single input instance could potentially result in different dialogs depending on the task. Hence, we consider _(instance, task)_ pairs and assign each to a leaf by traversing the tree according to the resulting dialog. At some point in the learning, the nodes become and stay 'pure' (the common trend among all _(instance, task)_ at the node stays constant till the end of training), at which point we can say that the agents have learned this dialog subsequence.

**Construction.** After constructing dialog trees at regular intervals, we identify 'concepts' at each node/leaf using the dialog tree of the completely trained model, which achieves a perfect accuracy on train set. A concept is simply the common trend among all the _(instance, task)_ tuples either assigned to a leaf or contained within the sub-tree with a node as root. Next, given a resultant concept for each of the node/leaf, we backtrack in time and check for the first occurrence when only tuples which satisfy the corresponding concept are assigned to that particular node/leaf. In other words, we compute the earliest time when a node/leaf is 'pure' with respect to its final learned concept. Finally, we plot these leaves/nodes and the associated concept with their backtracked time to get Fig. 2.

**Observations.** We highlight key observations below: (a) The agents ground most of the tasks initially at around epoch 20. Specifically, Q-bot assigns \(Y\) to both _(shape, style), (style, shape), (shape,color)_ and _(color, shape)_, while _(color, style)_ is mapped to \(Z\). Hence, Q-bot learns its first token very early into the training procedure. (b) The only other task _(style, color)_ is grounded towards the end (around epoch 170) using \(X\), leading to an immediate convergence. (c) We see a strong correlation between improvement in performance and when agents learn a language grounding. In particular, there is an improvement from \(40\%\) to \(80\%\) within a span of 25 epochs where most of the grounding is achieved, as seen from Fig. 2.

## 5 Conclusion

In conclusion, we presented a sequence of 'negative' results culminating in a 'positive' one - showing that while most invented languages are effective (_i.e_. achieve near-perfect rewards), they are decidedly _not_ interpretable or compositional. Our goal is simply to improve understanding and interpretability of invented languages in multi-agent dialog, place recent work in context, and inform fruitful directions for future work.

Figure 2: (a) Evolution of Language: timeline shows groundings learned by the agents during training, overlaid on the accuracy. Note that Q-bot learns encodings for all tasks early (around epoch \(20\)) except _(style, color)_. Improvement in accuracy is strongly correlated with groundings learnt. (b) Example dialogs for memoryless A-bot, minimal vocabulary setting §4.3.

 



## References

* Abbeel et al. (2017) Pieter Abbeel, Igor Mordatch, Ryan Lowe, Jon Gauthier, and Jack Clark. 2017. Learning to Communicate. [https://blog.openai.com/learning-to-communicate/](https://blog.openai.com/learning-to-communicate/).
* Bordes and Weston (2016) Antoine Bordes and Jason Weston. 2016. Learning End-to-End Goal-Oriented Dialog. _arXiv preprint arXiv:1605.07683_ .
* Crawford and Sobel (1982) Vincent Crawford and Joel Sobel. 1982. Strategic information transmission. _Econometrica_ 50(6):1431-51.
* Das et al. (2017a) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M.F. Moura, Devi Parikh, and Dhruv Batra. 2017a. Visual Dialog. In _CVPR_.
* Das et al. (2017b) Abhishek Das, Satwik Kottur, Jose M.F. Moura, Stefan Lee, and Dhruv Batra. 2017b. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning. _arXiv preprint arXiv:1703.06585_ .
* de Vries et al. (2017) Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C. Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In _CVPR_.
* Foerster et al. (2016) Jakob Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. 2016. Learning to communicate with deep multi-agent reinforcement learning. In _NIPS_.
* Havrylov and Titov (2017) Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In _ICLR Workshop_.
* Jorge et al. (2016) Emilio Jorge, Mikael Kageback, and Emil Gustavsson. 2016. Learning to play guess who? and inventing a grounded language as a consequence. In _NIPS workshop on deep reinforcement learning_.
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In _ICLR_.
* Kirby et al. (2014) Simon Kirby, Tom Griffiths, and Konny Smith. 2014. Iterated learning and the evolution of language. _Current opinion in neurobiology_ 28:108-114.
* Lazaridou et al. (2017) Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017. Multi-agent cooperation and the emergence of (natural) language. In _ICLR_.
* Lemon et al. (2006) Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew Stuttle. 2006. An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system. In _EACL_.
* Lewis (2008) David Lewis. 2008. _Convention: A philosophical study_. John Wiley & Sons.
* Lowe et al. (2015) Ryan Lowe, Nilsan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In _SIGDIAL_.
* Mordatch and Abbeel (2017) Igor Mordatch and Pieter Abbeel. 2017. Emergence of grounded compositional language in multi-agent populations. _arXiv preprint arXiv:1703.04908_ .
* Mostafazadeh et al. (2017) Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P. Spithourakis, and Lucy Vanderwende. 2017. Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation. _arXiv preprint arXiv:1701.08251_ .
* Nowak et al. (2000) Martin A. Nowak, Joshua B. Plotkin, and Vincent A. A. Jansen. 2000. The evolution of syntactic communication. _Nature_ 404(6777):495-498.
* Serban et al. (2016a) Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models. In _AAAI_.
* Serban et al. (2016b) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. _arXiv preprint arXiv:1605.06069_ .
* Sukhbaaatar et al. (2016) Sainbayar Sukhbaatar, Rob Fergus, et al. 2016. Learning multiagent communication with backpropagation. In _NIPS_. pages 2244-2252.
* Wang et al. (2016) Sida I Wang, Percry Liang, and Christopher D Manning. 2016. Learning language games through interaction. _Association for Computational Linguistics (ACL)_ .
* Weston (2016) Jason Weston. 2016. Dialog-based language learning. _arXiv preprint arXiv:1604.06045_ .
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_ 8(3-4):229-256.
* Winograd (1971) Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. Technical report, DTIC Document.