Title: Multi-Agent Cooperation and the Emergence of (Natural) Language
Authors: Angeliki Lazaridou
Alexander Peysakhovich
Marco Baroni
Venue: ICLR 2017
Keywords: Natural language processing
Reinforcement Learning
Games

Abstract: The current mainstream approach to train natural language systems is to expose them to large amounts of text. This passive learning is problematic if we are in- terested in developing interactive machines, such as conversational agents. We propose a framework for language learning that relies on multi-agent communi- cation. We study this learning in the context of referential games. In these games, a sender and a receiver see a pair of images. The sender is told one of them is the target and is allowed to send a message to the receiver, while the receiver must rely on it to identify the target. Thus, the agents develop their own language interactively out of the need to communicate. We show that two networks with simple configurations are able to learn to coordinate in the referential game. We further explore whether the “word meanings” induced in the game reflect intuitive semantic properties of the objects depicted in the image, and we present a simple strategy for grounding the agents’ code into natural language, a necessary step in developing machines that should eventually be able to communicate with humans.



Review discussion:
In this paper, a referential game is proposed between two agents. Both agents observe two images. The first agent, called the sender, receive a binary target variable (t) and must send a symbol (message) to the second agent, called the receiver, such that this agent can recover the target. The agents both get a reward, if the receiver agent can predict the target. The paper proposes to parametrize the agents as neural networks - with pretrained representations of the images as feature vectors - and train them using REINFORCE. In this setting, it is shown that the agents converge to  optimal policies and that their learned communications (e.g. the symbolic code transmitted from the sender to the receiver) have some meaningful concepts.
In addition to this, the paper presents experiments on a variant of the game grounded on different image classes. In this setting, the agents appear to learn even more meaningful concepts.
Finally, multi-game setup is proposed, where the sender agent is alternating between playing the game before and playing a supervised learning task (classifying images). Not surprisingly, when anchored to the supervised learning task, the symbolic communications have even more meaningful concepts.
Learning shared representations for communication in a multi-agent setup is an interesting research direction to explore. This is a much harder task compared to standard supervised learning or single-agent reinforcement learning tasks, which justifies starting with a relatively simple task. To the best of my knowledge, the approach of first learning communication between two agents and then grounding this communication in human language is novel. As the authors remark, this may be an alternative paradigm to standard sequence-to-sequence models which tend to focus on statistical properties of language rather than their functional aspects. I believe the contributions of the proposed task and framework, and the analysis and visualization of what the communicated tokens represent is a useful stepping stone for future work.
For this reason, I think the paper should be accepted.
Other comments:
- How is the target (t) incorporated into the sender networks? Please clarify this.
- Table 1 and Table 2 use percentage (%) values differently. In the first, percentages seem to be written in the interval [0, 100], and in the second in the interval [0, 1]. Please correct this. Perhaps related to this, in Table 1, the column "obs-chance purity" seems to have extremely small values. I assume this was mistake?
- "assest" -> "assess"
- "usufal" -> "usual"
Thanks for your constructive review.
Which image constitutes the target is implicitly signaled to the sender network by always presenting it with the target as the first image (whereas the receiver sees the images in random order). We will clarify this in the paper.
Thanks for noticing the inconsistency between the tables and the wrong chance purity scale: we will fix this.
To train natural language systems by putting multiple agents within an interactive referential communication game is very nice. As the authors mention, there has been some (although seemingly not much) previous work on using multi-agent games to teach communication, and it certainly seems like a direction worth pursuing.
Moreover, the approach of switching between these games and some supervised learning, as in the experiment described in Section 5 and suggested in Section 6, seems particularly fruitful.
Note: For “clarity”, I believe some of the network connections in Fig 1 have been omitted.
However, given the rather highly-customized architecture and the slightly hard-to-follow description in Section 3, the shorthand diagram only adds to the confusion. The diagram probably needs to be fine-tuned, but at the very least (especially if I am misunderstanding it!), a caption must [still] be added to help the reader interpret the figure.
Overall, the framework (Section 2) is great and seems quite effective/useful in various ways, the results are reasonable, and I expect there will be some interesting future variations on this work as well.
Caveat: While I am quite confident I understood the paper (as per confidence score below), I do not feel I am sufficiently familiar with the most closely related literature to accurately assess the place of this work within that context.
Thanks for the helpful review.
We agree that Fig 1 is currently confusing (some connections are indeed omitted). We will fix this by adding a more descriptive caption (adding more connections to the figure would probably make it too crowded).
Thank you for an interesting read.
Pros
- This paper tackles a very crucial problem of understanding communications between 2 agents. As more and more applications of reinforcement learning are being explored, this approach brings us back to a basic question. Is the problem solving approach of machines similar to that of humans.
- The task is simple enough to make the post learning analysis intuitive.
- It was interesting to see how informed agents made use of multiple symbols to transmit the message, where as agnostic agents relied only on 2 symbols.
Cons
- The task effectively boils down to image classification, if the 2 images sent are from different categories. The symbols used are effectively the image class which the second agent learns to assign to either of the images. By all means, this approach boils down to a transfer learning problem which could probably be trained much faster than a reinforcement learning algorithm.
Thanks for your supportive review.
Regarding your suggestion of using transfer learning, we would be grateful for further details. By using the same pre-trained CNN to provide image representations to both agents, we are arguably applying a simple form of transfer learning to our task, and this certainly speeds up convergence. We also see how learning to associate symbols to object names is a sensible strategy for our agents in the current setup.
However, we do not see how we could remove the reinforcement learning phase altogether while keeping the same setup.
Also, note that, although using pre-learned object names is a good strategy in our initial experiments, it is already challenged by the ReferIt experiment (where most images contain objects that were not seen in the supervised training phase), and it would not generalize to other setups we would like to test next (such as the interesting case you suggest where both images contain the same object).

Meta-review:
role: PC
data: comment: The authors present some initial findings on language emergence using multi-agent, referential games. The learning alternates between REINFORCE and supervised classification, which grounds the language. Pro- this is a relevant, novel paper. Con - experiments are somewhat simple/limited.

Decision: Accept (Oral)