On the interaction between supervision and self-play in emergent communication
Authors: Ryan Lowe*
Abhinav Gupta*
Jakob Foerster
Douwe Kiela
Joelle Pineau
Venue:  (2020)
Keywords: multi-agent communication
self-play
emergent languages

Abstract: A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training. However, recent work suggests that current machine learning methods are too data inefficient to be trained in this way from scratch. In this paper, we investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency: imitating human language data via supervised learning, and maximizing reward in a simulated multi-agent environment via self-play (as done in emergent communication), and introduce the term supervised self-play (S2P) for algorithms using both of these signals. We find that first training agents via supervised learning on human data followed by self-play outperforms the converse, suggesting that it is not beneficial to emerge languages from scratch. We then empirically investigate various S2P schedules that begin with supervised learning in two environments: a Lewis signaling game with symbolic inputs, and an image-based referential game with natural language descriptions. Lastly, we introduce population based approaches to S2P, which further improves the performance over single-agent methods.

Published as a conference paper at ICLR 2020
ON THE INTERACTION BETWEEN SUPERVISION AND SELF-PLAY IN EMERGENT COMMUNICATION
Ryan Lowe, Abhinav Gupta' MILA
Jakob Foerster, Douwe Kiela Facebook AI Research
Joelle Pineau
Facebook AI Research MILA
ABSTRACT
A promising approach for teaching artificial agents to use natural language involves using -uman-in-the-loop training. However, recent work suggests that current machine learning methods are too data inefficient to be trained in this way from scratch. In this paper, investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency: imitating human language data via supervised learning, and maximizing reward in simu- lated multi-agent environment via self-play (as done in emergent communication), and introduce the term supervised self-play (S2P) for algorithms using both of these signals. We find that first training agents via supervised learning on human bene- data followed by self-play outperforms the converse, suggesting that ficial to emerge languages from scratch. We then empirically investigate various S2P schedules that begin with supervised learning in two environments: Lewis signaling game with symbolic inputs, and an image-based referential game with natural language descriptions. Lastly, we introduce population based approaches to S2P, which further improves the performance over single-agent methods]
INTRODUCTION
Language is one of the most important aspects of human intelligence; it allows humans to coordinate and share knowledge with each other. is also crucial for human-machine interaction, as human language is natural means by which to exchange information, give feedback, and specify goals. A promising approach for training agents to solve problems with natural language is to have a "human in the loop", meaning collect problem- data from humans interacting directly with our agents for learning. However, human-in-the-long data is expensive and time-consuming to obtain as requires continuously collecting human data the agent's policy improves, and recent work suggests that current machine learning methods (e.g. from deep reinforcement learning) are too data-inefficient be trained in this way from scratch (Chevalier-Boisver et al., 2019). Thus, an important open problem is: how can we make human--n-the--eee in-the-loop training as data efficient as possible?
To maximize data efficiency, is important to fully leverage all available training signals. In this paper, we study two categories of such training methods: imitating human data via supervised learning, and self-play to maximize reward in a multi-agent environment, both of which provide rich signals for endowing agents with language-using capabilities. However, these are potentially competing objectives, as maximizing environmental reward can lead to the resulting communication protocol drifting from natural language (Lewis et al.) 2017 Lee et al.) 2019). The crucial question, then, is how do we best combine self-play and supervised updates? This question has received surprisingly little attention from the emergent communication literature, where the question of how to bridge the gap from emergent protocols to natural language is generally left for future work (Mordatch & Abbeel 2018 Lazaridou al. 2018 Cao al. 2018).
'These two authors contributed equally. Work done primarily while RL was at Facebook AI. Correspondence to: ryan.
abh nav@@nuu edu
Coode is available alttppp://gthhhhb..oo/bbberrrppp

---

Published as a conference paper at ICLR 2020
Phase 1: Collect seed dataset
Phase 2: Supervised learning
Phase 3: Self-play
Phase 4: Test agent
Object reconstruction (OR) game
33
APA
Pb Caa
O
OO
haae- referential (BB))
+
O
. play updates
Supervised updberze
Urterance
Prodiction
Olservation Vessige Pestuctio
(a)
(b)
Figure I: (a) Diagram of the supervised self-play (S2P) procedure (phases 1-3) and the testing procedure considered in this work (phase 4). (b) The environments considered in this paper (Sec.
Our goal in this paper is to investigate algorithms for combining supervised learning with self-play which we call supervised tasks: Lewis signaling game with symbolic inputs, and more complicated image-based referential game with natural language descriptions. Our first finding is that supervised learning followed by self-play outperforms emergent communication with supervised fine-tuning in these environments, and we provide three reasons for why this is the case. We then empirically investigate several supervised-fise S2P methods in our environments. Existing approaches this area have used various ad-hoc schedules for alternating between the two kinds of updates (I (Lazarioou et al. (017). but to our knowledge there has been no systematic study that has compared these approaches. Lastly, we propose the use of ropulation-basel methods for S2P, and find that it leads to improved performance in the more challenging image-based referential game. Our findings highlight the need for further work in combining supervised learning and self-play to develop more sample-efficient language learners.
(S2P) algorithms - using two classic emergent communication
2 RELATED WORK
In the past few years, there has been a renewed interest in the field of emergent communication (Sukhbaatar et al. 2016, Foerster et al. 2016 Lazaridou et al.) 2017 Havrylov & Titov| (017) culminating in NeurlPS workshops. Empirical studies have showed that agents can autonomously
evolve a communication protocol using discrete symbols when deployed in a multi-agent environment
which helps them to play a cooperative or competitive game (Singh et al. 2019 Cao et al. 2018
Choi et al. 2018, Resnick* al. 2019: Evtimova et al. 2018).
While the idea of promoting coordination among agents through communication sounds promising, recent experiments (Lowe et al.,, 2019 Chaabouni et al.) 2019 Kottur et al. 2017 Jaques et al., (019) have emphasized the difficulty in learning meaningful emergent communication protocols even with centralized training.
Apart from the above advances in emergent communication, there has been a long outstanding goal of learning intelligent conversational agents to be able interact with humans. This involves training the artificial agents in a way so that they achieve high scores while solving the task and their language is interpretable by humans close natural language. Recent works also add another axis orthogonal to communication where the agent also takes discrete action in an interactive environment (de Vries al.) 2018 Mul et al. 2019). Lewis al. (2017) introduced a negotiation task which involves learning linguistic and reasoning skills. They train models imitating human utterances using supervised learning and found that the model generated human-like captions but were poor negotiators. So they perform self-play with these pretrained agents in an interleaved manner and found that the performance improved drastically while avoiding language drift. [ee et al. (2019) also propose using auxiliary task for grounding the communication to counter language drift. They use visual grounding to learn the semantics of the language while still generating messages that are close to English.
A recent trend on using population based training for aulti-agent communication is a promising avenue for research using inspirations from language evolution literature (Smith al. 2003 Kirby
2

---

Published as a conference paper at ICLR 2020
2014 Raviv & Armon 2018). Cultural transmission is one such technique which focuses on the struc- ture and compression of languages, since a language must be used and learned by all individuals of the culture in which it resides and at the same time be suitable for a variety of tasks. [Harding Graesser eet al. 2019) shows the emergence of linguistic phenomena when a pool of agents contact each other giving rise to novel creole languages. Li & Bowling (2019); Cogswell al. (20T9); Tieleman et al.) (2018) have also tried different ways of imposing cultural pressures on agents, by simulating a large population of them and pairing agents solve a cooperative game with communication. They train the agent against a sampled generation of agents where the generation corresponds to the different languages of the different agent at different times in the history.
Our work is inspired from these works where we aim to formalize the recent advancements in using self-play in dialog modeling, through the lens of emergent communication.
METHODS
3.1 PROBLEM DEFINITION
Our agents are embedded in a multi-agent environment with N agents where they receive observations E O (which are functions of hidden state S) and perform actions A. Some actions Ag C A involve sending a message m E A5 over a discrete, costless communication channel (i.e. a cheap talk channel (Farrell & Rabin I996)). The agents are rewarded with a reward R for their performance in the environment. We assume throughout that the environment is cooperative and thus the agents are trained maximize the sum of rewards R Et=1p 1:T Eimisy Ti, across both agents. This can be thought of as a cooperative partially-observable Markov game (Littman] (1994)).
We define a target language L f. usually corresponding to natural language, that we want our agents to learn (we further assume L can be used to achieve high task reward). In this paper, we consider a language L E L to be simply a set of valid messages As and a mapping between observations and messages in the environment, L : O x Ar + [0, 1]. For example, in an English image-based referential game (Section 4 this corresponds to the mapping between images and image descriptions in English. We are given a dataset 'D consisting of [D] (observation, action) pairs, corresponding to No 'experts' (for us, N₂ = 2) playing the game using the target language L Our goal is to train agents to achieve high reward in the game while speaking language L* with an 'expert'. Specifically, we want our agents to generalize and to perform well on examples that are not contained in D.
To summarize, we want agents that can perform well on a collaborative task with seeaakin humans, and train them using a supervised dataset D and via self-play.
3.2 SUPERVISED SELF-PLAY (S2P)
In recent years, there have been several approaches to language learning that have combined su- pervised or imitation learning with self-play In this paper, we propose umbrella term for these algorithms called supervised self-play (S2P). S2P requires two things: (1) multi-agent environment where least one agent can send messages a reward function that measures how well the agents are doing at some task; and (2) a supervised dataset D of experts acting and speaking language L in the environment (such that they perform well on the task). Given these ingredients, we define S2P below (see Figure 2
a dedicated communication channel, along with
Definition 3.1. Supervised self-play (S2P). Supervised self-play is class of language learning algorithms that combines: (I) self-play updates in a multi-agent language environment, and (2) supervised updates on expert dataset D.
S2P algorithms can differ in how they combine self-play and supervised learning updates on D. When supervised learning is performed before self-play. we refer to the dataset 'D as the seed data. Why might we want to train our agents via self-play? Won't their language diverge from L*? One way to intuitively understand why S2P is beneficial is to think in terms of constraints. In our set-up, there are two known constraints on the target language L*: (1) it is consistent with the samples from the supervised dataset D, and (2) L" can be used to obtain high reward in the environment. Thus, finding L can be loosely viewed as a constrained optimization problem. and enforcing both constraints should clearly lead to better performance.
3

---

Published as a conference paper at ICLR 2020
3.3 ALGORITHMS FOR S2P
Here we describe several methods for S2P training. Our goal is not to exhaustively enumerate all possible optimization strategies, but rather provide a categorization combine self-play and supervised learning. To help describe these methods, further split the seed dataset into Dtrain. which is used for training, and Deat which used for early-stopping. also visualize the schedules in Figure 2
some well-known ways to
Emergent communication with supervised fine-tuning (sp2sup): We first perform self-play updates until the learning converges on the task performance. It then followed by supervised updates Dtrain until the listener performance converges the dataset Dval-
Training progress
sp2sup
sup2ap
Self play
Supervised learning with self-play (sup2sp): This the comple- ment of the above method which involves doing supervised updates until convergence on Deat followed by self- self-play updates until con- vergence on the task performance.
rand
Supervised
learning
sched
Speaker parameter
sched.
freezing
Random updates (rand); This the method used in (Lzaridou et al. 2017). At each time step, we sample a bernoulli random variable Bernoulli(g) where q is fixed. one supervised update, else we do one self- update, and repeat until both losses convergence on Dvat-
sched rand fiz
we perform
Figure A visual representa- tion of the different S2P meth- ods.
We first pretrain the listener and the speaker until convergence on Datt Then we create a schedule, where we perform self-play updates followed by supervised updates, and repeat until convergence the dataset.
Scheduled updates (sched);
Scheduled updates with speaker frreezing (sched frz): This method is based on the findings of Lewis et al. (017)), who do s sched S2P while freezing the parameters of the speaker during self-play to reduce the amount language drift. In our case, we freeze the parameters the speaker after the initial supervised learning.
Scheduled updates with random speaker freezing (sched rand frz): Experimentally. we noticed that s ached_frr frz didn't perform well in self-play. Thus, we introduce variation, we sample a bernoulli random variable = ~ Bernoulli(r) where is fixed. If = = 1, we freeze the parameters of the speaker during both self-play and supervised learning, else we allow updates the speaker as well.
3.4 PPPULATION-BASER S2P (POP-S2P)
0.85
As explained above, the goal of S2P is to pro- duce agents that follow dataset D while max- imizing reward in the environment. However, there are many such policies satisfying these criteria. This results in a large space pos- sible solutions, that increases the environ- ment grows more complex (but decreases with increasing (D)). Experimentally, we find that this can result in diverse agent policies. We show this in Figure training 50 randomly initialized agents on the image-based referential game (defined Sec. 4 the agents can often make diverse predictions for given image (Fig- ure Ba) and achieve variable performance when playing with other populations with slight pref- erence towards their own partner (the diagonal in Figure 3b).
5.0
SpeateriD
30.0
17.5
3.0
50
10.0
12.5
15.0
17.5
Prslcctoon
Lidomner
(a)
(b)
Figure 3: Results from training 50 S2P agents on the IBR game with [D] = 10000. (a) The agents have a range of predictions on many images. (b) When playing with each other, the agents exhibit uneven performance (color is mean reward, yellow higher), indicating policy variability.
Inspired by these findings, we propose to aug- ment S2P by training population of N agents, and subsequently aggregating them back into
4

---

Published as a conference paper at ICLR 2020
single agent (the 'student'). We call this invilaationbbssed S2P (Pop-S2P). While there many feasible ways of doing this, in this paper we train the populations by simply randomizing the initial seed, and we aggregate the populations using simple form of policy distillation (Rusu et al., 2016). Another simple technique to boost performance is via ensembling where we simply take the majority prediction each time step.
4 ENVIRONMENTS & IMPLEMENTATION DETAILS
We consider environments based on classical problems in emergent communication. These environ- ments are cooperative and involve the interaction between a speaker, who makes an observation and sends a message, and listener, who observes the message and makes a prediction (see Figure lb). Our goal to train a listener such that it achieves high reward when playing with an expert speaking the target language L on inputs unseen during training
Environment 1: Object Reconstruction (OR) Our first game a Lewis signaling game (Lewis) 1969) and a simpler version of the Task & Talk game from Kottur et al. (2017). with single turn and a much larger input space. The speaker agent observes an object with a certain set of properties, and must describe the object to the listener using sequence of words. The listener then attempts to reconstruct the object. More specifically, the input space consists of properties (e.g. shape, color) of types each (e.g. triangle, square). The speaker observes symbolic representation of the input, consisting of the concatenation of p one-hot vectors, each of length / = 10. The number of possible inputs scales tº, We define the vocabulary size (length of each one-hot vector sent from the speaker) as [V| = 60, and the number of words (fixed length message) sent to be T = 6.
For our target language L* for this task, we programatically generate a perfectly compositional language, by assigning each object unique word. In other words, to describe a 'blue shaded triangle', we create a language where the output description would be "blue, triangle, shaded", in some arbitrary order. By 'unique symbol', we mean that no two properties are assigned the same word. The speaker and listener policies are parameterized using a 2-layer linear network (results were similar with added non-linearity and significantly worse with 1-layer linear networks) with 200 hidden units. During both supervised learning and self pely, the listener is trained to minimize the cross-entrops loss over property predictions.
Environment Image-Based Referential game with natural language (IBR) Our second game is the communication task introduced in Lee al. (2018). The speaker observes target image d', and must describe the image using a set words. The listener observes the target image along with D distractor images sampled uniformly at random from the training set (for us, D = 9), and the message Yd- from the speaker, and is rewarded for correctly selecting the target image. For this game, the target language L English- we obtain English image descriptions using caption data from MS COCO and Flickr30k. We set the vocabulary size [V| 100, and filter out any descriptions that contain more than 30% unknown tokens while keeping the maximum message length T 15.
Similar to Mordatch & Abbeel 2018 Sukhbaatar et al. 2016), we train our agents end-to-end with of backpropagation. Since the speaker sends discrete messages, we use the Frraigh--Thoouge version (Jang et al., 2017 Maddison et al.) 2017) to allow gradient flow to the speaker during self-play (lay)) The speaker's predictions are trained on the ground truth English captions mn* using the cross entropy loss Typk-supervised The listener is trained using the cross-entropy loss Jisn-supervised where the logits are the reciprocal of the mean squared error which was found to perform better than directly minimizing MSE loss in Lee al. (2018). The mean squared error is taken over the listener's image representation bian of the distractor (or target) image and the message representation given input. The loss functions are defined as:
Jspk-supervisedl (d")
log Popk (milmee,,d))
D+1
Thnn Sun-supervisea (m", d' D)
Σ
log (egssoftmn (eg(softmax1//Po..g
bps (d))²)
d=1
Our approach could equally be used train a speaker of language T,"; we leave this to future work.
5

---

Published as a conference paper at ICLR 2020
play
Supervised loarning
Total samples to 93% kest acc xamples in wed
sup2ap
fine-tune
sumples in weed
schod
fine-tune
9k
100
10
91
100
20
10kc
Pop-sup22s sup2ap
sched
Tost accuraty (%6
800
Pop S2P
SUP
Supervised Optimal
Supersteed
Trariing Vie idation
0.70
44.9
150
palidttonn
50-
Pop S2P Pop Perfect EmComm
0.67
5000
10000
12 setrphes
106 serpples
50 Samples
75
100
(a)
(b)
(c)
Figure 4: (a) Left: In the OR game, best performance (number of total samples required achieve 95% test accuracy, lower is better) for S2P is achieved when all of the samples are in the seed. 0 on the x-axis corresponds sp2sup and Optimal is the actual (minimum) number of samples required to solve this optimization problem (see Appenix B). Right: This is also the case in the IBR game, where performance measured by the generalization accuracy using 10k total training samples (higher is better). (b) Adding more samples to initial supervised learning in the IBR game improves agents' generalization I,". (c) Even when we learn the perfect distribution with emergent communication in the OR game, it still performs worse than P2P (using sup2sp S2P).
D+1
Jsett-play (d, , D) =
Σ
log(s log(softmaz) log(softmaxx////nnn/u baso (d))2)
where yo- is the concatenation of one-hot vectors y ST- TTr--ummelSSffttaaapee
We use the same architecture as described in Lee et al. (2018). The speaker and listener are parameterized by recurrent policies, both using an embedding layer of size 256 followed by a GRU
(Cho et al. 2014) of size 512. We provide further hyperparameter details in Table i the Appendix.
Do SUPERVISED LEARNING BEFORE SELF-PLAY
A central question in our work is how to combine supervised and self-play updates for effective pre-training of conversational agents. In this section, we study this question by conducting experi- ments with two schedules: training with emergent communication followed by supervised learning (sp2sup), and training with supervised learning followed by self-play (sup2sp). We also inter- polate between these two regimes by performing the rand and sched on 0 < [D] samples, followed by supervised fine-tuning on the remaining (DI n samples.
Our first finding is that it is best to use all of your samples for supervised learning before doing self-play. This can be seen in Figure when all of the samples are used first for supervised learning, the number of total samples required to solve the OR game drastically, and in the IBR game the accuracy for fixed number of samples is maximized 4Fig While this may seem to be common sense, it in fact runs counter to the prevailing wisdom some emergent communication literature. where languages are emerged from scratch with the ultimate goal of translating them to natural language.
To better understand why it is best to do supervised learning first, we now conduct set of targeted experiments using the environments from Section Results of our experiments suggest three main explanations:
(1) Emerging a language is hard. For many environments, with emergent communication it's often hard find an equilibrium where the agents meaningfully communicate. The difficulty of eemergent language discovery" has been well-known in emergent communication (Lowe et al.) (017). we will only briefly discuss it here. In short, to discover useful communication protocol agents

---

Published as a conference paper at ICLR 2020
Ground-truth language
sup2sp
sp2sup
Sellypay
Supervised learning
Train
Val
Seed data
Supervised
self-play
Self-play
supervised
Type id
Word id
Figure 5: Results from the OR game with property and 10 types. When the supervised updates performed first (supervised data available for words 0 - 3), then the self-play updates make sensible predictions for the unknown words 4 - 7. When the self-play updates are performed first, the subsequent supervised updates merely correct the predictions for words 1 4, without enforcing the constraint that each word should result in a separate type to solve the task.
have to coordinate repeatedly over time, which is difficult when agents are randomly initialized, partieularly in environments with sparse reward. Compounding the difficulty that, if neither agent communicates and both agents act optimally given their lack of knowledge, they converge to Nash equilibrium called babbling equilibrium (Farelll & Rabin 1996). This equilibrium must be overcome to learn useful communication protocol. In S2P, the initial language supervision can help overcome the discovery problem, as it provides an initial policy for how agents could usefully communicate (Lewis et al. 2017).
(2) Emergent languages are different than natural language. Even if one does find an equilib- rium where agents communicate and perform well on the task, the distribution of languages they find will usually be very different from natural language. This is problem because, if the languages obtained through self-play are sufficiently different from L*, they will not be helpful for learning. This is seen for the OR game in Figure 4a where 17 samples are required in the seed before S2P outperforms the supervised learning baseline. We speculate that this due to the different pressures exerted during the emergence of artificial languages and human languages.
Thankfully, we can learn languages closer to L' by simply adding more samples to our initial supervised learning phase. We show this in Figure 4b) where we train populations of 50 agents on the IBR game and use Pop-S2P to produce a single distilled agent. With both 1K and 10K initial supervised samples, the distill agent generalizes to agents in the validation set of their population. However, the distilled agent trained with 10000 samples performs significantly better when playing with expert agent speaking L*, indicating that the training agents from that population speak languages closer to I.".
(3) Starting with self-play violates constraints. Even if you have 'perfect emergent communica- tion' that learns distribution over languages under which L has high probability, current methods of supervised fine-tuning do not properly learn from this distribution. What if we had all the correct learning pressures, such that we emerged a distribution over languages L with structure identical to L" and then trained a Pop-S2P agent using this distribution? Surprisingly, we find that S2P with all of the samples in the seed performs better than even this optimistic case, in terms of providing useful information for training Pop-S2P agent. We conduct an experiment the OR game where we programmatically define distribution over compositional languages L., of which our target language/ L* is a sample. Each language L E L. has the same structure and are obtained by randomly permuting the mapping between the word IDs and the corresponding type IDs, along with the order of properties in an utterance. Next, we compare two distilled policies using 50 populations: one is distilled from S2P populations (trained with X samples), and the other is distilled from 'perfect emergent communication' and fine-tuned on X samples. As can be seen in Figure 4c we show that when we train Pop-S2P agent on 50 of these compositional populations, samples than regular Pop-S2P (trained on 50 S2P agents with all of the samples in the seed) to reach 95% test accuracy
still need 3X more
Here the value of X is the corresponding number of samples on the horizontal axis in Figure 4c
7

---

Published as a conference paper at ICLR 2020
To understand why this happens, conduct case study in even simpler setting: single-agent S2P in the OR game with p = 1, t = 10, V = 10. We find that agents trained via emergent com- munication consistently learn to solve this task. However, as shown in Figure when subsequently trained via supervised learning on 'D to learn I,", the learned language no longer coherent (it maps different words the same type) and doesn't solve the task. On the other hand, agents trained first with supervised learning are able to learn a language that both solves the task and is consistent with D.
Intuitively, what's happening is that the samples in D are also valid for solving the task, since we assume agents speaking L can solve the task. Thus, self-play after supervised learning simply 'fills in the gaps' for examples not in D Emergent languages that start with self self-play, on the other hand, contain input-output mappings that are inconsistent with I.", which must be un-learned during subsequent supervised learning.
In theory, the above issue could be resolved using Pop-S2P; the distilled agent could use the population of emergent languages to discover structural rules (e.g. discovering that the languages in the OR game Figure 44 are compositional), could use the samples from DD to refine posterior distribution over target languages that is consistent with these rules (e.g. learning the distribution of compositional languages consistent with D). Current approaches to supervised fine-tuning in language, though, do not do this (Lazaridou et al. 2017 Lewis et al. 2017p. An interesting direction for future work is examining how to apply Bayesian techniques to S2P.
EXPLORING VARIANTS OF S2P
6.1 POPULATION-BASED S2P
In this section, we aim show that (1) S2P outperforms the su- pervised learning baseline, and (2) Pop-S2P outperforms S2P. We conduct our experiments the more complex IBR game, since the
80
71.73.35.2 35.2
Validation accuracy
60
54.1
agents must communicate in English, and measure performance by
40.
45.
calculating the accuracy at different (fixed) numbers of samples. Our
37.5
baseline is then the performance of a supervised learner fixed
Supervised
number of samples.
S2P
Pop-52P
We show the results in Figure We first note that, when both 1k and 10k samples are used for supervised learning, S2P (sched) out- performs the supervised learning baseline. We can also that the population- -based approach outperforms single agent S2P (sched) by significant margin. We also compare our distillation method to an ensembling method that keeps all 50 populations at test time, and find that ensembling performs significantly better, although it much less efficient. This suggests that there is room to push distilled Pop-S2P S2P to even better performance.
S2P ensemble
1k samples
10k samples
Figure 6: S2P (sched) out- performs the supervised base- line in the IBR game, and is in turn outperformed by Pop- S2P.
6.2 EXAMINING S2P SCHEDULES
In this section, we aim to: (1) evaluate several S2P schedules empirically on the IBR game; and (2) attain a better understanding of S2P through quantitative experiments.
Parameter freezing improves S2P We show the results comparing different S2P schedules in complex game, the sup2sp S2P performs much worse than
Figure 7a We find that in this the other options. We also see that adding freezing slightly improves the performance on the target language (Figure X the Appendix also shows that it converges more quickly). We hypothesize that this is because it reduces the language drift that is experienced during each round of self-play updates (eee al. 2019). Overall, however, the difference between different S2P schedules is relatively small, and it's unclear if the same ordering will hold in different domain.
In practice, we find that self-play updates can undo some of the learning of D, which is why we apply an alternating schedule.
8

---

Published as a conference paper at ICLR 2020
sched S2P
60
60
Consistent with D
Supervised
Val accuracy (%6
spsuup
50-
sup2sp
validation on
40
rad
sched
frz
.30
sched_rand_ff
rand_rrnd frz
Ripervieei karaing
20
High
Salf play
reward
10
10
self-play
Internadionn
1000
5000
10000
50
100
C
ROP
Samples
Sclf-play %
(a)
(b)
(c)
Figure 7: (a) Comparing test performances of different S2P methods the IBR game. For each method, we picked the model that gave the best performance on Drat. (b) 2D visualization of S2P (sched) performance over the course of training. in terms of performance on L (vertical axis) and performance in self-play (horizontal axis). The zig-zag patterns indicates that most self-play updates result in short-term decrease in target language performance. (c) Visualization of the role of the supervised and self-play updates in sched S2P.
Self-play acts as a regularizer What is the role of self-play in S2P? We can start to decipher this by taking a closer look the sched S2P. We plot the training performance of this method in Figure 76 Interestingly, we notice from the zig-za pattern that the validation performance usually goes down after every set of self-play updates. However, the overall validation performance goes up after the next round of supervised updates. This is also reflected in the poor performance of the sup2sp S2P Figure
This phenomenon can be explained by framing self-play as form of regularization: alternating between supervised and self-play updates a way to satisfy the parallel constraints of consistent with the dataset D and 'performs well on the task'. We visualize this pictorially in Figure 7bb while a set self-play updates results in poor performance on D, eventually the learned language moves closer to satisfying both constraints.
7 DISCUSSION
In this work, investigated the research question of how combine supervised and self-play updates, with a focus on training agents to learn a language. However, this research question is not only important for language learning; it is also a important in equilibrium selection and learning social conventions (Lerer & Peysakhovich) 2019) in general games. For example, in robotics there may be trade-off between performing task well (moving an object your policy be interpretable by humans (so that they will not stumble over you). Examining how to combine supervised and self-play updates in these settings an exciting direction for future work.
certain place) and having
There are several axes of complexity not addressed in our environments and problem set-up. First, we consider only single-state environments, and agents don't 't have to make temporally extended decisions. Second, we do not consider pre-training on large text corpora that are separate from the desired task (Radford et al., 2019 Devlin et al. 2018). Third, we limit our exploration of self-play to the multi-agent setting, which not the case in works such as instruction following (Andreas & Klein| 2015). Introducing these elements may result in additional practical considerations for S2P learning, which we leave for future work. Our goal in this paper is not determine the best method of S2P in all these settings, but rather to inspire others to use the framing of 'supervised self- eelf-pla algorithms' make progress sample efficient human-in-the-loop language learning.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
We are very grateful to Angeliki Lazaridou, with whom discussions at ICML 2019 and her simulta- neous work (Lazridouu et al. 2020) shifted the direction of this work considerably. We also thank Jean Harb, Liam Fedus, Amy Zhang, Evgeny Naumov, Cinjon Resnick, Igor Mordatch, and others at MILA and Facebook Research for discussions related to the ideas this paper. Special thanks to Arthur Szlam and Kavya Srinet for discussing their ongoing work with us. RL is supported in part by a Vanier Scholarship.
REFERENCES
Jacob Andreas and Dan Klein. Alignment- baaseeen-bbasedd compositional semantics for instruction following. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1165-11 1174, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.11653/v1/D15-111 1138.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In International Conference on Learning Representations, 2018. /uRLattteee/oenneevveeeeettttttt forum?? id=Hk 6WhagRW
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encod- ing emergent communication. H. Wallach, H. Larochelle, A. Beygelzimer, F. dltextquotesingle Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32. pp. 6290-6300 Curran Associates, Inc., 2019. URL http://tppppppppp..iiite -e5S-anti-efffiiiee---eeee ng- eme gge.t -tt.er/ggge.cccommmmnc.c.ooooooo
paper//
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL
httpss://peerreview..ee. htt.://weeeooouummooooooooooooo.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-175 1734, Doha, Qatar, October 2014, Association for Computational Linguisties. Linguutssss doi: 10.3115/v 4- 1179. URL btttps::/www.aclweettt
2ottholoyy/D44--o
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional communication learning from raw visual input. In International Conference on Learning Representations, 2018. URL BRRLhttps://opeeree///
Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of Compositional Language with Deep Generational Transmission. arXiv: arXiiv/904.000 00000906777 [cs, stat], April 2019. arXiv: 1904.09067.
Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 11..1180100000.. 2018.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communica- tion in multi-modal, multi-step referential game. In International Conference Learning Representations, 2018. / wtw.neet//orrmm22d=1000/0//0
Joseph Farrell and Matthew Rabin. Cheap talk. Journal of Economic Perspectives, 10(3): 103-118: September 1996. doi: (0.1257/jep.1033. URL http://ww.a.wweb.or rrt
10

---

Published as a conference paper at ICLR 2020
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to Communicate with Deep Aulti-Agent Reinforcement Learning. D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2137-2145. Curran Associates, Inc., 2016.
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3691-3701, Hong Kong, China, November 2019. 1006653/1/D11-13300 URL https://
Association for Computational Linguistics. doi:
www.aclweb.org/aae httprr////atttooooooooooeee 1384
Serhii Havrylov and Ivan Titov. Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems 30, pp. 2159-2159. Curran Associates, Inc., 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel- Softmax. In International Conference on Learning Representations, 2017. URL https
//openreview.ret//F ww..et/e..............e..e..o. http:///ooormmmmmddde......
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Dj Strouse, Joel Z. Leibo, and Nando De Freitas. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning. In International Conference on Machine Learning, pp. 3040-3049, May 2019. URL http:
wttp:/ssww...ppss/////.. jaquesiquusssseee 19.html
Simon Kirby. Iterated learning and the evolution of language. Current Opinion in Neurobiology, pp. 7, 2014.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2966-2967, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.186/3/1/D11113321. anthologyyDD17-132
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-Agent Cooperation and the Emergence of (Natural) Language. In International Conference on Learning Representations, 2017. URL hhtps: httpouRRttttt////eeeeeeeeee hattps::/openzevieew..ct www.::/weeeetttt/e/e// id- Hk-HHkk86
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel imput. In International Confer- ence on Learning Representations, 2018. URL https: http::ttt///pppee /tttt::://ppnreevieeeeett/tte wtw.:eewwwwee/oooom.i..e.e..
HJGv12 AW
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agen, communication meets natural language: Synergies between functional and structural language learning. In Proceedings of
the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7663-7674, Online, July 2020. Association for Computational Linguistics. URL httph: http::://wwwwweeee wwtp//wwwwwcccccoott/oo.o web.orgg
anthelogyy2222.ac11mma1... m55.665
Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela.
Emergent translation in multi- agent communication. In International Conference on Learning Representations, 2018. URL
ht https://cpenrevvee. hwtp://wwwwwee/oorrreeeom H1vExax
Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 4376 4386, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DD
Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observationally augmented self-play. In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 107-114, ACM, 2019.
11

---

Published as a conference paper at ICLR 2020
David Lewis. Convention: philosophical study. Harvard University Press, 1969.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end- to-end learning of negotiation dialogues. In Proceedings of the 20/7 Conference on Empirical Methods in Natural Language Prucessing, pp. 2443-2453, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/11/D17-1222 URL https
www.accweeb.orggg
Fushan Li and Michael Bowling. Efse-of- Teaching and Language Structure from Emergent Commu- nication. H. Wallach, H. Larochelle, A. Beygelzimer, dhtextuuotesingle Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 15825-15835.
Curran Associates, Inc., 2019. URL http://ss//ppppr.ppppsscpppppppp hwtp://ppppr9/1-1eaepeee-- teaching and a structure from cww.eeeeett-oomumnaaaa.o..o
Michael Littman. Markov games as a framework for multi-agent reinforcement learning. In International Conference Machine Learning, volume 157, pp. 157-163, 1994.
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- Environments. In I. Guyon, U. V.
Agent Actor-Critic for Mixed Cooperative-Competiee Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6379-6390. Curran Associates, Inc., 2017.
watp:nt--tttttt-ttt-eeeee URL Bt/.tpoe//ppeee..lcc///////////p -atp:mmmm--------aaa-ttoo-------
m. 1llec--cooerratteeee-omeetttteee
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of
measuring emergent communication. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pp. 693-701. International Foundation for Autonomous Agents and Multiagent Systems, 2019. ISBN 978 1 1------6303------ |
//ww.faanassor//rroeeeeiing wwwp:nnggaaaaaaaammaaammm amaa /ttp///aaaa00//aaaaaa////53/////
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Re- laxation of Discrete Random Variables. In International Conference on Learning Representations, 2017.
JE5L5q1
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAA/ Conference on Artificial Intelligence, 2018. URL http::ttp//// /aaaa.////////////////
Mathijs Mul, Diane Bouchacourt, and Elia Bruni. Mastering emergent language: learning to guide in simulated navigation. arXiv:1908.05/33 [cs], August 2019. arXiv: 1908.05135.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. Language models are unsupervised multitask learners. 2019. URL
Limor Raviv and Inbal Arnon. Systematicity, but not compositionality: Examining the emergence of linguistic structure in children and adults using iterated learning. Cognition, 181:160-173.
December 2018. ISSN 0010-0277.
Cinjon Resnick*, Abhinav Gupta*, Jakob N. Foerster, Andrew M. Dai, and Kyunghyun Cho. Capacity, bandwidth, and compositionality in emergent language learning. ArXiv, abs/1910.1 1424, 2019.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In International Conference on Learning Representations, 2016. URL https
95.pff
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Individualized controlled continuous communication model for multiagent cooperative and competitive tasks. In International Con- ference on Learning Representations, 2019. URL http::ttp//// /tppp:://ppeeeeeeee.eeett////e/e formmm?
rre rye7kkCgK7
12

---

Published as a conference paper at ICLR 2020
Kenny Smith, Henry Brighton, and Simon Kirby. Complex Systems In Language Evolution: The Cultural Emergence Of Compositional Structure. Advances in Complex Systems (ACS), 6(04):
537-558, 2003. doi: 10. 10.1142/5021192255000000113 /tt//acsxx//v00y20000 0 0T0.222011220000000000021011
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In D. D. Lee, M. Sugiyama, U. V. Luxburg. Guyon, and R. Gameett (eds.), Advances in Neural Information Processing Systems 29, pp.
Curran Associates, 631///63888811 1earnn--errnn------ tiagent
Inc., 2016. URL. http: http::///aaprrranppsscccpapprrre commun watp:mun-tttttoottt------eeeeeee wtt.::otoo.o...--oooooooooooooo.
Olivier Tieleman, Angeliki Lazaridou, Shibl Mourad, Charles Blundell, and Doina Precup. Shaping representations through communication. 2018. URL [tl/atttpsr//eeeneeeeeeeeee/////
13

---

Published as a conference paper at ICLR 2020
A HYPERPARAMETERS
We provide hyperparameter details in Table 1
Hyperparameter
Values
Learning rate
Ie-2, Ie-3, 2e-3, 6e-3, le-4, 5e-4, 6e-4 Linear, Bilinear, Non-Linear 1, 5, 10, 20, 50, 100, 200, 500, 1000 200, 500, 1000
Model architecture
Number of encoders (perfect emcomm)
Hidden layer size (Linear)
Number of encoders (Pop-S2P)
20, 40, 50, 60, 80, 100
Number of distractors GRU hidden size
1,4, 9
256
Word embedding size
512
Image embedding size (from pretrained Resnet50)
2048
Batch size
1, 1, 1000
Random seeds
0, 2, 3,4
Optimizer
Adam, SGD 0.3.
Dropout
Gumbel relaxation temperature
Vocabulary size
100, 200, 500, 1000, 5000 12, 15, 20, 30, 50 0, 1, 30, 40, 50, 70
Max sentence length
in sched
in sched
0, 30, 40, 50
q in rand
0.75
r in sched_rand__ frz
0.5
Number of initial supervised steps (pretraining)
0, 1000, 2000, 3000, 5000
Table 1: Hyperparameters considered in S2P training.
B CALCULATION OF OPTIMAL SAMPLE COMPLEXITY IN OR GAME
Here we provide quick calculation for how quickly a human might learn new compositional language L in the OR game in as few examples as possible, which we use as a baseline in Figure 10 types, T 6 words sent per message (concatenated together). and IV 60 vocabulary size. If this language L is compositional, then each word in the vocabulary assigned to 1 type. Thus, we need learn 60 total assignments. this analysis we assume we can construct (i.e. hand-design) the samples seen by the human, and thus the final number should considered something like a lower bound.
4a
We assume OR game with p properties.
Since T = 6, we get information about 6 word<-type assignments for every sample. However, this information is entangled as we don't know which word corresponded which type. Thus, we (I) divide the problem up by first constructing 9 (word sequence, object) sample pairs where none of the object types overlap between each sample. With this information, we are able to narrow down the word<-type assignments into 10 groups of (that is, in each group have 6 words corresponding to 6 types, but we don't 't know which type belongs to which word). Note we don't need 10 samples as the last one can be inferred by exclusion.
(2) We then construct more samples where each type belongs to separate group. We can do this because t > p. Because each type belongs to a separate group, cross-referencing the words observed from samples in (1) and (2) uniquely defines each word<-type assignment. Note again we don't 't need 6 samples as the last one can be inferred by exclusion. This gives us total of 5 14 samples.
14

---

Published as a conference paper at ICLR 2020
C ADDITIONAL PLOTS
We show training curves for various S2P schedules.
Performance samples
samples
40-
50-
30-
20-
30-
Val 80C on dataset
10-
50
100
150
200
25.
300
50
100
150
200
250
300
samples
samples
40-
100
200
250
350
100
150
200
250
300
sp2sup
sup2sp
rand
sched
sched frz
ached_rand
Figure Training curves for various S2P methods in the IBR game described in
15


Reviews:
[{"review_id": "SkeTV81RKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the effect of ordering supervised learning and self-play on the resultant language learnt between agents. The topic is of high relevance to the ICLR community and makes several interesting insights useful to anyone learning control of a multi-agent system where communication amongst agents is applicable. I have several suggestions for improvements below, but all I believe are feasible to make within the time period of the rebuttal with the most necessary being:\n\n1) The naming of methods in Section 5 is not consistent with those introduced in section 3.3. For example, in the first paragraph ec2supervised is presumably sp2sup and sched is presumably sup2sp? Similarly, on page 7 (S2P and Pop-S2P) and Figure 4. Please revise and ensure consistency throughout.\n\n2) Figures 4b, 6 and 7 only present single values. Are these average values from repeated runs? If so please quantify variance.\n\n3) The conclusion in Section 7 at the bottom of page 8 that \"S2P performs much worse than the other options\" is contrary to previous results. Can the authors please comment on what features of the environment caused this difference?\n\n4) Appendix A includes details of hyperparameters, but some details remain unclear. Specifically, hyperparameter ranges swept over are shown but how were they then chosen from? Are they optimised for each environment and algorithm? What does the bold text in the table represent? If it is chosen values, why do only some parameters have chosen values? These are important details to enable reproduction of the paper.\n\nMinor Comments:\nIn Section 3.3, if all these methods are \"well known ways to combine self-play and supervised learning\" can all be supported by an (or preferably multiple) exemplar publications that used these method previously. Directly linking each to the previous work will further clarify the contribution this specific paper makes and help readers new to the area gain insight across the multiple papers this work builds upon.\n\nFigure 3b, colour is representative of performance. Is this mean accumulated reward? Please clarify to increase how informative this visualisation is, as currently it is unclear if yellow or blue is the desired value.\n\nPage 5, small typo \"introduced in the Lee et al. (2017)\" should be \"introduced in Lee et al. (2017)\".\n\nFigure 4b, the legend is blocking two bars and their corresponding value. It looks like moving to the bottom right may help, or placing above the plot.\n\nFigure 4 caption refers to a subfigure (d) that is not included.\n\nOn Page 6, the reference to babbling equilibrium should include a citation for interested readers to learn more about this well established concept.\n\nOn Page 7 there is a reference to Figure r4b, is this intended to be a reference to Figure 4b right?\n\nFigure 6 appears after Figure 7. Maintaining ordered numbering would be preferable.\n\nOn Page 9 it is noted some experimental results are in the Appendix but as there is a page and a half of space remaining before the 10 page limit, I would encourage to include all results in the main body of the paper.\n\nMultiple references do not list a publication venue (e.g. Evtimova et al., Lazaridou et al. 2018, Tieleman et al. 2018) or cite Arxiv versions when the work has been later published (e.g. Jacques et al. 2018 was published at ICML 2018). \n\nFigure 9 caption should state the environment.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgVjdDRFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n---\n\n(motivation)\nTo develop language speaking agents we can teach them to mimic human language\nor to solve tasks that require communication. The latter is efficient, but\nthe former enables interpretability. Thus we combine the two in an attempt\nto take advantage of both advantages. This paper studies a variety of ways to\ncombine these approaches to inform future work that needs to make this tradeoff.\n\n(approach)\nThe trade-off is studied using reference games between a speaker and a\nlistener. Goal oriented _self-play_ and human _supervision_ are considered two contraints one\ncan put on a network during learning. This work considers algorithms that vary\nwhen self-play and supervision are used (e.g., training with self-play then supervision,\nor supervision then self-play, or alternating back and forth between the two).\nAdditional variations freeze the speaker or distill an ensemble of agents into one agent.\n\n(experiments)\nA synthetic Object Reference game (OR) and a Image-Base Reference game (IBR) with real images are used for evaluation. Performance is accuracy at image/object guessing.\n1. (OR) Like previous work, this work finds that emergent languages are imperfect at supporting their goals and cannot be understood by agents that only understand a human language like English.\n2. (OR) Pre-training with supervision then fine-tuning with self-play is superior to pre-training with self-play then fine-tuning with supervision. This is presented as surprising from the perspective of language emergence literature, which is though of as pre-training with self-play.\n3. (IBR) Distilling an agent from an ensemble of 50 independently trained agents outperforms training single agents from scratch, but is still not as good as the whole ensemble.\n\nSelf-play vs supervision schedules:\n4. (IBR) Supervision (using image captions) followed by self-play performs much worse than all other approaches.\n5. (IBR) Alternating between supervision and self play (e.g., randomly choosing supervision or self-play every iteration) performs best.\n\n\n\nStrengths\n---\n\nThe curricula considered by this paper seem to have a sigificant impact on performance. These are new and could be important for future work on language learning, which may have considered the sup2sp setting from figure 7a without considering the sched setting.\n\nThe diversity of experiments provided and the analysis help the reader get a better sense for how emergent communication models work.\n\nIt's nice to see experiments on both a toy setting and a setting with realistic images.\n\nFuture directions suggested throughout the paper are interesting.\n\n\nWeaknesses\n---\n\n\n* The 3rd point of section 5 is presented as a major conclusion of this paper, but it is not very surprising and I don't see how it's very useful. The perspective of language emergence literature is presented a bit strangely. The self-play to supervision baseline seems to be presented as an approach from the language emergence literature. I don't think this is what any of that literature promotes exactly, though it is close. Generally, I (and likely others) don't think it's too surprising that trying to fine-tune a self-play model with language supervision data doesn't work very well, for the same reasons cited in this paper (point 3 of section 5). I think the general strategy when trying to gain practical benefits from self-play pre-training is a translation approach where the learned language is translated into a known language like English rather than trying to directly align it to English as does the supervision approach in this paper. This particular baseline would be more useful if the paper considered learning some kind of translation layer on top of the self-play pre-trained model.\n\n* How significant are the performance differences in figure 7a, especially those between the frozen and non-frozen models? Is the frozen model really better or this performance difference just due to noise?\n\n* I'm somewhat skeptical that these trends will generalize to other tasks/models. The main goal of this paper is to inform future work. That makes it even more important than normal that the trends identified here are likely to generalize well. Are these trends likely to generalize well? Does the paper address when these trends are expected to hold anywhere?\n\n\nMinor Presentation Weaknesses:\n\n* Figure 4: I think the sub-figures are mis-labeled in the caption.\n\n* In the related work I'm not sure the concept of generations is right. I think it should refer to different languages of different agents across time rather than different languages of the same agent across time.\n\n\nMissing details / clarification questions:\n\n* What exactly does Figure 4c compare? Are both methods distilled from ensembles or is the blue line normal S2P while the other is distilled from an ensemble of compositional languages? It's not clear since point (3) in section 5 refers to the S2P result (not Pop-S2P) in that plot. I'm also assuming that PB-S2P means the same thing as Pop-S2P, but that's not made clear anywhere. Does PB stand for Population Based?\n\n* In the rand setting how is convergence defined? Do both objectives need to converge or just one?\n\n* In the sched_rand_frz setting what is r?\n\n* In the IBR how are the distractor images picked?\n\n\nSuggestions:\n\n* Can't both self-play and supervision be used at the same time (just use a weighted combination of the two objectives)? I don't think the paper ever did this but it seems like a very useful variation to consider.\n\n\nPreliminary Evaluation\n---\n\nClarity: The writing is fairly clear, though some details are lacking.\nSignificance: This work could help inspire some future models in the language emergence literature.\nQuality: Experiments are aligned with the paper's goals and support its conclusions.\nOriginality: The distillation approach and curricula are novel.\n\nOverall the work could prove to be an interesting and useful reference point inside the language emergence literature so I recommend it for acceptance.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxoajDCtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigated how two conflicting learning objectives; supervised and self-play updates could be combined with a focus on visual-grounded language tasks. With a different set of their combinations, the authors empirically found that alternating two learning updates may result in the best equilibrium state; consistency with samples in the supervised dataset and optimal state with high rewards in the task environment.\n\nThe paper is very well-written, and I really enjoyed reading it overall. There are some typos, presentation issues, and minor format issues (e.g., wrong naming) though. I do like this kind of simple but insightful result with enough empirical observations and discussions. Even though there is not that novel method proposed, the overall message found from the experiments, their interpretation by the authors, and meaningful comparisons to the past works in emergent communication are fair enough to learn high scientific values from it. The design of the experiment is again very simple (e.g., changing the size of data, switching two setups in different ways) but clear to understand. This work is a good example of how well-designed hypotheses and their empirical validation could contribute to the field. I also appreciate the large spectrum of literature surveys including from the recent advances (Lewis et al., 2017, Lee et al., 17) to the past literature in emergent communications such as Littman (1994) and (Farrell & Rabin, 1996). \n\nOne of my concerns is the lack of applications, especially on the tasks using more natural language. The two tasks; OR and IBR, seem to be very limited settings to evaluate how self-play operates with data supervision. As pointed out by the authors, supervision from the training data itself may include most of the unexplored cases of the task, leading a less chance to learn policies from the high rewards. I think more realistic tasks using natural language need to be considered: negotiation (e.g., Lewi’s task, “Decoupling strategy and generation in negotiation dialogues”), recommendation (e.g., “Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue”), and more. I agree with the point made by the authors that this work mainly focuses on investigation rather than exploitation. But, then it would be adding another emergent task where the self-play can learn many more policies than one in the supervised dataset. \n\nAdding to the point, I was expecting to see non-task related metrics to measure the effectiveness of their appropriate combinations. For example, it would be better to add language-side metrics (e.g., perplexity, fluency, consistency) to measure how language degeneration varies by the different combinations. This issue is not addressed in the paper, and I guess this is mainly because of the limited usage of language in the two limited tasks. If the paper is only focusing on emergent language which is related to specific tasks, it would be better to tone-down a little bit and state the major difference of it with natural language. \n\nThe population-based S2P seems to be a bit incremental and unrelated to the main theme of the paper. To me, the motivation of adding POP into S2P based on the policy variability is somewhat different from the original claim about the combination of supervised and selfplay. Also, the improvements on IBR in Figure 7 are incremental, making the major claim of this work little divergent. \n\nIn terms of presentation, if you like to show how performance changes over the different sizes of data, it would be better to show it by graphs over different variations instead of the bar charts only with 10k and 50k sizes. In addition, the figures and captions need to be improved for better interpretation. I think they are written in a hurry or changed a lot in the last minutes. Please see some minor formatting issues below. \n\n\nMinor comments:\nDuplicate reference of (Lewis et al., 2017)\nSome names defined in Section 3.3 and Section 5 are not exactly matched.\nFigures and fonts in Figures 4 and 7 are a little difficult to understand. Especially, I can’t understand the two upper figures in Figure 4a\nCaptions in Figure 4 are not matched with the sub-figures. \nFigure r4b -> Figure 4b\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how two means of learning natural language - supervised learning from labeled data and reward-maximizing self-play - can be combined. The paper empirically investigates this question, showing in two grounded visual language games that supervision followed by self-play works better than the reverse. 

The reviewers found this paper interesting and well executed, though not especially novel. The last is a reasonable criticism but in this case I think a little beside the point. In any case, since all the reviewers are in agreement I recommend acceptance.
Decision: Accept (Poster)