Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper investigated how two conflicting learning objectives; supervised and self-play updates could be combined with a focus on visual-grounded language tasks. With a different set of their combinations, the authors empirically found that alternating two learning updates may result in the best equilibrium state; consistency with samples in the supervised dataset and optimal state with high rewards in the task environment.

The paper is very well-written, and I really enjoyed reading it overall. There are some typos, presentation issues, and minor format issues (e.g., wrong naming) though. I do like this kind of simple but insightful result with enough empirical observations and discussions. Even though there is not that novel method proposed, the overall message found from the experiments, their interpretation by the authors, and meaningful comparisons to the past works in emergent communication are fair enough to learn high scientific values from it. The design of the experiment is again very simple (e.g., changing the size of data, switching two setups in different ways) but clear to understand. This work is a good example of how well-designed hypotheses and their empirical validation could contribute to the field. I also appreciate the large spectrum of literature surveys including from the recent advances (Lewis et al., 2017, Lee et al., 17) to the past literature in emergent communications such as Littman (1994) and (Farrell & Rabin, 1996). 

One of my concerns is the lack of applications, especially on the tasks using more natural language. The two tasks; OR and IBR, seem to be very limited settings to evaluate how self-play operates with data supervision. As pointed out by the authors, supervision from the training data itself may include most of the unexplored cases of the task, leading a less chance to learn policies from the high rewards. I think more realistic tasks using natural language need to be considered: negotiation (e.g., Lewi’s task, “Decoupling strategy and generation in negotiation dialogues”), recommendation (e.g., “Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue”), and more. I agree with the point made by the authors that this work mainly focuses on investigation rather than exploitation. But, then it would be adding another emergent task where the self-play can learn many more policies than one in the supervised dataset. 

Adding to the point, I was expecting to see non-task related metrics to measure the effectiveness of their appropriate combinations. For example, it would be better to add language-side metrics (e.g., perplexity, fluency, consistency) to measure how language degeneration varies by the different combinations. This issue is not addressed in the paper, and I guess this is mainly because of the limited usage of language in the two limited tasks. If the paper is only focusing on emergent language which is related to specific tasks, it would be better to tone-down a little bit and state the major difference of it with natural language. 

The population-based S2P seems to be a bit incremental and unrelated to the main theme of the paper. To me, the motivation of adding POP into S2P based on the policy variability is somewhat different from the original claim about the combination of supervised and selfplay. Also, the improvements on IBR in Figure 7 are incremental, making the major claim of this work little divergent. 

In terms of presentation, if you like to show how performance changes over the different sizes of data, it would be better to show it by graphs over different variations instead of the bar charts only with 10k and 50k sizes. In addition, the figures and captions need to be improved for better interpretation. I think they are written in a hurry or changed a lot in the last minutes. Please see some minor formatting issues below. 


Minor comments:
Duplicate reference of (Lewis et al., 2017)
Some names defined in Section 3.3 and Section 5 are not exactly matched.
Figures and fonts in Figures 4 and 7 are a little difficult to understand. Especially, I can’t understand the two upper figures in Figure 4a
Captions in Figure 4 are not matched with the sub-figures. 
Figure r4b -> Figure 4b

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: 
Summary
---

(motivation)
To develop language speaking agents we can teach them to mimic human language
or to solve tasks that require communication. The latter is efficient, but
the former enables interpretability. Thus we combine the two in an attempt
to take advantage of both advantages. This paper studies a variety of ways to
combine these approaches to inform future work that needs to make this tradeoff.

(approach)
The trade-off is studied using reference games between a speaker and a
listener. Goal oriented _self-play_ and human _supervision_ are considered two contraints one
can put on a network during learning. This work considers algorithms that vary
when self-play and supervision are used (e.g., training with self-play then supervision,
or supervision then self-play, or alternating back and forth between the two).
Additional variations freeze the speaker or distill an ensemble of agents into one agent.

(experiments)
A synthetic Object Reference game (OR) and a Image-Base Reference game (IBR) with real images are used for evaluation. Performance is accuracy at image/object guessing.
1. (OR) Like previous work, this work finds that emergent languages are imperfect at supporting their goals and cannot be understood by agents that only understand a human language like English.
2. (OR) Pre-training with supervision then fine-tuning with self-play is superior to pre-training with self-play then fine-tuning with supervision. This is presented as surprising from the perspective of language emergence literature, which is though of as pre-training with self-play.
3. (IBR) Distilling an agent from an ensemble of 50 independently trained agents outperforms training single agents from scratch, but is still not as good as the whole ensemble.

Self-play vs supervision schedules:
4. (IBR) Supervision (using image captions) followed by self-play performs much worse than all other approaches.
5. (IBR) Alternating between supervision and self play (e.g., randomly choosing supervision or self-play every iteration) performs best.



Strengths
---

The curricula considered by this paper seem to have a sigificant impact on performance. These are new and could be important for future work on language learning, which may have considered the sup2sp setting from figure 7a without considering the sched setting.

The diversity of experiments provided and the analysis help the reader get a better sense for how emergent communication models work.

It's nice to see experiments on both a toy setting and a setting with realistic images.

Future directions suggested throughout the paper are interesting.


Weaknesses
---


* The 3rd point of section 5 is presented as a major conclusion of this paper, but it is not very surprising and I don't see how it's very useful. The perspective of language emergence literature is presented a bit strangely. The self-play to supervision baseline seems to be presented as an approach from the language emergence literature. I don't think this is what any of that literature promotes exactly, though it is close. Generally, I (and likely others) don't think it's too surprising that trying to fine-tune a self-play model with language supervision data doesn't work very well, for the same reasons cited in this paper (point 3 of section 5). I think the general strategy when trying to gain practical benefits from self-play pre-training is a translation approach where the learned language is translated into a known language like English rather than trying to directly align it to English as does the supervision approach in this paper. This particular baseline would be more useful if the paper considered learning some kind of translation layer on top of the self-play pre-trained model.

* How significant are the performance differences in figure 7a, especially those between the frozen and non-frozen models? Is the frozen model really better or this performance difference just due to noise?

* I'm somewhat skeptical that these trends will generalize to other tasks/models. The main goal of this paper is to inform future work. That makes it even more important than normal that the trends identified here are likely to generalize well. Are these trends likely to generalize well? Does the paper address when these trends are expected to hold anywhere?


Minor Presentation Weaknesses:

* Figure 4: I think the sub-figures are mis-labeled in the caption.

* In the related work I'm not sure the concept of generations is right. I think it should refer to different languages of different agents across time rather than different languages of the same agent across time.


Missing details / clarification questions:

* What exactly does Figure 4c compare? Are both methods distilled from ensembles or is the blue line normal S2P while the other is distilled from an ensemble of compositional languages? It's not clear since point (3) in section 5 refers to the S2P result (not Pop-S2P) in that plot. I'm also assuming that PB-S2P means the same thing as Pop-S2P, but that's not made clear anywhere. Does PB stand for Population Based?

* In the rand setting how is convergence defined? Do both objectives need to converge or just one?

* In the sched_rand_frz setting what is r?

* In the IBR how are the distractor images picked?


Suggestions:

* Can't both self-play and supervision be used at the same time (just use a weighted combination of the two objectives)? I don't think the paper ever did this but it seems like a very useful variation to consider.


Preliminary Evaluation
---

Clarity: The writing is fairly clear, though some details are lacking.
Significance: This work could help inspire some future models in the language emergence literature.
Quality: Experiments are aligned with the paper's goals and support its conclusions.
Originality: The distillation approach and curricula are novel.

Overall the work could prove to be an interesting and useful reference point inside the language emergence literature so I recommend it for acceptance.


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper explores the effect of ordering supervised learning and self-play on the resultant language learnt between agents. The topic is of high relevance to the ICLR community and makes several interesting insights useful to anyone learning control of a multi-agent system where communication amongst agents is applicable. I have several suggestions for improvements below, but all I believe are feasible to make within the time period of the rebuttal with the most necessary being:

1) The naming of methods in Section 5 is not consistent with those introduced in section 3.3. For example, in the first paragraph ec2supervised is presumably sp2sup and sched is presumably sup2sp? Similarly, on page 7 (S2P and Pop-S2P) and Figure 4. Please revise and ensure consistency throughout.

2) Figures 4b, 6 and 7 only present single values. Are these average values from repeated runs? If so please quantify variance.

3) The conclusion in Section 7 at the bottom of page 8 that "S2P performs much worse than the other options" is contrary to previous results. Can the authors please comment on what features of the environment caused this difference?

4) Appendix A includes details of hyperparameters, but some details remain unclear. Specifically, hyperparameter ranges swept over are shown but how were they then chosen from? Are they optimised for each environment and algorithm? What does the bold text in the table represent? If it is chosen values, why do only some parameters have chosen values? These are important details to enable reproduction of the paper.

Minor Comments:
In Section 3.3, if all these methods are "well known ways to combine self-play and supervised learning" can all be supported by an (or preferably multiple) exemplar publications that used these method previously. Directly linking each to the previous work will further clarify the contribution this specific paper makes and help readers new to the area gain insight across the multiple papers this work builds upon.

Figure 3b, colour is representative of performance. Is this mean accumulated reward? Please clarify to increase how informative this visualisation is, as currently it is unclear if yellow or blue is the desired value.

Page 5, small typo "introduced in the Lee et al. (2017)" should be "introduced in Lee et al. (2017)".

Figure 4b, the legend is blocking two bars and their corresponding value. It looks like moving to the bottom right may help, or placing above the plot.

Figure 4 caption refers to a subfigure (d) that is not included.

On Page 6, the reference to babbling equilibrium should include a citation for interested readers to learn more about this well established concept.

On Page 7 there is a reference to Figure r4b, is this intended to be a reference to Figure 4b right?

Figure 6 appears after Figure 7. Maintaining ordered numbering would be preferable.

On Page 9 it is noted some experimental results are in the Appendix but as there is a page and a half of space remaining before the 10 page limit, I would encourage to include all results in the main body of the paper.

Multiple references do not list a publication venue (e.g. Evtimova et al., Lazaridou et al. 2018, Tieleman et al. 2018) or cite Arxiv versions when the work has been later published (e.g. Jacques et al. 2018 was published at ICML 2018). 

Figure 9 caption should state the environment.
review_title: Official Blind Review #3

Meta-review: This paper investigates how two means of learning natural language - supervised learning from labeled data and reward-maximizing self-play - can be combined. The paper empirically investigates this question, showing in two grounded visual language games that supervision followed by self-play works better than the reverse. 

The reviewers found this paper interesting and well executed, though not especially novel. The last is a reasonable criticism but in this case I think a little beside the point. In any case, since all the reviewers are in agreement I recommend acceptance.
Decision: Accept (Poster)