Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an analysis of emergent communication behavior at scale and discusses training techniques to reduce instability during RL optimization.
main_review: Strengths:
 - Emergent communication has classically focused on toy tasks, so a scaling paper is a much-needed contribution
 - Overall the paper is quite clear and well-written, with clear notation
 - Analysis of KL regularization in the emergent communication domain is interesting and important
 - Analysis of harder training tasks is also compelling; I wonder if you also add "hard negatives" to make the task more difficult
 - ETL is a useful proposed metric, albeit partially based on existing work
 - Best single-pair seed is indeed the correct metric for population-based work, which is missed in prior work (Cogswell, et al.)
 - Analysis of population size is interesting and a compelling rebuttal to previous work

Weaknesses:
 - Not clear to me that a lack of correlation between TopSim and |C| implies that TopSim is not a useful metric for generalization; although |C| does correlate with generalization, I think the usual argument is that TopSim correlates with compositional generalization to specific types of out-of-domain examples (i.e., those which have combinations of previously seen attributes of training examples). The finding is still interesting, but it might be worth tempering the claim here a bit.
 - Given that the paper argues for scaling up a field, more information about compute usage and hardware would be appreciated
summary_of_the_review: I strongly recommend accepting this paper, given the over-emphasis of toy tasks in the emergent communication community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper argues for scaling up experiments in emergent communication to better understand the evolution of human language and to construct more efficient representations. To achieve this, the authors demonstrate scaling up along dataset, task complexity, and population size in the context of a multi-agent Lewis game, evaluating the benefits and drawbacks of each using generalization, robustness to input noise, and ETL. They find that (1) KL regularization mitigates unstable optimization when scaling up; (2) increasing task complexity improves generalization and better separates performance of different algorithms; (3) topsim and generalization are uncorrelated, necessitating alternative metrics of compositionality at scale; (4) while scaling up population in itself is not obviously beneficial, imitation learning and voting in larger populations produce more "robust, productive, and easy-to-learn" languages. Overall, the paper makes a strong case to scale up emergent communication experiments supported by empirical contributions.
main_review: **Pros**

1. Thorough and balanced discussion of benefits/drawbacks among the scaling dimensions.
2. Research is very well-situated in the literature.
3. Experiments are well-motivated, novel to my knowledge, and support the authors' case to scale up emergent communication experiments.
4. Reads very clearly, especially the model and experiment implementation. The technical implementation details in Section 2 are precise.

**Cons**

1. Questions about statistical rigor, especially concerning few datapoints (1, 3 in additional comments)

**Questions and Additional Comments**

(1) The main paper considers |C| = 16, 64, 256, 1024 (the appendix considers also |C| = 4096) and N = 1, 10, 50. It is hard to justify conclusions such as "increased task complexity systematically improves generalization and ETL" and "there is no consistent pattern between TopSim and |C| with a non-significant Spearman correlation" with 4 datapoints. Were intermediate datapoints tested but not reported?

(2) In Sections 3.1 and 3.2 (Fig. 2, Table 1), several metrics such as accuracy, generalization, and ETL are shown as functions of |C| (task complexity). However, |C| grows exponentially while the performance improvement in these metrics grows roughly linearly. In Table 2, going from 10 to 50 pairs requires roughly 5x more training hours (max is 20 days), whereas their mean performance may not be significantly different. Do the authors think the cost of scaling up is worth the diminishing returns?

(3) In Section 3.2 (TopSim may fall short with natural images): "As the latter correlates... we expect to find a correlation between TopSim and |C| by transitivity." The authors imply that |C| ~ generalization and TopSim !~ |C| (Fig 3) ---> TopSim !~ generalization. In general, transitivity of correlation is not guaranteed. (For example, take X and Y are independent and X + Y = Z.) I would avoid this argument and replace it by directly computing the correlation between TopSim and generalization accuracy. 

In addition, "...TopSim and |C| with a ... between TopSim and generalization for ImageNet and CelebA respectively". I found this sentence hard to parse-- are the reported correlations between TopSim and |C| or TopSim and generalization?

(4) It would improve the focus of the paper to state the work's relationship to the cognitive science and the RL sides of emergent communication more explicitly. After reading the paper, it is clear that the authors focus on the robotics/modelling aspect of emergent communication rather than the evolutionary origins of human language. However, the authors begin by discussing both [cognitive science/evolutionary linguistics] and RL as reasons to scale up. It's not clear that starting with complex tasks is the best model of human language emergence. Work in cognitive science theorizes that from both an evolutionary and a first language acquisition perspective, complex language develops from simple communication such as pointing and pantomiming in simple settings rather than from immediately solving difficult tasks (see Barrett and Skyrms 2017, Deacon 1998, Stern 1974). Given the authors partially motivated their work with researching the origins of human language, it would be nice to see more discussion on how their experiments mirror (or contradict) hypothesized settings of language development. For example, in Section 3.3's voting, this is a technique that improves model performance but is likely not how language originated in humans. Alternatively, this can be avoided if the authors state that their work just focuses on the RL side.
summary_of_the_review: **Recommendation**

I recommend to accept this paper based on its valuable experimental results, which demonstrate that scaling up emergent communication can find more efficient and generalizable representations. Supported by empirical contributions, the paper steps beyond toy experiments that are common in emergent communication literature and towards increasingly complex and realistic settings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to scale up the usual Lewis game language emergence setup. Instead of simple synthetic objects, the authors consider scaling up to image datasets such as ImageNet and CelebA. Another aspect they scale up is the task difficulty, by increasing the number of candidates the listener has to choose the correct answer from. Third, they also attempt scaling up the population size. To deal with the instability arising from a large number of candidates, the authors use established stabilization techniques such as minimization of KL divergence between the online policy and an exponentially moving average target policy. The paper also introduces a new metric called Ease and transfer learning (ETL) to measure how well an emergent language transfers to a novel task as an alternative to topographic similarity. Finally, the authors explore some techniques such as imitation learning and inference-time voting to try to leverage a population of agents.
main_review: Strengths:
- This is a timely paper for a problem worth solving.
- I think measuring how well an emergent language transfers to distinct tasks is a very useful metric to determine the utility of an emergent language.
- I appreciate the "best 1 pair" baseline against the population results, it is a very sensible baseline to compare population-based methods against.
- Leveraging population for language emergence is difficult and the presented methods help scale up here while providing some benefits in generalization and robustness.

Weaknesses:
- Although ETL is a useful metric to understand if a language can transfer to a new task, it is useful to measure and understand the structure in the emergent language itself. As indicated in the paper, topographic similarity is probably not the best metric; what about something like TRE [1]? In simple Lewis games, a high topo sim gives a strong clue about what the structure of the language is, but in this case, I have no idea.
- The authors construct test sets that have overlapping identities for CelebA. While this makes sense for the ETL experiment they set up, in general, this is ensuring that the train and test splits are more in-distribution, making the evaluation task much easier. One of the most important questions w.r.t. emergent languages is if they can generalize systematically. To this end, I would prefer a systematic split where one can study how the emergent language performs in a systematically out-of-distribution setup. An example of something like this would be including CelebA samples that contain a larger proportion of certain types of correlations (e.g. bald with beard) in the train set but have uncorrelated attributes for distinct identities in the test set.

Questions:
- You observe that scaling up the task difficulty entails unstable RL optimization. Did you try other ways to backpropagate the gradients to the speaker such as Gumbel straight-through estimation?
- What is the significance of "ease" in "Ease and transfer learning"? Do you measure ease of learning, e.g. through acquisition speed or something else?

Suggestions:
- Section 1: Typo in the second to last line of Introduction: "emergent" -> "emergence".
- I would suggest adding a period (.) at the end of paragraph titles (between the title and the paragraph content).

References:  
[1] Andreas, J., 2019. Measuring compositionality in representation learning. arXiv preprint arXiv:1902.07181.
summary_of_the_review: The commonly used Lewis game setups with synthetic data for language emergence are indeed quite limited for real-world applicability. Thus, scaling up the game for the emergence of communication protocols is an important problem. I think this paper is a great contribution, but I would appreciate some more understanding of the structure within the emergent languages. Alternatively (or additionally), demonstrating an ability to systematically generalize can also be very valuable.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **After rebuttal**: I am keeping my score

**Before rebuttal**: The paper conducts a large-scale study on RL-based emergent communication. The scale is extended in three dimensions: task complexity, data naturalness, and population size. Scaling up gives new findings that were not observed in previous (small-scale) work. Important findings are (1) evaluating with a large number of candidates is essential for observing the advantage of training with a large number of candidates (2) when experimenting with real images, TopSim, a widely used metric for measuring compositional, no longer correlates with generalization (3) larger population does not necessarily enhance generalization; more clever leverage of the population is needed. 
main_review: The paper is very well-written. The overall motivation of the work is very clear, and the motivation of each experiments are supported by observations from previous work and theories in cognitive science. The findings are novel and interesting, bringing out the need for conducting large-scale experiments. A minor drawback is that the work largely presents quantitative results and lacks qualitative analyses. At a large scale, are there anything particular about the learned communication that makes it generalize better?

(I didn't aim to write such a short review, but this is a standard exploratory paper that scales up a well-established setting. The experiment setups are well-designed and I don't find any ambiguous point that needs clarification. I am personally not strongly interested in this fully RL communication-learning direction, but researchers that have good reasons for pursuing this topic may find the results very interesting.)
summary_of_the_review: Overall, I find this work thorough and well-motivated. The results would have significant impact in this area. I recommend acceptance. However, I do not work extensively this area and may not follow the latest work.
review_title: null

Meta-review: This manuscript expands the range of recent work in reinforcement learning for language games to much larger and more realistic datasets. A timely and relevant contribution and one that is well evaluated. Further work in stabilizing RL approaches for such large-scale problems is likely to have other far-reaching consequences. Reviewers were unanimous that this is a strong submission after the author discussion period.
Decision: Accept (Spotlight)