Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the expressivity, complexity and unpredictability of emergent languages in referential games. The authors demonstrate that the expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context that the languages are used in. They also introduce a contrastive loss based training method for referential games that alleviates the collapse of message types often seen when using other standard training methods.
main_review: I like the overall idea of the paper on exploring the relationships between expressivity, complexity and unpredictability in emergent languages. 

**Strengths:**

- The paper properly motivates with examples, the study of expressivity, complexity and unpredictability in the framework of deep learning based language games as a way to gain further insights into natural languages which I find really interesting.
- The paper introduces a novel measure of expressivity for emergent languages that is defined through its generalization performance across different tasks.
- The paper studies the usage of contrastive loss in referential games and this loss helps alleviate the collapse of message types in language games which is interesting and novel.
- The experiments are extensive which is great to see.
- Further discussions in the appendix is also nice. Particularly, those around mutual information.

**Weaknesses:**

I have included actions that can be taken to strengthen this paper. If these actions are addressed, I am willing to increase my score.

- **Section 3.1/4.1:**
    - **Complexity/unpredictability:** I don't fully understand what equations 2, 3 mean, both are of the form $f(x_t) = E_{x_t}[...]$. So, $x_t$ is both an argument to the function and the expectation. How is the expectation over $x_t$?
        - **Action:** Clarify equations 2 and 3.
    - **Complexity/unpredictability:** I am a little confused at how the closed form solutions for equations 2 and 3 were arrived at. Equation 2 for example, might be correct only if it is sampled with replacement, not without. Both are of the following form: the probability that a set A contains an object from set B where A, B are sampled from X without replacement. The solution to this is 1 - (|X| - |B|  \choose |A|) / (|X| \choose |A|).
        - **Action:** Correct or clarify solutions to equations 2 and 3.
- **Section 3.2:**
    - The paper does not fully motivate why Definition 3.1 is a good quantifiable metric for studying expressivity in emergent languages. How does generalization that is studied in emergent languages like [2] relate to this particular definition of expressivity?
        - **Action:** Compare expressivity with the notion of generalization studied in past works.
- **Section 4.1:**
    - Predictions in section 4.1 depends on the expressions being accurate in Section 3.1. It is also not clear to me how the definitions of unpredictability and complexity leads to predictions over expressivity.
        - **Action:** Clarify how we can expect these predictions to be true from the definitions provided.
- **Section 5.3:**
    - Looking at Figure 4 and the corresponding Table 4 (in the appendix), it does not seem that referY is statistically worse than referX for Y > X as claimed in this section.
        - **Action:** Including these comparisons with a statistical significance test would be needed to draw these conclusions.

**Comments:**

- **Section 2.2:**
    - What are source and target games? They are defined in a later section and pointing the readers to section 4.2 would make it easier to follow.
    - Section 4.2 is a little difficult to follow. This section defines ReferX and ReferX/f which is used in the rest of the paper. The definition for ReferX/f is particular is not very clear. The following statement is also not very clear: "since language games with fixed context are not good simulation of human communication , we keep only the ones having varying batches from $G_s$ in $G_t$."
        - **Action:** Include further discussions into the definitions for ReferX, ReferX/f, $G_s$ and $G_t$.

[1] Kirby, S., Tamariz, M., Cornish, H., & Smith, K. (2015). Compression and communication in the cultural evolution of linguistic structure. *Cognition*, *141*, 87-102.

[2] Chaabouni, R., Kharitonov, E., Bouchacourt, D., Dupoux, E., & Baroni, M. (2020). Compositionality and generalization in emergent languages. *arXiv preprint arXiv:2004.09124*.
summary_of_the_review: The paper properly motivates the study of expressivity, complexity and unpredictability in the framework of deep learning based language games as a way to gain further insights into natural languages. The paper also studies the usage of contrastive loss in referential games and this loss helps alleviate the collapse of message types in language games. The key contributions of the paper is predicated on the definition and closed form expressions for unpredictability and complexity. However, I do not believe that those expressions are sound. Although, there are extensive experiments, some conclusions are not well supported. Therefore, as it stands, I would not recommend the paper to be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the properties of emergent languages in DL-based language games. In particular, they look at referential games where speakers emit messages and listeners need to identify the target object observed by the speaker from some set of candidate objects.  

They define the expressivity of an emergent language as the amount of discriminatory information required to encode the inputs so that a listener can correctly decode them. In this setting, complexity refers to the similarity between different objects in a given context. In a context with higher complexity, more similar objects will be present, and more discriminatory information will have to be encoded so that a listener is capable of identifying the correct target. In their definition, the notion of unpredictability can be thought of as how stable the information necessary for encoding is across different trials.  

Their primary contribution is to support the hypothesis that the expressivity of emergent language is determined by (and a trade-off between) the complexity and unpredictability of context in language games. The authors introduce a new measure to evaluate expressivity based on partial ordering between languages in terms of their generalization across tasks. They argue that mutual information is not the most appropriate measure to evaluate the expressivity of languages. They propose a contrastive loss which they show helps mitigate the issue of the collapse of message types. 
main_review: In my opinion, the biggest strength of the paper is the empirical evaluation conducted to support the claims. The experimental design and setup look sound. The definitions and different notions introduced in the paper seem natural and reasonable. At the same time, given the notions of complexity and unpredictability, a number of conclusions seem very natural based on intuition.
 
The paper is very well written and the arguments are backed up with reasonable experiments.  

The proposed metric seems to work well in the given setting. Isn't the metric is dependent on the transfer learning abilities of the model to a certain extent? Will the partial ordering be consistent across different types of models?  

The idea of applying contrastive loss is not particularly novel. It seems like to address the issue of space efficiency, they simply confined the previous method to batches rather than the entire set. 
summary_of_the_review: Overall, it seems like a strong paper with a clear hypothesis and brings in a novel perspective which could be a worthwhile contribution in the subarea of emergent languages. I see no obvious issues with the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors investigate what aspects of scenario design affect the resultant communication protocol in emergent communication. Specifically, the authors investigate whether scenario complexity (in terms of having many similar distractors) and scenario unpredictability (in terms of future batches not containing items to previous batches) affect the expressivity (measured in terms of learnability) of the resulting protocol.

Further, the authors show that defining a softmax loss over the entire batch of possible references (dubbed the 'contrastive' loss) outperforms the 'referential' loss used in previous works. 
main_review: The authors attempt to tackle the important issue of how functional pressures affect the properties of resulting emergent communication protocols. In particular, the are interested in quantifying the 'expressivity' of the resulting communication protocol, and give an operational definition of expressivity in terms of learnability by a randomly initialised listener. The authors note that mutual information is a poor measure for expressivity, and I like their proposed metric. It reminds me a little bit of various definitions of probing in the NLP literature, and perhaps a connection can be drawn here.

The resulting partial order in language transferability is described in text on page 8, but I think it would be better to draw the poset as a diagram, to make it easier to see what is comparable to what. In addition, I think the target games should be the x-axis of Figure 4, and the colours should be used for the source games. This way, we would be able to read off expressivity dominance immediately from the plot, as which curve was higher overall than the other. Further, it's unclear whether the generalisation performance is measured using a held-out set of language games, or whether the same games are used to train and evaluate the meta-suite of models.

The authors investigate the effect of two factors on language expressivity: scenario complexity and scenario unpredictability. Scenario complexity is defined as how many similar items each batch of data contains, on the assumption that items close in perceptual space are more difficult to distinguish. Scenario unpredictability is defined as the probability that the next batch of data contains an item not contained in the current batch.

I feel like complexity as the authors have defined is a very sensible metric. However, I'm not convinced by the unpredictability measure. It only considers bigram transitions, so considering a game with 4 inputs a, b, c, d and cycling the inputs endlessly as (a, b), (c, d) would be maximally unpredictable under this metric, when clearly it shouldn't be. I feel like the right notion of unpredictable should consider some kind of online learning scenario, such as prequential prediction. In addition, the human experiments used to motivate unpredictability rely on the fact that humans can remember past contexts when playing the game. In comparison, the models considered in the paper are memoryless, and the only effect of data presentation order is implicit in the optimisation dynamics. I understand that the intuition is that unpredictable environments force the speaker to be maximally informative in its utterances, but it's not clear that the proposed metric is the right way to capture this intuition.

Another contingent issue is that, at least as presented, complexity and informativeness are dependent variables, and the true controlled variable is batch size (indeed, the x axis of all the plots is batch size, and the formulae used to define complexity and unpredictability are not used anywhere else in the paper). This is because data points for a batch are sampled i.i.d. This means that the two axes of variation are not independent, which leads to conclusions which can't tease apart the effect of either measured variable, and reduces the overall impact of the paper as the empirical results suffer from confounding. I believe that alternative data sampling schemes would help tease apart the effect of both variables.

One final thing is that the 'contrastive' loss presented in equation 1 is contrasted with a 'referential' loss which is never explicitly given in the paper. To make the exposition self-contained, it would at least help to see how the 'referential' loss is computed, so that the reader can make a better comparison. 

(As an aside: Wittgenstein did use the phrase 'language game', but his concern was not acquisition, but rather how meaning is modulated by context. The first sentence of the introduction therefore misrepresents his statements, and I feel should be removed.)

=============

Post author response:

> Memory of human participants also decay, thus the context is always limited in a finite number of time steps. Defining unpredictability on a longer history simply changes the power of Equation 3 in our setting, but doesn’t influence the growth relationship between unpredictability and context size. Therefore, it won’t influence our conclusion if we extend Equation 3 to a longer history.

This is only assuming a uniform bigram/trigram/n-gram distribution. Often in the real world, sequential data has long-term structure which isn't well captured by any n-gram transition model (which is why RNN/Transformer language models so dramatically outperform count-based ones), and so it's not obvious that using more sophisticated transition schemes would result in similar findings.

To be honest, I find the unpredictability measure more opaque after author response. Section 5.1 varies complexity while keeping unpredictability constant using something called a 'fixed batch', but it's not very well explained what this is, and why it makes minimises unpredictability. Again, I wonder why predictability of the future given the past is important for these communicating agents, considering they are memoryless across episodes. I brought up this point in my original review, and it is not addressed in the author response.

> We would be happy to include more useful definition of unpredictability if there’s other helpful definition you have in mind. We’ll also be happy to see more exploration on how unpredictability of context implicitly influence the training dynamics of agents.

I gave a definition: online learnability using prequential coding (see, e.g. The Description Length of Deep Learning Models, Blier and Ollivier 2018 for a description). There is a large literature relating unpredictability and online learning, and I believe that many ideas in that field will be applicable here.

> We would be happy to include a new data sampling scheme, if there’s some other useful method you have in mind.

Any kind of non-independent sampling would decouple complexity from unpredictability, such as stratified sampling of contexts according to similarity given a target. I believe that there is much scope to precisely control complexity and unpredictability using such schemes, which I believe would make a future revision of this paper much more solid.

After seeing the author resposne, I don't feel that many of the issues I raised in my review have been substantially addressed, and I have not changed my score. Most notably, the issue of the definition of unpredictability and how it affects the communication are still unclear, and I believe this needs to be clarified substantially in future revisions. 
summary_of_the_review: Some interesting ideas in this paper (measuring the effect of the environment on the effectiveness of the language protocol, introducing a new and sensible definition of effectiveness), but unfortunately I think there are issues outlined in the main review which preclude acceptance in the current state.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the "expressivity" of emergent language in language games. To my understanding, "expressivity" is empirically the transferring ability of the language to unseen data. In this paper, the authors propose two factors of the underlying language game that can affect the expressivity of emergent language: context complexity and unpredicability. In referential games, the context is the distractor candidates for a sample. Complexity denotes how likely a "close" distractor will be included in the candidates. Unpredictability is how likely the context for a single sample will be different among epochs. This paper proposed hypotheses that both context complexity and unpredicability can improve language expressivity, but these two factors are "contradictory" and we need to have a trade-off between them. The hypotheses are supported by empirical experiment results.
main_review: #### Strength
- This paper provides novel and interesting insights to the language emergence analysis by introducing the concepts of expressivity, context complexity and unpredictability. These concepts provide a deeper understanding of the behaviors of emergent languages from (referential) language games.

- The hypotheses are clearly stated, reasonable, and well supported by the experiment results.

- Besides the main results, the paper also proposes a simple update on training loss, contrastive loss, which improved the model behavior.


#### Weakness
- The introduced concepts (context complexity and unpredictability) are finally simplified to be only dependent on the candidate set size $|C|$. Also, experiments and analysis design are focused on $|C|$. The scope might be narrow (but also specific, so not necessarily a weakness).

- Although the hypotheses are clearly stated and supported, they are not very surprising based on intuition.

- Regarding the proposed training loss, to my understanding, it simply replaces the original candidate set with the batch. In this way, the contribution does not look significant.


#### Questions & Suggestions
- The definition of key concepts, context complexity and unpredictability, do not seem fully accurate. In text, the context complexity is defined as the expectation of the probability that "$C(x_t)$ includes an object from $N_k(x_t)$". However, equation 2 means the expectation of the probability that "a random object from $C(x_t)$ is in $N_k(x_t)$". Nonetheless, the simplified form $1-(\frac{|X|-|N_k(x_t)}{|X|})^{|C|}$ is according with the text definition. Maybe equation 2 needs to be modified. For unpredictability, the text definition is the expectation of "the probability that $C^{e+1}(x_t)$ contains an object that is not from $C^e(x_t)$", which seems wrong because this is clearly not monotonically decreasing on |C|. Based on the simplified form, I guess it should be the expectation of "the proportion of $C^{e+1}(x_t)$ that are not from $C^e(x_t)$".
summary_of_the_review: This paper studies "expressivity" of emergent language, which is empirically the transferring ability of the language. The paper also defines "context complexity" and "unpredictability" of the underlying language game. The study shows that these two factors both contribute to the language expressivity, but they are contradictory on the candidate set size $|C|$ so a trade-off is needed. Generally, the analysis is novel and interesting. Although I have some concerns about the scope and significancy, the work still provides useful insights for language emergence study.
review_title: null

Meta-review: This paper studies the expressivity, complexity and unpredictability of emergent languages in referential games. The authors defined measures of complexity and unpredictability and empirically showed that the expressivity of emergent languages is a trade-off between the complexity and unpredictability of the context that the languages are used in. They introduced a contrastive loss based training method that alleviates the collapse of message types seen using standard referential loss functions.

The paper is controversial among the reviewers. On the positive side, most liked how the paper has a clearly stated hypothesis and extensive evaluations which makes a clear contribution to the field of emergent languages. On the negative side, the paper only shows the results in an artificial setting where the key variables are highly simplified (e.g. size of candidates). The main negative review argue the authors used an inappropriate definition of unpredictability and that the batch size is actually the key independent variable instead of what is claimed. The paper does somewhat equate batch size with the candidate size that is so important to their results (after eq (1)), but they seem to measure candidate size in the key figures. Perhaps an experiment controlling for batch size independently of the candidates size can address this issue. On the point of defining unpredictability, the other reviewers and I find the given definition to be reasonable and at least defensible. However, the reviewer remained unconvinced. More generally, the paper relies on one definition of the concepts measured in one setting to make a general claim, which is at risk of missing other important variables. Overall, most reviewers found the scope to be sufficient, and two improved their scores after the discussion.

Recommendation: accept
Decision: Accept (Poster)