Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper proposes a stochastic process, FILEX, to abstract out the essence of deep learning based emergent language systems. FILEX follows the intuition that the more a word is used, the more it will be used in the future. On four experimental settings, the authors show that the correlation between parameters of FILEX and the lexicon entropy is similar to the correlation between hyper parameters of neural networks and the lexicon entropy.
strengths_and_weaknesses: strength 

- This work attempts to construct a simple enough theory for understanding emergent language. Such attempt is generally valuable for the overall community as the filed of emergent language itself is, emerging.
- The proposed method is simple enough to mimic the model behavior thus providing a level of abstraction/ simplification for understanding. Additionally, I believe similar observation is also discussed in the previous VAE literature, just for the authors information[1].
- This work is inspiring to me and I believe such work should also be inspiring to future work. Though I am not fully convinced with the existing experiments (see below), I am happy to see either future work may reinforce its conclusion or turn them over.

weakness 

- The proposed methods is too simple to miss important details of many aspects of neural networks. When the authors link the parameters to of FILEX to the neural network hyper parameters, their reasons are more about intuition rather than rigorously mathematically discussions.
- The experimental settings are too simple and may be hard to generalize to more complicated setting. One direct generalization is whether the conclusion will hold if the vocabulary size is large (say 10K) and compositional (say a sentence of length 10).

[1] Alemi et. al. ICML 2018. Fixing a Broken ELBO
questions: How would the conclusion generalize to setting when the vocabulary is large and when the setting is compositional? 
limitations: The authors have properly discussed the limitations
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes a stochastic process FILEX developed from the Chinese restaurant process, to mathematically model the lexicon entropy of emergent language between multiple agents (the speaker and the listener) in ELS (emergent language system) environments. The authors make correspondences between the FILEX and ELS to evaluate FILEX in real ELS environments. Experimental results on four ELS environments show that FILEX can correctly predict the correlation between hyperparameters and the lexicon entropy of well-trained emergent language in real ELS systems.
strengths_and_weaknesses: In this paper, the authors propose a mathematical model of lexicon entropy of emergent language between agents. Moreover, the authors conduct extensive experiments to verify the model, and the results show its effectiveness.
Strengths:
1.	The paper provides an inspiring idea of mathematically modeling the properties of the emergent language without training in real environments. Besides, the mathematical description is more precise and testable than natural language.
2.	The experimental results demonstrate the effectiveness of the proposed FILEX, which verifies the feasibility of the idea.
Weaknesses:
1. The mathematical model may be not rigorous enough.
----a) The proposed FILEX is based on the assumption that word use is reinforced in emergent language in sec 3.1. However, the paper only provides evidence in human language but not emergent language between agents. The paper does not prove the assumption fits all ELS environments either.
----b) The correspondences of hyperparameters between FILEX and ELS in line 193 are based on analogy, thus the relationships between FILEX and ELS maybe not strong enough to support the alignments. Besides, beta in FILEX is corresponding to both buffer size and temperature in ELS, which may break the independence of the two hyperparameters despite the reason given in the paper.
2. The experiments seem to be not well-organized and insufficient somewhat.
----a) In sec 3.4, the meaning of the hyperparameter-entropy correlation and the reason why it can be used to evaluate whether FILEX matches ELS is not given.
----b) It is better to add the experimental evidence of the self-reinforcing assumption of emergent language in the four ELS environments, which is the basic assumption of the proposed FILEX.
----c) In line 173, as a testable mathematical model, FILEX is expected to provide more precise information, thus the equality of sign of correlation as the metric seems a little too weak. It would be better if the authors add some stronger metrics.
3. The organization and presentation of the paper could be improved.
----a) The introduction section misses necessary background introductions to the key concept “emergent language” and “lexicon entropy”, which may confuse readers from a broad domain.
----b) The paper lacks a formal definition of the problem or task. Besides, it is better to give the input, output, and goal of FILEX before the technical details in sec 3.1.
----c) In sec 3.4, it is better to provide more explanations about the obscure hyperparameter-entropy correlation, which may be unfamiliar to many readers.
----d) The paper may be not well-organized. For example, the environments in sec 3.2 and result analysis in sec 5.1 should be included in the experimental parts.
----e) There are some typos and small mistakes in the paper. For example, line 9 in Algorithm 1 is not correctly initialized and used elsewhere; “Section 3.3” -> “in Section 3.3” in line 102.

questions: 1. Does the self-reinforcing assumption of emergent language in line 79 make sense in all ELS environments? Are there any evidences in the literature or experimental results?
2. What’s the idea or intuitions behind the hyperparameter-entropy correlation in line 162? Why can it be used to evaluate whether FILEX matches the real ELS system?

limitations: According to the authors, there are weaknesses in this work that have not been addressed. Details are as above.
This paper does not have any potential negative societal impact.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper fits into the Emergent Language literation and argues that prior work has been insufficiently rigorous when presenting their hypotheses. This work is presented as a starting point for mathematical methodology in developing and testing models of emergent language. 

Their main contribution is an approach "FiLex" to understanding models of emergent language influenced by the Chinese Restaurant process (CRP). The relation here is to describe words as having a similar property to the tables in the CRP in that their usage is self-reinforcing. This approach is detailed in Section 3.1 with Algorithm 1 and the Formulation on page 3.

Their hypothesis, presented in Section 3.4, is that hyperparameters in FiLex correlate with hyperparameters commonly found in emergent language setups: time steps, lexicon size, learning rate, buffer size, and temperature. Their specific statement is that the sign of the correlation between those hyperparameters and entropy will be the same for FiLex as it is for the ELS setups. 

They experimentally test this hypothesis by comparing FiLex against 4 ELS setups in Table 2 and Figure 1 and then make claims in Sec 5 (discussion) that this model lets practitioners more rigorously understand confounding factors.
strengths_and_weaknesses: Originality:

This paper is original. It's taking a well-known idea (CRP) and applying it to a domain (emergent language) where the procedure is potentially beneficial for attaining understanding. It's unclear to me that this is actually the right approach for this domain and that's something that warrants greater discussion in the paper, but it's still an interesting direction that has merit.


Quality:

I don't think that this approach makes sense to do in ELS. I love the motivation to have better mathematical foundations in the space, but I disagree that this approach, or at least as presented, is the right path.

First, why should it be the case that methods with same sign correlations in individual hyperparameters are actually indicative of each other? This is much too weak of a statement to be predictive as we could manufacture this artificially without much issue. The argument given at the end of section 3 that then qualitatively associates the variables doesn't fix this issue; changing just the algorithm used from PPO to something else that doesn't have a buffer size shows how fragile that is. 

The explanation in 3.4 (L173-L180) points to this as well, suggesting that there are too many factors unaccounted for (due to FiLex's simplicity) to do much else besides this. But that's exactly what the paper set out to do at the beginning after discussing how prior work skipped out on this important step. Perhaps they did so for this same reason?

Sec 5, the evaluation, talks about making predictions ("[... FiLex] makes the correct prediction 20 out of 20 times.") and how that wouldn't happen if it wasn't predictive. But then the graphs (and the discussion) shows how tenuous that is. It's not very helpful for my research to run this method a bunch of times and hope that the correlation remains positive versus just varying a hyperparameter in my actual model and seeing what that does. Perhaps the authors are thinking that I cannot run my actual model that much because, say, it's a real robot. Okay that's fair, but I have no idea if this method would actually work in that setting and little belief that my model in totality would really be correlated given that a handful of hyperparameters are.

Clarity:
The paper is clear enough. The one area which would have been more helpful is to coalesce Table 2 and Figure 1 to be more apparently connected. It takes longer than I would have liked to understand what was going on in them (which are the results) and the stories they told.

Significance:

This paper is only significant in the world where the simple model is predictive of much bigger and more interesting models. The ones examined are not that, nor is it even assessed as predictive of anything in those settings.

For it to get to a place where we can confidently say that this is interesting for real-world settings, it would need to be aligned with actual (emergent) language learning. That is unfair to evaluate wrt learning new words in this setting (as neither FiLex nor most ELS systems are doing that), but it is fair to ask whether the learned language distributions are the same. Is that true? Is FiLex learning similar word distributions to the ELS models? More to the point, because FiLex's learned distribution won't ever be different given its simplicity, is it Zipfian? We expect that to if it's ever going to be able to model a full language. 


questions: 1. It's not clear in this paper whether we are asking whether the mathematical model that this process produces can answer questions of current ELS systems or are we asking whether it can help us illuminate where these systems are weak. Which one is it? The second is much more significant imo.

2. Why should the sign of the per-hyperparameter entropy correlation be predictive about what my approach will do? Is there some foundation for that in other literature?

3. What is the resulting word distribution of the CRP at steady state?

4. My suggestion is to return to the models you've made (the 4 methods NoDyn Recon Sig and Nav) and try and see what, if anything, is invariant and unit-independent amongst these. I think that will bear more interesting fruit than this approach starting w CRP.
limitations: Societally, this is fine. Wrt limitations, see the above S&W section.
review_title: null

Meta-review: metareview: This paper proposes FiLex -- a mathematical model to capture lexicon entropy in emergent language systems. The paper tackles an important and interesting problem in a field (emergent language) where relatively less theory currently exists. However, the reviewers find the experiments not convincing enough (e.g. they do not evaluate actual emergent language and instead use human languages) and lacking in scale. I do think the paper has some merits and can be strengthened further by addressing the reviewer comments, but the current version unfortunately seems below bar for acceptance. 
confidence: Certain
Decision: Reject