Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a new paradigm for evaluating the degree to which artificial emergent languages resemble natural languages. They test the predictions of Harris' articulation scheme (HAS) on the messages produced by converged agents trained in a Lewis signaling game, where a speaker agent is trained by reinforcement learning and a listener agent is trained by supervised learning.
The authors develop a simple word segmentation algorithm designed on the assumptions of HAS. They also propose several predictions which they would expect to hold of the resulting vocabulary / utterance structures based on the nature of the reference game. They find only weak partial support for these predictions in the resulting emergent languages, and claim that this demonstrates a gap between the structure of emergent language and natural language.
strengths_and_weaknesses: Significance: I am not certain of the significance of the weak negative results of the paper. Some ways that I could imagine this paper would be more significant: 1) if there were a more concretely demonstrated gap with human natural language (see question #1); 2) if there were a positive result (e.g. resulting from implementing one of the ideas in L322-324); 3) if there were adequate qualitative examination (e.g. demonstrating that the TopSim positive result is meaningful, or showing the emergent meanings of hypo-words grounded in the reference game, or showing qualitative examples of compositional utterances).

Clarity: the paper provides *too much* detail, I think -- specifically, I don't see a reason for all of the figures and results to be repeated across different threshold settings. If there is one threshold setting/range that yields intuitive results across analyses, then it'd be best to present just that. If there isn't, that might suggest deeper problems with the procedure! But it seems like 0 < threshold < 0.5 would be sufficient for all the main results?

Quality+originality: Good! Clear and sensible operationalization of abstract ideas in experimental predictions (though see questions).
questions: 1. What is the baseline / foil for these experiments? The authors claim that HAS is a "statistical universal in natural language," but greedy unsupervised segmentation algorithms based on entropy alone are known to be insufficient for word segmentation tasks / lexicon induction tasks. (The authors cite some sub-100% F-scores themselves in L112-114). This means it's unclear what to expect from this procedure even if we were to stumble upon an emergent language with natural language structure. If humans played the same reference game posed to the communicating agents, what would we expect the outputs of the analysis to look like?
2. What is the intuition behind the ZLA result? This makes sense in systems where the speaker has a length penalty, but not otherwise to me.
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Harris' hypothesis (or articulation scheme, "HAS"; Harris, 1955) suggests that the linguistic boundaries of words can be detected by a count that is unrelated to meaning. However, it is concerned with natural human languages. In this paper, the authors explore if emergent languages between agents share this property with natural languages. 

They do this by using HAS for unsupervised word segmentation, i.e., the segmentation of text (in emergent languages) into smaller units. Then, analyzing the results, they ask: Do emergent languages have meaningful segments?

They find that emergent languages somewhat follow HAS, but not in all aspects. Similar to natural languages is that: (1) the conditional entropy decreases monotonically and (2) the branching entropy repeatedly falls and rises. However, in contrast to natural languages, HAS-based boundaries in emergent languages aren't always meaningful. Overall, this is a mixed, but nevertheless interesting result.
strengths_and_weaknesses: Strengths:
- This is a very clear paper with an easy-to-understand yet interesting research question. 
- The experiments are solid and suitable to answer the research question.

Weaknesses:
- The experiments are limited with regards to the settings in which the emergent languages emerge, but the paper is written quite general. This might be worth clarifying.
questions: - Do you believe your results are generally applicable to all/most emergent languages? If not, this might be worth clarifying in the relevant passages.
- Except for the above, where do you see the limitations of your work?
limitations: There are no obvious negative societal impacts of this work. The authors do not explicitly address limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates whether HAS also holds in emergent languages, which are languages emerging among artificial agents in a simulated environment. HAS is a universal property in natural languages which says that articulartory boundaries can be obtained from statistical information of phonems without knowing what words mean. The experiments show mixed results---emergent languages satisfy some preconditions for HAS, but the boundaries are not necessarily semantically valid.
strengths_and_weaknesses: Strengths:
- Originality: I am not super familiar with this research area but the idea seems quite original. It also seems like a scientifically interesting research topic to me to find the linguistic differences between natural and emergent languages.
- Quality: The experiments seem to be well-motivated, and the background is explained with good attention to details. 
- Clarity: The paper writing is quite good.
- Significance: The paper presents extensive experiments. While the experimental results do not fundamentally change the field of NLP, it is a nice focused contribution.

Weaknesses:
- It could be even better to explore emergent languages in multiple settings in addition to Lewis's signaling game. 
- Scope of the paper is quite narrow but I do not think this takes away from the merit of the paper.
questions: Do you think such results will extrapolate to emergent languages in settings other than Lewis's signaling game. Are there better ways to quantify how "meaningful" the word boundaries may be?
limitations: Seems fine to me
review_title: null

Meta-review: metareview: Harris' hypothesis (or articulation scheme, "HAS"; Harris, 1955) suggests that the linguistic boundaries of words can be detected by a count that is unrelated to meaning. In this paper, the authors explore if emergent languages between agents share this property with natural languages. They test the predictions of HAS on the messages produced by converged agents trained in a Lewis signaling game, where a speaker agent is trained by reinforcement learning and a listener agent is trained by supervised learning. 
The results are mixed (they find only weak partial support for these predictions in the resulting emergent languages) but the idea is novel and interesting.
All reviewers suggest acceptance, but the paper is IMO just below the decision boundary. It is likely to be of interest to only a small number of people and it is unclear whether it will spark any follow-on work since the Harris' hypothesis doesn't seem to fully hold.
confidence: Less certain
Decision: Reject