ELCC: the Emergent Language Corpus Collection

Authors: Brendon Boldt, Charis Graham, David R Mortensen
Primary area: datasets and benchmarks
Keywords: emergent language, emergent communication, dataset, resource

Abstract:
We introduce the Emergent Language Corpus Collection (ELCC): a collection of corpora generated from open source implementations of emergent communication systems across the literature. These systems include a variety of signalling game environments as well as more complex tasks environments like a social deduction game and embodied navigation. Each corpus is annotated with metadata describing the characteristics of the source system as well as a suite of analyses of the corpus (e.g., size, entropy, average message length, performance as transfer learning data). Currently, research studying emergent languages requires directly running different systems which takes time away from actual analyses of such languages, makes studies which compare diverse emergent languages rare, and presents a barrier to entry for researchers without a background in deep learning. The availability of a substantial collection of well-documented emergent language corpora, then, will enable research which can analyze a wider variety of emergent languages; this more effectively uncovers general principles in emergent communication rather than artifacts of particular environments. We provide some quantitative and qualitative analyses with ELCC to demonstrate potential use cases of the resource in this vein.

Decision: reject
Reviewer scores: 6, 6, 3, 1

Reviews and discussion:
<thread>
creation date:
1730022500327

thread_type:
official_review

<thread_object>
creator: reviewer_hvvr

<content>
summary:
This paper is about providing a collection of artificial corpora resulting from "emergent communication systems". These systems generate artificial language by forms of interactions between agents (sender, receiver). Having such corpora is important for studying similarity of artificial language to human language - which in turn could be useful for obtaining pre-training data. 

The main contribution of this the paper is the data resource and code for reproducibility.

strengths:
This resource could be valuable for the niche community on artificial languages. The authors also identify reproducibility issues in related work.

weaknesses:
While I do agree that these could be useful corpora and the paper should be published, I do not think that  ICLR is the right venue

- 1) it's mainly a resource contribution for which there are other venues, e.g., LREC or a corresponding journal

- 2) the whole field of study is quite niche

- 3) the substance is a little thin for an ICLR contribution, e.g., compiling approaches from other research

- 4) I would have hoped to read something in the context of recent LLM-based multi-agent interaction. However, mostly older approaches seemed to have been considered 

- 5) There were some typos (minor), and I would also have liked to see a more central illustrating examples or an illustration figure show-casing agent-based simulation and language generation (e.g., the signalling game)

questions:
As a suggestion, I would like to see an illustrating example. What are we supposed to be doing with the example in Figure 3? A sequence of integers is hard to read, interpret and understand.

What should we do with the numbers in Table 2?

Do I really understand correctly that you merely assemble the 73 corpora from other research and conduct a bit of analysis on it in Section 5?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730119638520

thread_type:
official_review

<thread_object>
creator: reviewer_98vb

<content>
summary:
This paper presents a system and approach called Emergent Language Corpus Collection (ELCC) for the standardized storage of languages produced by emergent communication systems. It further discusses several commonly used coordination games, comparing the languages they generate primarily using the XferBench evaluation metric.

strengths:
There is a clear need to standardize various aspects of emergent communication (EC) experimental setups, and efforts in this direction such as the suggested ELCC, are valuable. The authors provide a compelling critique of the difficulties in reproducing past EC research and identify missing elements that hinder accelerated progress. They call for a more robust framework to facilitate easier reproducibility, comparison, and competition in the field, which could significantly accelerate its development.

weaknesses:
The motivation behind this work is underdeveloped. Although standardization is beneficial, it is unclear what specific problems this work aims to solve. A clear example of ELCC’s utility, such as its potential to generate new insights, would enhance its justification. Additionally, the static nature of the ELCC collection raises questions about the kinds of research it enables and what insights it may provide beyond those already published by the corresponding papers. A leaderboard with well-defined datasets, metrics, and results might be more impactful than a static collection of generated languages.

Furthermore, while the ELCC corpus could have been used to evaluate the soundness and completeness of various EC evaluation metrics proposed in the literature, the analysis is mostly limited to XferBench. For example, although the authors mention the HAS metric (Lines 464-468), they only discuss it theoretically rather than demonstrating how ELCC could aid in evaluating this metric across languages.

In conclusion, the paper lacks innovative contributions, and the provided resource appears too limited to address key challenges in the field, such as establishing standardized EC benchmarks with readily available datasets and evaluation metrics.

questions:
Could the Hugging-Face platform, API, and cloud infrastructure be leveraged to host and promote ELCC to the broader research community?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696632814

thread_type:
official_review

<thread_object>
creator: reviewer_e7aw

<content>
summary:
This paper introduces a collection of "corpora" called Emergent Language Corpus Collection (ELCC), generated from open-source emergent communication systems (ECS). In Emergent Communication (EC), researchers are generally required to be familiar with deep learning, reinforcement learning, and machine learning frameworks such as PyTorch to simulate the emergence of language and communication. However, not all of the previous EC studies provide well-documented, high-quality, and open-source implementations, which prevents us from performing the meta-analysis of the EC studies and non-ML researchers from joining the EC field. To mitigate such problems, this paper introduces ELCC, collecting several appropriate open-source implementations, conducting reproduction experiments, summarizing the experimental results (i.e., emergent languages) as "corpora," and performing some basic evaluations (e.g., n-gram entropy, XferBench). The author(s) suggest that ELCC can also be used for further analyses (e.g., word boundaries, syntax).

strengths:
- Contribution to the reproducibility of the EC field and the ease of entry to the field.
- The novelty of the idea of creating corpora of emergent languages, which had not been conceptualized before, probably due to the artificial nature of emergent languages.
- Most EC papers mention previous work in the "Background" section or "Related Work" section regarding problem setting, methodology, and evaluation metrics. Interestingly, this paper additionally focuses on the quality and availability of their source code implementations and summarizes them comprehensively, which will be helpful for EC researchers.
- The corpus collection process is carefully planned and conducted.
- The paper is clearly written and easy to read.

weaknesses:
- ELCC is not annotated with meanings (or semantics, inputs) corresponding to messages (sentences). This would be somewhat problematic when the EC researchers want to evaluate other properties, such as compositionality.
- The basic analyses provided (e.g., XferBench) are indeed valid in showing the "negative result" (i.e., that emergent language is not similar to human language). However, I am not sure if the provided metrics are still valid in showing the "positive result" in the future (i.e., that emergent language is similar enough to human language), recalling some (computational) linguistic literature claiming that even monkey-typing sequences follow a Zipf-like distribution [1], that even pre-training with simple artificial language ($\neq$ emergent language) has a positive influence on downstream task performance [2], etc. This is another reason I think ELCC should be annotated with semantic information to allow future EC researchers to perform more detailed analyses.
- I feel references (only) to Ueda et al. (2023) and van der Wal et al. (2019) in some parts of the Introduction and Discussion sections are somewhat abrupt, considering a number of  EC papers have studied human language-ness of emergent languages from various points of view. I know typical topics in the field, like TopSim-based compositionality, are out of this paper's scope. Still, I'd be glad if you mention a few more human-language-ness papers somewhere in this paper, e.g., in the Limitation section.
- (minor) Figure 4 has no caption (other than sub-captions a, b).
- (minor) When mentioning Zipf's law, you may cite Zipf (1949) [3]. Likewise, you may cite Harris (1955) [4] and Tanaka-Ishii (2021) [5] for Harris' articulation scheme.

[1] Piantadosi, S.T. Zipf's word frequency law in natural language: A critical review and future directions. Psychon Bull Rev 21, 1112–1130 (2014). https://doi.org/10.3758/s13423-014-0585-6

[2] Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7302–7315, Dublin, Ireland. Association for Computational Linguistics.

[3] Zipf, G. K. (1949). Human behavior and the principle of least effort. Addison-Wesley Press.

[4] Zellig S. Harris. From phoneme to morpheme. Language, 31(2):190–222, 1955. URL http://www.jstor.org/stable/411036.

[5] Kumiko Tanaka-Ishii. Articulation of Elements, pp. 115–124. Springer International Publishing,
Cham, 2021. URL https://doi.org/10.1007/978-3-030-59377-3_11.

questions:
- Is it possible to (easily) update ELCC with semantic annotations in the (near) future?
- With ELCC, is it possible to evaluate language similarity (or language synchronization), which is sometimes considered in the context of populated signaling games?
- What can we read from Table 2? Does it indicate either language-ness or unlikeness of emergent language?
- (minor; just curious) As EC researchers, how can we contribute to this corpora creation project after deanonymization (e.g., by sending a pull request)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700501240

thread_type:
official_review

<thread_object>
creator: reviewer_cvc2

<content>
summary:
The paper introduces the Emergent Language Corpus Collection (ELCC), a collection of corpora generated from open-source implementations of emergent communication systems (ECS). The authors perform quantitative and qualitative analyses with ELCC to demonstrate potential use cases of the resource.

strengths:
1. The sources and statistics of the data are well documented.
2. The data can enable broader engagement and new research directions in ECS.

weaknesses:
1. As the authors already mentioned in limitations, the data are not annotated, having not reference to the semantics of the communication. This limits the scope of possible analysis.

questions:
1. "it is not comprehensive collection of all of the open source implementations let alone all ECSs in the literature" -- to your best knowledge, what are some ECS categories having no representative systems in ELCC?

2. Typo: line 169: syntehtic -> synthetic
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission presents an interesting contribution to the community, and the proposed corpora have potential utility. However, several concerns raised by the reviewers remain unresolved:

- The paper lacks significant innovative contributions.
- The benchmark provided does not adequately address key challenges in the field, such as the need for standardized EC benchmarks with readily accessible datasets and evaluation metrics.
- The substance of the work appears limited; the benchmark is assembled from existing datasets without introducing new approaches.
- It is unclear which specific research directions the benchmark facilitates and what novel insights it offers beyond those already discussed in the referenced papers.

While we appreciate the authors' efforts to address these concerns during the rebuttal process, the reviewers agree that the contributions of this submission fall short of the standard expected at ICLR.