Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates whether agents evolve a shared language when equipped with a sensorimotor system that can produce and perceive drawings. The paper introduces a variant of the referential game, (called GREG) in which two agents must communicate via drawings. Agent A (the speaker) produces a drawing (a continuous graphical utterance) to indicate the ‘name’ of an object in a shared visual scene and communicates this to agent B. Agent B (the listener) then has to select the object indicated by agent A.   The agents switch roles between rounds. The drawings are produced by the agents involved using basic actions over a sketching environment, and so produce apparently continuous signals instead of discrete symbols (as would be the case in written language). Note that the drawings are not constrained to visually reflect the referents. 

The paper then presents a way to train agents to succeed in GREG. The method (Contrastive utterance-reference associative scoring) uses contrastive learning to represent the alignment between utterances (drawings) and referents (objects in the environment), and the generation of utterances maximises energy shared with the target referent to make  the utterance as identifiable (to the speaker, not to the listener) as possible.

The paper examines performance in this game, analyses the resulting learned language using coherence and compositionality metrics and claims that the learned language demonstrates componsitional properties.

strength_and_weaknesses: 
*Strengths:*
- The experiments are well designed (one-hot vs visual-shared vs visual-unshared). This difference in perspective between the speaker and listener is a nice addition to this game.

- In the main text there is a balanced set of conclusions around compositionality which match the evidence: that the presence/absence of compositionality in the learned language is difficult to assess and conclude anything about. However (see weaknesses) the abstract makes a different claim.

Weaknesses:

- Despite the lack of clear evidence for compositionality in the main text, the abstract claims in the final sentence “we show that our method allows the emergence of a shared, graphical language with compositional properties”. This claim is unsubstantiated and should be removed.

- It wasn’t clear to me how the drawn image is actually perceived by the listener. Is there a CNN in here somewhere to instantiate this perception, that I am missing? If the listener’s perception happens instead in the action space of the speaker then is it really a continuous signal? The perception of the image by the listener needs to be clarified in the text.

- Compositionality is a hypothesis which is generally difficult to falsify empirically. It is not clear what data would have yielded a clear compositional vs not-compositional result either way. It is therefore not clear what this section of the paper on measuring compositionality contributes to the literature.

- Figure 4 is a bit hard to parse as it has multiple axes going on at the same time. These results would be clearer if there was another way to present them.

clarity,_quality,_novelty_and_reproducibility: The descriptions in the text are quite clear, however the presentation of data in Figure 4 is quite hard to read. There are some details around perception by the listener that I could not easily find in the text which would be essential for reproducibility. 
summary_of_the_review: This paper proposes a variant on the referential game, which creates a new medium for communication: drawings. This results in a continuous space of output to be perceived by the listener, which bring the game setting closer to verbal communication between agents. While the game itself is interesting, the method to solve it does not seem particularly general (which is not a big drawback, just a small one) and the section on compositionality is a weak point.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new variation of the signalling game for emergent language with 2-dimensional sketch-based messages produced via dynamical motor primitives.
To address this new environment, the authors also introduce contrastive learning-based method which aims to align the message embeddings with that of the observation in a joint space.
Finally, the paper provides an analysis of the performance, message characteristics, and compositionality of the emergent language.

strength_and_weaknesses: ### Strengths
- (major) The paper presents a new, well-formulated emergent language environment and learning method.
- (minor) The experiments give an adequate characterization of the empirical behavior of the system.
### Weaknesses
- (major) The contributions of the paper is introducing a new environment, yet this environment is not strongly motivated.
  Simply differing from previous work is not motivation enough.
- (major) The field of emergent language does not stand to benefit that much from new environments if they do not address a relevant issue in the field.

clarity,_quality,_novelty_and_reproducibility: ### Clarity
- The explanation of Greg/Curves is good and well-illustrated.
- Figure placements that break the text (e.g., Figure 4) are distracting.
  I generally recommend $\LaTeX$'s automatic figure placement.
- Figure 7 is a great visualization.
### Quality
- The experiments are a good illustration of some of the properties of the environment, but it does not seem there are any broader hypotheses about the properties of emergent language being presented or tested.
- Was Hausdorff distance the only image similarity metric you tried?
  It is a reasonable choice, but it would still be good to argue briefly as to why it is suited to this application.
### Novelty
- This paper is similar to Mihai & Hare (2021) in that it tackles emergent language with sketch-based messages, but it is sufficiently distinguished based on its unconstrained objective (no referent-utterance similarity constraint), different method of generating sketches, and different environment.
- The introduction of the new environment is not sufficiently motivated.
  Although, I see how it is different from previous environments, the paper does not argue convincingly enough for why the changes made for the environment are important in some way and not simply different.
  - From my perspective, emergent language as a field is most in need of contributions which make steps towards clear advancements and well-motivated applications.
    While in the earlier stages of the field, it was acceptable to publish more exploratory work which simply tests the waters with new environments seeing if the emergence of language was possible at all.
    But now it is pretty clear that some sort of language will always emerge and what needs to be demonstrated and discovered is how emergent languages can actually become more like human languages and how this can be used to solve real problems.
### Reproducibility
- The explanations of the method are clear, and further detail are presented in the appendix.
- Code contains dependencies and README with information on how to run an experiment.
  - Did not attempt to run code.
  - Providing dependencies with versions is even better for reproducibility (especially once the code ages for a year or five).

summary_of_the_review: I am recommending marginal reject because while I find the environment to be well-formulated, I do not see the novelties as being significant from a research perspective.
I think that given where the field of emergent language is at, the introduction of new environments needs to be focused on more tangible advancements in the field; research which is primarily exploratory is not sufficient for this.


### Misc.
- "emergent language is as close to natural language" -> add "as possible"
- "no number overlap" -> "no number overlaps" 
- "that quantifies how an utterances denoting ..." -- check for typos
- Equation 1: What is $u^\star$?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study emergent machine language. They propose a game and then a system for solving the game. In the game, two agents, a speaker and a listener, must communicate the identity of a referent symbol. In the simplest version of the game, the speaker receives a target referent symbol from a set of context symbols. The speaker produces an utterance, a 2D curve in the plane, which the listener must interpret to predict the target. The authors propose a system for playing this game in which both speaker and listener use a contrastive objective to learn associations between utterances and referents. The proposed system is able to play the game better than chance. Finally, the authors propose a topographic similarity metric to study the compositionality of the emergent symbols. They find that the proposed metric makes suggestive, but not conclusive, findings about the language of their proposed system.
strength_and_weaknesses: # Strengths
- Exploring compositionality using an automatic topographic metric is an interesting idea. It would be nice to see what this metric gives for real world ideograms/pictograms to get a sense of its trustworthiness and interpretability.
- The continuous setting is a very nice departure from the typical setup

# Weaknesses
- The setting is limited. In the single-target case, the alphabet has 5 symbols. In the joint-target case, there are 5 choose 2 = 10 symbols.
- Studying compositionality in this setting is thus limited to exploring how conjunctive symbols may be encoded, as opposed to studying negation, disjunction, adjectives, adverbs, hierarchy, etc.
- In this work, compositionality is studied by looking at the form of produced symbols. For instance, to study the compound symbol $u(r_{ij})$, the authors use a topographic score to compare the compound symbol to the corresponding individual constituent symbols. However, there are many ways for a writing system to distinguish between semantics without dramatically distinguishing between forms and vice versa. It would make more sense to do this type of analysis in terms of the symbols decodability. Look at what sort of mistakes a decoding model makes. If the symbol $u(r_{ij})$ only encodes the joint symbol, and nothing about the constituent pieces, then we would expect no correlation among the incorrectly decoded symbols. But if compositionality is employed, we would expect to see predictions of the form $u(r_{ik})$ and $u(r_{ki})$ at higher frequency.
- The emergent language is not conclusively shown to be compositional (section 4) using the proposed topographic measure
- I don't think the setting can be rightly called a "language game" in which the speaker and listener are jointly learning to communicate. From section 3, pgs. 5 and 6, as best I can tell, the listener's loss is directly optimized to learn the association between the ground truth referent and the utterance. Please correct me if I'm wrong (see clarification question in section below). I believe the correct thing to do would be for the listener to learn an association between the predicted referent and the utterance. As it stands, this is information that is typically not available to the listener during the language game

clarity,_quality,_novelty_and_reproducibility: # Questions
- Some of the claims could be better supported, and the information available to each agent should either be clarified or modified (see below)
- Section 2. pg. 5 "Agents could, for instance, denote compositional referents using newly invented signs." I don't think I follow the argument here. If the agents invent new symbols at evaluation time, how can they perform better than chance? Moving to the domain of joint symbols essentially introduces a new alphabet. Is there a way to communicate referents in this alphabet on-the-fly without using compositions of existing symbols?
- The notation and terminology are sometimes difficult to follow. The use of the terms "referent", "context", and "feature" could be clarified, especially in section 2. As far as I understand it, a "referent" can refer to a single MNIST instance (caption of figure 1), but it can also refer to a collection of instances (section 2). In section 2, the term "feature" always refers to a single MNIST instance. And in the caption of figure 1, "context" always refers to a collection of instances. Have I understood correctly? I suggest disambiguating the use of the terms "referent" and "context".
- In section 2, referents $\mathcal{R}_m$ are defined terms of vectors. Why not define a referent as a set? Indeed, the notation $r\in \mathcal{R}_m$ is used in the same paragraph.
- Can I get clarification on the loss function (section 3, pg. 5-6)? I'm a little confused as to how the loss for the listener is constructed, and I have two hypotheses for what is happening. (1) The loss for the listener is optimized to learn high similarity between the uterrance embedding and the listener predicted referent. (2) The loss for the listener is optimized to learn high similarity between the utterance embedding and the ground truth referent. Do either of these match what is being described? If so, should this phrase "make sure that its selection matches the speaker’s referent" instead say "make sure that its selection matches the predicted referent"
- Along the same lines, on page 5, what is $r_L^i$? Is it the listener's prediction for the referent, or is it the ground truth label?
- Figure 4. What do the error bars indicate?
- Figure 5. What do the axis labels mean? The caption refers to a "group" of agents? Do A[0] and A[1] refer to separate groups?
summary_of_the_review: The proposed system solves a game in a very limited setting (5 symbols) and the proposed metric for studying compositionality does not conclusively show compositionality for the given emergent language. A revised version of the work that shows why this metric should be trusted, perhaps by giving validation on real-world symbols, would be a much stronger submission.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a sensory-motor shared language task between two agents, metrics for evaluating the coherence and compositionality of the language that the agents develop, and an initial algorithm for solving this task using a contrastive multi-modal encoder coupled with a generative model. Experiments conducted on this solution show that the languages that develop are coherent, but not very compositional. 
strength_and_weaknesses: Strengths:
* The proposed approach addresses an important and timely problem of agents learning to generate a shared language under realistic sensory-motor constraints. As discussed in the paper, several approaches have shown promising results without such constraints, but these are not realistic models of human language development because they lack them.
* The proposed problem formulation and framework appear to be well suited to evaluating this problem. They are simple enough to be tractable, but sophisticated enough to capture the important aspects of the problem.
* The proposed metrics seem reasonable in measuring performance
* The proposed baseline shows promising results for coherence, but not compositionality. It shows that solutions to the task are possible and that the metrics can capture the performance of solutions.
* The paper is clearly written and explained.

Weaknesses:
* The abstract states, "We show that our method allows for the emergence of a shared, graphical language with compositional properties." I'm not necessarily convinced that this is true. The proposed solution does not seem to include much in the way of compositionality. The proposed metrics seem to allow for compositionality, but it is difficult to judge the extent to which they would successfully identify a compositional language if one developed. This statement also misled me into thinking that the proposed solution was compositional, although on re-reading, it does not actually state this.
clarity,_quality,_novelty_and_reproducibility: Clarity:
* Very little detail is given as to the nature of the sensory motor model that creates the sketches. As this is an important part of the system, it would be useful to include more information about how it works exactly. The reference to Schaal (2006) and Dynamic Movement Primities seems to be a very general framework. How is it instantiated here?
* Otherwise, the paper is generally very clearly written with clear justifications and relevant citations to the literature.

Novelty:
* The proposed problem builds upon, but is significantly different from, recent studies simulating the development of language between agents.

Reproducibility:
* The authors provide their code in the supplementary material for review along with model architectures and hyperparameters in an appendix
summary_of_the_review: This paper poses an interesting and important problem and appears to provide reasonable means of evaluating solutions, although it is difficult to say for sure without an actual full solution. It does provide one solution, but it seems to fall short in terms of compositionality. The paper could provide slightly more detail on the actual sensory-motor aspect of the utterance production process, which is a crucial aspect of its novelty.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors introduce a graphical referential game to study the emergence of visual conventions between communicating agents. The framework is based on recent computational approaches exploring the emergence of language through language games. In particular, the authors exploit a “Listener / Speaker” architecture, where the goal of the speaker is to produce an utterance that can be perceived by the listener to identify a target referent. Utterances are generated by feeding command vectors to a differentiable sketching model, with a contrastive learning objective promoting the alignment between referent and utterance embeddings. The authors also explore the compositional properties of the emerging lexicon.
strength_and_weaknesses: A strength of the paper is that it tackles a non-trivial problem by combining recent ideas from different disciplines (e.g., language games, speaker/listener communication architectures, contrastive representation learning, differentiable computer graphics…). The proposed framework is well-motivated, and the work stimulates further thinking.

At the same time, however, I see several weaknesses that might downplay the significance of the work:
- Besides an increase in ecological / cognitive realism, it is unclear what could be the advantage of using motor primitives to create utterances in the graphical space, rather than directly optimizing the continuous graphical representation (as done in previous work). Introducing a “differentiable motor system” might be relevant to better understand how sensory-motor constraints shape our ability to create written symbols, but this aspect has not been investigated in the present work.
- The authors also argue that a possible advantage of exploiting a drawing framework is to increase the interpretability of the emerging lexicon; however, also this aspect has not been emphasized in the simulations (and, in fact, the emergent signs appear quite arbitrary).
- It is not clear what is the relationship between the complexity of the concept space (e.g., similarity between objects and/or number of different objects to represent) and the emerging graphical conventions; further analyses would be needed to clarify this important aspect.
- What is the role of the dimensionality of the graphical space in the emerging lexicon? I guess this parameter might have a major impact on the type of signs that could emerge, but it seems that the dimensionality was arbitrarily chosen.
- From my understanding, the contrastive learning mechanism should (indirectly) promote an increase in similarity between the referent images and the generated utterances. However, this does not seem to happen in practice. Am I missing something?

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is readable, but I think clarity could be improved in several aspects. For example, there are often repetitions within the text (e.g., it is said several times that “for a given context R the agents are randomly assigned their roles”; also the objectives / contributions / scope of the paper are often redundant). The authors initially mention that agents can perceive referents “from different perspectives” without explaining what it means (the notion of “perspective” is introduced later). Furthermore, the authors discuss the distinction between discriminative vs. descriptive versions of the game, but such distinction is treated in a confused way during the presentation of the results.

Quality: The methodology seems appropriate, and the results appear robust. At the same time, I think that further analyses are required to better clarify the weaknesses mentioned above.

Novelty: Some ideas presented in this paper are original, but the overall framework seems to be partially incremental with respect to previous work (e.g., Mihai and Hare, 2021). One key innovation introduced here is the use of contrastive learning to promote a better alignment between the representations of referents and utterances; however, in my opinion the implications of this choice have not been systematically evaluated.

Reproducibility: I think the results are reproducible.
summary_of_the_review: I enjoyed reading this paper and I found it stimulating. At the same time, I think that the overall clarity of the manuscript should be improved, and the results should be extended to clarify several aspects of the proposed architecture. The feeling is that part of the results is more “preliminary” than ready to be published in a top-tier venue like ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: An agent must produce a drawing that allows another agent to identify a target object among N distractors.
Strengths: A novel setting for emergent language that builds on prior work to provide more flexibility while avoiding sharing gradients between speaker and listener. A novel contrastive method.
Weaknesses: The manuscript, even after edits in response to the reviewers, continues to highlight compositionality as a key feature; as the objective it lists "This paper investigates how a group of two agents can agree on a shared compositional
language to denote referents...". Having set this objective, the manuscript does not address it conclusively. Very weak evidence is provided; evidence that is far from sufficient for such a critical claim. Even the status of the resulting symbol sets as a language is unclear, as one reviewer put it "that part of the results is more “preliminary” than ready to be published in a top-tier venue like ICLR". The manuscript would be stronger if it simply never mentioned compositionality at all, as this claim is not supported. Some reviewers did not see the novel setting as a plus, questioning the need for including a drawing-based sensory system, and some questioning the complexity of the setup. Authors did simplify the setting, but more could be done here. Reviewers also pointed out that the setting is very narrow, having an alphabet of only five symbols.
justification_for_why_not_higher_score: Even after revisions in response to reviewer comments, and the authors admitting that compositionality is not at all addressed either in the method or in the evaluation that the manuscript provides, the manuscript continues to make statements about compositionality.

Reviewers were unconvinced by the setting, authors did not meaningfully explain why a system for drawing sketches is a meaningful addition to the study of the emergence of language. The setting is still rather complex and could be further simplified to get at the core of the problem being addressed. Reviewers pointed out that scaling the setting up may reveal more, beyond the five symbols that are in the alphabet today, rather than focusing on a more complex output system. A similar simpler environment already exists in the literature, the manuscript should engage with that work more convincingly.

This could be a valuable contribution, but it needs more polish, precision, and direction.
Decision: Reject