Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes symbolic mapping. The main point of this method is to associate objects with words in processing before making pragmatic decisions about what should be said. THis is then part of a larger recurrent architecture that produces words to describe or discrimate objects. The architecture is trained using REINFORCE - in different scenarios. Another point the paper is trying to make is that training the symbolic mapping on a simple scenario and the explanding complexity helps. The method is evaluated for success but also in terms of its ability to create compositional and symmetric language.
main_review: 
I have a number of larger issues with the paper. Some of this may stem from me not being able to understand the paper. The writing isn't very clear. Concepts are seemingly introduced in a random order. The authors move around between architectures, tasks, training regimes - which made it very hard to follow for me.

The paper is trying to make too many different points including a "new" architecture, a "new" game, a "new" curriculum approach. Some of these seem trivial or known in prior work. 

It's quite known in RL more generally and in language emergence more broadly that curriculums are helpful. I don't think the paper really adds anything interesting to that discussion.

Unfortunately, I did not understand well how the agent architecture works. It's not clear how the word bank works and what the speaking networks and reconstruction networks are. But from what I understand - the symbolic mapping is basically an MLP that looks up words for objects. It's these words then that are fed to the recurrent sentence generator. This seems like a prior that works in the particular setting here. However, it also seems like a trivial extension.


Important baselins such as IL are not explained in the paper.

Some major details are missing. E.g. the different MLPs such as the reconstruction network - how many layers etc

Other Comments and Questions

was originated -> originated

The definitions around objects in 3.1 don't seem to make sense. If a \in {1,2,...,n} and there are n attribues in each object - are you saying that all objects are the same? What's m? It seems to be the number of values per attribute. but you already said that the value of a is one of these {1,2,...,n}. That's quite confusing
Are you saying that all attributes have the same number of possible values in your experiments?

It's not clear to me what is the word bank and what does it consist of. 

What is a reconstrution network? The term is used once and never reappears. Is it the rec networks in your figure? Are they per attribute?

What aren the dimensions for your different MLPs

Why do you use $s$ and $w_i$ for the word?

"then both agents success" -> "then both agents succeed"?

I think it's not useful to use the term discrimination game and then suggest that it's more difficult since it requires dialogue. Discrimination games have been introduce in the 90s and they are referential and they do not necessarily require dialogue

Similarly people have used the term "decscription game" in emergent communciation research since the 90s and it is precisely not a referential game in the classical sense.
summary_of_the_review: 
The paper unfortunately suffers from some unclear writing as well as some lengthy unimportant descriptions and some important aspects that are missing

PRO
* interesting domain with lots of problems
* main claim of language emergence from simple to complex is illustrated by one example. But this is such a general claim - it would require a lot more experiments
* the main claim of the paper is not really a technical claim 

CONS
* some key explanations are unclear.
* many less important explanations are lengthy and convoluted making it difficult to follow
* somewhat trivial domain (that is pretty common though in this line of research)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates methods for emergent communication. Specifically, it looks at inductive biases that can contribute to the emergence of compositional and symmetric languages. The main environment the paper studies is a multi-turn dialog game, where two agents are shown objects that differ by 0 or 1 attribute and have to agree on the differing attribute.

The paper makes 2 main contributions:

1) It investigates the role of curriculum learning, and shows that first training on a simpler task -- a referential description game (task transfer), or the discrimination game with fewer properties (vocabulary expansion) -- helps agents learn compositonal languages on the broader discrimination game.

2) It proposes a new agent architecture, 'symbolic mapping', which can help agent performance and increase compositionality.
main_review: Papers like this are hard for me to evaluate. There has been quite a bit of previous work on looking at what environmental pressures and inductive biases lead to the emergence of compositional communication. In these papers, the environments considered are so simple that the systems developed are unlikely to be practical for any near-term applications. The contributions in these papers are instead scientific -- do the experiments help us understand something new about how simulated languages develop? Are the experiments rigorously performed?

I would say this is not a bad paper. The paper uses an environment that, while simple, is in fact a bit more complicated than environments in previous emergent communication papers. I would lump their 'task transfer' and 'vocabulary expansion' results under the general banner of curriculum learning -- if you first train on a simpler task, then train on the more complicated task, you will do better at the more complicated task. This is pretty intuitive, I'm a bit surprised that no previous paper to my knowledge has made this specific point (though it may exist), but I think this is a positive contribution of the paper.

The paper also proposes a fairly specific LSTM-based architecture (SM) that further improves performance. I don't find this result particularly compelling -- given that it is mostly an engineering contribution, I'd want to see it tried on a broader range of tasks that we care about. One of the claimed benefits of SM from section 3.3 is that the architecture allows us to do 'vocabulary expansion', but the results indicate that vocabulary expansion seems to help the base LSTM policy just as much as the SM policy. I don't find the remaining arguments and intuitions in 3.3 general or compelling, but I may be missing something.

Ultimately, if we take the main contribution of the paper to be 'forms of curriculum learning help compositional languages to emerge', I don't think this paper presents experiments that are quite extensive enough for the reader to really understand *how* curriculum learning helps. I'd want to see more ablations, e.g. looking at multi-step curricula (the paper only considers 2 steps), effect of initial task difficulty, etc.. 

Overall, I think the paper is not quite at the level of acceptance, and would currently recommend rejection.
summary_of_the_review: While the paper shows that curriculum learning can help compositional language emergence, which is interesting, I'd want to see more extensive experiments and ablations to recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper talks about introducing a new learning paradigm to train compositional and symmetric emergent languages while the tasks increase in complexity as the training progresses. The main motivation comes from works in psychology that show that human languages are also evolved from a simpler task to gradually increasing the task complexity. The proposed symbolic mapping module is shown to perform better than training agents directly on the complex task, thereby showing the importance of using past strategies to learn a solution for the new complex task.
main_review: The paper is well motivated and is easy to understand. The authors perform an extensive search over the relevant metrics used for computing compositionality and systematic generalization in an emergent communication setting. Although, the authors show promising results on the small 'toyish' dataset, there is no evidence or discussion on whether the findings would scale to more complex domains. The two different games used in the paper still only use symbolic data with only a limited number of training samples.

The task transfer experiment is not clearly stated. The authors state that symbolic mapping provides knowledge about the learned language implicitly. It would be better to demonstrate this with a small example on what exactly transfers from learning the simple solution to solving a more complex task. Moreover, the authors refer to increasing the complexity of the task to increasing the number of types of an attribute. How would this approach scale if we add a new dimension of complexity? Would it require fewer game interactions or samples than training an agent from scratch? More empirical evidence is needed to support the claim that the proposed module would scale to any complex task, or state how such a complex task should be designed.

The authors state that training agents in the discrimination game is harder than expected and the effect of symbolic mapping is not really pronounced in this game. A qualitative example would be helpful to highlight the possible issues with this game as compared to the description game.


summary_of_the_review: The authors propose a novel module that helps in training better compositional and symmetric language using a curriculum from simple to complex tasks. The paper is well-written and the experiments show that the proposed method indeed supports the hypothesis although the experiments are only performed on a simple environment. The complexity of the tasks can be varied along different axis but only a few options are explored to extend this.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper has several main contributions: (1) A novel architecture, symbolic mapping (SM), (2) Showing SM outperforms vanilla LSTM and NIL in terms of success rate, degree of compositionality, and degree of symmetry on the description game and discrimination game after bootstrapping,(3) Showing SM aids vocabulary expansion moving from a smaller to a larger vocabulary and attribute space. 

SM, a linear module that maps an object to a subset of the symbol vocabulary, is intuitively an agent's internal object representation or "word bank". A recurrent speaking module then selects the speaker agent's utterance from their word bank. SM associates object to symbols in a task and speaker-agnostic way. This not only facilitates the transfer of these object-symbol mappings to different tasks but also expanded vocabularies.

Theories of language emergence posit that human communication evolved to be more complex by adapting to increasingly complex settings. Following this hypothesis, the authors define two emergent communication games, a description game which is easier and discrimination game which is harder, and use SM to facilitate the transition between the two. They quantify performance using training and test success rate, as well as refdis (compositionality) and refdiv (symmetry). They find that SM outperforms a vanilla LSTM and NIL in the description game in all metrics. They show that all three models perform poorly trained from scratch in the discrimination game, then that their performance improves after pre-training speaker agents on the description game. Notably, when bootstrapping from the description game, SM achieves a higher success rate while also exhibiting a higher degree of compositionality and symmetry than the other two models. This supports the effectiveness of SM as an internal object representation for the agent.

Finally, the authors conduct a vocabulary expansion experiment. They first show that both an LSTM and SM agent learn poorly  from scratch on a large attribute space and vocabulary size. Then, they show that all agents perform well on the same task after bootstrapping from a smaller attribute space and vocabulary size. In particular, SM (whether or not reinitialized) outperforms LSTM after transfer.  
main_review: **Weaknesses:**
1. Unclear that SM, refdis/refdiv, and the reinforcement learning tasks are sufficiently original to stand alone.
2. Missing significant literature.
3. General issues with clarity 

**Strengths:**
1. Well-motivated and well-designed experiments. 
2. Results make a strong case for further work in intermediate symbolic representations and support prior research!

**Recommendation:**
Based on the above reasons, especially due to lack of citations/originality, I vote to reject the paper in its current form. However, since there are strong empirical contributions, if the authors revise the paper to include the relevant prior work (details below), I'd willing to change my vote to accept the paper.

**Questions and Additional Comments:**
(organized by originality, technical quality, and clarity)

*Originality============================================================*

In general, the paper can benefit from a more thorough discussion on how the new research differs from past work. There are several major areas in the paper where the previous literature is inadequately cited (see points 1-4). It's unclear that the authors' modelling contribution and reinforcement learning tasks are sufficiently original to stand alone. However, the experiments showing that SM is beneficial in task transfer are important in that they support prior findings, and the experiments on vocabulary expansion are novel, to my knowledge.

(1) The main idea of symbolic mapping, which is mapping raw input to an intermediate symbolic representation, is not new and should be discussed in relevant work. See work such as "Towards Deep Symbolic Reinforcement Learning" (Garnelo, et al 2016) and "Reconciling deep learning with symbolic artificial intelligence: representing objects and relations" (Garnelo, et al 2019) for a starting point.  

(2) The idea of bootstrapping in emergent communication from simpler tasks to harder tasks is not new and should be properly cited in section 2 (I suggest removing the statement "orthogonal to existing work"). See "Developmentally motivated emergence of compositional communication via template transfer" by Korbak, et al (2019) and its references for a starting point.

(3) The literature in Section 1 on language evolving from simple to complex tasks can be greatly expanded. See Deacon's "The Symbolic Species" and other references in Korbak, et al (2019) for a starting point.

(4) The concept of refdis is very similar to context independence introduced in "Emergence of Communication in an Interactive World with Consistent Speakers" by Borgin et al (2018) and discussed in "On the Pitfalls of Measuring Emergent Communication" by Lowe et al (2019). It would be good to differentiate context independence from refdis, discuss with a citation in section 3.4.

Nit:
(5) Rewording the explanation of Eq. (1) to be less similar to the explanation of Eq. (1) in Chaabouni, et al (2020). 

*Technical details ============================================================*

Overall, the authors explain technical details and modelling choices thoroughly, and well-motivate the training scheme/experiments. In particular, the LSTM ablation results are strong in that they suggest SM produces a good intermediate object representation. While the big-picture is technically sound, I disagree with some of the claims in the paper and would like to see a deeper discussion of referential disentanglement. I recommend editing the verbiage to be more task-specific instead of general (see point 1).

(1) Section 3.3: "symbolic association which is nearly orthogonal to tasks…" I disagree. For example, in grounded language games, symbolic association is closely linked to the task. It is important to specify that this claim applies to *this specific setting* rather than in general tasks as implied.

(2) The discussion of compositionality in Section 3.4 needs further development. Why does referential disentanglement, which is similar to context independence, capture compositionality in this setting? What are the limitations of referential disentanglement? Note that referential disentanglement works in this setting because the semantic composition rule of the attributes is intersective and not order-dependent. 

(3) Section 4.1: "agent cannot predict… when the dialog will be terminated." Is this problem solved when fixing the number of turns instead of letting agents terminate the game?

(4) Section 4.3: was NIL tested in vocab expansion?

*Clarity ============================================================*

The implementation of the model and experiments are mostly clear. However, there are some statements that I found vague, some claims that can be better argued, and some adjustments that can improve the flow of the paper. 

(1) In Section 3.1, it's unclear what the turn taking structure is until Section 3.2. It would help to include a line saying players explicitly take turns unless the game is terminated.

(2) The first two lines on pg. 4 are hard for me to understand. Why does detecting the difference between objects imply the agents may not be clear what their partner's symbols mean? I would think that it's never guaranteed that the agent understands the meaning of their partner's symbols.

(3) In Section 3.1, Description Game, reconstruction model is used first and defined later. I would include a line explaining what a reconstruction model is. 

(4) Section 3.3: "though the mapping is simple… help maintain language properties across tasks." What is intended by "language properties"?

(5) Section 3.3: in the explanation of vocabulary expansion, the word "vocabulary" is overloaded. It's not clear upon initial reading that vocabulary expansion is increasing the size of the word bank rather than the base vocabulary.

Nit:
(6) We fix the message length to n (unclear upon first reading why it's n) -> We fix the message length to n, corresponding to one symbol per attribute.

(7) Please state both bounds for both refdis and refdiv.

(8) Table 1: second and third column -> first and second column

(9) Section 4.3: "We have empirically proved that language can evolve from simple tasks to difficult tasks" is too general. Change to "We have shown that our agents' language/communication protocol evolved in task transfer,.." or something more specific.

(10) Figure 3: it's hard to tell blue/green and red/orange apart.

(Not important for this review, but please clean up grammatical mistakes for future iterations.)

summary_of_the_review: The current version of the paper greatly overstates the novelty of its technical contributions (SM, refdis, task transfer). To me the primary contribution is therefore the synthesis of prior technical contributions, the strong experimental result, and the case for further research into symbolic representations. Only on the condition that the authors substantially revise their discussions of SM, refdis, and task transfer to credit prior work, I would recommend the paper for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper models a new aspect of emergent language: developing language to complete a simple task before transferring the emergent language to a new, more complex task.
It does this by creating curriculum of a simple referential game followed by a slightly harder dialog task.
A traditional architecture does not perform well on this task transfer so the paper introduces _symbolic mapping_ in order to encourage a compositional language which generalizes well.

The following empirical evaluations are performed:
- discrimination (dialog) game only
- description (referential) game then discrimination game with and without symbolic mapping
- expanding vocabulary by transferring to a new task with new attributes to describe

The contributions of the paper are:
- Introducing experiments which look at the progression from a simpler game to a more complex one
- Developing a novel architectural component called _symbolic mapping_ which aids in both transferring to new tasks and in developing a compositional communication protocol
- Introducing a metric referential disentanglement based off of positional disentanglement

main_review: ## Post-rebuttal Edit
In light of the added clarity regarding symbolic mapping, I am upgrading my review from 5 to 6 and correctness from 1 to 3.

## Strengths
- It presents an intuitive yet novel approach to emergent language: transfer from a simpler to more complex task.
- The work is complete in itself but clearly points the way towards future directions.
- The structure of experiments provides the right kind of empirical support for the premise.

## Weaknesses
- The explanation of the core method, symbolic mapping, is unclear, which makes a difficult to evaluate exactly what is happening with the task transfer.
- The experiments use a small number of trials (3) in very small environments which raises concerns about the robustness of the observed effects.
- It seems as if refdis is selected as the metric for compositionality and then discarded in Section 4.2.2 Paragraph 3 with the claim that the communication protocol is compositional anyway despite the low refdis.

## Recommendation
I recommend "reject" in the current form of the paper because even with careful reading, I am not able to clearly understand what symbolic mapping is, a key component of the paper.
I would be happy to change my recommendation with revisions which clarify the explanation of symbolic mapping.
Less important, but running a greater number of trials and giving a rigorous qualitative explanation when compositionality does not line up with refdis would strengthen the paper.

## Justification
The paper up to Section 3.2 is well written and motivated, but Sections 3.2 and 3.3 (and Figure 2) are critical yet unclear.
An understanding of symbolic mapping bridges the conceptual gap from task transfer as a motivated method of doing emergent language to concrete instantiation of the agents in neural networks.
As a result, I cannot determine if the experimental results are trivial or informative.
Direct references to the location of issues in the paper are given in `Additional Comments`.

Specific criteria:
- Correctness: 1
    - Although I cannot judge the experimental design fully until I understand symbolic mapping, I believe that the experiments largely support the claims (i.e., 3).
    - A larger number of trials and a clearer analysis of refdis (in the context of the experiments -- it's formulation is clear) would be needed in addition for a 4.
- Technical Novelty and Significance: 3
- Empirical Novelty and Significance: 2
    - As mentioned above, a larger number of trials is needed.
    - Additionally, the empirical significance is lessened by the fact that only the only environments that work are of the scale $(3,3,3)$ and $(4,4)$.

## Questions
- What exactly is symbolic mapping? Equations and examples can always make things clearer.
- In what way is refdis an inadequate metric (if at all)? Is there a better alternative that could be used for this paper?

## Additional Comments
- `s1`: Contributions should explicitly stated in the introduction
- `s1`: "curriculum learning" should at least be mentioned somewhere since it is a common concept in machine learning
- `s3.1 discrimination game p1`: "is comprised of" -> "comprises"
- `s3.1 discrimination game p2`: use $m_a$ or $m^{(a)}$ instead of $m^a$ -- it took me a minute to realize it was indexing and not an exponent
- `s3.1 description game p1`: Is there any reason the simpler game is second? It would seem to make a bit more sense going first.
- `Figure 2`: This figure is very unclear insofar as the most important details are left unspecified.
    - Do the triangle and the symbolic mapping box have any relation? Or do they just happen to be adjacent?
    - How is the green triangle represented numerically, since it is presumably being concatenated with a vector?
    - What is the symbolic mapping box? What is coming out of it?
    - What are the inputs and outputs to the word bank?
    - Is the word bank generating multiple $s$'s? Why is that?
    - Why are there three copies of the speaking network and why do they appear to feed into the same softmax?
    - What is the purpose of the decision network and what $v_t$?
    - Is the robot face in the top right corner supposed to represent anything?
    - What is the rec network?
    - What is $m$ in the listening model?
- `s3.2 p1`: What does it mean to "sample symbols according to the probability given by the output of the sigmoid function"?
    Please be clear with what the symbols and the word bank are because they are critical to understanding symbolic mapping.
    What are the values in the vector and why do they correspond to "relevance"?
    What is the probability distribution being used here, is it a Bernoulli distribution for each element of the vector or a categorical distribution over the whole vector?
- `s3.2 p2`: Initializing with Gaussian noise seems a bit more typical and could maybe aid in robustness slightly. Just speculation on my part, though.
- `s3.2 p2`: "id" -> "identity"
- `s3.2 p3`: What does it mean for a symbol to be encoded into a one-hot embedding?
- `s3.2 p3`: What is the distribution $pi^i_\text{sp}$?
- `s3.2 p3`: What is the action $v_{t+1}$?
- `s3.2 p3`: "an one-hot" -> "a one-hot"
- `s3.2 p4`: "In description game" -> "In the description game"
- `s3.2 p4`: What does it mean for the speaker to be represented by the "bag-of-words model"?
    I am familiar with BoW in the context of text features, but I am not sure how it is applied here.
    Does that just mean that the listener receives a sum of vectors which the speaker has generate?
- `s3.2 p5`: 3 independent runs seems very small. On the order of 10s seems like a minimum while 100s is even better.
- `s3.4 Compositionality`: I see how positional disentanglement is inappropriate for this task, but BoW disentanglement from the same paper might work just fine -- it is very similar to referential disentanglement as given here. Furthermore, 15 minutes of notebook scribbles leads me to think that it might not suffer from the issue of redundant symbols for a given attribute (i.e., bosdis is still high unlike refdis).
- `s4.1 p1`: "unfixed" -> "not fixed"
- `s4.2.1 p2`: If the experiments are using a shared listener in the dialog setting, wouldn't this undermine the concept of refdis measuring _symmetry_? It seems like it would be measuring something more like consensus or inter-speaker agreement since the agents are not properly symmetric (i.e., independent speaking and hearing models).
- `Table 2`: What is the meaning of "protocol" and "mapping"?
- `s4.2.2 p3`: It seems like the point is being made that compositionality is still present despite the low refdis.
    This undermines the usefulness of the metric.
    If the claim is that the metric does not capture compositionality in this case, a well-illustrated qualitative argument should be made to this end.

summary_of_the_review: The paper presents a novel and motivated setting of task transfer and has the right start for experiments.
On the other hand, the explanation of symbolic mapping, central to linking the setting and experiments, is very unclear.
I intend to change my "reject" to an "accept" if the symbolic mapping is sufficiently clarified.
review_title: null

Meta-review: This manuscript presents a novel approach to learning a shared language between multiple agents.

In general, reviewers had difficulty understanding the symbolic mapping component. For such a critical part of the manuscript, questions by multiple reviewers were extremely basic, asking what symbolic mapping even is. Authors did clarify this in the discussion and updated the manuscript, but further improvements to the manuscript are warranted.

Reviewers had concerns about the novelty of the approach. Including being confused about whether this is just an application of curriculum learning. Reviewers were also concerned about the lack of ablations.

Reviewers also had concerns about the fact that this is a toy domain. Symbolic mapping as defined in the manuscript appears to be possible only for such toy domains. It fundamentally wouldn't scale to simple language games with real images. This significantly limits the scope of the work. More broadly, reviewers wanted to see symbolic mapping exercised much more. If this is a useful idea, they wanted to see the authors apply it to other domains.

Reviewers were confused about many other details in the manuscript. For example, about the fact that refdis is later discarded as a metric, which the authors answered is due to redundant symbols ("the symbolic mapping is not a highly compositional representation here because of the redundant symbols"). Why redundant symbols lead to less compositional representations seem unclear.

With significant additional improvements to the clarity of the manuscript, a demonstration of how symbolic mapping is useful in another domain, and additional experiments suggested by multiple reviewers this could be a strong submission in the future.
Decision: Reject