Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method to learn meaningful emergent languages with a continuous communication channel. The game consists of two agents that perform a unidirectional communication where the agents are trained using deep RL. The training follows an iterated procedure where one agent acts as a 'teacher', who has the actual meaning of the word, and the other as a 'student', who is trained at a given iteration by using information obtained form the teacher during training. The authors show that this type of grounding allows generalization and helps in learning a compositional language.
main_review: The paper explores a two-agent communication problem where the Sender agent sends a continuous waveform to the Listener who is tasked to reconstruct the original concept vector. There are two settings of concepts used, single concepts (or a flat vector) and double concepts (consists of a combination of two separate meanings).

However, there are many important details missing in the main text. It is not clear how big is the alphabet size that is used by the Speaker to construct the waveform. It is an important parameter as it affects the compositional generalization of the agents (Chaabouni et al. 2020, Kottur et al. 2017, Resnick et al. 2020, Li and Bowling 2019). 
Regardless of the alphabet size used, the problem setting used is quite 'toyish'. Even in the double concepts game the total possible combination of all concepts is 16. In most of the previous literature in this field, even the minimal setup uses 3 concepts of 5 types each (Chaabouni et la. 2020) and then results are validated using a more realistic problem setting either using real images as input or increasing the input space. The question here is would this method scale to larger problem settings.

The idea of using signalling to make one agent learn about the underlying concept and active listening to help the listener solve the given downstream task is already explored in Eccles et al 2019. Unfortunately, there has been no comparisons made with this work that seems to be the most relevant to the proposed method.

Moreover, many relevant are not cited in this work (for example the above works). I would encourage the authors to do a thorough literature survey of the recent work done in the space of compositionality and generalization in emergent languages.

The hypothesis around compositionality is only evaluated qualitatively. There have been some good progress done to compute domain independent quantitative compositionality metrics that are well established and grounded (Chaabouni et al. 2020, Lazaridou et al. 2018, Resnick et al. 2020, Andreas 2019). In addition, the notion of compositionality captured in the paper is more aligned to combinatorial generalization.

Good-enough compositional data augmentation. Andreas 2019  
Entropy minimization in emergent languages. Kharitonov et al. 2020  
Compositionality and generalization in emergent languages. Chaabouni et al. 2020  
Ease-of teaching and language structure from emergent communication. Li and Bowling 2019  
Capacity, bandwidth, and compositionality in emergent language learning. Resnick et al. 2020  
Natural language does not emerge ‘naturally’ in multi-agent dialog. Kottur et al. 2017  
Emergence of linguistic communication from referential games with symbolic and pixel input. Lazaridou et al. 2018  
summary_of_the_review: The paper explores an interesting research question to train agents that can communicate using a continuous communication channel and be able to solve the given downstream task thereby exhibiting compositional generalization. Although the underlying hypothesis is interesting, the idea is not completely novel and not evaluated extensively using well-established metrics in the literature. This work is a good preliminary work in investigating this research direction and I would encourage the authors to refine the evaluation and make explicit comparisons with prior work in the future submissions of this work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tackles the well-established signalling emergent language game with the innovation of using a continuous, audio-based channel for communication as opposed to the more typical discrete, symbolic channel.
This continuous channel takes the form of a sender generating phones which are then synthesized into an audio waveform; this audio is then passed to the receiver after being transformed into a spectrogram.
The experiments then evaluate some general characteristics of the aforementioned referential game.

The following empirical evaluations are performed:
- Basic referential game
- 2-attribute ref. game to test for generalization and compositionality
- Varying the noisiness of the channel to determine effects on generalization
- Grounding the sender in English and letting the receiver learn 
- Grounding the receiver in English and letting the sender learn 

The primary contributions of the paper are:
- Presenting a phone-based, continuous-channel referential game
- The effect of channel noise on generalization
- Ability to learn a grounded language from a pretrained conversation partner

main_review: ## Strengths
- The environment is novel and provides a good basis for studying certain traits of continuous-channel referential games.
- Excellent clarity and presentation of ideas

## Weaknesses
- The experiments provided do not analyze the unique characteristics of the environment introduced; instead, the experiments are similar to the typical gamut for a discrete symbol-based referential game.
- The analysis of compositionality given is insufficient.
    It appears that a single example was analyzed qualitatively.
    In the presence of definitive results, this might be acceptable, but in its absence, as in this paper, there needs to be quantitative analysis across multiple runs to demonstrate the robustness of the phenomenon.


## Recommendation
I recommend rejecting the paper.
While I appreciate the environment introduced and believe it to be promising, the experiments presented fail to highlight the novelty of the environment.
It is not the case that I find the experimental results inadequate, rather, the experiments run in the first place are generic and do not illustrate the points of interest with a continuous-channel referential game.
It is possible that I could be convinced otherwise as I truly appreciate the environment, but I would need to see an argument as to how the paper properly supports its introduction of a new environment.


## Justification
As my primary critique concerns the experiments, I will address each individually mentioning any deficiencies and potential improvements.
- unconstrained, single-concept:
    This is adquate to demonstrate that a minimal environment works (as stated in the paper).
- unconstrained, multi-concept:
    This needs a direct comparison to a traditional discrete-channel referential game.
    Thus, the experiment could answer questions such as:
    - Does compositionality emerge at the same rate in continuous- and discrete-channel games?
    - What is it about an audio channel model that makes compositionality easier or more difficult to learn?
    - Does the interaction between various phonemes restrict the means by which compositional codes can be communicated (e.g., /aio/ and /aeo/ are harder to distinguish than /abo/ and /ato/)?
- multi-concept and noise:
    The noise you add to the channel has a number of different components; there should be an ablation study to illustrate the effects of these components.
    If the experiment does not peer inside the structure of the noise and treats it as a gray box instead, it is not much different than a discrete-channel environment where random edits are made to the message.
- grounded language learning:
    In both of these experiments, there is analysis provided on what aspects of the audio-based communication channel make the problem harder, easier, or just different from the same experiment with a discrete channel.
    For example, experiments could investigate if using words which are phonologically similar (e.g., "boat" and "moat") is harder to distinguish than dissimilar words.
    Or an experiment could address if using the phonotactics from a natural language results is more effective than using a randomly selected phonemes due to necessity for natural languages to have acoustically distinct words.


Specific criteria:
- Correctness: 4
    - The claims are supported, but I do not think the claims go far enough (e.g., "noise has an effect on generalization" is claimed when instead what that effect is needs to be characterized).
- Technical Novelty and Significance: 3
- Empirical Novelty and Significance: 1

## Questions
_What_ is unique about the environment has been presented well, so my primary question is _how_ is it unique?
In other words, I see that the structure of the environment is unique, but what are the consequences which this structural difference begets?

## Additional Comments
- `s1`: Introduction is paced well
- `s2p2`: what is T? Is it just every tick of the audio signal?
- `s3.2p2`: What is the specific reasoning for using a bidirectional GRU? Using a model capable of streaming would make sense from the point of view of inductive biases. 
- `s3.2 p2`: Due to the argmax, is it the case that the "confidence" of the predictions is not used in optimization? This seems like it would artificially make optimization harder.
- `Table 2`: Are these values computed over a single or multiple runs (in which case include stddev/confidence intervals).
- `s4.3 Findings p4`: "the channel ... influences generalisation" -- Without further explication, this potentially interesting point does not gain any traction.

summary_of_the_review: The paper presents a well-designed, novel referential game where the communication channel is continuous and audio-based.
The paper fails to demonstrate with its experiments the actual effects of using this continuous channel.
As a result, my decision is "reject"; I think a new set of experiments highlighting the novel setup will yield a strong paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes some experiments to simulate language acquisition in which a speaker talks to a listener over a continuous voice channel.  The speaker learns to generate a phone sequence for each input concept that it wants to convey and the listener learns to recognise the input phone sequences to decode the embedded concept(s).  The speaker and listener use RL to learn. When the listener correctly identifies a concept both the speaker and listener receive a positive reward.  The authors claim that the system demonstrates the ability to exploit composition and generalisation. 

main_review: This paper describes some experiments to simulate language acquisition in which a speaker talks to a listener over a continuous voice channel.  The speaker uses a GRU conditioned by one or more concepts to generate a sequence of sounds selected from an inventory of 160 English phonemes released by eSpeak.  The listener applies a convolutional net to the spectrogram of received sounds, then decodes sequential frames to generate a distribution over phonemes from which an argmax generates the decoded concept.   Learning uses a simple 1-step DQN.  The channel between speaker and listener has additive noise and time/pitch shifts applied.

The experiments include the unconstrained transmission of 1 and 2 concepts and the same but constrained so that either the speaker knows English words or the listener knows how to recognise English words.  Claims made include the ability to use composition and generalisation when moving from 1 to 2 concept transmission.

There are many things that I do not understand about this paper.

Firstly, when the speaker is generating a sequence of M phonemes, each phone is regarded as an action but the optimisation treats this an M-armed bandit problem and simply averages the Q values over all actions.  Why is this not treated as an M-step trajectory and solved using the standard DQN optimisation?

Secondly, it appears that all phoneme sequences are exactly M long, rather than a maximum of M long (see eg Table 1).  Is this true?   Wouldn't it be better to allow variable length and perhaps use the reward to encourage brevity.

Thirdly, how are the input concepts coded?  The examples suggest that each concept is coded as a 1-hot vector.  For multiple concepts, the natural assumption would be that the concepts are simply concatenated.  If this is the case, how does the listener work since it is only capable of forming a distribution over 1 set of concepts.  How does the listener distinguish multiple component concepts?

The claim that the results in Table 1 demonstrate the ability to learn a compositional language protocol are unconvincing.  Quoting 0.25 as the chance mean reward ignores the fact that the agents could simply recognise one of the concepts whenever the combination was unseen.

Adding noise will often improve the robustness of a classifier, why is the generalisation implied by Table 2 any different?  What is the statistical significance of the numbers in this table?

On page 9, it is claimed that using DTW for the 2 word listener case is impractical.  Why is this? If the Listener only has to consider a vocabulary of 50 words, there are only 2500 two word combinations which should be tractable for a DTW style recogniser with pruning.

summary_of_the_review: I am not convinced by the motivation of this paper, the presentation is poor and the experimental results are unconvincing.  The experimental design appears to involve some unnecessary approximations.  The paper claims to be an advance on Gao, but Gao addresses the problem of segmenting a speech stream into words when the listener does not have a pronouncing dictionary.  Here the use of the continuous channel appears to me to be an unnecessarily complicated way of adding channel noise to what is otherwise a discrete system.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to move in the direction of emerging oral communication, instead of written communication. They create an architecture where the sender agent first outputs a string of symbols, which map statically to a fixed table of phonems. The sender uses an off-the-shelf pre-trained text-to-speech synthesizer to convert these phonemes into sounds. The sounds pass through a channel with Gaussian white noise added, along with Gaussian-sampled time-shift and pitch-shift. The receiver converts the input sounds into a mel-spectrogram, which is a sequence of vectors over time representing the frequency distribution. The receiver passes these vectors into a CNN, an RNN, and finally a linear classifier.

The paper motivates using the pre-trained speech-to-text synthesizer as a necessary initial baby-step, to keep the problem tractable in the context of RL.

The paper uses an encoder-decoder task, where the input is a symbolic concept comprising one dimension of size 16, or two dimensions of size 4. The output is a prediction of the original input concept.

The paper shows in a first experiment that their model can learn to predict the input concept correctly with a meaning space of 1 dimension of size 16. Then, in a second experiment, the concepts are extended to two dimensions, each of size 4. In addition, in this second experiment, the effect of communication channel noise is measured: adding noise to the channel decreases performance on concepts seen during training, but improves performance on a holdout set of novel input concepts.

In the final experiment, where the speaker or the listener are fixed, and grounded in English. In this case, the other agent will learn a language that is close to English.
main_review: Good points:
- pushes emergent communication in a new direction: that of oral communications

Bad points:
- some key missed references, i.e. Foerster et al 2016 section 6.4 uses a continuous channel, and finds that adding noise to the channel is necessary in order to effectively 'discretize' the outputs. I also feel that Lazaridou et al 2018 is a better reference than Chaabouni et al 2020, since Lazaridou et al 2018 was more or less the first work to use a referential task in the recent wave of emergent communication literature.
- a lot of the space in the paper appears to me to be used to describe relatively vanilla concepts, such as the architecture of the sender and receiver RNN models, what a replay buffer is, and some of the earlier experiments are I feel more like 'debugging'/smoke-test experiments, that show the architecture learns
- I wasn't sure what I was supposed to learn from the grounded experiments. I felt that it was unsurprising that grounding one agent in English would cause the other agent to learn a language that learns English
- the text to speech synthesize is pretrained, which goes I feel against the spirit of emergent communication. I understand that motivation for this was provided, and that motivation makes sense to me, but listing this for completion
- the meaning/object spaces being used are very small (1 dimension, of size 16; or two dimensions, each of size 4; for example)
- code is not provided with the submission
- I didn't come away with any sense of 'oh that was interesting/surprising; I learned something'

## Notes

Notes I made as I read through the paper:

### Abstract

I like the idea of using an accoustic channel :) It's something I've been intending to write a paper on sometime, just didn't get around to it yet :)

It seems clear to me that spoken language preceded written language, so starting with accoustic language, for emergent communication, seems like a great idea to me :)

I'm a little surprised/unclear why we map from symbols to accoustic. why not just keep everything continuous?

noisy communication channel being needed for compositionality and/or generalization has been shown in a few previous works.

### 1. Introduction

Mising reference "Learning to Communicate with Deep Multi-Agent Reinforcement Learning", Foerster et al 2016, which in section 6.4, uses a continuous channel, and discovers that by adding noise, the channel effectively becomes discretized.

Citing Chaabouni et al 2020 for 'referential signaling game' seems I feel not the original reference. "Emergence of linguistic communication from referential games with symbolic and pixel input", Lazaridou et al, 2018 was one of the first works to bring the idea of a referential game into emergent communication domain. Prior to that, the Lewis signaling game, Lewis, 1969, is, as far as I know, the original reference.

Why use deep Q-learning? Not that it's not a good choice, but curious why you choose this, when the vast majority of emergent communication works use simple REINFORCE (or Gumbel).

### 2. Environment

Using a pre-trained text-to-speech synthesizer seems to go against the goals of 'emergent' communication?

The explanation for why this is (later in the section) does make sense to me though. Baby steps :)

### 3. Learning to speak and hear using RL

A large amount of space (~1.5 pages) is being spent describing what appear to be quite vanilla sender and receiver models. I'm not suggesting that the models should be made 'fancier', but that that space could perhaps be filled with additional experiments, or analysis, etc.

### 4. Experiments

I feel discussing eSpeak vs Festival gives a sort of 'engineer-y' feel to the paper; and could be omitted.

#### 4.2 Unconstrainted communication of single concepts

I realize at this point that somehow I missed what are the actual input data. Ok, skimming right back to page 3, it looks like there is only a single meaning dimension, of size N? This could represent eg 'color'. From 4.2, it looks like N is 16, though it's not stated clearly. This is I feel an extremely small meaning/concept/object space.

On the whole, 4.2 seems like a superfluous debugging section, which doesn't really provide any information/signal, and could be omitted.

#### 4.3 Unconstrained communication generalizing to multiple concepts

I was briefly excited by the mention of Kirby's ILM, but I feel that the citation of Kirby's ILM is superfluous here: the experiment is simply changing from a 1-dimensional meaning space of size 16, to a 2-dimensional meaning size, where each dimensino is of size 4.

Again, this experiment seems to me to be quite vanilla.

I feel that in addition to providing a table of utterances to eye-ball, it could be slightly more rigorous to use objective measures of compositionality such as topological similarity.

I feel that the effect of channel noise in Table 2 appears quite weak to me. In addition, it could be argued that the noise is simply providing training time regularization, which we see as a drop in performance of the training codes, rather than leading to a qualitative shift in the emergent communication, as Foerster et al 2016 showed, in their section 6.4.

#### 4.4 Grounding emergent communication

I feel that 'grounded emergent communication' is a contradiction in terms: either the language is grounded, or it emerges. If there is any grounding provided, then it is a grounded task, not an emergent task, I feel?

I'm not really sure of the purpose of this set of experiments. The various agents are being grounded in english in various ways, and end up outputing some English-like language. I'm not really sure what I can learn from this?

I feel that using English-language grounding in a paper positioning itself as emergent communication is somewhat risky. With sufficient motivation and analysis, I don't have any fundamental objection to doing this though. Preference to come away with some sense of 'that was interesting; I learned something'. This might mean making the experiments challenging enough so that they do *not* actually work, and then digging into why, and what we can learn from that, perhaps?
summary_of_the_review: On the whole, I like that the work pushes towards moving emergent communication into a new direction of oral communication. I feel that this work as it stands is an excellent start in this direction. However, I feel that the experiments and analysis could be extended somewhat, into scenarios where it feels somewhat less predictable what the outcomes will be. For example, what happens if the meaning space is larger? Does that work? If it doesn't work, how does it fail and why?
review_title: null

Meta-review: While a lot of previous work on emergent communications studies discrete protocols, this work explores a continuous and audio-based channel for learning multi-agent communication. Reviewers have commented positively on the novelty of the topic. At the same time, there are a number of concerns raised with respect to experimental design and implementation (6auy) and general approach of the topic which, as reviewers t576 and 42Xh point,  doesn't really go deep into the analysis and understanding of the particular experimental setup and findings. So, unfortunately as the papers stands I cannot recommend acceptance at this time. However, given that continuous communication in emergent communication is a somewhat overlooked topic, I would encourage the authors to use the reviewers' feedback and strengthen their manuscript.
Decision: Reject