Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channel
Authors: Kevin Michael Eloff
Arnu Pretorius
Okko Räsänen
Herman Arnold Engelbrecht
Herman Kamper
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning
language acquisition
emergent communication
acoustic communication
continuous signalling

Abstract: While multi-agent reinforcement learning has been used as an effective means to study emergent communication between agents, existing work has focused almost exclusively on communication with discrete symbols. Human communication often takes place (and emerged) over a continuous acoustic channel; human infants acquire language in large part through continuous signalling with their caregivers. We therefore ask: Are we able to observe emergent language between agents with a continuous communication channel trained through reinforcement learning? And if so, what is the impact of channel characteristics on the emerging language? We propose an environment and training methodology to serve as a means to carry out an initial exploration of these questions. We use a simple messaging environment where a "speaker" agent needs to convey a concept to a "listener". The Speaker is equipped with a vocoder that maps symbols to a continuous waveform, this is passed over a lossy continuous channel, and the Listener needs to map the continuous signal to the concept. Using deep Q-learning, we show that basic compositionality emerges in the learned language representations. We find that noise is essential in the communication channel when conveying unseen concept combinations. And we show that we can ground the emergent communication by introducing a caregiver predisposed to "hearing" or "speaking" English. Finally, we describe how our platform serves as a starting point for future work that uses a combination of deep reinforcement learning and multi-agent systems to study our questions of continuous signalling in language learning and emergence.

Under review as a conference paper at ICLR 2022
TOWARDS LEARNING TO SPEAK AND HEAR THROUGH MULTI-AGENT COMMUNICATION OVER A CONTINUOUS ACOUSTIC CHANNEL
Anonymous authors Paper under double-bling blind review
ABSTRACT
While multi-agent reinforcement learning has been used as an effective means to study emergent communication between agents, existing work has focused almost exclusively on communication with discrete symbols. Human communication often takes place (and emerged) over a continuous acoustic channel; human infants acquire language in large part through continuous signalling with their caregivers. We therefore ask: Are we able to observe emergent language between agents with continuous communication channel trained through reinforcement learning? And if so, what is the impact of channel characteristics on the emerging language? We propose an environment and training methodology serve as means carry out an initial exploration of these questions. We use a simple messaging environment where "speaker" agent needs convey concept is equipped with a vocoder that maps symbols to a continuous waveform, this is passed over a lossy continuous channel, and the Listener needs map the continuous signal to the concept. Using deep Q-learning, we show that basic compositionality emerges in the learned language representations. We find that noise is essential in the communication channel when conveying unseen concept combinations. And we show that we can ground the emergent communication by introducing caregiver predisposed to "hearing" or "speaking" English. Finally, we describe how our platform serves as a starting point for future work that uses a combination of deep reinforcement learning and multi-agent systems study questions of continuous signalling in language learning and emergence.
"listener". The Speaker
INTRODUCTION
Reinforcement learning (RL) increasingly being used as a tool to study language emergence
(Mordatch & Abbeel 2017 Lazaridou al.) 2018 Eccles al.) 2019: Chaabouni et al. 2020
Lazaridou & Baroni 2020). By allowing multiple agents to communicate with each other while solving common task, communication protocol needs to be established. The resulting protocol can be studied to see if adheres to properties human language, such as compositionality (Kirby 2001] Geffen Lan al. 2020 Andreas 2020 Resnick et al. 22200). The tasks and environments themselves can also be studied, to see what types of constraints are necessary for human-like language to emerge (tteels 1997). Referential games often used for this purpose (Kajic al., 2020 (Havrylov & Titov 2017 Yuan et al., 2020). While these studies open up the possibility of using computational models to investigate how language emerged and how language is acquired through interaction with an environment and other agents, most RI. studies consider communication using discrete symbols.
Spoken language instead operates and presumably emerged over a continuous acoustic channel. Human infants acquire their native language by being exposed to speech audio in their environ- ments (Kuhl (2055): by interacting and communicating with their caregivers using continuous signals, infants can observe the consequences of their communicative attempts (e.g. through parental re- sponses) that may guide the process of language acquisition (see e.g. Howard & Messum (2014) for discussion). Continuous signalling is challenging since agent needs to be able to deal with different acoustic environments and noise introduced by the lossy channel. These intricacies are lost when agents communicate directly with discrete symbols. This raises the question: Are we able

---

Under review as a conference paper at ICLR 2022
Lossy communication channel
Speaker Agent
f(-)
Listener Agent
Figure I: Environment setup showing a Speaker communicating to Listener over a lossy acoustic communication channel f.
to observe emergent language between agents with continuous communication channel, trained through RL? This paper is our first step towards answering this larger research question.
Earlier work has considered models of human language acquisition using continuous signalling between a simulated infant and caregiver (Oudeyer) 2005; Steels & Belpaeme, 2005). But these models often rely heuristic approaches and older neural modelling techniques, making them difficult to extend; e.g. it isn't 't easy to directly incorporate other environmental rewards or interactions between multiple agents. More recent RI. approaches would make this possible, but noted, has mainly focused on discrete communication. Our work here tries to bridge the disconnect between recent contributions in multi-agent reinforcement learning (MARL) and carlier literature in language acquisition and modelling (Moulin-Frier & Oudeyer 2021).
One recent exception which do continuous signalling within a modern RL framework the work of Gao et al. (2020). In their setup, Student agent is exposed to a large collection of unlabelled speech audio, from which builds up a dictionary of possible spoken words. The Student can then select segmented words from its dictionary to play back to Teacher, which uses trained automatic speech recognition (ASR) model classify the words and execute a movement command a discrete environment. The Student is then awarded for moving towards a goal position. We also propose Student-Teacher Teacher setup, but importantly, our agents can generate their own unique audio waveforms rather than just segmenting and repeating words exactly from past observations. Moreover, in our setup an agent is not required to use a pretrained ASR system for "listening".
Concretely, propose the environment illustrated in Figure I which is extension of referential signalling game used in several previous studies (Lewis) 1969: Lazaridou et al.] 2018 8 Chaabouni al. 2020 Rita et al. 2020). Here represents communicate to Listener agent. Taking this concept as input, the Speaker produces a waveform as output, which passes over a (potentially lossy) acoustic channel. The Listener "hears" the utterance from the speaker. Taking the waveform as input, the Speaker produces output 8. This output is the Listener's interpretation of the concept that the Speaker agent tried to communicate. The agents must develop a common communication protocol such that s = s. This process encapsulates one of the core goals of human language: conveying meaning through communication (Dor 2014). To train the agents, we use deep Q-learning (Mnih et al. 2013).
of set of possible concepts the Speaker must
Our bigger goal to explore the question of whether and how language emerges when using RI. to train agents that communicate via continuous acoustic signals. Our proposed environment and training methodology serves as a means to perform such an exploration, and the goal of the paper is to showcase the capabilities of the platform. Concretely, we illustrate that valid protocol is established between agents communicating freely, that basic compositionality emerges when agents need to communicate combination of two concepts, that channel noise affects generalisation, and that one agent will act accordingly when the other is made to "hear" or "speak" English. At the end of the paper, we also discuss questions that be tackled in the future using the groundwork laid here.
2

---

Under review as a conference paper at ICLR 2022
phone sequence
andio waveform
mel-stectoggamm
Whas
X
daun
Speaker Ageni
Synthesiser
Channel
Listener Agent
Q-network Dictionary Lookup
eSpeak Festival
Noise
Q-network
Time Pitch warping
DTW
masking
Figure 2: Example interaction of each component and the environment in single round.
2 ENVIRONMENT
We base our environment on the referential signaling game from Chaabouni et al. (2020) and Rita et al. (020) which itself is based on Lewis (1969) and Lazaridou al. (2018) -weee sender must convey a message to a receiver. In our case, communication takes place between Speaker and a Listener over a continuous acoustic channel, instead sending symbols directly (Figure []] each game round, Speaker agent is tasked with conveying single concept. The Speaker needs to explain this concept using a speech waveform which transmitted over a noisy communication channel, and then received by Listener agent. The Listener agent then classifies its understanding of the Speaker's 's concept. If the Speaker's 's target concept matches the classified concept from the Listener, the agents rewarded. The Speaker then presented with another concept and the cycle repeats.
Formally, in each episode, the environment generates 8, a one-hot encoded vector representing one of N target concepts from set S. The Speaker receives and generates sequence of phones c= (c. CM), each G E P representing phone from a predefined phonetic alphabet P. The phone sequence then converted into waveform Wraw. audio signal sampled 16 kHz. For this we use trained ext-to-speech model (Black & Lenzo 2000: Duddington, 2006). A channel noise function then applied the generated waveform, and the result win (( is presented as input to the Listener. The Listener converts the input waveform to mel-scale spectrogram: a sequence of vectors over time representing the frequency content of an audio signal sealed to mimic human frequency perception (Davis & Mermelstein) 1980p. Taking the mel-spectrogram sequence X = (x) #2..... ØT) T' acoustic frames as input, the Listener agent outputs vector representing its predicted concept. The agents are both rewarded if the predicted word is equal to the target word s = &.
c2,...,
To make the environment a bit more concrete, we present brief example in Figure 2 For illustrative purposes, consider a set of concepts S {up, down, left, right). The state representation for down would be
[0. 1, O, TO A possible phone sequence generated by the Speaker would be (d, a, ú, n, 
> This would be synthesised, passed through the channel, and then be interpreted by the Listener agent. If the Listener's prediction is á = [0, 1, 0, 0] then it selected the correct concept of down. The environment would then reward the agents accordingly:
if & {
r=
(1)
otherwise
In our environment we have modelled the task of the Speaker agent as discrete problem. Despite this, the combination of both agents and their environment is a continuous communication task; in our communication channel, we apply continuous signal transforms which can be motivated by real acoustic environments. The Listener also needs to take in and process a noisy acoustic signal. It is true that Speaker outputs discrete sequence; what we have done here to equip the Speaker with
and </> respectively represent the start-of-sequence and end-of-sequence tokens.
3

---

Under review as a conference paper at ICLR 2022
articulatory capabilitiee that these do not need to be learned by the model. There studies that consider how articulation can be learned (Howard Messum 2014; Asada 2016 Rasilo & Räsänen 2017), but none of these do in RL environment, rather using form of imitation learning. In Section | we discuss how future work could consider learning the articulation process itself within environment, and the challenges involved in doing so.
LEARNING TO SPEAK AND HEAR USING RL
To train our agents, we use deep Q-learning (Mnih et al. 2013). For the Speaker agent, this means predicting the action-value of phone sequences. The Listener agent predicts the value of selecting each classification target & E S.
3.1 SPEAKER MODEL
The Speaker agent is tasked with generating a sequence of phones describing concept or idea. The model architecture is shown in Figure 3 The target concept is represented by the one-hot input state s. We use gated recurrent unit (GRU) based sequence generation as the core of the Speaker agent, which generates sequence of Q-values, a distribution over phones per output-step from I to M. The input state s is embedded as the initial hidden state hg of the GRU. The output phone of each GRU layer embedded input to the next GRU layer We also make use of -art-of-sequence (SOS) and end-of-sequence (EOS) tokens, <8> and 
 respectively, appended the phone-set. These allow the Speaker to generate arbitrary length phone sequences up to maximum length M.
3.2 LISTENER MODEL
The Listener agent may be viewed as classification task with the full model architecture illustrated in Figure 4 The model is roughly based on (Amodei al. 2016). Given input mel-spectrogram X, the Listener generates a set of state-action values. These action-valuce represent the expected reward for each classification vector 8.
We first apply a set of convolutional layers over the input mel-spectrogram, keeping the size of the time-axis consistent throughout. We then flatten the convolution outputs the filters and feature axis, resulting in a single vector per time step. We process each vector through a bidirectional GRU, feeding the final hidden state through linear layer arrive our final action-value predictions. An argmax of these action-valle gives greedy prediction for &.
No gradients flow through argmax: connection indicates to network which phone was selected at the previous GRU step.
<
C2
CM
argmaax
argmax
argmax
logits
logits
logils
ho
har
GRU
GRU
GRU
embedding
embeddiig
embedding
Figure 3: The Speaker agent generates an arbitrary length sequence of action- values given an input concept represented by
4

---

Under review as a conference paper at ICLR 2022
mel-spectoogram
fatten
batch, sime]
batch, s,fsiters,
Sime]
00
GRU
lincar
argmax
batch, 2iltbe festere, Sima
[batch, bustze]
jaath, n. concepts]
Figure 4: The Listener agent Q-network generates action-values given an input mel-speetrogram
3.3 DEEP Q-LEARNING
The Q-network of the Speaker agent generates sequence of phones in every communication round until the EOS token is reached. The sequence of phones may be seen as predicting an action sequence per environment step, while standard RL generally only predicts single action per step. To train such a Q-network, we therefore modify the general gradient-descent update equation from Sutton & Barto 1998). Since only have single communication round, update the model parameters as follows:
1 M Σ
- 0
am (S, A: (q(S,A:0),
where the reward is given in (T. S is the environment state, A is the action, is the learning rate, and q = (q1, 92.-... .as). For the Speaker, 9m the value of performing the action Cm at output m. For the Speaker, the environment state would be the desired concept S and the actions would be A = c (2,, CM), the output of the network in Figure
The Listener also trained using (2). but here this corresponds the more standard case where the agent produces a single action, i.c. M = 1. Concretely, for the Listener this action is A = 8, the output of the network Figure The Listener's environment the mel-spectrogram S X. The Speaker and Listener each have their own independent learner and replay buffer (Mnih et al. 2013). replay buffer is a storage buffer that keeps track of the observed environment states, actions and rewards. The replay buffer is then sampled when updating the agent's Q-networks through gradient descent with (2). We may see this two-agent environment as multi-agent deep Q-learning [Tampuu) et al. 2017), and therefore have to take careful consideration of the non-stationary replay bufter: we limit the maximum replay buffer size to twice the batch size. This ensures that the agent learns only from its most recent experiences.
EXPERIMENTS
4.1 IMPLEMENTATION
The lossy communication channel has Gaussian white noise with signal-to-noiss ratio (SNR) of 30 dB, unless otherwise stated. During training, the channel applies Gaussian cusssian-samped time stretch and pitch shift using Librosa (McFee al. 2021), with variance 0.4 and 0.3, respectively. The channel also masks up to 15% of the mel-spectrogram time-axis during training. We train our agents with e-greedy exploration, where is decayed exponentially from 0.1 to 0 over the training steps.
We use eSpeak (Duddington) 2006) as our speech synthesiser. eSpeak is a parametric text-to-speech software package that uses formant synthesis to generate audio from phone sequences. Festival (Black)

---

Under review as a conference paper at ICLR 2022
1.0
1.0
Evaluation Reward
Evaluation Reward
0.6
0.6
0.2
0.2
comm.
acoustic comm.
discrete comm.
un.seen codes
unseen codes
0.0
0.0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Training Episode
Training Episode
(a) Mcan evaluation reward of the Listener agent interpreting a single concept over 20 runs.
(b) Mean evaluation reward of the Listener agent interpreting two concepts each round.
Figure 5: Results for unconstrained communication. The agents are evaluated every 100 training episodes over 20 runs. Shading indicates the bootstrapped 95% confidence interval.
Lenzo 2000) was also tested, although eSpeak is favoured for its simpler phone scheme and multi- language support. We use eSpeak's 's full English phone-set of 164 unique phones and phonetic modifiers. The standard maximum number of phones the Speaker is allowed generate in each communication round is M = 5, including the FOS token. All GRUs have 2 layers with a hidden layer size of 256. All Speaker agent embeddings (Section 3.1p are also 256-dimensional. The Listener (Section 3.2) uses 4 convolutional layers, each with 64 filters and a kernel width and height of 3. The input to the first convolutional layer is sequence of 128-dimensional mel-spectrogram vectors extracted every 32 ms. We apply zero padding of size at each layer to retain the input dimensions. Additional experimental details are given Appendix A
4.2 UNCONSTRAINED COMMUNICATION OF SINGLE CONCEPTS
Motivation We first verify that the environment works expected and that valid communication protocol emerges when no constraints are applied to the agents.
Setup The Speaker and Listener agents trained simultancously here, described in Section 3.3 The agents are tasked with communicating 16 unique concepts. We compare our acoustic commu- nication to discrete baseline based on RIAL (Foerster al., 2017). In this setup, the CNN of the Listener agent is replaced by an embedding network, allowing the discrete symbols of the Speaker to be directly interpreted by the Listener. The Speaker's 's discrete alphabet size of setup is equal the phonetic alphabet size of 164. Improvements have been made to RIAL e.g. (Eccles et al. 2019 Chaabouni et al. 2020) although RAIL itself proves sufficient as a comparison to our proposed acoustic communication setting.
Findings Figure 5a shows the mean evaluation reward of the Listener agent over training steps. (This is also an indication of the Speaker's performance, since without successful coordination between the two agents, no reward given to either.) The agents achieve final mean reward 0.917 after 5000 training episodes, successfully developing valid communication protocol for roughly 15 out of the total communication which reaches mean evaluation reward of 0.959. What does the communication sound like? Since there are no constraints placed on communication, the agents can easily coordinate use arbitrary phone sequences communicate distinct concepts. The interested reader listen We next consider a more involved setting in order to study composition and
concepts This is comparable to the performance
the purely discrete
to generated samples. 4l generalisation.
4.3 UNCONSTRAINED COMMUNICATION GENERALISING TO MULTIPLE CONCEPTS
To study composition and generalisation, we perform experiment based on (iiby 200Tp. They used an iterative language model (ILM) to convey two separate meanings (a and b) in single string. This ILM was able generate structured compositional mappings from meaning to strings. For example, in one result they found og + q and by - da. The combination the two
Motivation
"The maximum evaluation reward in all experiments is 1.0.
Audio samples for all experiments are available http:://ccc2022-15444...thubbbb.

---

Under review as a conference paper at ICLR 2022
Table 1: Mean evaluation reward of the two- concept experiments with varying channel noise. The results for no lossy communica- tion channel is also shown. The 95% confi- dence for all values falls within 0.01.
Table 2: Output sequences from a trained Speaker. Each entry corresponds to combination of two con- cepts, $1 and $2, respectively. The bold combinations were unseen during training.
Average SNR (dB)
Training Codes
Unseen
Codes
no channel
0.966
0.386
nnyyyx
ÖsAgge
Isspxx
mnssss
40
0.878
0.389
aORRCC
WWWXXX
888339
30
0.931
0.402
20
0.895
0.413
ijA8n
ÖpAAj:
Öwppcx
enysss
10
0.731
0.361
HA::
yödp:
yss!:
0
0.654
0.366
meanings was therefore (ao, bo) + qda. Similarly, (a),b)) + bguda with ai + bgu. Motivated by this, we try to test the generalisation capabilities in continuous signalling in our environment.
Setup Rather than conveying a single concept in each episode, we now ask the agents to convey two concepts. The target concept and predicted concept & now become $1, $2 and $1. $2, respectively. We also make sure that some concept combinations are never seen during training. We then see if the agents are still able to convey these concept combinations at test time, indicating how well the agents generalise to novel inputs. The reward model is adjusted accordingly, with the agents receiving 0.5 for each concept correctly identified by the Listener. Here 81 can take on 4 distinct concepts while 82 can take on another 4 concepts. Out of the 16 total combinations, we make sure that 4 are never seen during training. The unseen combinations are chosen such that there remains an even distribution of individual unseen concepts. We also increase the maximum phone length to M = 7. To encourage compositionality (Kottur et al., 2017), we limit the size of the phonetic alphabet to 16.
As an example, you can think of 81 as indicating an item from the set of concepts Si = {up, down, leftt, right) while $2 indicates and item from S2 [fast, medium, regular, slow) and we want the agents to communicate concept combinations such as up+fast. Some combinations such as righ never given the target concept combination during training (but e.g. right+fast and seow+ would be), and we see if the agents can generalise to these unseen combinations at test time and how they do it.
The results are shown in Figure 5b We see the mean evaluation reward of the acoustic Listener agent reaches 0.931 on the training concept combinations. This is slightly lower than the discrete case which reaches a mean of 0.965. The acoustic communication agents achieve a mean evaluation reward of 0.402 on the unseen combinations, indicating that they are usually able to successfully communicate at least one of the two concepts. The discrete agents do marginally better on unseen combinations, with slightly higher variance. The chance-level baseline for this task would receive mean reward of 0.25. The performance on the unseen combinations is thus better than random.
Findings: Quantitative
Table shows the mean evaluation reward of the same two-concept experiments, but now with The goal here is to evaluate how the channel
varying degrees of channel noise expressed in SNR 5 influences the generalisation of the agents to unseen input combinations. In the no-channel case, the Speaker output is directly input to the Listener agent, without any time stretching or pitch shifting. The no channel case does best on the training codes as expected, but does not generalise as well to unseen input combinations. We find that increasing channel noise decreases the performance of the training codes and increases generalisation performance on unseen codes, up to a point where both decrease. This is an carly indication that the channel specifically influences generalisation.
Lazaridou al. (2018) reported the structural similarity of the emergent communication in terms of Spearman correlation between the input and message space, known as topographic similarity or topism (Brighton& & Kirby 2006). Chaabouni et al.) (2020) extended this metric by introducing two new metrics. Positional disentanglement (poxdis) measures the positional contribution of symbols to
'The SNR is calculated based on the average energy in signal generated by eSpeak.
7

---

Under review as a conference paper at ICLR 2022
Table 3: Compositionality metries of the unconstrained multi-concept Speaker agents. The mean evaluation metrics and 95% confidence bounds are shown
topism
posdis
bosdis
acoustic comm.
0.265 (+0.041)
0.103 (=0.)15)
0.116 (+0.018)
discrete comm.
0.244 (=0.032)
0.087 (=0.017)
0.118 (+0.017)
meaning. Bag-of-symbols disentanglement (bosdis) measures distinct symbol meaning but does so in a permutation-ine inviiiiann language way. We record all 3 metrics for the case where the average SNR is 30 dB, taking measurements between the input space and the sequence of discrete phones. The results are shown in Table 3 For topism, we average 0.265, which is comparable to the results of (Lazaridou al. 2018). For posdis and bosdis, average 0.103 and 0.116, respectively. This falls within the lower end of the results of (Chaabouni et al. 2020). All three metrics yield similar results for both acoustic and discrete communication.
Findings: Qualitative Table shows examples of the sequences produced by trained Speaker agent for each concept combination, with the phone units written using the international phonetic alphabet. Ideally, we would want each row and each column to affect the phonetic sequence in unique way. This would indicate that the agents have learnt a compositional language protocol, combining phonetic segments together to create a sequence in which the Listener can distinguish the individual component concepts. We see this type of behaviour to some degree in our Speaker samples, such the [x] phones for 81 or the repeated [s] sound when 81 = 3. This indicates at least some level of compositionality in the learned communication. More qualitatively, the realisation from eSpeak of [X] sounds very similar to [n] for $2 0. (We refer the reader the sample page linked Section
The bold phone sequences in Table 2 were unseen during training. The agents correctly classified one combination (81, 82 = 3, 0) out of the 4 unseen combinations. For the other unseen combinations, the agents correctly predicted least 81 or #2 correctly. These sequences also show some degree of compositionality. such as the [j4] sequence where 81 = O. We should note that the agents are never specifically encouraged to develop any sort of compositionality in this experiment. They could, for example, use a unique single phone for each of the 16 concept combinations.
4.4 GROUNDING EMERGENT COMMUNICATION
Motivation Although the Speaker uses an English phone-set, up to this point there has been no reason for the agents to actually learn to use English words to convey the concepts. In this subsection, either the Speaker or Listener is predisposed to speak or hear English words, and the other agent needs to act accordingly. One scientific motivation for this setting is that it can be used to study how an infant learns language from caregiver (Kuhl 2005). To study this computationally. several studies have looked at cognitive models of early vocal development through infant-caregiver interaction; Asada (2016) provides comprehensive review. Most of these studies, however, considered the problem of learning to vocalise (Howard & Messum, 2014: Moulin-Friee et al., 20155] Rasilo & Räsänen (017). which limits the types of interactions and environmental rewards that can be incorporated into the model. We instead simplify the vocalisation process by using an existing synthesiser, but this allows us to use modern MARL techniques to study continuous signalling.
We first give the Listener agent the infant role, and the Speaker will be the caregiver. This mimics the setting where an infant learns to identify words spoken by a caregiver. Later, we reverse the roles, having the Speaker agent assume the infant role. This represents an infant learning to speak their first words and their caregiver responds to recognised words. Since here one agent (the caregiver) has an explicit notion of the meaning of word, this process can be described as "grounding" from the other agent's 's perspective (the infant).
Setup We first consider setting where we have single set of 4 concepts S {up, down, left, right). While this is similar to the examples given in preceding sections, here the agents will be required to use actual English words convey these concepts. In the setting where the Listener acts as an infant, the caregiver Speaker agent speaks English words; the Speaker consists simply of dictionary lookup for the pronunciation of the word, which is then generated by eSpeak.
8

---

Under review as a conference paper at ICLR 2022
In the setting where the Speaker takes on the role of the infant, the Listener is now static entity that can recognise English words; we make use of a dynamic time warping (DTW) system that matches the incoming waveform to a set of reference words and selects the closest one as its output label. 50 reference words are generated by eSpeak. The action-space of the Speaker agent is very large ((P) ), and would be near impossible to explore entirely. Therefore, we provide guidance: with probability e (Section[4.) . choose the correct ground truth phonetic sequence for 8. We also consider the two-concept combination setting of Section 4.3 where either the Speaker or Listener now hears or speaks actual English words; DTW is too slow for the static Listener in this case, so here we first train the Listener the infant role and then fix it as the caregiver when training the Speaker.
Findings: Grounding the Listener Here the Listener is trained while the Speaker is a fixed caregiver. The Listener agent reached mean evaluation reward of 1.0, indicating the agent learnt to correctly classify all 4 target words 100% of the time (full graphs given in Appendix [B.)) The Listener agent was also tested with vocabulary size of 50, consisting of the 50 most common English words including the original up, down, left, and right. With this setup, the Listener still reached mean evaluation reward of 0.934.
Grounding the Speaker We now ground the Speaker agent by swapping its role to that of the infant. The Speaker agent reaches a mean evaluation reward of 0.983 over 20 runs, indicating it is generally able to articulate all of the 4 target words. Table 4 gives samples of one of the experiment runs and compares them to the eSpeak ground truth phonetic descriptions. Although appearing very different to the ground truth, the audio generated by eSpeak of the phone sequences qualitatively similar. The reader can confirm this for themselves by listening to the generated samples (again we refer the reader to the (ample page linked in Section 4.2)
Findings:
Findings: Grounding generalisation in communicating two concepts Analogous to Sec- tion 4.3 we now have infant and caregiver agents in setting with two concepts, specifically S tup, down, left, right} and S {fast, medium, regular, slow). Here, these sets don't 't simply serve as an example as in Section 4.3 but the Speaker would now actually say "up" when it is the caregiver and the Listener will now actually be pretrained to recognise the word "up" when it is the caregiver. combinations are unseen during training: up-slow, down-regulat left- left-medium, and right-fast. Again we consider both role combinations of infant and caregiver. Figure (oa shows the results when training a two-word Listener agent. The agent reaches a mean evaluation reward of 1.0 for the training codes and 0.952 for the unseen code combinations. This indicates that the Listener agent learns npar-optimal generalisation. As mentioned above, for the case where the Speaker is the infant, the DDDeed fixed Listener was found be impractical. Thus, we static Listener agent pre-trained to classify 50 concepts for each 81 and 82. This totals to 2500 unique input combinations. The results of the two-word Speaker agent are shown in Figure 6b The Speaker agent does not perform as well as the Listener agent, reaching a mean evaluation reward of 0.7199 19 for the training word combinations and 0.425 for the unseen.
We have replicated the experiments in this subsection using the Afrikaans version of eSpeak, reaching similar performance to English. This shows our results are not language specific.
5 DISCUSSION
The work we have presented here has gone further than Gao et al. (2020), which only allowed segmented template words to be generated: our Speaker agent has the ability to generate unique audio waveforms. On the other hand, our Speaker can only generate sequences based on a fixed
Table 4: Table of the target word, ground truth phonetic description, and trained Speaker agent's predicted phonetic description.
Target word
Ground truth
Predicted phones
up down left right
AP daon left
Avb dau lr JaIfjn
aart

---

Under review as a conference paper at ICLR 2022
1.0
1.0
Evaluation Reward
Evaluation Reward
0.6
0.6
0.2
training
0.2
training
unseen codes
unseen codes
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Training Episode
Training Episode
(a) Mcan evaluation reward of word Listener agent over training runs.
(b) Mean evaluation reward of two word Speaker agent over training runs.
Figure 6: Evaluation results of the grounded swo-word Speaker and Listener agent during training. The mean evaluation reward of the unseen word combinations are also shown.
phone-set (which is then passed over continuous acoustic channel). This is in contrast to earlier work (Howard & Messum 2014 Asada 2016 Rasilo & Räsänen, 2017) that considered a Speaker that learns a full articulation model in effort come close as possible in imitating utterance from a caregiver; this allows Speaker to generate arbitrary learnt units. We have thus gone further than Gao et al. (2020) but not as far these older studies. Nevertheless, our approach has the benefit that it is formulated in modern MARL setting: it can therefore be easily extended. Future work can therefore consider whether articulation can be learnt as part of our model posssibly using imitation learning to guide the agent's exploration of the very large action-space of articulatory movements.
In the experiments carried out in this study, we only considered a single communication round. We also referred to our setup as multi-agent, which is accurate but could be extended even further where single agent has both speaking and listening module, and these composed agents then communicate with one another. Future work could therefore consider multi-round communication games between or more agents. Such games would extend our work to the full MARL problem, where agents would need "speak" to and "hear" each other to solve common task.
Finally, in terms of future work, we saw in Section 4.3 the importance of the channel for generalisation. Adding white noise is, however, not good enough simulation of real-life channel acoustic channels. But our approach could be extended with real background noise and more accurate models of environmental dynamics. This could form the basis for a computational investigation of the effect of real acoustic channels in language learning and emergence.
We reflect our initial research question: Are we able to observe emergent language between agents with continuous acoustic communication channel trained through RL? This work has laid only a first foundation for answering this larger question. We have showcased the capability of environment and training approach which will serve as means of further exploration in answering the question.
ETHICS STATEMENT
We currently do not identify any obvious reasons to have ethical concerns about this work. Ethical considerations will be made taken into account in the future if some of the models compared to data from human studies or trials.
REPRODUCIBILITY STATEMENT
We provide all model and experimental details in Section and additional details in Appendix A The information given should provide enough details to reproduce these results. Finally, our code will be released on GitHub with an open-source license upon acceptance.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, J. Chen, Chen, Z. Chen, M. Chrzanowski, A. Coates, Diamos, K. Ding, N. Du, E. Elsen, J. Engel, W. Fang. L. Fan, C. Fougner, L. Gao, C. Gong, A. Hannun, T. Han, L. V. Johannes, B. Jiang, C. Ju, B. Jun, LeGresley, L. Lin, J. Liu, Y. Liu, W. Li, X. Li, D. Ma, S. Narang, A. Ng, S. Ozair, Y. Peng, R. Prenger, S. Qian, Z. Quan, J. Raiman, V. Rao, S. Satheesh,
D. Seetapun, S. Sengupta, K. Srinet, A. Sriram, H. Tang, L. Tang, C. Wang. J. Wang, K. Wang, Y. Wang, Z. Wang, Z. Wang, S. Wu, L. Wei, B. Xiao, W. Xie, Y. Xie, D. Yogatama, B. Yuan, Zhan, and Z. Zhu. Deep Speech 2: End-to-end speech recognition in English and Mandarin, In Proc. ICMI., pp. 173-182, 2016.
Andreas. Good-enough compositional data augmentation. Proc. ACL, 2020.
M. Asada. Modeling early vocal development through infant- caregiver interaction: A review. IEEE Transactions on Cognitive and Developmental Systems, pp. 128-138, 2016.
A. Black and K. Lenzo. Building voices in the festival speech synthesis system. wnwphhiedduuiibiiidddddddcooo document, 2000. URL URL http:: cstrr cstr ed. ac. http:/pjeeceeeeeaaaaaa////////te
H. Brighton and S. Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life, 2006.
R. Chaabouni, E. Kharitonov, D. Bouchacourt, E. Dupoux, and M. Baroni. Compositionality and generalization in emergent languages. In Proc. ACL, pp. 4427-4442, 2020.
Davis and P. Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions Acoustics, Speech, and Signal Processing, pp. 357-366, 1980.
D. Dor. The instruction of imagination: language and its evolution communication technology. pp. 105-125. Princeton University Press, 2014.
J. Duddington, eSpeak text to speech, 2006. URL htLt::////// sow.ceeearsouucceor....
Eccles, Y. Bachrach, G. Lever, A. Lazaridou, and T. Graepel. Biases for emergent communication in multi-agent reinforcement learning. In Proc. NeurIPS, 2019.
J. Foerster, N. Nardelli, G. Farquhar, T. Afouras, P. H. S. Torr, P. Kohli, and S. Whiteson. Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. Proc. ICML, 2017.
S. Gao, W. Hou, Т. Tanaka, and T. Shinozaki. Spoken language acquisition based reinforcement learning and word unit segmentation. In Proc. ICASSP, pp. 6149-6153, 2020.
N. Geffen Lan, E. Chemla, and S. Steinert-Threlkelle On the Spontancous Emergence of Discrete and Compositional Signals. In Proc. ACL, pp. 4794-4800, 2020.
S. Havrylov and I. Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. Proc. NeurIPS, 2017.
I. S. Howard and P. Messum. Learning to pronounce first words in three languages: An investigation of caregiver and infant behavior using a computational model of infant. PLOS ONE, pp. 1-21, 2014.
I. Kajic, E. Aygün, and D. Precup. Learning cooperate: Emergent communication in multi-agent navigation. arXiv e-prints, 2020.
S. Kirby. Spontaneous evolution of linguistic structure: iterated learning model of the emergence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, pp. 102-110, 2001.
Kottur, J. Moura, S. Lee, and D. Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proc. EMNLP, 2017.
11

---

Under review as a conference paper at ICLR 2022
P. K. Kuhl. Early language acquisition: cracking the speech code. Nature Reviews Neuroscience, pp. 831-843, 2005.
A. Lazaridou and M. Baroni. Emergent multi-agent communication in the deep learning era. CoRR, 2020.
A. Lazaridou, K. Hermann, K. Tuyls, and S. Clark, Emergence of linguistic communication from referential games with symbolic and pixel input. Proc. ICLR, 2018.
D. Lewis. Convention. Blackwell, 1969.
B. McFee, A. Metsai, M. Vccicar, S. Balke, C. Thomé, C. Raffel, F. Zalkow, A. Malek, Dana, K. Lee, O. Nieto, D. Ellis, J. Mason, E. Battenberg, S. Seyfarth, R. Yamamoto, vikorrandrevichmonzooy K. Choi, Moore, R. Bittner, S. Hidaka, Z. Wei, nullmightybofo, D. Hereñú, F.R.. Stöter, P. Friesch, A. Weiss, M. Vollrath, T. Kim, and Thassilo. librosa/librosa: 0.8. 1rc2, 2021. URL
https:/ owt.:///oo..o//.///oo www///22o000/// 4792298
Mnih, K. Kavukcuoglu, Silver, Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. In NIPS Deep Learning Workshop, 2013.
I. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent popula- tions. In Proc. AAAI, 2017.
C. Moulin-Frier and P.-Y. Oudeyer. Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historieal Context and Future Challenges. In Proc. AAAI, 2021.
C. Moulin-Frier, Diard, J.-L. Schwartz, and P. Bessière. Cosmo ("communicating about ob- jects using censory-motor operations"): A bayesian modeling framework for studying speech communication and the emergence of phonological systems. Journal of Phonetics, pp. 5-41, 2015.
P.-Y. Oudeyer. The self-organizztion of speech sounds. Journal of Theoretical Biology, pp. 435-449, 2005.
H. Rasilo and O. Räsänen. An online model for vowel imitation learning. Speech Communication, pp. 1-23, 2017.
C. Resnick, A. Gupta, J. Foerster, A. Dai, and K. Cho. Capacity, bandwidth, and compositionality in emergent language learning. Proc. AAMAS, 2020.
M. Rita, R. Chaabouni, and E. Dupoux. "LazImpa": Lazy and impatient neural agents learn to communicate efficiently. In Proc. ACL, pp. 335 343,343,440 2020.
I.. Steels. The synthetic modeling of language origins. Evolution of Communication, pp. 1-34, 1997.
L. Steels and T. Belpaeme. coordinating perceptually grounded categories through language: case study for colour. Behavioral and Brain Sciences, pp. 469-489 2005.
R. S. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
A. Tampuu, Т. Matüsen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLOS ONE, pp. 1-15, 2017.
L. Yuan, Z. Fu, J. Shen, I.. Xu, J. Shen, and S.-C. Zhu. Emergence of pragmatics from referential game between theory of mind agents. In Proc. NeurIPS, 2020.
12

---

Under review as a conference paper at ICLR 2022
APPENDICES
A EXPERIMENT DETAILS
A.1 GENERAL EXPERIMENTAL SETUP
Here we provide the general setup for all experimentation.
Parameter
Value
Optimiser
Adam
Batch Size 128
Replay size
256
Training Episodes
5000
Evaluation interval
100
Evaluation episodes
25
Runs (varying seed)
20
GPU
Nvidia RTX 2080 Super
Time (per run)
R 30 minutes
A.2 EXPERIMENT PARAMETERS
Here we provide specific details on per-experiment basis. The phone sequence length M in the grounded experiments is chosen such that the full ground truth phonetic pronunciation could be made by the speaker agent.
Experiment
Agent
Learning Rate
Phone length (M
GRU hidden size
Unconstrained Single-Concept
Speaker
x 10
256
Listener
x 10-5
256
Unconstrained Multi Multi-Concept
Speaker
10
512
Listener
5 x 10-5
512
Grounded Single-Concey
Speaker
10
256
Listener
5 x 10-5
256
Grounded Calti-Concept
Speaker
10-6 5
Listener
5 x 10-5
512
B RESULTS
B.
GROUNDING EMERGENT COMMUNICATION
1.0
1.0
Evaluation Reward
Eraluation Return
0.6
0.6
0.2
0.2
500
1000
1500
2000
2300
3000
1000
2000
3000
4000
5000
Training Episode
Training Episode
(a) Mean evaluation reward of Listener agent over 20 training runs.
(b) Mean evaluation reward of Speaker agent over 20 training runs.
Figure 7: Evaluation results of the grounded Speaker and Listener agent during training. Shading indicates the bootstrapped 95% confidence interval.
13


Reviews:
[{"review_id": "zcZ8axtzxfq", "reviewer": "Reviewer_6auy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper describes some experiments to simulate language acquisition in which a speaker talks to a listener over a continuous voice channel.  The speaker uses a GRU conditioned by one or more concepts to generate a sequence of sounds selected from an inventory of 160 English phonemes released by eSpeak.  The listener applies a convolutional net to the spectrogram of received sounds, then decodes sequential frames to generate a distribution over phonemes from which an argmax generates the decoded concept.   Learning uses a simple 1-step DQN.  The channel between speaker and listener has additive noise and time/pitch shifts applied.\n\nThe experiments include the unconstrained transmission of 1 and 2 concepts and the same but constrained so that either the speaker knows English words or the listener knows how to recognise English words.  Claims made include the ability to use composition and generalisation when moving from 1 to 2 concept transmission.\n\nThere are many things that I do not understand about this paper.\n\nFirstly, when the speaker is generating a sequence of M phonemes, each phone is regarded as an action but the optimisation treats this an M-armed bandit problem and simply averages the Q values over all actions.  Why is this not treated as an M-step trajectory and solved using the standard DQN optimisation?\n\nSecondly, it appears that all phoneme sequences are exactly M long, rather than a maximum of M long (see eg Table 1).  Is this true?   Wouldn't it be better to allow variable length and perhaps use the reward to encourage brevity.\n\nThirdly, how are the input concepts coded?  The examples suggest that each concept is coded as a 1-hot vector.  For multiple concepts, the natural assumption would be that the concepts are simply concatenated.  If this is the case, how does the listener work since it is only capable of forming a distribution over 1 set of concepts.  How does the listener distinguish multiple component concepts?\n\nThe claim that the results in Table 1 demonstrate the ability to learn a compositional language protocol are unconvincing.  Quoting 0.25 as the chance mean reward ignores the fact that the agents could simply recognise one of the concepts whenever the combination was unseen.\n\nAdding noise will often improve the robustness of a classifier, why is the generalisation implied by Table 2 any different?  What is the statistical significance of the numbers in this table?\n\nOn page 9, it is claimed that using DTW for the 2 word listener case is impractical.  Why is this? If the Listener only has to consider a vocabulary of 50 words, there are only 2500 two word combinations which should be tractable for a DTW style recogniser with pruning.\n", "summary_of_the_paper": "This paper describes some experiments to simulate language acquisition in which a speaker talks to a listener over a continuous voice channel.  The speaker learns to generate a phone sequence for each input concept that it wants to convey and the listener learns to recognise the input phone sequences to decode the embedded concept(s).  The speaker and listener use RL to learn. When the listener correctly identifies a concept both the speaker and listener receive a positive reward.  The authors claim that the system demonstrates the ability to exploit composition and generalisation. \n", "summary_of_the_review": "I am not convinced by the motivation of this paper, the presentation is poor and the experimental results are unconvincing.  The experimental design appears to involve some unnecessary approximations.  The paper claims to be an advance on Gao, but Gao addresses the problem of segmenting a speech stream into words when the listener does not have a pronouncing dictionary.  Here the use of the continuous channel appears to me to be an unnecessarily complicated way of adding channel noise to what is otherwise a discrete system. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "k6abGf5DQz", "reviewer": "Reviewer_yr56", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The environment is novel and provides a good basis for studying certain traits of continuous-channel referential games.\n- Excellent clarity and presentation of ideas\n\n## Weaknesses\n- The experiments provided do not analyze the unique characteristics of the environment introduced; instead, the experiments are similar to the typical gamut for a discrete symbol-based referential game.\n- The analysis of compositionality given is insufficient.\n    It appears that a single example was analyzed qualitatively.\n    In the presence of definitive results, this might be acceptable, but in its absence, as in this paper, there needs to be quantitative analysis across multiple runs to demonstrate the robustness of the phenomenon.\n\n\n## Recommendation\nI recommend rejecting the paper.\nWhile I appreciate the environment introduced and believe it to be promising, the experiments presented fail to highlight the novelty of the environment.\nIt is not the case that I find the experimental results inadequate, rather, the experiments run in the first place are generic and do not illustrate the points of interest with a continuous-channel referential game.\nIt is possible that I could be convinced otherwise as I truly appreciate the environment, but I would need to see an argument as to how the paper properly supports its introduction of a new environment.\n\n\n## Justification\nAs my primary critique concerns the experiments, I will address each individually mentioning any deficiencies and potential improvements.\n- unconstrained, single-concept:\n    This is adquate to demonstrate that a minimal environment works (as stated in the paper).\n- unconstrained, multi-concept:\n    This needs a direct comparison to a traditional discrete-channel referential game.\n    Thus, the experiment could answer questions such as:\n    - Does compositionality emerge at the same rate in continuous- and discrete-channel games?\n    - What is it about an audio channel model that makes compositionality easier or more difficult to learn?\n    - Does the interaction between various phonemes restrict the means by which compositional codes can be communicated (e.g., /aio/ and /aeo/ are harder to distinguish than /abo/ and /ato/)?\n- multi-concept and noise:\n    The noise you add to the channel has a number of different components; there should be an ablation study to illustrate the effects of these components.\n    If the experiment does not peer inside the structure of the noise and treats it as a gray box instead, it is not much different than a discrete-channel environment where random edits are made to the message.\n- grounded language learning:\n    In both of these experiments, there is analysis provided on what aspects of the audio-based communication channel make the problem harder, easier, or just different from the same experiment with a discrete channel.\n    For example, experiments could investigate if using words which are phonologically similar (e.g., \"boat\" and \"moat\") is harder to distinguish than dissimilar words.\n    Or an experiment could address if using the phonotactics from a natural language results is more effective than using a randomly selected phonemes due to necessity for natural languages to have acoustically distinct words.\n\n\nSpecific criteria:\n- Correctness: 4\n    - The claims are supported, but I do not think the claims go far enough (e.g., \"noise has an effect on generalization\" is claimed when instead what that effect is needs to be characterized).\n- Technical Novelty and Significance: 3\n- Empirical Novelty and Significance: 1\n\n## Questions\n_What_ is unique about the environment has been presented well, so my primary question is _how_ is it unique?\nIn other words, I see that the structure of the environment is unique, but what are the consequences which this structural difference begets?\n\n## Additional Comments\n- `s1`: Introduction is paced well\n- `s2p2`: what is T? Is it just every tick of the audio signal?\n- `s3.2p2`: What is the specific reasoning for using a bidirectional GRU? Using a model capable of streaming would make sense from the point of view of inductive biases. \n- `s3.2 p2`: Due to the argmax, is it the case that the \"confidence\" of the predictions is not used in optimization? This seems like it would artificially make optimization harder.\n- `Table 2`: Are these values computed over a single or multiple runs (in which case include stddev/confidence intervals).\n- `s4.3 Findings p4`: \"the channel ... influences generalisation\" -- Without further explication, this potentially interesting point does not gain any traction.\n", "summary_of_the_paper": "This paper tackles the well-established signalling emergent language game with the innovation of using a continuous, audio-based channel for communication as opposed to the more typical discrete, symbolic channel.\nThis continuous channel takes the form of a sender generating phones which are then synthesized into an audio waveform; this audio is then passed to the receiver after being transformed into a spectrogram.\nThe experiments then evaluate some general characteristics of the aforementioned referential game.\n\nThe following empirical evaluations are performed:\n- Basic referential game\n- 2-attribute ref. game to test for generalization and compositionality\n- Varying the noisiness of the channel to determine effects on generalization\n- Grounding the sender in English and letting the receiver learn \n- Grounding the receiver in English and letting the sender learn \n\nThe primary contributions of the paper are:\n- Presenting a phone-based, continuous-channel referential game\n- The effect of channel noise on generalization\n- Ability to learn a grounded language from a pretrained conversation partner\n", "summary_of_the_review": "The paper presents a well-designed, novel referential game where the communication channel is continuous and audio-based.\nThe paper fails to demonstrate with its experiments the actual effects of using this continuous channel.\nAs a result, my decision is \"reject\"; I think a new set of experiments highlighting the novel setup will yield a strong paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "jusANKgQMQi", "reviewer": "Reviewer_42Xh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good points:\n- pushes emergent communication in a new direction: that of oral communications\n\nBad points:\n- some key missed references, i.e. Foerster et al 2016 section 6.4 uses a continuous channel, and finds that adding noise to the channel is necessary in order to effectively 'discretize' the outputs. I also feel that Lazaridou et al 2018 is a better reference than Chaabouni et al 2020, since Lazaridou et al 2018 was more or less the first work to use a referential task in the recent wave of emergent communication literature.\n- a lot of the space in the paper appears to me to be used to describe relatively vanilla concepts, such as the architecture of the sender and receiver RNN models, what a replay buffer is, and some of the earlier experiments are I feel more like 'debugging'/smoke-test experiments, that show the architecture learns\n- I wasn't sure what I was supposed to learn from the grounded experiments. I felt that it was unsurprising that grounding one agent in English would cause the other agent to learn a language that learns English\n- the text to speech synthesize is pretrained, which goes I feel against the spirit of emergent communication. I understand that motivation for this was provided, and that motivation makes sense to me, but listing this for completion\n- the meaning/object spaces being used are very small (1 dimension, of size 16; or two dimensions, each of size 4; for example)\n- code is not provided with the submission\n- I didn't come away with any sense of 'oh that was interesting/surprising; I learned something'\n\n## Notes\n\nNotes I made as I read through the paper:\n\n### Abstract\n\nI like the idea of using an accoustic channel :) It's something I've been intending to write a paper on sometime, just didn't get around to it yet :)\n\nIt seems clear to me that spoken language preceded written language, so starting with accoustic language, for emergent communication, seems like a great idea to me :)\n\nI'm a little surprised/unclear why we map from symbols to accoustic. why not just keep everything continuous?\n\nnoisy communication channel being needed for compositionality and/or generalization has been shown in a few previous works.\n\n### 1. Introduction\n\nMising reference \"Learning to Communicate with Deep Multi-Agent Reinforcement Learning\", Foerster et al 2016, which in section 6.4, uses a continuous channel, and discovers that by adding noise, the channel effectively becomes discretized.\n\nCiting Chaabouni et al 2020 for 'referential signaling game' seems I feel not the original reference. \"Emergence of linguistic communication from referential games with symbolic and pixel input\", Lazaridou et al, 2018 was one of the first works to bring the idea of a referential game into emergent communication domain. Prior to that, the Lewis signaling game, Lewis, 1969, is, as far as I know, the original reference.\n\nWhy use deep Q-learning? Not that it's not a good choice, but curious why you choose this, when the vast majority of emergent communication works use simple REINFORCE (or Gumbel).\n\n### 2. Environment\n\nUsing a pre-trained text-to-speech synthesizer seems to go against the goals of 'emergent' communication?\n\nThe explanation for why this is (later in the section) does make sense to me though. Baby steps :)\n\n### 3. Learning to speak and hear using RL\n\nA large amount of space (~1.5 pages) is being spent describing what appear to be quite vanilla sender and receiver models. I'm not suggesting that the models should be made 'fancier', but that that space could perhaps be filled with additional experiments, or analysis, etc.\n\n### 4. Experiments\n\nI feel discussing eSpeak vs Festival gives a sort of 'engineer-y' feel to the paper; and could be omitted.\n\n#### 4.2 Unconstrainted communication of single concepts\n\nI realize at this point that somehow I missed what are the actual input data. Ok, skimming right back to page 3, it looks like there is only a single meaning dimension, of size N? This could represent eg 'color'. From 4.2, it looks like N is 16, though it's not stated clearly. This is I feel an extremely small meaning/concept/object space.\n\nOn the whole, 4.2 seems like a superfluous debugging section, which doesn't really provide any information/signal, and could be omitted.\n\n#### 4.3 Unconstrained communication generalizing to multiple concepts\n\nI was briefly excited by the mention of Kirby's ILM, but I feel that the citation of Kirby's ILM is superfluous here: the experiment is simply changing from a 1-dimensional meaning space of size 16, to a 2-dimensional meaning size, where each dimensino is of size 4.\n\nAgain, this experiment seems to me to be quite vanilla.\n\nI feel that in addition to providing a table of utterances to eye-ball, it could be slightly more rigorous to use objective measures of compositionality such as topological similarity.\n\nI feel that the effect of channel noise in Table 2 appears quite weak to me. In addition, it could be argued that the noise is simply providing training time regularization, which we see as a drop in performance of the training codes, rather than leading to a qualitative shift in the emergent communication, as Foerster et al 2016 showed, in their section 6.4.\n\n#### 4.4 Grounding emergent communication\n\nI feel that 'grounded emergent communication' is a contradiction in terms: either the language is grounded, or it emerges. If there is any grounding provided, then it is a grounded task, not an emergent task, I feel?\n\nI'm not really sure of the purpose of this set of experiments. The various agents are being grounded in english in various ways, and end up outputing some English-like language. I'm not really sure what I can learn from this?\n\nI feel that using English-language grounding in a paper positioning itself as emergent communication is somewhat risky. With sufficient motivation and analysis, I don't have any fundamental objection to doing this though. Preference to come away with some sense of 'that was interesting; I learned something'. This might mean making the experiments challenging enough so that they do *not* actually work, and then digging into why, and what we can learn from that, perhaps?", "summary_of_the_paper": "The paper proposes to move in the direction of emerging oral communication, instead of written communication. They create an architecture where the sender agent first outputs a string of symbols, which map statically to a fixed table of phonems. The sender uses an off-the-shelf pre-trained text-to-speech synthesizer to convert these phonemes into sounds. The sounds pass through a channel with Gaussian white noise added, along with Gaussian-sampled time-shift and pitch-shift. The receiver converts the input sounds into a mel-spectrogram, which is a sequence of vectors over time representing the frequency distribution. The receiver passes these vectors into a CNN, an RNN, and finally a linear classifier.\n\nThe paper motivates using the pre-trained speech-to-text synthesizer as a necessary initial baby-step, to keep the problem tractable in the context of RL.\n\nThe paper uses an encoder-decoder task, where the input is a symbolic concept comprising one dimension of size 16, or two dimensions of size 4. The output is a prediction of the original input concept.\n\nThe paper shows in a first experiment that their model can learn to predict the input concept correctly with a meaning space of 1 dimension of size 16. Then, in a second experiment, the concepts are extended to two dimensions, each of size 4. In addition, in this second experiment, the effect of communication channel noise is measured: adding noise to the channel decreases performance on concepts seen during training, but improves performance on a holdout set of novel input concepts.\n\nIn the final experiment, where the speaker or the listener are fixed, and grounded in English. In this case, the other agent will learn a language that is close to English.", "summary_of_the_review": "On the whole, I like that the work pushes towards moving emergent communication into a new direction of oral communication. I feel that this work as it stands is an excellent start in this direction. However, I feel that the experiments and analysis could be extended somewhat, into scenarios where it feels somewhat less predictable what the outcomes will be. For example, what happens if the meaning space is larger? Does that work? If it doesn't work, how does it fail and why?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H0sINJzJodV", "reviewer": "Reviewer_GZXE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper explores a two-agent communication problem where the Sender agent sends a continuous waveform to the Listener who is tasked to reconstruct the original concept vector. There are two settings of concepts used, single concepts (or a flat vector) and double concepts (consists of a combination of two separate meanings).\n\nHowever, there are many important details missing in the main text. It is not clear how big is the alphabet size that is used by the Speaker to construct the waveform. It is an important parameter as it affects the compositional generalization of the agents (Chaabouni et al. 2020, Kottur et al. 2017, Resnick et al. 2020, Li and Bowling 2019). \nRegardless of the alphabet size used, the problem setting used is quite 'toyish'. Even in the double concepts game the total possible combination of all concepts is 16. In most of the previous literature in this field, even the minimal setup uses 3 concepts of 5 types each (Chaabouni et la. 2020) and then results are validated using a more realistic problem setting either using real images as input or increasing the input space. The question here is would this method scale to larger problem settings.\n\nThe idea of using signalling to make one agent learn about the underlying concept and active listening to help the listener solve the given downstream task is already explored in Eccles et al 2019. Unfortunately, there has been no comparisons made with this work that seems to be the most relevant to the proposed method.\n\nMoreover, many relevant are not cited in this work (for example the above works). I would encourage the authors to do a thorough literature survey of the recent work done in the space of compositionality and generalization in emergent languages.\n\nThe hypothesis around compositionality is only evaluated qualitatively. There have been some good progress done to compute domain independent quantitative compositionality metrics that are well established and grounded (Chaabouni et al. 2020, Lazaridou et al. 2018, Resnick et al. 2020, Andreas 2019). In addition, the notion of compositionality captured in the paper is more aligned to combinatorial generalization.\n\nGood-enough compositional data augmentation. Andreas 2019  \nEntropy minimization in emergent languages. Kharitonov et al. 2020  \nCompositionality and generalization in emergent languages. Chaabouni et al. 2020  \nEase-of teaching and language structure from emergent communication. Li and Bowling 2019  \nCapacity, bandwidth, and compositionality in emergent language learning. Resnick et al. 2020  \nNatural language does not emerge ‘naturally’ in multi-agent dialog. Kottur et al. 2017  \nEmergence of linguistic communication from referential games with symbolic and pixel input. Lazaridou et al. 2018  ", "summary_of_the_paper": "The paper proposes a method to learn meaningful emergent languages with a continuous communication channel. The game consists of two agents that perform a unidirectional communication where the agents are trained using deep RL. The training follows an iterated procedure where one agent acts as a 'teacher', who has the actual meaning of the word, and the other as a 'student', who is trained at a given iteration by using information obtained form the teacher during training. The authors show that this type of grounding allows generalization and helps in learning a compositional language.", "summary_of_the_review": "The paper explores an interesting research question to train agents that can communicate using a continuous communication channel and be able to solve the given downstream task thereby exhibiting compositional generalization. Although the underlying hypothesis is interesting, the idea is not completely novel and not evaluated extensively using well-established metrics in the literature. This work is a good preliminary work in investigating this research direction and I would encourage the authors to refine the evaluation and make explicit comparisons with prior work in the future submissions of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While a lot of previous work on emergent communications studies discrete protocols, this work explores a continuous and audio-based channel for learning multi-agent communication. Reviewers have commented positively on the novelty of the topic. At the same time, there are a number of concerns raised with respect to experimental design and implementation (6auy) and general approach of the topic which, as reviewers t576 and 42Xh point,  doesn't really go deep into the analysis and understanding of the particular experimental setup and findings. So, unfortunately as the papers stands I cannot recommend acceptance at this time. However, given that continuous communication in emergent communication is a somewhat overlooked topic, I would encourage the authors to use the reviewers' feedback and strengthen their manuscript.
Decision: Reject