Neural Agents Struggle to Take Turns in Bidirectional Emergent Communication
Authors: Valentin Taillandier
Dieuwke Hupkes
Benoît Sagot
Emmanuel Dupoux
Paul Michel
Venue: ICLR 2023 poster (2023)
Keywords: language emergence
turn-taking
conversation
communication
neural agents
cooperative game
reinforcement learning

Abstract: The spontaneous exchange of turns is a central aspect of human communication. Although turn-taking conventions come to us naturally, artificial dialogue agents struggle to coordinate, and must rely on hard-coded rules to engage in interactive conversations with human interlocutors. In this paper, we investigate the conditions under which artificial agents may naturally develop turn-taking conventions in a simple language game. We describe a cooperative task where success is contingent on the exchange of information along a shared communication channel where talking over each other hinders communication. Despite these environmental constraints, neural-network based agents trained to solve this task with reinforcement learning do not systematically adopt turn-taking conventions. However, we find that agents that do agree on turn-taking protocols end up performing better. 
Moreover, agents that are forced to perform turn-taking can learn to solve the task more quickly. 
This suggests that turn-taking may help to generate conversations that are easier for speakers to interpret.

Published as a conference paper at ICLR 2023
NEURAL AGENTS STRUGGLE TO TAKE TURNS IN BIDIRECTIONAL EMERGENT COMMUNICATION
Valentin Taillandier
Dieuwke Hupkes
ENS PSL,, Inria
Meta AT Research di ewwkehupee fb. com
va lentinn.nnnn tai tai aadieerggaaaaai com
Benoît Sagot
Emmanuel Dupoux
Paul Michel" ENS PSL. Inria @miche1314150ppp com
Inria benoit. wagottinriiiaaii....... fr
Meta AI Research dpx@ fb. com
ABSTRACT
The spontaneous exchange of turns is central aspect of human communication. Although turn-taking conventions come to us naturally, artificial dialogue agents struggle to coordinate, and must rely on hard-coded rules to engage in interactive conversations with human interlocutors. In this paper, we investigate the condi- tions under which artificial agents may naturally develop turn-taking conventions in a simple language game. We describe a cooperative task where success is con- tingent on the exchange of information along shared communication channel where talking over each other hinders communication. Despite these environmen- tal constraints, neural-network based agents trained to solve this task with rein- forcement learning do not systematically adopt turn-taking conventions. However, we find that agents that agree on turn-taking protocols end up performing better. Moreover, agents that are forced to perform turn-taking can learn to solve the task more quickly. This suggests that turn-taking may help to generate conversations that are easier for speakers to interpret.
INTRODUCTION
Natural conversations involve a rapid exchange of ut- terances where speakers coordinate on-the-fly avoid talking over each other. This tun-ttaking phenomenon ubiquitous across cultures (Stivers al. 2009) and is even found in some forms of animal communication (Pika et al.] 2018; Demartsev et al. 2018). The abil- ity to engage in spontancous turn-tite wornttaking develops early in humans, even before linguistic competence (Nguyen et al., (021) and allows us to hold fluent conversations with very little downtime between utterances (Heldner & Edlund 2010). In contrast, fluid turn-taking is difficult to replicate artificial dialogue systems. Modern con- versational agents often rely on explicit cues, for instance pressing "eetter" in text-based chatbots, the use of spe- cific wake-words (Gao al. 2020), long silences of pre-determined length (Skantze|
Time
Illustration of our proposed game. Both agents can exchange utter- ances through a shared communication channel. At each step of the conversa- tion, agents can decide to either speak or stay silent. However, information can- not be transmitted if both agents decide to speak at the same time.
Figure 1:
The goal of this paper provide testhed for studying the conditions under which artificial agents may develop a turn-taking convention resolve a cooperative task. We describe a simple two- wwo-player game where agents observe partial views on an object which they must reconstruct. Agents can exchange information by emit- ting symbols across shared communication channel over multiple rounds. The game exhibits two
Now at DeepMind.

---

Published as a conference paper at ICLR 2023
key features: first, at each round agents can decide to either talk or stay silent. However, if both agents decide to talk at the same time, they are not able to hear the other agent's 's message. Second, agents get higher score if they solve the task in fewer rounds. This creates explicit pressure towards a protocol that allows the agents to solve the task quickly while minimizing overlap.
In experiments, we find that simple seural-network-baseee agents trained with reinforcement learning do not consistently develop natural turn-taking strategies. However, agents that do develop a turn- taking protocol are able to achieve much higher score, sometimes solving the task perfectly. To shed light on this finding, we perform an in-depth analysis of an asymmetric version of the game, where one agent has all the information. We show that in this case, there is an optimal strategy that does not rely on turn-taking. However, we demonstrate empirically that agents fail to solve the game when they are forced to use this strategy. contrast, agents that are forced to use strategies in multiple turns rapidly achieve almost perfect accuracy.
2 RELATED WORK
Schegloff (2000) attributes the first description of turn-taking to Goffman (1955), although the term "turn-taking" itself was coined much later (Yngve 1970). Since then, and following early seminal work in the 70s (Duncan 1972 Sacks et al. 1978). turn-taking has been the subject of considerable
study under the umbrella of "conversation analysis" (eevinson) 1983). For example, researchers have sought to characterize linguistic and paralinguistic cues involved in organizing turns (Stephens & Beattie 1986 Kendon| 1967| Clancy et al. 1996), to understand the time-scales involved in turn changes (Stivers et al.J 2009; Heldner & Edlund] 2010) or to identif'y turn-taking conventions in
non-human primates (Rossano 2013 Rossano & Liebal 2014 Demartsev et al. 2018).
Research in automated dialogue systems dates back to early efforts the 60s and 70s (Weizenbaum) 1966 Bobrow et al. 1977). Particularly relevant to our work is line of research on task- based meth- ods, which formulates dialogue as a reinforcement learning problem to satisfy some user-relevant task (Walker 2000 Levin et al., (2000). After the success of early deep-learning based models (Vinyals Le 2015|] ei (016), most state-of-the-arr systems (Adiwardana al., 2020 Zhang et al. 2020) are now generally built on top of large pretrained models (Radford et al. Lewis et al., 2020 Research in dialogue systems focuses primarily on text-based models, and spoken dialogue scaffolding around text-based core (Chen et al. 2018). However, there have been recent on addressing phenomena specific to spon- taneous spoken dialogue such as filled pauses (Székely et al.) 2019) non-verbal vocalizations and turn changes (Nguyen et al. 2021).
systems are generally implemented as speech recognition/generaa
Our work belongs long line of research devoted to the use of computational models for sim- ulating the emergence of language (Batali 1998 Cangelosi & Parisi] 2002). With the advent of deep learning, there has been surge of interest emergent communication among neural-network agents trained with reinforcement learning (Lazaridou et al. [017 7| Foerster et al. 2016 Lazaridou & Baroni 2020). The overwhelming majority of research in this area focuses on unidirectional sender-
receiver communication (see Havrylov & Titov 201 Bouchacourt & Baroni (2018) Chaabouni
al. (220)) inter alia) based on variants of the Lewis signaling game (Lewis)] [ewis][1661 1969; Skyrms (2010).
Although several authors have studied bidirectional communication (Kottur et al.
2017
Graesser
jt al. 2019), alternating turns are usually hard-coded into the game.
METHOD: A TESTBED FOR THE EMERGENCE OF
There is evidence in the conversational analysis literature that turn-taking systems in human lan- guages almost universally tend to favor (1) few instances of overlapping speech and (2) minimal pauses between turns (Stivers et al.] 2009). In this section, we describe a communication game based on variant of the Lewis signaling game (Lewis) 1969) which encapsulates these features.
3.1 GAME DESCRIPTION
The game is played between two agents, Agent 1 and Agent 2. Before every episode of the game, an object is drawn according to distribution p from pool X and both agents observe partial views on the object, it and &z. The goal of the game is for the agents to communicate enough information to reconstruct the original input Throughout the paper, we will experiment with two versions of
2

---

Published as a conference paper at ICLR 2023
the game: symmetrie variant where both agents observe partial views, and asymmefric variant where one agent (by convention, Agent 2) has access to the full object (0z = 2).
3.1.1 CONVERSATION OVER A SHARED CHANNEL
During an episode, conversation between both agents proceeds over communication steps. At every step & {I....,,T) T), each agent E [1, must choose one of two actions:
S SILENCE: the agent says nothing
SPEAK: the agent emits a symbol st taken from a finite vocabulary V {wr,.... w vI}- The agent chooses which symbol to send (there are therefore [V] possible SPEAK actions).
Agents choose their respective actions synchronously and independently: they cannot coordinate choose their action based on the other agent's 's action at the same time-step.
To emulate the presence of single auditory channel, we introduce an additional environmental constraint depending on the actions chosen by the agents. Two situations may occur:
1. if at least one agent stays S SILENT, communication succeeds. In this casc, both agents are able observe each other's actions, including any symbol sent.
2. both agents may attempt to SPEAK at the same time. We call this situation overlap: in this case, we do not allow the agents to perceive each other's s actions directly, to simulate failure to transmit information. We detail specific strategies for modeling overlap in Section 3.1.2
After each step of the conversation, agents can make guess to the original object based on the communication history they have observed and receive a reward for this communication step. Importantly, agents are trained to maximize their total reward R Σt= Tt. This means that agents encouraged to reconstruct the correct object (and thus get maximal reward) soon possible, thus creating pressure towards efficient communication (Rita et al. 2020).
3.1.2 MODELING OVERLAPS
Central to our game the special handling of overlaps: agents cannot simply communicate infor- mation simultaneously. We implement this constraint by perturbing the symbols emitted by each agent. We experiment with several ways to corrupt the emitted symbols to represent a failure of each other:
communication that may result from agents speaking
Noise: instead of observing the symbol emitted by their communication partner, each agent ob- serves special <noise: symbol. Both agents are aware that they have attempted to communi- cate but they do not observe the exact message sent by each other at this specific time-step.
Misunderstanding: another possibility is that agents misunderstand each other in case of overlap. We implement this by substituting the emitted symbols with random symbols from the vocabulary.
Walkie-Talkin in this more restrictive setting, the communication channel behaves akin to a walkie-talkie device: when agents speak simultancously, they perceive the other's action as s SILENT. This is a more punitive case of overlap because the agents are not able detect that they are overlapping.
In most experiments, we adopt the Noise model. However, we will show that the choice of the overlap model can have large consequences on the emergent protocol.
3.1.3 OBJECTS AND PARTIAL VIEWS
To facilitate analysis and enable fine- fine-grained control of the difficulty of the game, in the remaining of this paper we consider games where the underlying objects & are simple attribute- value vectors Kottur et al. 2017 Chaabouni et al. 2020 [Specifically, each object x: is vector of N,, attributes, each which can take N, distinct values, for total of N.W. possible combinations. We represent
Although the game could be performed on other objects, such as partially obstructed images Graesser 2019)
3

---

Published as a conference paper at ICLR 2023
partial view è object by masking Nn of its attributes. At every episode of the game, sample an object z and randomly mask some of its attributes according to a distribution 6m((12) to obtain partial views 2.
In the asymmetric version of the game, only one partial view sampled (#1) since Agent observes the original object. In the symmetric variant, both partial views d1 and Sgg are obtained by sampling masks independently for both.
3.2
MODELING THE AGENTS
3.2.1 GENERAL ARCHITECTURE
During an episode of the game, agents must achieve two tasks: they must exchange information across the communication channel, and they must be able to reconstruct the original object æ based on their conversation history. For this purpose, each agent must implement two functions:
(a, S1-1), action policy which defines distribution over one of the [V| + 1 possible actions (cither stay silent or emit one of the V symbols)
1)-1); reconstruction function which defines distribution over possible reconstruc- tions of the original input
Both distributions conditioned S the state of the conversation up until step / from the point of view of agent i. S encapsulates (I) the original partial view observed by the agent, (2) the history of all past actions of the agent a'z, and finally (3) the symbols received from the other agent j. &2z-- Recall that the latter may not correspond exactly to the symbols emitted by the other agent j if the agents overlapped.
In practice we implement these functions with a shared conversation encoder ((()))) which maps the conversation state S (a<e, &t,) onto continuous representation (for instance by means of a recurrent neural network). Both the action policy and the reconstruction function are class-based soft- max (Goodman 2001), where we first predict the type of action, either SILENT or SPEAK. If the latter is chosen, separate softmax operator predicts the emitted symbol. We factor the reconstruc- tion function into N. independent classifiers, one for predicting the value of each attribute (out of No candidates).
functions of this encoding (see Figure 2 The action policy is parameterized
3.2.2 STANDARD TRAINING OBJECTIVES
During training, each agent is then trained to maximize two objectives, conversation objective Jcomv with respect to its action policy and a reconstruction objective Jnee with respect to its reconstruction function:
J Jonnv
gmp E
S. ((RR)
(((S)+RR(S)
(112))
T
Jree,
zmp fig
(f (112))))
Σ
log o S
2m
where Es-re, (121))=5 (00)))==,,11 (+122) denotes the expectation taken over all conversations sampled from the agents' policies conditioned on &1 and i... Both agents learn to maximize the joint reward R' + +R,, making the game fully cooperative. We will refer to this objective as the Standard objective.
To encourage agents to communicate about their missing information, we experiment with an al- ternative objective where agents must only reconstruct their masked attributes, but also predict which attributes are masked for their interlocutor. We call this objective Reciprocal, since agent must be able reconstruct & and not
4 EXPERIMENTAL SETTINGS
4.1 GAME PARAMETERS
4

---

Published as a conference paper at ICLR 2023
In all experiments, inputs vectors of Na 10 attributes with N₂ = 16 possible values, re- sulting in x 10 total combinations. Unless specified otherwise, agents are optimized on training set of 10 examples. We also sample validation and test set of 1,000 examples each. We set the number of masks Nm = During training. the positions of the masked attributes are sampled uniformly on-the-fly. At evalua- tion time, we exhaust all possible combinations of masks for the first agent. In the symmetric setting, we sample another pair of mask inde- pendently for the second agent.
S
Agent
Agent
h1
x1
S
S
Figure 2: Overview of step of the game from the point of view Agent The agent takes in S and tries to reconstruct the object, yield- ing reward r!- Simultaneously, the agent sam- ples an action at and observes symbol & emitted by agent 2. The conversation state S is updated based on both agent's actions.
We set the size of the vocabulary to - 8. Importantly, we are careful to choose a com- munication channel that is small enough to dis- qualify trivial strategies where each agent simply describes the entirety of its partial view. Indeed, a single agent needs N₂ log [1]+1] Na steps to fully describe vector of N,, attributes with N, values using a vocabulary (26 in the symmetric setting). Therefore limit the size of the channel to T = 8 in the asymmetric variant and 16 in the symmetric variant. This means that to solve the task, agents must leverage the fact that they share information about the problem if they are to solve the task successfully.
size [V| With vocabulary size of 8 this corresponds to Rs 13 steps per agent
4.2 EVALUATION METRICS
We evaluate agents by their ability to reconstruct the original object at the end of the conversation. We report the average accuracy over all masked attributes. Note that the symmetric version of the game, it is possible that some attributes are masked for both agents, meaning that the original value cannot be reconstructed. We exclude these occurrences from the computation, which means that the maximum accuracy is always 100%.
We also analyze the resulting conversations in terms of number of turns and overlaps, following standard conversation analysis taxonomy (Sacks et al. 1978). Specifically, turn starts when an agent i is speaking and the other agent) j isn't. The turn lasts up until agent i becomes silent. the next agent speaking is still agent i, the turn continues and any silence is counted pause". However, if the next agent speaking is agent j. the turn ends before the first silence, and agent j's turn starts whenever it starts speaking. If there are intermediate steps between the two turns where both agent are silent, they are counted as a "gap" in the conversation. Therefore, each timestep between the start of the first turn and the end of the last turn can be counted either as agent i's turn, agent) j's turn, or gap, Additionally, we also record all timesteps where agents speak over each other (overlap).
"intra-turn
4.3 AGENT IMPLEMENTATIONS
We implement the conversation encoder h; of each agent single layer LSTM networks (Hochreiter & Schmidhuber 1997) with hidden dimension 512 and input dimension 256. At each step t, agent i encodes the state of the conversation hi LSTM (M)M(()-)
11 + where hi_ is the hidden state corresponding to the previous timestep and fat_, and exy_, are embeddings of the previous action at-1 and the symbol 8{_1 received from the other agent at step ( 1.
At the beginning of the conversation, the values of the partial views on the inputs, à are encoded as N₂ 256-dimensional embeddings, one for each attribute. For each attribute we learn N. + 256-dimensional embeddings, one for each values and one to represent the masked attribute. The concatenated N₂ x 256 dimensional representation à is then fed into a fully connected layer mapping to 512-dimensionnl encoding. This input encoding is used to initialize the hidden state of the recurrent network.
2The basis of the logarithm is [V| + 1 because agents transmit information not only by emitting one of the [V] symbols, but also by staying silent.
5

---

Published as a conference paper at ICLR 2023
Accuracy
Overlaps
#Turns
Accuracy
Overlaps
#Turns
Stand.
92.63 12.55
1..00
1.00
Stand. 16.24
1.42 10.66
7.00 5.34
Recip.
90. 10.17
0.77
1..5
Recip. 83.63 16.91
3.07 0..89
(a) Metrics for the asymmetric game.
Metries for the symmetric game.
Table 1: Metrics for both asymmetric and symmetric game with both standard and reciprocal objective. We report standard deviation across 20 runs.
4.4 AGENT TRAINING
Agents are trained to maximize their objectives using gradient descent with minibatching. At each step of the optimization process, we sample objects z uniformly from the training set, and partial views & and 2 according to the masking distribution 6m æ). We then unroll the conversation by repeatedly sampling from each agent's policy until the maximum number of steps have been reached, yielding conversation = (S) 5?) (S)::)))=.. I=..TT
We obtain gradient estimates for Jre by automatic differentiation. For maximizing Jeon with spect to the policies n", we must resort to reinforcement learning. Specifically, we estimate Vo, Jeome using REINFORCE (Williams) T992). policy gradient algorithm. We set the rewards to be the
construction log probability of both agents Rig E log of (:) We use the average reward in minibatch as baseline reduce the variance of the REINFORCE updates (Sutton al. 1999). and add an entropy maximization term to encourage exploration (Williams & Peng 1991). ) Our implementation is written in Pytorch (Paszke et al. 2019), and is based on the EGG framework Kharitonov et al. 202I).
4.5 TRAINING HYPERPARAMETERS
Agents trained using the Adam optimizer (Kingma & Baj 2015) with learning rate of 0.001, a batch size of 2048 and a weight of 0.001 for the entropy term. We train for a total of 600, 000 steps, and keep the pair of agents with the highest accuracy on the validation set. Unless indicated otherwise, all experiments are run for 20 different random seeds and we report both the mean and the standard deviation of each metric across all runs. See Appendix 7 for additional reproducibility.
RESULTS
5.1 ASYMMETRIC GAME: AGENTS DEFAULT TO UNIDIRECTIONAL COMMUNICATION
We first train agents to perform the asymmet- ric version of the game where Agent 2 has all the information. In Table la we report recon- struction accuracy (of Agent 1), the number of overlaps and the number of turns when agents
Pearson p: -0.83
Pearson p: 0.77
100
Accuracy
95
are trained with both the standard and recip- rocal objective. We find that agents trained with the standard objective adopt unidirectional communication strategies where only the agent that has all the information communicates (only one turn, overlap). However, agents are not able to solve the game completely, consistently converging about 92% accuracy.
Overlaps 1.5
2.0 Turns
2.5
Figure Correlation between turn-taking met- rics and accuracy in conversations developed by agents for the asymmetric game (p-value < 10*).
On the other hand, agents trained with the re- ciprocal objective are encouraged to communi- cate both ways to exchange information. We observe that agents generally develop protocols in more than one turn, albeit with some amount of overlap. Although agents reach lower accuracies on average, the variant across runs is very high. In fact, we find that more than half of the runs see agents achieve an accuracy greater than 98% (by comparison, in the standard mode the best accu- racy across all runs is 97.7%). To understand the source of this variability, in Figure we report the accuracies reached by different agent pairs as a function of the number of turns and overlaps their conversations. Conversations involving turn-taking (two
turns, minimal overlap) correlate

---

Published as a conference paper at ICLR 2023
Accuracy
Overlaps
Turns
80
60
40
20
Misunderstanding Walkie kic
Standard
Reciprocal
Standard
Reciorocal
Standard
Reciprocal
Figure Influence of the overlap model the emergent protocols for the symmetric game.
strongly with high accuracy. However, there several outliers where agents efficient turn-taking and accuracy is much lower. In other words, while agents do not systematically agree on turn-taking protocols, those that do consistently perform almost perfectly the game.
able develop
Appendix C showcases a successful "question-answer" based strategy, where Agent first declares the masked positions and Agent 2 responds with the corresponding values.
5.2 SYMMETRIC GAME: NoT ALL TURN-BASED STRATEGIES ARE EQUAL
In the symmetric version of the game both agents observe masked inputs. Therefore it is not possible for the agents to rely solely on unidirectional communication. In Table [b] we observe that agents achieve much lower accuracy than the asymmetric game, In particular with standard training objective agents reach below 50% accuracy. Interestingly, agents develop protocols involving many short turms (== on average) with few overlaps.
Training with the reciprocal objective results in much higher accuracy. Agents still engage in turn taking. but the conversation involves fewer, longer turns (== turns). We postulate that in this case agents adopt a similar strategy as in the asymmetric setting, first declaring their masked position and then announcing the missing values (see Appendix for qualitative example).
Overall, these results suggest that successful turn-taking does not necessarily entail success at the task (as for agents trained with the standard objective). Winning strategies rely on question-answers style conversations (agents trained with the Reciprocal objective).
5.3 INFLUENCE OF OVERLAP MODELING CHOICES
Up to this point, all experiments were performed with the "Noise" implementation of overlaps. However, stands reason that different models of overlaps may lead to different strategies. Fig- ure reports results for the symmetric game with all three implementations (see Appendix [ for asymmetric game). Although there some variance, both Noise and Misunderstanding result in approximately the same behavior. On the other hand, agents trained in the Walkie-Talkie mode avoid overlaps all costs: in this mode overlapping is much more punitive since agents that talk over each other not aware that they overlapped. Consequently find that agents struggle to perform conversations with multiple turns, and accuracy is much lower.
ANALYSIS
In this section we investigate why some agents adopting turn-taking conventions achieve higher accuracy. For the sake of simplicity, we focus our analysis on the simpler, asymmetric setting where one agent (namely Agent 2) has all the information.
6.1 OPTIMAL STRATEGIES
We describe two theoretical strategies which agents may adopt to solve the game while avoiding overlaps. For each, we can compute the length of conversation L needed for the agents to fully recover the original object (see Appendix for derivations).
Question-answers (QA): Agent first communicates the position here masks to Agent (the "question"). Based on this information, Agent (who knows the full object æ) answers with the

---

Published as a conference paper at ICLR 2023
values of the corresponding attributes. This strategy requires two turns, and agents must agree turn-taking convention. The total length of the conversation is LQA log (N... +n log No. where the first term corresponds to the length of the question and the second to the length of the answer. As shown in Section we find that agents sometimes learn to adopt this strategy when they are trained with the reciprocal objective.
Na
Error-correcting code (ECC): more efficient approach is based on Reed-Solomon error- correcting codes (Reed & Solomon| 1960). In this strategy, both agents can agree on Nm well- chosen linear encodings of the objects such that it is sufficient for Agent to send the evaluation of these linear functions on 2. Agent can then provably reconstruct by evaluating the same linear functions on & and comparing the result with the message from Agent 2 (see Appendix B for detailed explanation). The length of the conversation is then only LEce Nm log PN.,Nos where PN...N is the smallest prime number greater than max(Na, NN).
In our setting, the corresponding lengths (rounded to the next integer) are LQA = and LEce = 3. On the surface, ECC is much more efficient than QA: it requires 2 fewer steps to accomplish, and it only requires unidirectional communication.
6.2
TURN-TAKING FACILITATES LEARNING
We have established that in the asymmetric setting, turn- taking is not the optimal strategy for solving the game as early as possible given the environmental constraints of shared communication channel. This seems to contradict our results in Section 5.1 where we found that agents en- gaging in turn turn-taking performed better than agents relying on unidirectional communication. To understand this dis- crepancy, we compare the performance of the agents us- ing any of these communication strategies. We compare four protocols: the two hard-coded strategies QA and ECC, and two strategies learned by agents through re- inforcement learning. We select one policy where agents engage in turn-taken tunntaaing (RL-TT) and one where only uni- directional communication occurs (RI-Uni).
100
80
Policy
60
QA
ECC
40
20
RL-Unl
50k
100k
150k
200k
Training steps
Figure 5: Validation accuracy through- out training with fixed policies. Conver- sations involving turn-taking (dashed lines) are learned much faster.
Figure 5 depicts the evolution of the reconstruction accuracy for pairs of agents trained purely to solve the task using fixed policy. We find that turn-taking strategies not only enable the agent to reach the highest accuracy, but convergence is dramatically faster. In contrast, agents forced to adopt RL- learn much slower. Finally, agents are able learn solve the game using ECC, even though it is theoretically the most efficient strategy.
6.3 DISCUSSION
These findings seem to discredit purely functional explanations for turn-taking as the most efficient communication strategy through shared channel. Indeed, our theoretical analysis asserts that turn- taking is not optimal under these constraints. Nevertheless, according to Figure 5 conversations involving turn-taking Moreover, agents learn to decode these conversations much earlier. This begs the question whether tum-takking may have developed information. More future work is warranted to determine whether these findings generalize beyond our simple setting.
casier to interpret for agents than more sophisticated alternatives like ECC.
response extraneous pressures beyond the efficient transfer of
7 ADDITIONAL REPRODUCIBILITY DETAILS
All experiments are single Nvidia V100 GPUs (16 or 32GB VRAM). Training takes about 4 hours (resp. 6 hours) for 100k steps for the asymmetric (resp. symmetric) experiments. pair of single GPU with a batch size of 2048.
agents totals 8,792k parameters and
Training parameters (learning rate and entropy coefficient) were chosen via a preliminary grid search in the asymmetric setting.
8

---

Published as a conference paper at ICLR 2023
The code used for the analysis in this paper is publicly available GitHub repository
8 LIMITATIONS
Regarding agent modeling, we resort to simple LSTMs mostly for reasons of computational effi ciency. However, neural architectures (e.g. Transformers; Vaswani et al. 2017) may imbue agents with different inductive biases and affect the resulting strategy. On similar note, we only use very simple reinforcement learning algorithm (REINFORCE with a baseline and entropy penalty). There more sophisticated methods dedicated to multi-agent reinforcement learning in the literature (Pa-) poudakis et al., 2021) such as counterfactual policy gradient (Foerster et al. 2018) or multi-agent variants of actor-critic algorithms (Yu et al., 2021). The of such approaches may for instance enable agents to discover elaborate turn-based strategies (involving multiple questions and answers)
without the use of an explicit auxiliary reciprocal guessing objective.
Modeling overlaps is the central feature of our proposed game. For the sake of simplicity, we only considered "discrete" implementations of overlap, replacing one symbol with another. A more elaborate alternative would be to represent symbols as actual speech signals (e.g. phonemes) and overlaps the superposition of two signals. There is also some skepticism of the assumption that overlap significantly perturbs the original signal (see, e.g. Levinson 2016). Moreover, sign lan- guages also exhibit turn-taking, even though communication is not constrained by a shared auditory channel (Vos et al. 2016). This suggests that we could also model the effect of overlap (intrinsic to the agents) of producing and perceiving simultaneously (rather than a perturbation the perceived signal), although it is not clear how this should be implemented. We leave the exploration of these alternatives to future work.
difficulty
CONCLUSION
In this paper, we studied the emergence of turn-taking conventions two-way communication. We proposed a simple language game where two agents must exchange information using shared communication channel to solve a cooperative task. This enabled us to make several observations: first, neural network agents trained play this game struggled to agree effective turn based strategies. Despite this fact, agents that did engage in turn-taking tun-takiig tended to achieve higher accuracy across the board. Detailed analysis showed that while turn-based communication is not necessarily optimal in theory, in practice it is much easier for agents to learn than more efficient alternatives. We hope that this work can foster future research I. understanding the origins of turn-taking human communication and 2. developing conversational agents that can achieve turn-taking naturally.
REFERENCES
Daniel Adiwardana, Minh- Than-Thang Luong, David So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, al. Towards a human- open- domain chatbot. arXiv preprint arXiv:2001.09977 2020.
John Batali. Computational simulations of the emergence of grammar. Approaches to the evolution of language: Social and cognitive bases, pp. 405, 1998.
Daniel G Bobrow, Ronald M Kaplan, Martin Kay, Donald Norman, Henry Thompson, and Terry Winograd. Gus, frame-driven dialog system. Artificial intelligence, 8(2):155-173, 1977.
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emer- gent language game. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 981-985. 2018
Angelo Cangelosi and Domenico Parisi. Computer simulation: A new scientific approach to the study of language evolution. In Simulating the evolution of language, pp. 3-28. Springer, 2002.

---

Published as a conference paper at ICLR 2023
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4427-4442, 2020.
Yun-Nung Chen, Asli Celikyilmaz, and Dilek Hakkani-Tur, Deep learning for dialogue systems. In Proceedings of the 27th international conference on computational linguistics: Tutorial abstracts, pp. 23-31, 2018.
Patricia M Clancy, Sandra A Thompson, Ryoko Suzuki, and Hongyin Tao. The conversational use of reactive tokens in english, japanese, and mandarin. Journal of pragmaties, 26(3):355-387, 1996.
Vlad Demartsev, Ariana Strandburg-Peshkin, Michaela Ruffner, and Marta Manser. Vocal turn- taking in meerkat group calling sessions. Current Biology, 28(22)3361-3666, 2018.
Starkey Duncan. Some signals and rules for taking speaking turns in conversations. Journal of personality and social psychology. 23(2):283, 1972.
Pierre Dusart. Explicit estimates of some functions over primes. The Ramanujan Journal, 45(1): 227-251, January 2018. ISSN 1572-9303. doi: 10.1007/s11139-016-
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Yixin Gao, Yuriy Mishchenko, Anish Shah, Spyros Matsoukas, and Shiv Vitaladevuni. Towards data-efficient modeling for wake word spotting. In ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7479-7483, IEEE, 2020.
Erving Goffman. On face-work: An analysis of ritual elements in social interaction. Psychiatry, 18 19:211-233,, 1955.
Joshua T Goodman. bit of progress in language modeling. Computer Speech & Language, 15(4): 403-434, 2001.
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3700 0710, 2019.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning com- municate with sequences of symbols. Advances in neural information processing systems, 30, 2017.
Mattias Heldner and Jens Edlund. Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4)5555-56, 2010.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Adam Kendon. Some functions of gaze-direction in social interaction. Acta psychologica, 26:22-63, 1967.
Eugene Kharitonov, Roberto Dessi, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on Emergence of lanGuage in Games. https:://itthut com/ Facebokkesearcc/EEe EGG, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
10

---

Published as a conference paper at ICLR 2023
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'nat- urally'in in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 2962-2967, 2017.
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprini S.Kiv:2000.024111 2020.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations (ICLR), 2017.
Esther Levin, Roberto Pieraccini, and Wieland Eckert. A stochastic model of duman-machine in- teraction for learning dialog strategies. IEEE Transactions on speech and audio processing, 8(1); 11-23, 2000.
Stephen C Levinson. Conversational Structure, Chapter 6. Cambridge university press, 1983.
Stephen C Levinson.
Turn-taking in human communication-oriiins and implications for language processing. Trends in cogmitive sciences, 20(1):6-14, 2016.
David Lewis. Convention: A philosophical study. John Wiley & Sons, 1969.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequencc pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 7871-7881 2020.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, 2016.
Vivian Nguyen, Otto Versyp, Christopher Cox, and Riccardo Fusaroli. A systematic review and bayesian meta-analysis of the development of turn taking in adult-child vocal interactions. Child Development, 2021.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round l), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Simone Pika, Ray Wilkinson, Kobin H Kendrick, and Sonja C Vernes. Taking turns:
bridging the gap between human and animal communication. Proceedings of the Royal Society B, 285(1880): 20180598, 2018
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training.
Irving S Reed and Gustave Solomon.
Polynomial codes over certain finite fields. Journal of the society for industrial and applied mathematics, 8(2):300-304, 1960.
Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "lazimpa": Lazy and impatient neural
agents learn to communicate efficiently. In Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 335-343 2020.
Federico Rossano. Sequence organization and timing of bonobo mother-infant interactions. Inter- action studies, 14(2):160-189 2013.
Federico Rossano and Katja Liebal. Requests' and 'offers' in orangutans and human infants. Re- questing in social interaction, pp. 333-362, 2014.

---

Published as a conference paper at ICLR 2023
Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn taking for conversation. In Studies in the organization of conversational interaction, pp- 7-55. Elsevier, 1978.
Emanuel A Schegloff. Overlapping talk and the organization of turn-taking for conversation. Lan- guage sociery, 29(1):1 -63, 2000.
Claude Elwood Shannon. mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
Gabriel Skantze. Turn-taking conversational systems and human-robot interaction: a review. Computer Speech & Language, 67:101178, 2021.
Brian Skyrms. Signals: Evolution, Learning, and Information. Oxford University Press, 2010.
Jane Stephens and Geoffrey Beattie. On judging the ends of speaker turns in conversation. Journal of Language and Social Psychology, 132)4119-134 1986.
Tanya Stivers, Nicholas J Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter De Ruiter, Kyung-Fun Yoon, et al. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, (06(26)::0557-105555 2009.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. 12, 1999.
Éva Székely, Gustav Eje Henter, Jonas Beskow, and Joakim Gustafson. Spontaneous conversational speech synthesis from found data. In Interspeech, pp. 4435 4439,,449,,,0 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Oriol Vinyals and Quoe Le. neural conversational model. arXiv preprint 1rXiv:1506.05800 2015.
C de Vos, Francisco Torreira, and Stephen C Levinson. Turn-timing in signed conversations. coordinating stroke-to-stroke turn boundaries. 2016.
Marilyn A Walker. An application of reinforcement learning to dialogue strategy selection a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387-416 2000.
Joseph Weizenbaum. Eliza-a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36-45, 1966.
Ronald J Williams. Simple statistical gradient-followiin algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255 1992.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Victor H Yngve. On getting word in edgewise. In Chicago Linguistics Society, 6th Meeting, 1970, pp. 567-578, 1970.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.0/995 2021.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. Dialogpt: Large-scale generative pre-training for conver- sational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, pp. 270-278, 2020.
12

---

Published as a conference paper at ICLR 2023
A STRATEGIES FOR ASYMMETRIC GUESSING
Let us focus on the asymmetric version of the game, where only one agent (hereafter the "Guesser") must reconstruct the object from partial view è, while the other agent (hereafter the "Oracle") has access to the full object x.
In this simplified scenario, we can describe theoretical conversation strategies that allow for solving the game without overlaps. We characterize each strategy by the effective amount of bits /uesssrr and oracie of information that agents have exchange after which the guesser accurately reconstruct æ.
Naive strategy: simple protocol is for the oracle describe the entirety of the input z. This strategy requires little coordination: the guesser does not need to speak, she must simply parse the message to reconstruct æ. Given that there are N. possible objects (distributed uniformly), classical information theory (Shannon 1948) tells us that the number of bits needed to encode the objects / = log, 1/N5" -Na log, Ng Tguesser
Traall N₂ vlg N
Question-answer: Naturally the naive strategy is sub-optimal: it does not leverage the fact that the guesser only needs to uncover the N,, masked attribute. If the oracle knows which attributes the guesser is missing, she could simply send their values, which can be encoded in Nm log2 No bits of information.
However, the guesser must first communicate the position of her Nm missing attributes. The number of possible combinations of masks is given by the binomial coefficient (N N../N...N.. No..., and masks are chosen uniformly at random. Consequently the guesser's message needs to be (Na) bits long. This is particularly efficient when Nm, is small in comparison to Na. However, this strategy requires bidirectional communication: both agents must learn to take turns, and the oracle must learn to act conditioned on the guesser's 's message.
log2 (N...
Na
Tguesser loggvi++
NNm
loracle Nm log N.
Error Correcting Code (ECC): Although the turnttaking strategy outlined above appears as a natural solution, there exists taking whatsoever. Consider the simplest case where there is only N,, assume that both agents agree on a numerical encoding of each value (the integers from 1 N,). In this case it is sufficient for the oracle to send the sum of the encoding of all values in x, modulo N,. Upon reception, the guesser can simply compare this value with the sum of all N, 1 attributes observes in å to retrieve the missing value. In this case the oracle only needs to send log, No bits of information for the guesser be able to reconstruct å.
more efficient solution which doesn't necessitate any turn-
masked attribute, and
This strategy can be extended to arbitrary number of masks, borrowing ideas from the literature on error correcting codes. In particular, there is an encoding scheme based on Reed-Solomon codes (Reed & Solomon, 1960) in which the oracle evaluates well-chosen linear functions modulo N...N. the smallest prime number greater than both N, and N.. Upon reception, the guesser can provably retrieve the values of the missing attributes by solving a system of Nm linear equations in a finite field with PN..N, elements. We refer to Appendix B for a detailed explanation of this code.
ECC is the most efficient strategy far: it only necessitates communicating Nm, numbers of value up to PN..N.. leading to a total effective amount of bits of = Nm log, PN..N.- Moreover, doesn't necessitate any turn taking all: only the oracle needs to speak. However, the agent to agree upon a more sophisticated convention: in particular recovering the missing attributes from the message is not trivial.
Iguesse: =0
Toracle Nm logz PN..N.
13

---

Published as a conference paper at ICLR 2023
Moreover, the proposition 5.4 in (Dusart 2018) shows that for any 2 89693, find prime number p so that 2 2 201 +1++) //n2)). This allows us to bound PN..N, and shows that foracle Nm. logg NN,,N, is never really far from N,, log2 max(NN... NN.).
Computing the length of a communication: Knowing that agents are using discrete tokens compute the effective length L of
in vocabulary of size [V] conversation with
(including silence),
I gueesser
Loracte
L=
log ([V) +
log [IV++1)
This analysis inform choice of game parameters: for instance choose [V] and T such that the naive strategy does not allow the agents to solve the game.
B ERROR CORRECTING CODE (ECC) STRATEGY
We describe in more details on the construction of the ECC protocol. Let's assume that á is the partial view of æ, including a number Nm masks at the positions ..,...
The idea for the oracle agent is to compute Nm values, U1,-.-UN,
UN,. defined as
Uy (z) = Σ xge x- (mod PN..N.)
(I)
where PN..NN the smallest prime number so that P2 max N₂). All the values are computed modulo PN,.NN which means that we only keep the remainder of the division by PNG.N,- The amount of bits of information to transmit the values is therefore Loracle = Nn loga PN..N.-
Let show that the Nm values vi (x),.. .............................. UN... (x) are sufficient to reconstruct knowing á. Let Vn ...),......... (f) be the values computed by Agent I by applying Equation to ă, counting masks O. By comparing 2n (z) and 24 (.), obtain the following linear system
(x) 21 (f)
Tki
= W
(2)
UN.. (x) UN (i)
Ik
where
1
ko
W
(3)
Nm
le Nm N.
matrix with elements in the field Z/PN,,N, N. Z of integers modulo PN..N.- W is known Vandermonde matrix, and its determinant is given by
det(W)=
II ki) (mod PN..N.) ISRJSN..
Note that 0 < (ky- k2) < PN..N. for any í,j. This means that det(W) cannot be a multiple of PN,,,. since it is prime number. Consequently, del(W) A 0 modulo PN_,N,, which means that W is invertible in E PN.,,N Z. To find the missing values, the guesser agent can simply solve the system by inverting W.
C EXAMPLE OF A LEARNED QUESTION- STRATEGY
Figure (shoocassss a successful "question-ansee answer" based strategy where Agent first declares the masked positions and Agent responds with the corresponding values.
14

---

Published as a conference paper at ICLR 2023
Tum
100%
75%
50%
Agent Agent 2
25%
0%
Mask Prediction Accuracy (Agent 2)
100%
0%
Accuracy (Agent 1)
100%
75%
50%
25%
0%
3
7
Communication steps
Figure 6: Example of question-answer strategy developed by pair of agents in the asymmetric game with the reciprocal objective. During the first steps, Agent sends the position of the masked attributes: Agent 2 is able to correctly predict the masked positions. Afterward, Agent responds with the missing values, and Agent I's accuracy increases accordingly.
Accuracy
Overlags
Turns
1.0
1.50
3.25
3.00
1.5
0.50
0.4
Natw
0.2
Misunderstandnng
Wh Kre-Thildee
-0.25
Standard
Recipocal
Shandard
Reclorecal
Standand
Rec Recipronal
Objective
Objective
Objective
Figure 7: Influence of the overlap model on the emergent protocols for the asymmetric game.
D INFLUENCE OF OVERLAP MODEL IN THE ASYMMETRIC GAME
In Figure we depict the influence of the choice of overlap model on the performance of the agents in the asymmetric game. Similarly the symmetric game, Noise and Misunderstanding are relatively similar. Unsurprisingly, with the Walkie-Talkia model agents develop strategies without overlaps. However, contrary to the symmetric game here agents are able to develop a strategy in two turns even with this more punitive implementation of overlaps. We hypothesise that this is due to the smaller number of turns required to solve the game (only versus least 4 in the symmetric games). With only one turn change, it is easier avoid overlaps.
E QUALITATIVE EXAMPLE OF QUESTION-ANSWER STRATEGY IN THE SYMMETRIC GAME
Figure shows an example strategy developed by pair of agents trained to play the symmetric game with the reciprocal objective. Agents follow similar strategy as the asymmetric case (Figure 6 in the main text): in the first few steps, they exchange information about the positions of their masks. Based on this information, the rest of the conversation is devoted to communicating the values of these masked attributes.
15

---

Published as a conference paper at ICLR 2023
Tum
100%
75%
50%
Agent Agent 2
25%
0%
Mask Prediction Accuracy (Agent 2)
100%
75%
50%
25%
0%
Accuracy (Agent 1)
100%
75%
50%
25%
0%
Mask Prediction Accuracy (Agent 1)
100%
75%
50%
25%
0%
Accuracy (Agent 2)
100%
75%
50%
25%
0%
2
3
5
6 7 9 10 11 12 13 14 16
Communication steps
Figure 8: Example question-answer strategy developed by pair of agents the symmetric game with the reciprocal objective. During the first 4 steps, Both agents communicate the positions of their masks. Afterwards, they each respond with the corresponding missing values.
16


Reviews:
[{"review_id": "wkyZXoSwXAC", "reviewer": "Reviewer_D8G5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a setting and game by which they study the emergence of conversational strategies when eliciting and discovering information. In particular, the authors show that commonly observed (among humans and animals) turn-taking behavior does not automatically arise in agents trained on the game, but that it does result in higher performance and that turn-taking is more easily adaptable and discoverable compared to other optimal strategies.", "summary_of_the_review": "This is an interesting paper that investigates how turn-taking can evolve as a means of conversational strategy. The setting is clear and the authors investigate several paradigms of interference/penalties that map to realistic conversational situations. The results are interesting and contradict some explanations for the rise of turn-taking, and merit further study.", "strength_and_weaknesses": "Strengths:\n- The authors clearly establish the experimental setting and the possible methods of modeling overlap/interference are well justified and detailed (3.1.2).\n- Interesting discussion on the difficulty of learning the optimal strategy in a variation of the conversational game and a demonstration that turn-taking is easier to adapt to\n\nWeaknesses:\n- A small comment: in the Misunderstanding setting, using garbled random symbols seems unnecessarily punitive, as misunderstandings seem often rooted in truth and at least snatches of what is meant to be communicated.\n- It would be nice to see a comparison of how the studied effects may vary depending on vocabulary size (proxy for how complicated the language or modality of communication is) and model size", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cPJDIDUdBO", "reviewer": "Reviewer_31zg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have designed a simple language game to develop turn-taking conventions for neural agents. The game is a co-operative game, where two agents get a partial view of an object and the objective of the game is to reconstruct the original object with the lowest number of turns through mutual communication. Communication is done through a single channel that prevents both agents to speak at the same time, thus encouraging them to take turns while communicating. The partial view is constructed by masking some aspects of the original object. Overlaps were mostly modeled as noise, but other cases like misunderstanding and walkie-talkie were also experimented with.\n\nThe LSTM architecture was used to generate embeddings of the partial views of the agents. REINFORCE was used as the policy gradient algorithm for setting rewards. As for metrics, accuracy is used to measure if the prediction of the masked features. Experimental results show that agents trained under standard training objective tend to lean towards the agent with more information speaking most of the time and agents are able to solve the game 92% times, whereas with reciprocal objective, it is 98%.", "summary_of_the_review": "1. It would be nice to see how the experiments perform if BERT is used to get embeddings of the agents. BERT is one of the state of the art models in NLP for generating embeddings.\n\n2. Add more bottom padding to figure 3, as it hampers in reading.", "strength_and_weaknesses": "1. Created a simple and effective game to model the problem.\n\n2. Showed experimental results for situations that both worked and did not work.\n\n3. Could have used BERT instead of LSTM to generate embeddings.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FGZVB0-LFn", "reviewer": "Reviewer_qXss", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper takes a look at emergent turn-taking in communication. To study this, the paper sets up a language game, which is a cooperative task, and information needs to be shared over a communication channel to be successful at the task. The authors introduce a condition that disrupts communication when turn-taking is violated. Through different task settings and reward engineering, the authors show how agents can be biased towards taking turns and how taking turns might improve the efficiency of learning a task.", "summary_of_the_review": "The papers claims are not sufficiently supported by the tasks /agents tested. The tasks are too simplistic to improve our understanding of human communication or improving conversational agents.", "strength_and_weaknesses": "### Strengths:\n\n- **************************Novelty:************************** The paper is unique in its selection of its problem to understand the emergence of turn-taking.\n- ******************Clarity:****************** The paper, the methods, the metrics were quite easy to read and understand.\n- The paper provides a good framework for how a communication channel might behave when overlapping messages are sent.\n\n### Weaknesses:\n\n- My main concern with the paper and the conclusions it draws are weak because of the simplistic task setting. For instance, it is unclear how the work might extend to future directions that the authors mention: 1. understanding the origins of turn-taking in human communication and 2. developing conversational agents that can achieve turn-taking naturally.\n- The paper's conclusions (agents struggle to agree on turn-based strategies and that turn-taking agents achieve higher scores) seem pretty narrow. The paper did not convince me that they extend beyond the simple language game proposed by the authors.\n- I don’t think the speculative claims in section 6.3 of functional vs cultural transmission are sufficiently supported by the analysis.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F7anZvV2Wu", "reviewer": "Reviewer_Nn7U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors investigate the the ability of agents to engage in turn taking through a simple language based game. In this game, agents can emit information through a shared channel and using the information the agents needs to solve the task. The authors find that agents that develop turn taking achieve higher score\n\nContributions:\n1. Exploration the turn taking strategies between agents through a language based game. \n2. Initial results indicate agents that do turn taking are able to solve tasks quicker than non-collaborative agents.", "summary_of_the_review": "In this work, the authors investigate the the ability of agents to engage in turn taking through a simple language based game.", "strength_and_weaknesses": "Strength and Weakness:\n1. The basis of this work is interesting and studying turn taking in collaborative tasks in an understudied area. However, this work seems very preliminary and exploratory work and needs more analysis to be performed.\n2. Why do the agents take lesser turns with partial information in the symmetric game? Shouldn't  higher accuracy indicate more communication/exchange of informations between the agents in Table 1(b)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster