Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations
Authors: Kalesha Bullard
Franziska Meier
Douwe Kiela
Joelle Pineau
Jakob Nicolaus Foerster
Venue:  (2021)
Keywords: emergent communication
multi-agent communication
multi-agent reinforcement learning

Abstract: Effective communication is an important skill for enabling information exchange and cooperation in multi-agent settings. Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels. One limitation of this setting is that it does not allow for the emergent protocols to generalize beyond the training partners. 
 Furthermore, so far emergent communication has primarily focused on the use of symbolic channels. In this work, we extend this line of work to a new modality, by studying agents that learn to communicate via actuating their joints in a 3D environment. We show that under realistic assumptions, a non-uniform distribution of intents and a common-knowledge energy cost, these agents can find protocols that generalize to novel partners. We also explore and analyze specific difficulties associated with finding these solutions in practice. Finally, we propose and evaluate initial training improvements to address these challenges, involving both specific training curricula and providing the latent feature that can be coordinated on during training.

Under review as a conference paper at ICLR 2021
EXPLORING ZERO-SHOT EMERGENT COMMUNICA- TION IN EMBODIED MULTI-AGENT POPULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Effective communication an important skill for enabling information exchange and cooperation in multi-agent settings. Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels. One limitation of this setting is that it does not allow for the emergent protocols to generalize beyond the training partners. Furthermore, so far emer- gent communication has primarily focused the use of symbolic channels. In this work, we extend this line work to a new modality, by studying agents that learn communicate via actuating their joints in 3D environment. We show that under realistic assumptions, a non-uniform distribution of intents and a common- knowledge energy cost, these agents can find protocols that generalize novel partners. We also explore and analyze specific difficulties associated with finding these solutions in practice. Finally, we propose and evaluate initial training im- provements to address these challenges, involving both specific training curricula and providing the latent feature that can coordinated on during training.
INTRODUCTION
The ability communicate effectively with other agents is part of necessary skill repertoire of intelligent agents and, by definition, can only be studied in multi-agent contexts. Over the last few years, number of papers have studied emergent communication in multi-agent settings (Lazaridou
et al. 2016 Havrylov & Titov| 2017: Cao et al. 2018 Bouchacourt & Baroni 2018 Eccles et al., 2019 Graesser 2019] Chaabouni et al. 201 Lowe et 2019b.. This work typically assumes symbolic (discrete) cheap-talk channel, through which agents can send messages that have no impact on the reward function or transition dynamics. A common task the so called referential game, in which sender observes an intent needing be communicated to a listener via message.
In these cheap-talk settings, the solution space typically contains many equivalent but mutually in- compatible (self-play) policies. For example, permuting bits in the channel and adapting the receiver policy accordingly would preserve payouts, but differently permuted senders and receivers are mu- tually incompatible. This makes difficult for independently trained agents utilize the cheap-talk channel at test time, a setting which is formalized as zero-shot (ZS) coordination (Hu et al., (020).
In contrast, study how gasture-based communication emerge under realistic assumptions. Specifically, this work considers emergent communication in the context of embodied agents that learn to communicate through actuating and observing their joints in simulated physical environ- ments. In other words, our setup is referential game, where each message is a multi-step process that produces an entire trajectory of limb motion (continuous actions) in a simulated 3D world.
Not only does body language play a crucial role in social interactions, but furthermore, zoomorphic agents, robotic manipulators, and prelingual infants are generally not expected use symbolic lan- guage to communicate at all. From a practical point of view, it is clear that our future Al agents will need to signal and interpret the body language of other (human) agents, e.g., when self-driving cars decide whether it is safe to cross an intersection.
With that, there has been work on the emergence of grounded language for robots (Steels al. 2012, (rranger),0000) To the best of our knowledge however, we are first to explore deep reinforcement learning for emergent communication in the context embodied agents using articulated motion.

---

Under review as a conference paper at ICLR 2021
Moreover, while cheap-talk is great proxy for symbolic communication across dedicated channels, communication through articulated motion means agents have to control their joints to signal. One universal feature of this phys- ical actuation is it requires the expenditure of energy, which is a scarce resource both for bio- logical agents and man-made robots.
Discriminator network
receiver
x
Loss(s, x?
Intent library
Another ubiquitous factor of the physical world (and many other domains) is that communica- tive intents are not distributed uniformly. In particular, the Zipf distribution (Zipi) (016) is known to be a good proxy for many real-world distributions associated with human activity.
x
motion trojectories
sender
Policy network
consequence of combining energy cost with non-uniform distribution over intents in the context of referential games is that, in principle, it allows for ZS communication: Trajectories requir- ing lower energy exertion should be used for encoding more common intents, while those associated with higher energy encode less common ones.
Figure I: Embodied Referential Game
In contrast, superficially related, auxiliary losses such as entropy penalties do not allow for ZS coordination without further assumptions. While these auxillary losses are design decisions, energy cost is example of universal ccommon-knoweddee cost grounded in the environment, which can be exploited for ZS communication.
Unfortunately, training agents that successfully learn these strategies is difficult problem for current state-of-the-a machine learning models. There are three major challenges: 1) Local optima associated with the lock-in between sender and receiver: The interpretation of message depends on the entire policy, not just the state and action. While recent methods have been developed to address this in discrete action spaces (Foerster al.) 2019). to the best of our knowledge none have been proposed for costly, continuous action spaces. 2) The latent structure underlying the protocol, in our case energy, that can be coordinated on, needs to be discovered, requiring agents to ignore other (redundant) degrees of freedom. 3) Even when this structure is provided, optimization is difficult since contains a large number of local optima. As a consequence, on top of the continuous optimization problem there is combinatorial problem of ordering the energy values for each intent.
We explore and analyse these difficulties, suggesting initial steps for addressing them in our setting. First, show that providing the latent variable (energy) training time does indeed allow for some amount of ZS coordination. To do so, we adapt our method two fold: 1) During training, we change We add an observer agent that trains on an entire population of fully trained Self-Play (SP) agents. To evaluate 7S performance, we test this observer on an independently trained of SP agents. The ZS performance in this setting is around 35% for 10 intents, comparable with the most frequent class for the Zipf-distribution (34%).
the observation to only the energy value of a given trajectory. 2)
Next, we pretrain sender agents to minimize the energy associated with each intent, but ZS per- formance remains around 34% (10 intents). This is intuitive due to the challenges associated with re-ordering energy values on a 1D line without incurring a high cross-entropy while the different distributions are overlapping. Finally, we show that using the entire trajectory during SP, but only the energy value for the external observer, in combination with pretraining, consistently achieves a much higher 7S performance of around 56%.
All of this illustrates that learning an optimal 7S policy given specific set of assumptions about the problem setting, which are common knowledge to all parties, is a challenging problem, even in seemingly simple instances. Furthermore, since this work focuses on the two extreme ends this problem (self-play and ZS), our ideas are relevant for broad range of intermediate settings as well.
2

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND
2.1 MULTI-AGENT REINFORCEMENT LEARNING
We formalize the protocol learning problem decentralized partially observable Markov decision process with N agents (Bernstein et al. 2002), defined by tuple (S.A,T,R, 0, y). S is the set of states, A1 -y the set of actions for each of N agents in the population, and transition function S AxAI.... .....AA + S mapping each state and set of agent actions taken to a distribution over next states. This work assumes all agents have identical state and action spaces. In partially observable setting, no agent can directly observe the underlying state s, but each receives private observation of E Ω correlated with the state. O a, ) is the probability that o is observed, given action a led to state s'. Agent reward r; f2 x A Av - IR is a function of state and actions taken. The objective is to infer a set of agent policies that maximize expected shared return R. We employ centralized training regime with execution decentralized (Foerster et al. 2016 Lanctot et al. 2017 Rashid et al. 2018).
Policy gradient algorithms /Williams 1992: Sutton et al. 2000) are widely used for reinforcement
learning domains with continuous action spaces. Similar (Mordatch & Abbeell 2018), we use model-based policy gradient approach with a fully differentiable dynamics model for our emergent communication framework. In particular, we employ a stochastic value gradient approach, SVG- infinity (Heess et al. 2015). SVG methods compute the policy gradient through backpropagation.
2.2 ZERO-SHOT COORDINATION
Zero-shot (ZS) coordination the problem setting of agents coordinating at test time with novel partners, iee. other independently trained agents (Hu et al.) 2020). In cooperative multiagent set- tings, a key challenge is for agents to learn general skills for coordinating and communicating with other agents. Nonetheless, just as in single-agent settings, agents can overfit to their environment, in multiagent settings, agents can co-adapt with and overfit to their training partners. Thus ZS coordi- nation is useful for evaluating how well the learned agent policies generalize to unseen agents they may need to later coordinate with (e.g. new human partners).
PROBLEM SETTING
This work is framed in the ZS Communication problem setting. ZS communication is a specific instance of the 7S coordination setting, where the task for the agents is to communicate effectively; there is no additional cooperative task being solved concurrently, in the environment. The protocol learning problem then for all independently trained sender-receiver agent pairs is to converge upon optimal zero-shot policies, for communicating intent to novel partners at test time. Sender-receiver pairs engage in self-play to train communication protocols, then we use cross-play as our evaluation metric for testing ZS coordination. Cross-play (CP) is an instance of diitt---iitiittiion testing, whereby senders are paired with receivers they have never encountered nor influenced during train- ing (i.e. agent pairs tested in CP have been trained independently of one another).
In embodied problem formulation, populations are composed of spatially articulated agents. Concepts represent communicative intents and messages are instantiated as motion trajectories. In- tents are represented as discrete-valued symbols, defined priori from an intent library. The policy state space 5 consists of the joint configuration of an agent and an intent to be communicated, where the joint configuration of the agent is defined by three-dimensional position (Px, Py- Py.Pt) and three- dimensional rotation (Tx.Ty.Tz) r2) of each agent joint j in the agent's 's set of joints J. The policy action space A consists of the angular velocities for each joint E J: (rr..rry.. Ar:)). All a A are commu- nication actions which can be observed by other agents but have no effect on the environment.
We employ referential games for generating communicative motion through paired-play. At each time step r, the policy inputs the joint configuration of the agent, (Ps.Py. Fy.. Vj con- catenated with a fixed intent embedding. It outputs an action a, Ary, Vj E Transitions T: (s,,a) + 5++1 are deterministically computed through differentiable forward kinematics (FK) module, ensuring kinematically valid trajectories are generated. After T steps, the episode termi- nates, and the sequence of joint states visited and velocity actions taken is concatenated to compose a motion trajectory. The observer model takes the motion trajectory as input and predicts the actor's s intent. The shared return for the episode is the cross-entropy loss between the observer's prediction
3

---

Under review as a conference paper at ICLR 2021
and the ground truth intent; this return R is backpropagated from observer to actor, since we use centralized training regime for the mulli--gent system. The FK module has been adapted from an existing motion library (Holden et al., 2017) to be differentiable. Figure Iillustatte the embodied referential game. Algorithm| provides an overview of training for the game.
Importantly, for continuous domains, the injection of noise is critical. During trajectory generation, Gaussian noise is added to each action degree of freedom, both to represent imperfect observability and to induce exploration. Noise helps regularize the communication channel, in that finite noise level essentially provides a quantization of the continuous space, a "coarse graining', such that tra- jectories mapped different intents are distinguishable. This prevents collapse of the trajectories.
4 METHODOLOGY
Given that we aim to develop agents with general communication skills (policies that are not simply locally optimal to their current communication partner), it is useful to first consider why converging on an embodied protocol that generalizes to novel partners is challenging problem.
Since embodied communication is physically instantiated, it occurs through manipulating the ve- locities of agent joints; thus, communication through motion generation implies high-dimensional continuous action space. Combining the training criterion with such high dimensional action space results in a highly non-convex optimization surface. This means the optimization landscape has many local optima (reasonable solutions) for inferring protocol. It can be difficult for local optimization algorithm to navigate this type of landscape in search of a global optimum. would also be sensitive to where in the policy space policy parameters are initialized. Because of these chal- lenges, our approach proposes to induce latent structure during protocol training, to provide similar grounding for how independent actors generate communication. This structure can subsequently be exploited to improve 7S coordination at test time.
4.1 INDUCING IMPLICIT LATENT STRUCTURE THROUGH PHYSICAL ENERGY EXERTION
To implicitly induce latent structure in the policy learning process (and thus trajectory generation), we propose to use Energy Regularization coupled with a Zipf distribution over intents. The regular- izer enforces minimal energy trajectories for the protocol, and Zipf imposes a monotonic ordering upon intents, based upon likelihood. Coupled, they incentivize an inverse relationship between en- ergy exertion and intent frequency, assigning minimum energy trajectories to maximally occurring intents. Moreover in principle, there exists a global optimum for the protocol: energy values can be ordered to be strictly increasing, and intents can be ordered to be strictly decreasing. Then presum- ably, a 1:1 mapping between energy values and intents can be induced.
In practice however, this is a very challenging optimization problem, primarily because the opti- mization surface has many local optima for inferring protocol. Given the latent structure, there is additionally the problem of strictly ordering energy values, which is combinatorial in the number of intents. Different local optima will solve this problem in different ways, and because energy values are continuous, reasonable solutions can be obtained (and achieve low loss) without matching the correct ordering. This especially true as the number of concepts to be learned grows and increas- ing numbers of intents have very similar likelihoods of occurring. Additionally, because the learning problem for each agent is to infer one policy that effectively communicates multiple intents, if the optimization algorithm gets stuck in local optimum, finding better local optima likely requires non-local traversal through the policy space. This because the algorithm may need to change the actions for multiple intents concurrently, to locate a better solution in the space.
For inducing implicit structure, there are two objectives traded off: maximization of communication success (Equation I) and minimization of energy exertion (Equation 2). Let us denote the set of intents (goals) to be communicated G and motion trajectory 7 E T, the set of all trajectories. We employ L.2 torque loss where is the moment of inertia and the angular velocity, for all agent joints. The total loss is a linear combination of prediction (cross-entropp and energy (torque) losses.
Lpred logg po (8 g" r)
Lengy 1 (Ø). @D.T 11
(2)
4

---

Under review as a conference paper at ICLR 2021
The latent structure is implicit because agents never made aware that any predefined, exploitable structure exists, nor are they directly incentivized to discover a more compact representation. This means if agents fail to autonomously discover the induced structure, they will be relegated to overfit- ting to trajectory input with their current partners and though they might perhaps find some structure, it will be insufficient for ZS coordination with novel partners. For this reason, in our experiments, we examine two ways of giving imput to observer models: (1) provide actor trajectories generated, as this ideally the goal, that the observer can autonomously discover any latent structure and exploit it for decoding messages, and (2) directly provide the latent energy values, as this allows decou- pling of successfully induced structure in the policy learning from the successful discovery of that structure for the interpretation of messages. Both are necessary for successful coordination.
4.2 PROVIDING EXPLICIT LATENT STRUCTURE
For explicitly providing structure, we define a set of latent features on trajectories Φ(T). This structure is intended to reduce dimensionality of messages passed, while preserving informativeness of messages. Thus we aim to induce a relationship ((;;)) ==tt))/(f ((;;((()))) >> 0, to achieve high mutual information between G and db This relationship can be expanded as I(G; H(G) (G) However, the intent distribution is given as Zipfian, so HG)) is held constant. Thus, as is generally true in protocol learning, maximizing mutual information between goals and messages implies:
((t) +1 g =
min ()
(p(g +0 else
This implication is critical for ZS coordination, in the absence of data to bias how agents learn to generate trajectories. It suggests that even a new observer does not know how a particular actor communicates, i.e., it has not learned a mapping between actor trajectories and intents, if it can successfully infer the latent structure for encoding trajectories, it can still decode the message.
4.3 THIRD-PARTY OBSERVER EVALUATION
We evaluate policies learned using third-party (external) observer, not trained on the protocol. Learned policies are frozen and the external observer initialized randomly. The population is split into disjoint training and test sets. At each iteration, trains by only observing actors in the training set, and tests on its ability to correctly interpret intent of actors in the test set. Since all actors are trained independently, the ability understand these unseen partners represents ZS communication.
EXPERIMENTS AND RESULTS
Our experiments analyze: (1) the value of inducing implicit latent structure embodied protocol learning, (2) the impact discovery the latent structure by observer agents - in the ZS coordi- nation setting, and (3) challenges associated with optimizing in continuous space for a monotonic ordering on energy values. We also provide some qualitative insights about learned policies. The experimental setup is detailed in the Appendix.
5.1 EFFICACY OF INDUCING IMPLICIT STRUCTURE IN PROTOCOL LEARNING
This first set of experiments in Figure 2 reports communication success during training, on task with 10 intents. Results for our experimental condition (Energy Zipf) compared against all three ablations. Figure 2a shows all conditions successfully converge on a protocol with their train- ing partners (SP), though conditions trading off two objectives converge more slowly. Figure 2b shows communication success for each actor paired with wnseen observer agents in the population (all <actor, observer> pairs that did not train together) (CP). This represents an diiittiiiiiiitoo test case. It shows that none of the models are able to immediately generalize to new partners (at test time), elucidating the need for additional third-party observer training, to increase the likelihood of communication generalization. It also shows there is value in using nonuniform distribution, as it enables independent agents to agree a priori on how to prioritize intents for future interactions. Lastly, from this experiment, found that while adding energy objective does slow convergence, reduced energy exertion by several orders of magnitude, which can critical in reducing resource consumption and enabling prolonged operation for practical embodied systems (e.g. robots).
5

---

Under review as a conference paper at ICLR 2021
engy
Lnif
200 ongs
Gome 20ccess
engy
0.4
0.2
02
30
20
30
40
10
20
30
40
50
1amm Epochs (In thoustndds)
Num Esochs Cn miousnnds
(a) Training Curves (SP)
(b) Out-of-Distriuttin Test Curves (CP)
Figure 2: Learning Curves, associated with the four experimental conditions described. Subfigure 2a shows communication success with training partners (SP). Subfigure 2b shows communication success of each actor with all unseen observers in the population (CP), as actor policies are being trained. The latter out of distribu- tion, since (actor, observer) pairs train independently. Plot illustrates that while protocol training converges to near perfect performance, generalization of protocol to novel partners is extremely challenging.
5.1.1 EFFICACY OF IMPLICIT STRUCTURE IN DISCRETE- CHECTELL PROTOCOLS
To further investigate the value of coupling Zipfian intent distribution with an energy cost objective for learning ZS policies, we evaluate a simple discrete-channel communication domain. In this simplified domain, we consider protocols with 5 intents and examine two learning tasks. In the first task, there are 10 discrete actions each sender agent can take, each assigned own unique energy value: (aa) id).. Examples of learned protocols for this task are visualized in Figure 3 For achieving both maximal prediction accuracy and minimal energy exertion, each intent must be mapped to the lowest energy value it can be assigned, but two intents can be mapped to the same action. The bottom protocol Subfigure 3b illustrates a globally optimal solution for this domain in the ZS setting, where discriminability is maximized and energy minimized.
For quantitative analysis on the value of inducing energy costs on top of Zipfian distribution, trained five independent sender-receiver pairs, both without energy penalty and using the penalty to induce the energy-based latent structure. For both conditions, we computed SP and CP performance between and across agent pairs, respectively. While SP performance protocol training is comparable for the conditions, CP performance is 0. 0.12 12 and 0.58 + 0.22 for the zipf (only) condition zipf energy conditions, respectively. Although this suggests significant variance in performance across pairs for CP evaluation, the key finding is still clear. Additional results for both learning tasks in this domain are discussed in Appendix Sections A.4 and A.5| We also contribute a colab notebook an instructive tool for exploring protocol learning in this simple ! Overall, findings are consistent across tasks: coupling a Zipfian distribution with the energy objective is important for learning more general (ZS) communication skills. We continue our continuous-action protocol experiments using the implicit energy-based latent structure.
the end of
5.2
IMPACT OF LATENT STRUCTURE FOR ZERO-SHOT COORDINATION
While the previous set of experiments highlight the potential value of inducing the proposed im- plicit latent structure, we now analyze proof-of-concept task, to extract insights about solving the problem setting of ZS coordination using a high-dimensional continuous communication channel. Table shows 7S results given by the external observer evaluation, for a N = concepts task.
N = Intents Task
N = 2 Intents Task
Test Input
Test Input
((t)
t
Φ(r)
Train
0.75
0.97 0.997
Train
t (Tr)
0.66
0.995
Input
((T)
0.67
Input
0.67
0.999
(a) Curriculum
(b) Torque Curriculum
Table 1: CP with External Observer. Proof-of Conceet Task: 2 Concepts. In both settings (with without curricllum) near-perfect ZS coondination can he achieved.
Colab notehook for experimenting with simple Diserete Domain http://shorurllaallHHPP

---

Under review as a conference paper at ICLR 2021
(a) Zipf (2 Runs)
(b) Zipf + Engy (2 Runs)
Figure 3: Protocols Learned in Discrete- Chrrnee-hhhnnel Domain. Two independently trained agent pairs (top, bottom) per condition. Left column shows sender policy mapping given intents to communication actions (messages). Middle column shows receiver policy mapping sender actions to predicted intents. Right column shows SP Performance (p(predicted - intent true intent)) at the end of Protocol Training. Illustrates both conditions train sender policies communicate effectively with training partners, but only the condition with the energy-basee structure (3b) produces policies sufficiently similar for generalization beyond training partners.
In the tables, rows represent the type of input given to the observer model at training time and columns represent the type of input given at test time. Inducing Zipf distribution on the intents, the likelihoods are [0.67, 0.33] on concepts and 2, respectively. Thus using a max class classifier as a reasonable baseline for comparison, would expect success rate of 0.67. Bold figures highlight conditions that achieved communication generalization, success beyond that of the baseline. The results show perfect ZS coordination only when providing the external observer with the explicit latent structure (energy values), independent of what information was provided during training.
To gain more insights about why agents able to generalize well, given the latent energy vari- able, we visualize a distribution over energy values for N = intents, for the best performing agent (Figure 4). Only given energy values test time, the only way for an external observer to per- form with such high accuracy is with very little overlap between the intent-conditioned Gaussian distributions. Looking at Figure this is exactly what we see. The left plot shows the condition given trajectory input at training time and the right plot given the latent variable input. The trend is consistent and very clear. Given only these energy values cessfully decode the message, independent of the way the specific actor qualitatively communicates through motion. This means the implicitly induced latent structure successfully creates a maximally informative relationship between intent and energy exertion, characterized by Equation
time, an external observer can suc-
Figure illustrates snapshots of sample behaviors learned for one agent in the population on the N=2 task. They visualized using the FAIR Motion Library (Gopinath & Won 2020). From this visualization, we observe qualitatively distinct communicative behaviors for the two intents.
Teggo Coetton
Enegge Sumtion
Trajectory (Training Input)
T.TT Latent Variable (Training Input)
Figure 4: Energy Exertions of Intents 1 (blue) and 2 (orange). Task = Intents. Shows intent-conditooned Gaussian distributions approximate energy exertion of one agent, given both types of Training Input. High ZS coordination is achieved this lask because there is minimal overlap between intent energy distributions.

---

Under review as a conference paper at ICLR 2021
(a)=1
(=)
(c) 12
t=) 18
(=) 24
t= t= 30
g=)=
t=
(1)== 12
()) t 18
(k) t= 24
(= 30
Figure 5: Comparison of T Learned Behaviors for N=2 Intents Task. Illustrates one instantiation of how an agent learns generate communication for distinct intents in the protocol. Top Intent Bottom Intent 2. Note: Although the time herizon 7 for policy learning, trajectories are upsampled visualization purposes to T = 30. So visualized snapshots are evenly spaced throughout an entire generated trajectory.
The primary insight from this proof-of-coneey experiment is that dination with this energy-based implicit structure underlying the protocol. However, coordination with novel partners remains improbable unless the latent structure is first discovered by the agents.
indeed achieve ZS coor-
5.3 IMPROVING COMMUNICATION GENERALIZATION WITH LEARNING CURRICULUM
= Intents Task
N 5 Intents Task
Test Input
Test Input
t
+t())
t
<((T)
Train Input
0.37
0.76
Train
0.44
0.70
((t)
0.44
0.60
Input
((r)
0.44
0.70
No Curriculum
(b) Torque Curriculum
N
10 Intents Task
N 10 Intents Task
Test Input
Test Input
Φ(t)
t
Φ(7)
Train
0.30
0.61
Train
t
0.56
0.56
Input
+(t)
0.34
0.55
Input
(())
0.35
0.56
(c) No Curriculum
(d) Torque Curriculum
Table 2: CP with External Observer. For tasks of increasing complexity: 5 and 10 intents. ZS Coordination is maximized when using energy curriculum and providing the explicit latent structure.
Observing such high ZS coordination success on the proof-of-concept task is very promising. How- ever, we also aim to characterize performance the complexity of the communication task in- creases. With this in mind, we investigated generalization with an external observer on tasks with 5 and 10 communicative intents, respectively. The left side of Table ordination results after training protocols with the implicit latent structure. The most likely intents sampled with likelihood of approximately 0.44 and 0.34 for the N 5 and N 10 concepts tasks, respectively; so these probabilities form baselines for comparison against a naive max class classifier. Bold figures highlight conditions with some communication generalization. In the case of the N = 5 task, we see a consistent trend as observed in our proof-of-concept task: the external observer can successfully generalize to unseen actors only when given the explicit latent structure. Though notably, ZS coordination success is substantially diminished with this more difficult com- munication task. Consistently for all three tasks, when given trajectory input, the external observer cannot outperform simply selecting the most likely intent. Furthermore, with an increase in task complexity to 10 concepts, the external observer never performs better than this baseline strategy.
(2 and 2ch shows ZS co-
In analyzing these trends more closely, it is important to consider that as the number of concepts increases, the number of ways to order the corresponding energy values increases combinatorially. It follows that the complexity of the task increases, the optimization algorithm becomes more susceptible to getting stuck a local optimum. We conduct analysis on optimizing with the energy objective, to better understand the behavior of the energy optimization and accordingly decide how to address the most salient challenges arising. Figures ba and 6b show the energy loss over time on the N 5 concepts task, given both types of input. The plots subsample energy loss values over the
8

---

Under review as a conference paper at ICLR 2021
Energy Oct on over
Energy motim over
3200
Veeaten Mantbr
10000
Laraton Mambbar
6000
20000
30000
25000
12000
30000
00
0.7
00
0.4 175 Ty 0/52
2.3
0.7
0015 TJy C.52
(a) Trajectory Input (No Curriculum)
(b) Latent Input (No Curriculum)
Figure 6: Energy Costs by Intent, sub-sampled throughout duration of protocol training. Energy cost for each intent proceeds over time- from top to boltom. Task concepts. Training with no curriculum. Shows cor- rectly mapping intents by rank to continuous-valin valuud energy costs is challenging for the optimization algorithm. Even sufficiently good solutions converged upon can still be suboptimal (see intents 4 and 5, subfigure (b).
entire duration of training, for each run. Each intent is represented by different colored line and shaded horizomtal lines show the variance of the energy loss, at a particular point in training.
key observation from this analysis repeated repositioning of intent ordering as training traverses, and there is noticeably more repositioning of intents when trajectory input is given. This can be ex- plained by the dimensionality of the trajectory input being orders of magnitude larger than the scalar latent input; thus there are many more degrees of freedom in the search space for the optimization algorithm to manipulate the positioning of the intents. Yet for both input types, there is very little repositioning of intent 1. The Zipfian imposes the following approximate distribution over five ele- ments: [0.44, 0.22, 0. 14,0.11, 01,0099. Notably, the difference between intent 1 and any other intent is much larger than the difference between any other pair of intents. This suggests that the optimization algorithm would incur too much cost if it orders intent incorrectly, so energy cost of intent 1 gen- erally remains lowest throughout training. By contrast, the positions of much less likely intents (e.g. 4, 5) continue to shift (particularly given trajectory input) and at many points during training, are incorrectly ordered as the optimization algorithm searches for sufficiently good solution. This im- plies that the constant shifting of intent-conditionn conditioned energy values is bound to impact coordination between actor and observer, since influences trajectories generated for communication.
From this insight, we decided to pretrain actor policies on energy, by first minimizing torque in trajectory generation for all intents uniformly. Our hypothesis was that training the protocol with an energy-based curriculum may help to reduce the amount of intent repositioning necessary to find the optimal ordering, since the intents will all start with comparable energy exertions. The right side of Table 2 (2b) and 2d) shows ZS coordination results after training protocols with the latent structure and torque- torque-based curriculum. The torque curriculum seems to only improve generalization notably when the external observer is given trajectory input for the hardest communication task (N 10).
Overall, two key insights emerge from our experiments and analyses: (1) it is advantageous to induce an energy- latent structure for learning generalizable communication protocols and (2) explicit discovery of the latent structure is separate yet critical piece for ZS coordination to occur.
CONCLUSION AND FUTURE WORK
We have presented a first exploration into emergent non-verbal communication in the context of embodied agents in high-dimensional simulated environments. We show that under mild assump- tions, the grounding provided by the physical environment should allow our agents to learn protocols that can generalize to novel partners. Yet we also find that the current approaches are more brittle than one might hope. We hypothesize that better robustness could be achieved by methods that can discover the maximum possible coordination from a given level of common knowledge ('ground- ing") in the environment. Specifically, this may require distinguishing between those aspects of the optimization problem that are idiosyncratic to a given agent (or might even be shared knowledge) compared to those that common knowledge (and therefore be used to coordinate). Overall, while there are many interesting open challenges that remain, this work opens up exciting avenues for exploring continuous action communication protocols in virtually or physically embodied agents.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emer- gent language game. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 998, 2018.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In International Conference Learning Representations, 2018.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient en- coding in emergent communication. In Advances in Neural Information Processing Systems, pp- 6293 6303, 2019.
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emer- gent communication in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 13111-13121, 2019.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent communication in a multi-modal, multi-step referential game. In International Conference on Learning Repre- sentations, 2018.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-21444 2145, 2016.
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent re- inforcement learning. In International Conference on Machine Learning, pp. 1942-1951, 2019.
Deepak Gopinath and Jungdam Won. Fairmotion tools to load, process and visualize mo- tion capture data. Github, 2020. URL https::/github..co httpttuw..co/fcccbbooreeeeeeeeee
rmot Fairmot ion
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing
pp. 3691-3701, 2019.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning com- municate with sequences of symbols. In Advances in Neural Information Processing Systems, pp- 2149-2159, 2017.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944 2952, 2015.
Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character con- trol. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. Other-play for zero-shot coor- dination. In International Conference Machine Learning, 2020.
Satwik Kottur, José Moura, Stefan Lee, and Dhruy Batra. Natural language does not emerge 'nat- urally'in in multi-agent dialog. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 2962-2967, 2017.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified thee-the approach to multiagent inforcement learning. In Advances in Neural Information Processing Systems, pp. 4190-4203, 2017.
10

---

Under review as a conference paper at ICLR 2021
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182, 2016.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations, 2018.
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 693-701. International Foundation for Au- tonomous Agents and Multiagent Systems, 2019a.
Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In International Conference on Learning Representations, 2019b.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp. 4295-4304, 2018.
Michael Spranger. The evolution of grounded spatial language. Language Science Press, 2016.
Michael Spranger and Lue Steels. Emergent functional grammar for space. Experiments in Cultural Language Evolution, --207-232, 2012.
Luc Steels. Language games for autonomous robots. IEEE Intelligent systems, 16(5):16-22, 2001.
Luc Steels and Michael Spranger. Emergent mirror systems for body language. Experiments in Cultural Language Evolution, pp. 87-109, 2012.
Luc Steels, Michael Spranger, Remi Van Tripp, Sebastian Höfer, and Manfred Hild. Emergent action language on real robots. In Language grounding in robots, pp. 255-2766 Springer, 2012.
Richard Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- lion processing systems, pp. 1057-1063, 2000.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
George Kingsley Zipf. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio Books, 2016
A APPENDIX
A.1 EMBODIED AGENT MORPHOLOGIES
Embodied agents explored in this work have articulated bodies. An articulated body is collection of rigid bodies (links), connected by set of joints e The body of agent is constrained kinemati- cally, such that links cannot detach from one another. Topology defines a tree structure for the body, where each joint is node and edges denote link between joints. The root joint, jo- is the root of the tree. Only the root joint has six degrees of freedom (DOF): Pe: Ty. Since the body must move as one entity and thus links cannot detach from one another, given no further joint limits or constraints, all other joints have the ability only to rotate freely in Euclidean space. So all deriva- tive (child) joints j; where o have three DOFs: (v.).. For simplicity, we constrain all agent joints with only the DOF' to rotate in three-diment ftree-dimensional space. This means agents cannot navigate through the environment; their bodies can only move in place. Moreover, a predefined topological structure implies trajectories generated must adhere the kinematic constraints of the agent given a priori. We employ a forward kinematics (FK) transition model to ensure valid transitions between joint states. The FK module is adapted from (Holden al. 2017) to be differentiable.
rz).

---

Under review as a conference paper at ICLR 2021
A.2 PROTOCOL TRAINING
Algorithm Training Overview
Let < set of communicative intents (goals)
2: L f population of agents
3: Randomly Initialize 0, #(policy and discriminator models, for all agents)
4: for ir Iterations do
5: # *Train Sender ++ Receiver
6: B B~G #(sample batch of intents, according to predefined distribution)
7: for g E B do
8:
< Pb #(agent reference pose)
9:
for T do
(tss 8) < Zss, gl #(compute policy action given state and target intent)
11:
a; + (xs, 8) + ~() #(add noise for exploration and imperfect actuation)
12:
$;41 a)(); #analytically compute next state using given agent kinematics)
13:
end for
14:
(2, 57 #(compose actor trajectory sequence of joint states and actions)
15:
8 g (o(=) #(compute observer prediction intent)
16:
R=
-(g.gg #(compute shared return)
17:
Update model parameters θ +
18:
end for
19:
end for
A.3 EXPERIMENTAL SETUP
We design an Intent Recognition referential task for our experiments and consider a library of 10 communicative intents (concepts) given priori (e.g. "yes", "no", "turn left", "come here"). Each population instantiated with a size of agents. The agents are not situated in any physical or virtual environment. At each iteration of the game, batch of 1024 intents is sampled, based upon the specified distribution (uniform or Zipf). Each sender agent generates a batch 1024 intent-conditioned trajectories. Receiver agents able directly observe positions and orien- tations of sender joints. However, noise added to each action, to represent imperfect observ- ability. Noise emission is modeled through unimodal Gaussian distribution N(O, σ), with fixed {o, 2.0, a, = 0.4) values for joint positions and rotations, respectively. Here, position values are assumed be measured in feet and rotation values in radians. Both policy and discriminator models are 3-layer feed-for forward neural networks, with ReLU activations on all layers except the final. The discriminator network uses a SoftMax for the final layer, and the policy network uses no activation in the final layer, since it is allowed to output a broad range of continuous action values.
We experiment with robot arm morphology in this work, motivated by potential application do- mains where non-verbal communication is most relevant, i.e. robotics and graphics. The agents in the population each have articulated joints. No constraints were imposed on agent joints, so as to allow maximal range of motion. We did however employ action equivalences, whereby the modulus operator was applied to joint angles predicted by the policy network; this confined all joint angle values to the range of [0, 2r). We employed the same learning rate for both the policy and discriminator networks, le - 3. As simplifying assumption, we consider a small time horizon (T 5), but upsample the number of frames for visualization.
For the external observer evaluation, we randomly initialize new discriminator model and split the original community, with 80% of the agents assigned to training set and 20% assigned to the test set. In each iteration, the training actors each generate a batch of 1024 intent-conditioned trajectories, based upon the sampled Zipf intent distribution. The external observer is also tested at each iteration with the trajectories generated by the test actors, who also each generate a batch of 1024 trajectories. The external observer experiments were run for 500 iterations on N and N 5 concepts tasks and 1000 iterations on the N = 10 task, though learning stabilizes after approximately 100 iterations.
A.4 DISCRETE DOMAIN: TASK 1 ADDITIONAL RESULTS
The first task in this simple discrete-channel domain considers 10 discrete actions each sender agent can take, each assigned its own unique energy value: Φ(aa) = =i)()). Examples of three independently
12

---

Under review as a conference paper at ICLR 2021
trained protocols for baseline (Zipf only) and experimental (Zipf + Energy Penalty) conditions of this task are visualized in Figures 7a and Sa respectively. Visual illustrations of pairing off the senders with unseen receivers from the set in CP (for evaluation of ZS coordination) are shown in
Figures 7b and 8b These are juxtaposed with SP pairings and protocols trained (Figures and and 8a
A.5 DISCRETE DOMAIN: TASK 2 (PROTOCOL LEARNING GIVEN ACTION DEGENER
The second task in the simple discrete communication domain also considers protocols with intents and only five distinct energy values. However, there are now 17 discrete actions, which implies action redundancy. In particular, one action has unique, lowest energy value; the other energy values are each mapped to four different actions.
In Physics, a degeneracy denotes the number of distinct quantum states a system that have a given energy. Analogously, this experiment investigates what happens in the degenerate case where several actions can be mapped to the same energy cost. Thus even in optimizing for minimal effort protocols, there many ways converge upon a globally optimal protocol, and consequently. ZS coordination becomes more challenging. This is because a sender-receiver pair can both maximize disriminabilitty and minimize energy by assigning different actions (but with the same low energy exertion) to different intents.
This gets closer challenges faced with continuous action protocols, where as the likelihoods of different intents grow arbitrarily close together, a sender may decide to employ visually distinct trajectories, but with comparable (low) energy exertions, different intents. When this happens, becomes exceedingly difficult to differentiate intents (for new receiver agent) based upon the latent energy values, and ZS communication performance degrades.
Examples of learned protocols for baseline (Zipf only) and experimental (Zipf + Energy Penalty) conditions of this task are visualized and zoomed in on in Figures g and 10 respectively. Those same protocols are juxtaposed with visual illustrations of pairing off senders with unseen receivers
for CP, Figures [1 and 12
For quantitative analysis on the value of inducing energy costs on top of Zipfian distribution, for both conditions, we computed SP and CP performance between and across agent pairs, respectively. Like the simpler (first) task in this discrete domain, while SP performance at the end of protocol training is comparable for the two conditions, CP performance is notably different between them. In particular, CP success was 0 and 0.35 + 0.2 for the zipf (only) condition zipf + energy conditions, respectively. As an additional point of comparison, the most frequencly occuring class for N = intents occurs with approximately 0.44 probability. So even in the experimental condition which performed better, ZS coordination capability does not exceed simply guess the maximally occurring class; this speaks the inherent difficulty of learning ZS policies, even in seemingly simple domains. The empirical findings show evidence that on this harder communication task (with action degeneracies), ZS coordination is more difficult achieve than as compared the simpler task where all actions were assigned unique energy values. This was expected, but still useful to confirm. Lastly and importantly, these findings suggest that even when action degeneracies exist, coupling a nonuniform distribution with the energy objective is helpful for maximizing generality of communication skills learned. This finding is also consistent with what we observed on the simpler task in this discrete communication domain.
13

---

Under review as a conference paper at ICLR 2021
(a) Zipf: Training Pairs (SP)
Zipf: Pairings Tested (CP)
Figure 7: Protocols Learned and Tested for 7S Coordination in Discrete-Chanenn Channell Domain (Task 1). Zipf only Baseline Condition. Protocol Training (SP) and Evaluation (CP). Illustrates how challenging ZS communica- tion (7b) right column) is compared communication success with training partners (7al right column). It is made harder by the senders (7al left column)
policies learned during initial protecol training phase are very different, across
14

---

Under review as a conference paper at ICLR 2021
(a) Zipf Engy: Training Pairs (SP)
(b) Zipf Engy: Pairings Tested (CP)
Figure Protocols Learned and Tested 7S Coordination Discrete-Channel Domain (Task 1). Zipf + Energy Experimental Condition. Protocol Training (SP) and Evaluation (CP). Illustrates that while 7S communication (8b) right column) is somewhat more challenging than success with training partners (8a right column), ZS coordination from protocols built upon energy-based latent structure noticeably more successful than ZS coordination baseline condition with no energy penalty (&b right column). This can observed by higher prediction probabilities (shown as yellow boxes) occurring with significantly greater frequency along the diagonal (meaning sender and receiver agree on intent) than in baseline condition.
15

---

Under review as a conference paper at ICLR 2021
Comnlitz 1208)
ataraa111 Recdiir molic
Teatig 33591
Sonden Pollcy
Rocoker
Trarep 22291
Sander
eNciPtrr cende1
(a) Zipf Runs)
Figure 9: Protocols Learned in Discrete-Channel Domain, with Action Degeneracies. Three independently trained agent pairs (individual rows) in baseline condition. Left column shows sender policy mapping given intents communication actions (messages). Middle column shows receiver policy mapping sender actions predicted intents. Right column shows Performance (p(predicted intent true - intentt) of Protocol Training. Illustrates condition can train sender policies to communicate effectively with training partners, but there is significant variance across sender policies learned. This would make finding common structure and exploiting it generalize novel partners very difficult (perhaps impossible since common structure is evident).
end
16

---

Under review as a conference paper at ICLR 2021
Recdket Polic
Thenii Coutit- 3339
Sender Ps kry
Ieserta Recoberiiiity
Trankng Cooelite 35991
Sander Policy
(a) Zipf Engy (3 Runs)
Figure 10: Protocols Learned Discrete-Channe Domain, with Action Degeneracies. Three independently trained agent pairs (individual rows) in experimental condition. Left column shows sender policy mapping given intents to communication actions (messages). Middle column shows receiver policy mapping sender actions predicted intents. Right column shows SP Performance (p(predicted intent the end of Protocol Training. Illustrates condition using energy- latent structure both train effective protocols between training partners and induce similar structure in policies leamned, across sender agents, for enabling better communication generalization (to unseen partners).
intent))
17

---

Under review as a conference paper at ICLR 2021
(a) Zipf: Training Pairs (SP)
(b) Zipf: Pairings Tested (CP)
Figure 11: Protocols Learned and Tested for 7S Coordination in Discrete-Chamme Domain given Action Degeneracies (Task 2). Zipf only Baseline Condition. Protocol Training (SP) and Evaluation (CP). Illustrates how challenging ZS communication [Tbb right column) compared to communication success with training partners [Tla] right columa). made harder by the that policies learned during initial protocol training phase are very different, across senders [Ta left column).
18

---

Under review as a conference paper at ICLR 2021
(a) Zipf + Engy: Training Pairs (SP)
(b) Zipf + Engy: Pairings Tested (CP)
Figure 12: Protocols Learned and Tested for ZS Coordination in Discrete-Channel Domain given Action Degeneracies (Task 2). Zipf + Energy Experimental Condition. Protocol Training (SP) and Evaluation (CP), Illustrates ZS communication (12b right column) substantially more challenging than success with training partners [12a] right column). Nonetheless, ZS coordination from protocols built upon energy-based latent structure slightly more successful compared against ZS coordination in baseline condition with no energy penalty (12bl right column).
19


Reviews:
[{"review_id": "vmD3HxBQ8q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the zero-shot emergent non-verbal communication in this paper. Different from most papers on emergent communication. this paper uses the motion of three-joint agents. The agents meet partners that they have never seen in the training phase, presenting the challenge of the universal protocol. To make a universal protocol possible, the authors study intents sampled from Zipf distribution and energy regulation. The authors conducted experiments on tasks with 2, 5, 10 intents. The results show that providing latent energy feature is essential for zero-shot coordination. To achieve better than chance accuracy on tasks with 10 intents, a torque curriculum is needed. \n\nPros:\n1. The setting of ZS coordination with non-verbal emergent communication is novel and interesting. \n2. The use of energy regulation and Zipf distribution is natural and intuitive. The results also support the intuition. \n\nCons:\nWhat is the aim of using motion as a communication protocol? In human communication, actions are often used as an iconic gesture to describe intents that are not well described by words. In this sense, humans can perform zero-shot non-verbal emergent communication since the iconic gestures are always universal. In other words, a grounded world should be used to study ZS coordination. This paper seems to study a problem that is not realistic. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "GmeiePmBaS8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how gesture-based (non-verbal) communication can emerge in embodied multi-agent population.\nIn their problem setting, there is a set of agents, and an observer. Given an intent sampled from a Zipf distribution, the agents generate energy-efficient trajectories, and the observer predicts the intent given these trajectories. Zero-shot coordination is enabled by combining energy cost with Zipf distribution (which is monotonic), since energy cost is grounded in the environment. The authors evaluate zero-shot coordination performance by evaluating a third-party observer trained and tested on two separate subsets of agents from the population respectively.\n\nPros: \n1. The idea of using energy cost and combining energy cost with a non-uniform distribution for emergent communication is new.\n2. The idea of exploiting common-knowledge grounded in the environment for zero-shot coordination is new and interesting. \n\nCons:\n1. The paper is sometimes hard to follow with missing details.\n2. The authors failed to discuss options other than energy cost for ZS coordination, and why energy cost is preferred compared with other options.\n3. The assumption of a monotonic ordering for intent distribution is strong. The assumption that energy cost is available during training/testing is also strong.\n\nI like the idea of exploiting a universal (common-knowledge) cost grounded in the environment for zero-shot coordination.\nHowever, is energy cost the only option here? what about trajectory length (spatial), cumulative angular velocity change, etc. I understand energy cost may be a better option, but the authors should discuss and compare with other options. Meanwhile, in many multi-agent environment, the only available inputs are visual inputs and energy cost may not available. Finally, even if the learned policy achieved zero-shot coordination with a third-party observer, it is not human interpretable since humans cannot directly observe energy cost. Any thoughts on learneing human-interpretable gesture-based language?\n\nWhile Figure 1 is \"Overview of Learning System\", it does not really helps the readers to get the key idea of the paper. It is better to have a more detailed figures explaining your problem setting and motivation.\n\nThe assumption that the intent is from a monotonic distribution seems a little bit strong to me. It is better to have experiments with other distributions (not uniform). e.g. a subset of intents has equal probabilities to be selected.\n\nIn Section 3 Problem Setting, you mentioned that \"After T steps, the episode terminates, and the sequence of ...\". So the number of steps per trajectory is fixed? Why?\n\nIn 5.1 where are the \"unseen observer agents in the population\" come from? My understanding is that there is one observer and a set of agents during the training. But here it implies that there are more than one observer during the training?\n\nAlgorithm 1 in A.2 is not clear. For example, Line 2 says \"Let A<-population of agents\". However, \"A\" never showed up again in the remaining of the algorithm; The term discriminator (line 3), receiver (line 5), and observer (line 15) are used interchangeably in Algorithm 1.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9bA2ptmMasO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper deals with agents that communicate non-verbally via actuating their joints in a 3D environment. The authors show that the agents should be able to learn protocols that can generalize to novel partners. Furthermore, the authors find that the current training approaches are brittle, and they propose and evaluate approaches to address this challenge.\n\n##########################################################################\n\nReasons:\n\nOverall, I vote for accepting. The authors develop and evaluate an algorithm that allows for emergent communication which generalizes to novel partners. Furthermore, the authors openly communicate and address the brittleness of the training.\n\n##########################################################################\n\nPros:\n\n* Novel algorithm to perform emergent communication via joint actuations\n* Algorithm allows for generalization to novel partners\n* Suggestions to improve training stability\n\n##########################################################################\n\nCons:\n\n* It would be desirable to train the agents N times with different initializations and report mean and std of the performance\n* More experiments and evaluation results with various joint numbers, intents, potentially different communication strategies would be helpful \n\n##########################################################################\n\nMisc:\n\nThe text in Figure 1 is too small.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4LmOLgxI-_P", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*** Summary ***\n\nThe paper investigates emergent gesture-based communication in Embodied Multi-Agent Populations. A noticeable feature of the paper is that it investigates emergent communication in the case of non-uniform distribution of intents and costly communication (i.e. agents are penalized for effort). The authors find that in certain scenarios, these conditions may lead to communication generalization of learned communication strategies to new, previously unseen agents.\n\n*** Relevance, Originality, Novelty *** \n\nI believe that the paper meets ICLR conference standards when it comes to the importance of the problem considered, the originality of the approach and the novelty of the proposed ideas.\n\n*** Clarity ***\n\nThe paper is, in general, clearly written and is a pleasure to read. Figure 4 is deeply confusing, however. As mentioned in the appendix, the actual number of timesteps is equal to 5 and is upsampled during visualization. I believe that it is important to mention it right away (i.e. in the image description).\nOtherwise, the reader can easily misunderstand the setup after seeing this picture.\n\n*** Quality ***\n\nThe quality of the experimental support is, unfortunately, the weakest part of the contribution.\n\nThe authors introduce a number of exciting ideas, all of which make the communication learning setting more realistic. Specifically, the authors are studying communication in an embodied setting, Zipfian intent distribution, and the idea of effort-based action cost.\n\nAt the same time, while the ideas sound very promising in theory, in practice, the complications introduced by the embodied setting complicate the interpretation of the obtained results when it comes to the effects of Zipfian intent distribution and effort-based action costs.\n\nIn order to mitigate the difficulties, authors propose additional measures: providing explicit feature information to the outside observer (i.e. the observer directly gets the action effort); using \"torque pretraining\" to make the model more familiar with the reward landscape, dramatically reducing the number of actions.\nThese measures, in my opinion, are significantly detrimental to the potential impact of the paper. Providing effort information directly they makes the setting less realistic, defeating the purpose of the embodied approach.  Torque curriculum makes the results very specific to the considered setting.\n\nThere are other significant limitations of the experimental results that are probably indirectly induced by the embodied setting. Specifically, figure 2 and table 1 report no variability measures for the obtained result, and it's not clear, how many runs were made to obtain these results. Moreover, the agent population considered in these experiments is 10 (mentioned in the appendix). With such a small population, it is not surprising that the observer can simply memorize the individual patterns, and is not pressured to infer the underlying structure.  It seems very natural, therefore, to increase the number of agents in the simulation.\nI suspect that these limitations are a consequence of the fact that training in the multidimentional setting is very costly and time-consuming.\n\nMoreover, some of the limitations I describe in the \"technical soundness\" section could have been controlled for in a simpler setting.\n\nOverall, all things considered, it's not clear what are the insights gained from the embodied approach.\n\n*** Technical soundness ***\n\nThere are certain technical problems with the contribution:\n\nFirstly, the justification for why Zipfian intents, together with effort-based penalty should lead to generalization is not rigorous (paragraph 1, section 4.1), and not completely convincing.\n\nOne of the sentences in the intuitive justification raises particularly serious concerns. The authors say \"Coupled, [Zipfian intents and Energy regularization] incentivize an inverse relationship between energy exertion and intent frequency, assigning minimum energy trajectories to maximally occurring intents.\". This assumes that energy levels are \"occupied\" by a limited number of actions. Why that would be true is entirely unclear to me.\n\nFor example, for any action with a certain energy level, there is a mirror action (with all joint torques multiplied by -1) with exactly the same energy consumption. Even if for some reason such specific pair of distinct actions with exactly the same effort \nis impossible in a given setting, given how high the degree of freedom of the system is, it is still easy to imagine that every energy level offers a plethora of actions to choose from.\n\nThis raises a question of why then we see good performance in 2-action scenario. There is an alternative explanation that was not explored: there is a unique action with zero effort: doing nothing.\nI suspect that it may be the only special case, where energy level is mapped uniquely onto an action (i.e. there is no different action with the same (0) energy penalty).\n\nAnother alternative explanation that must be addressed is the exposure bias. Rare intents are sampled less frequently, therefore the model has less time to optimize its actions for that intent. This may explain some of the differences in energy costs between the rare and frequent actions.\n\nLastly, the architecture choice and problem setting formalization also raise certain concerns. Specifically, the sender only receives joint positions and angles at time t, with no history. This does not seem to be an appropriate state specification. Coupled with the fact that the network is a simple feedforward architecture (no recurrence), this \nlimits the number of possible jestures that can be generated.\n\n*** Conclusion ***\n\nThe paper addresses a highly relevant problem and proposes a number of interesting ideas. Unfortunately, in my opinion, \nsome of the proposed ideas (studying communication in an embodied setting, Zipfian intent distribution, costly actions), when introduced together, are not interacting well and are hindering result interpretation.\n\nAs authors mention, using an embodied setting with multidimentional continuous action and observation spaces makes the problem extremely challenging. Therefore, when we see only partial success or no success in learning, it becomes difficult to identify the true source of the problem. That is, it is unclear if we are hitting a fundamental limit of what can be achieved by the combination of the Zipfian intent + costly actions, or is whether it a just a limitation of the architectures/algorithms that the authors used, when those algorithms are applied in a challenging multidimentional setting.\n\nThe latter concern is exacerbated by some technical questions. In particular, the choice of a feedforward architecture seems extremely limiting, when combined with the state description that the authors used. The state space for the policy (sender) agent only includes the current joint configuration, without the history. This renders a large number of actions impossible (i.e. the \"pendulum-like\" wave of a hand becomes impossible, since when the hand returns to the middle after the first half-swing, the policy would act as if the action just started).\n\nLastly, the logic behind the main hypothesis is not fully theoretically supported, and the alternative explanations are not addressed.\n\nOverall, I believe that the contribution is extremely promising, but it still requires some polishing and potential restructuring in order to be up to the standards of the ICLR conference. I do deeply hope to see this paper improved and published in the future, as I believe that after some improvements, it will be of interest to a large portion of the community.\n\n*** Suggestions ***\n\nI believe that the paper can be made much stronger if the authors take a step back and start with a simplified communication game setting. E.g. instead of actually producing an action trajectory, agents could pick among a number of actions with pre-defined effort levels, or they can separately pick the gesture and the amount of effort to allocate to that gesture (i.e. a subtle hand-wave vs flailing one's arms in the air).\nThis would allow to explore the benefits and limitations of the proposed Zipfian intent+Costly actions approach. In that case, demonstrating that the approach also works in an emboddied setting would be a beautiful cherry on top. Alternatively, it is possible to \"boost\" the embodiment part of the paper, \ne.g. by pre-training the agents on some other tasks, thus making certain actions more natural, etc. Currently, it just seems that embodiment itself is not adding additional insights, but complicating the experiments.\n\nAdding noize (mentioned in the appendix) is a very good idea as intuitively, it seems that it is the only reason why the agents don't converge on an array of extremely subtle actions with near-zero effort. I believe that it was not discussed enough in the paper and maybe its importance is somewhat overlooked.\n\n*** Questions *** \n\n- \"Furthermore, with an increase in task complexity to 10 concepts, the external observer never performs better than this baseline strategy\" I thought the baseline accuracy was 0.34, so it seems that with curriculum the external observer does outperform the baseline.\n- Apart from the zero-effort action, why can't the agents converge on minimally distinguishable actions with the same effort level? \n\n*** Typos and other minor suggestions ***\n\n\"execution decentralized\" -> \"decentralized execution\"\n\"Agent reward\" verbally described as a \"function of state and actions taken\", but the formula is written as if that it is a function of observation and actions taken.\n\nTable 1 is slightly confusing to read, it may be better to make it broader so that the phrase \"Train Input\" does not break into two lines.\n\nConflicting notation: N is used as the number of agents on page 3, but later N refers to the number of concepts. 10 agents - not enough to infer the latent variable, especially if no randomness is added to the agents.\n\nI personally believe that the use of the term \"zero-shot\" in this setting is not ideal. The observer is trained on a sample of communication protocols and is then tested on a held-out set. \nIn my view, it's analogous to \"generalization performance\", not \"zero-shot\" performance. I understand that the authors are referring to another study that did use \"zero-shot\", so this consideration did not affect my score.\nBefore the name for this problem setting is thoroughly established, it may be possible to use \"out of sample coordination\" or some other option.\n\n*** Update after rebuttal ***\n\nI have read the rebuttal and I deeply appreciate the detailed response by the authors. I especially appreciate the introduction of a number of experiments on a simple domain that help to illustrate the main point of the paper.\n\nAt the same time, I believe that some serious issues still remain unresolved. For example, my main concern remains: I believe that wrapping the problem in the embodied setting is not introducing additional insights. I must clarify that I fully agree that studying how embodiment affects cognition/behavior is an extremely important and exciting area of research. But in the present paper, the models have no chance to benefit from embodiment (since there is no prior / shared embodied experience), but rather have to solve the task despite being placed in an embodied setting.\n\nThe main insight (in my opinion) is the observation that Zipfian intent distribution together with energy costs could be a good basis for zero-shot communication. I think that additional experiments that the authors introduced help to strengthen this point, although more experiments could still be beneficial (e.g. systematically varying the population size), as well as a more thorough theoretical discussion. At the same time, the limitations of the main \"embodied\" experiment remain (most importantly, the fact that fairly high accuracy can be achieved because of the unique \"do nothing\" action trajectory). \n\nIn short, a large part of the paper (on embodiment) contributes relatively little in terms of its impact and conclusions that can be drawn from it, which necessarily limits the extent to which the main insightful point can be explored. The main point is truly interesting, however, which makes the paper borderline.\n\nOverall, I believe that the paper is extremely promising and I would love to see an expanded version published. I feel very torn about the decision, but at the moment, I believe that the paper is still below the acceptance threshold, although only marginally. I am happy to adjust the score up, and I regret that I can not switch it to an \"accept\" recommendation.\n\nAs a minor aside - the newly introduced Colab Notebook does not fully run and crashes at the cell #4 (model loading), so I can not fully explore the newly introduced experiments. That being said, I think that after fixing, this resource can be very useful in the future. This minor issue did not affect my evaluation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received borderline scores, R1, R3, R4 gave a score of 6 and recommended a borderline acceptance. R2 provided by far the most detailed review and recommended a score of 5 (i.e., borderline reject). After the rebuttal, R2 comments, "I believe that the paper is still below the acceptance threshold, although only marginally". Overall, I concur with R2. The reasons are detailed below: 

The paper proposes a method for communication between two agents, wherein one agent actuates its joints to communicate intent. Intuitively, this resembles making a gesture. The paper considers the setting of a discrete number of intents. The sender agent is modeled as a neural network that takes as input the intent and outputs a trajectory of joints. The receiver observes a noisy version of the trajectory and outputs the intent. The parameters of the sender policy and receiver discrimination network are optimized to maximize classification accuracy. It is shown that if the intents are sampled from Zipf distribution and trajectories are penalized based on their energy, then a receiver agent initialized from scratch is better at inferring the intent from a pre-trained speaker agent, as opposed to when the distribution is uniform or when the energy regularization is not used (Figure 2).

Further, section 5.2 shows that when the listener is provided with the energy of the trajectory then it is better at recognizing the intent as opposed to being provided with the entire trajectory when a number of intents are small. With a larger number of intents (N=10), the performance is at chance accuracy.
 
The biggest challenge with the paper is that it is very poorly written. Large parts of the method and experimental setup are in the appendix (A.2 / A.3), which makes it hard to understand the paper. Section 4.2 is rather confusing because the ideas introduced are not used for training, but simply for evaluation. Further, the authors point out in the rebuttal that torque curriculum is not required, but it is still there in the paper and makes it more confusing. I recommend the authors to substantially rewrite the paper and focus on relevant parts instead of philosophical arguments. Lastly, I am confused by results in Table 2 -- the authors mention in the text that with 10 intents, intent identification is at chance (i.e. 34% accuracy), but the table shows 56% accuracy. A clarification would be helpful here. 

The problem of communicating intents via gestures, when the agents are unaware of mapping from intents to gestures is an exciting area of research. From the perspective of emergent gestures, this paper has a novel contribution. However, the settings are toy and even in such a setup, the results are underwhelming. The assumptions that make the setup toy are: the listener agent knows about all joint locations of the sender (with some noise) and also has access to the energy exerted by the agent. Without access to energy, the performance is poor. In real-world scenarios, these are big assumptions. Furthermore, even when the energy is known For instance, even when the number of intent is small (i.e.,  N=10,) the performance is bad. The authors argue that is due to local minima in the optimization -- but that's exactly where the contribution could have been. 

I will reiterate, that the authors claim their contribution is in using energy minimization + Zipf intent distribution as a mechanism for communicating intent -- which I agree to. However, as pointed out earlier, the paper is not well executed or written and therefore I recommend rejection. 


Decision: Reject