Communication in Multi-Agent Reinforcement Learning: Intention Sharing
Authors: Woojun Kim
Jongeui Park
Youngchul Sung
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-agent reinforcement learning
communication
intention
attention

Abstract: Communication is one of the core components for learning coordinated behavior in multi-agent systems.
In this paper, we propose a new communication scheme named  Intention Sharing (IS) for multi-agent reinforcement learning in order to enhance the coordination among agents. In the proposed IS scheme, each agent generates an imagined trajectory by modeling the environment dynamics and other agents' actions. The imagined trajectory is the simulated future trajectory of each agent based on the learned model of the environment dynamics and other agents and represents each agent's future action plan. Each agent compresses this imagined trajectory capturing its future action plan to generate its intention message for communication by applying an attention mechanism to learn the relative importance of the components in the imagined trajectory based on the received message from other agents. Numeral results show that the proposed IS scheme outperforms other communication schemes in multi-agent reinforcement learning.

Published as a conference paper at ICLR 2021
COMMUNICATION IN MULTI-AGENT REINFORCEMENT LEARNING: INTENTION SHARING
Woojun Kim, Jongeui Park, Youngchul Sung" School of Electrical Engineering, KAIST
Daejeon, South Korea avoojun, jun.kin kim, jongeui, park,, ycsung)kkaist.aas
ABSTRACT
Communication is one of the core components for learning coordinated behavior in multi-agent systems. In this paper, we propose a new communication scheme named Intention Sharing (IS) for multi-agenn reinforcement learning in order enhance the coordination among agents. the proposed IS scheme, each agent generates imagined trajectory by modeling the environment dynamics and other agents' actions. The imagined trajectory is a simulated future trajectory of each agent based on the learned model of the environment dynamics and other agents and represents each agent's future action plan. Each agent compresses this imag- ined trajectory capturing its future action plan to generate its intention message for communication by applying an attention mechanism to learn the relative impor- tance of the components in the imagined trajectory based on the received message from other agents. Numeral results show that the proposed IS scheme significantly outperforms other communication schemes in multi- agget reinforcement learning.
INTRODUCTION
Reinforcement learning (RL) has achieved remarkable success various complex control problems such as robotics and games (Gu et al. (); Mnih et al. (2013): Silver et al. (2017)). Multi-agent reinforcement learning (MARL) extends RL to multi-agent systems, which model many practical real-world problems such connected cars and smart cities (Roscia et al. (013)), ). There exist sev- eral distinct problems MARL inherent the nature of multi-agent learning (Gupta et al.) 2017); Lowe et al. (2017)). One such problem is how to learn coordinated behavior among multiple agents and various approaches to tackling this problem have been proposed (Jaques et al. (2018); Pesce & Montana 2019); Kim et al. (2020)). One promising approach to learning coordinated behavior is learning communication protocol among multiple agents (Foerster et al. (2016); (Sukhbaatar et al. 2016); Diang & Lu (018); Das al. (2019). The line of recent researches communication for MARL adopts end-to-end training based on differential communication channel (Foerster et al. 2016): Jiang & Lu (0018; Das et al. (209)). That is, message-generation network is defined at each agent and connected to other agents' policies or critic networks through communication chan- nels. Then, the Dessage-generation network trained by using the gradient of other agents' policy network is conditioned on the current observation
or critic losses. Typically, the or the hidden state of recurrent network with observations as input. Thus, the trained message encodes the past and current observation information minimize other agents' policy or critic loss. has been shown that due to the capability of sharing observation information, this kind of commu- nication scheme has good performance compared to communication-fferr MARL algorithms such as independent learning, which is widely used in MARL, in partially observable environments.
In this paper, consider the following further question for communication in MARL:
"How to harness the benefit of communication beyond sharing partial observation."
We propose intention of each agent as the content of message to address the above question. Sharing intention using communication has been used in natural multi-agent systems like human society.
Cooresponding author

---

Published as a conference paper at ICLR 2021
For example, drivers use signal light to inform other drivers of their intentions. A car driver may slow down a driver in his or her left lane turns the right signal light on. In this case, the signal light encodes the driver's 's intention, which indicates the driver's 's future behavior, not current or past observation such as the field view. By sharing intention using signal light, drivers coordinate their drive with each other. In this paper, we formalize and propose a new communication scheme for MARL named Intention sharing (IS) in order to go beyond existing for communication in MARL. The proposed IS scheme allows each agent to share its intention with other agents in the form of encoded imagined trajectory. That is, each agent generates an imagined trajectory by modeling the environment dynamics and other agents' actions. Then, each agent learns the relative importance of the components in the imagined trajectory based on the received messages from other agents by using an attention model. The output of the attention model an encoded imagined trajectory capturing the intention of the agent and used as the communication message. We evaluate the proposed IS scheme in several multi-agent environments requiring coordination among agents. Numerical result shows that the proposed IS scheme significantly outperforms other existing communication schemes for MARL including the state-of-the-an algorithms such as ATOC and TarMAC.
dnservation-sharine schemes
2 RELATED WORKS
Under the asymmetry in learning resources between the training and execution phases, the frame- work of centralized training and decentralized execution (CTDE), which assumes the availability of all system information in the training phase and distributed policy in the execution phase, has been adopted in most recent MARL researches (Lowe et al. (2017): Foerster et al. (2018): [Iqbal (ha Kim et al. (2020). Under the framework of CTDE, learning communication proto- col has been considered to enhance performance in the decentralized execution phase for various multi-agent tasks (Foerster et al. (016); Jiang & Lu (2018); Das et al. (0019). For this pur- pose. Foerster et al. (2016) proposed Differentiable Inter-Agent Learning (DIAL). DIAL trains gessage-gennrattoon network by connecting it to other agents' Q-networks and allowing gradient flow through communication channels in the training phase. Then, in the execution phase the mes- sages are generated and passed to other agents through communication channels. Jiang & Lu (2018) proposed an attentional communication model named ATOC learn when to communicate and how to combine information received from other agents through communication based on attention mechanism. Das al.] (2019) proposed Targeted Multi-Agent Communication (TarMAC) to learn the message-generation network in order to produce different messages for different agents based on a signature- based attention model. The message-generattin
Sha (2018);
networks in the aforementioned al- gorithms are conditioned on the current observation or a hidden state of LSTM. Under partially observable environments, such messages which encode past and current observations are useful but do not capture any future information. In our approach, we use not only the current information but also future information to generate messages and the weight between the current and future information is adaptively learned according to the environment. This yields further performance enhancement, as we will see in Section
In our approach, the encoded imagined trajectory capturing the intention of each agent is used the communication message in MARL. Imagined trajectory was used in other problems too. Racanière al. (2017) used imagined trajectory to augment it into the policy and critic for combining model- based and model-free approaches in single-agent RL. It is shown that arbitrary imagined trajectory (rolled-out trajectory by using random policy or own policy) is useful for single-agent RL in terms of performance and data efficiency. Strouse et al. (2018) introduced information-regulann regularizer to share or hide agent's intention to other agents for a multi-goal MARL setting in which some agents know the goal and other agents do not know the goal. By maximizing (or minimizing) the mutual information between the goal and action, an agent knowing the goal learns to share (or hide) its intention other agents not knowing the goal in cooperative (or competitive) tasks. They showed that sharing intention is effective in the cooperative case.
In addition to approach, Theory of Mind (ToM) and Opponent Modeling (OM) use the notion of intention. Rabinowitz et al. (2018) proposed the Theory of Mind network (ToM-net) to predict other agents' behaviors by using meta-learning Raileanu et al. (0018) proposed Self Other-Modellng (SOM) to infer other agents' goal in an online manner. Both ToM and OM take advantage of pre- dicting other agents' behaviors capturing the intention. One difference between our approach and
2

---

Published as a conference paper at ICLR 2021
the aforementioned two methods is that inference. That is, the agents in our approach allow other agents to know their intention directly through communication, whereas the agents in ToM and OM should figure out other agents' inten- tion by themselves. Furthermore, the messages in our approach include future information by rolling the policy, whereas ToM and CM predict only the current just next time-step information.
communication to share the intention instead of
SYSTEM MODEL
We consider partially observable N-agent Markov game (Littman (194)) and assume that com- munication among agents is available. At time step t, Agent i observes its own observation of, which is a part of the global environment state st, and selects action aj E A; and message mj M, based on its own observation oj and its own previous time step message mi-1 1 plus the received messages from other agents, i.e., m = (m/_1,"" ... .m We assume that the message m of Agent is sent to all other agents and available at other agents at the next time step, i.c., time step È E+ 1. The joint actions a, = [a).... aa²) ..,,"""""""" yield the next environment state 5c+1 and rewards fritmm according to the transition probability T S x A S + [o, 1] and the reward function R : S x A IR, respectively, where S and A I A' are the environment state space and the joint action space, respectively. The goal of Agent i is to find the policy x' that maximizes its discounted return R = Lt' 7" rt. Hence, the objective function of Agent i is defined as J((x') = E, Ri]. where = (œ) """,""""" and y E [0, are the joint policy and the discounting factor, respectively.
THE PROPOSED INTENTION SHARING SCHEME
The key idea behind the IS scheme is that multiple agents communicate with other agents by send- ing their implicit future plans, which carry their intention. The received messages capturing the intention of other agents enable the agent to coordinate its action with those of other agents. We now describe the architecture of the proposed IS scheme. At time step t, Agent selects an action af ~ >'(110% m+-1) and a message m = MGN'(o,, m,-1, x') based on its own observation of and received messages m,-1, where MGN is the message-generation network (MGN) of Agent i. The MGN consists of two components: Imagined trajectory generation module (ITGM) and at- tention module (AM). Each agent generates an imagined trajectory by using ITGM and learns the importance of each imagined step in the imagined trajectory by using AM. The output of AM is an encoded imagined trajectory reflecting the importance of imagined steps and is used as the commu- nication message. The overall architecture of the proposed IS scheme is shown in Fig. In the following we describe the detail of each module.
4.1 IMAGINED TRAJECTORY GENERATION MODULE (ITGM)
The role of ITGM is to produce the next imagined step. ITGM takes the received messages, obser- vation, and action as input and yields the predicted next observation and predicted action as output. By stacking ITGMs, we generate imagined trajectory, as shown in Fig. T For Agent time step t, we define an H H-length imagined trajectory as
(F), f #1+
(1)
where #4++ (6t+.. (i+x) is the imagined step at time step t + k. Note that + (of, at) is the true values of observation and action, but the imagined steps except are predicted values.
ITGM consists of roll-out policy and two predictors: Other agents' action predictor (o() (we will call this predictor simply action predictor) and observation predictor fo af, a, a7). First, we model the action predictor which takes the observation as input and produces other agents' predicted actions. The output of the action predictor is given by
fo = (a) ... a.-a aatt
aN) =: a-
(2)
Note that the action predictor can trained by the previously proposed opponent modeling method (Rabinowitz et al. 20188; Raileanu al. (2018)) and can take the received messages as input. Next,
3

---

Published as a conference paper at ICLR 2021
1
MarMul
m
5 Ponin
AM
Seatie
6
We
Wg
Wy
7p-1 [T " Tc=
MGN
ITGM
m1 o a2+2
π0
m
Tp+1
ITGM
mc-1
m
Mc-1
0
d++1
Figure I: Overall architecture of the IS scheme from the perspective of Agent i
model the observation predictor fo at, ') which is conditioned on the observation oj, own action ai. and the output of the action predictor a," Here, we adopt the dynamics function that predicts the difference between the next observation and the current observation, i.c., of+1 of
instead of the next observation of+1 proposed in (Nagabandi et al. (2018)) in order to reduce model bias in the early stage of learning. Hence, the next observation be written as
6e = of fh at,4,"
(3)
By injecting the predicted next observation and the received messages into the roll-out policy in ITGM, we obtain the predicted next action ä = T (0( m,-1). Here, we use the current policy as the roll-out policy. Combining 0 and ā'+1, we obtain next imagined step at time step t + 1, The (6 1.+ In order to produce an H H-length imagined trajectory, we inject the output of ITGM and the received messages m into the input of ITGM recursively. Note that we use the received messages at time step 1, m4-1, in every recursion of ITGM
4.2 ATTENTION MODULE (AM)
Instead of the naive approach that uses the imagined trajectory [TE...., - directly as the message, apply an attention mechanism in order to learn the relative importance of imagined steps and encode the imagined trajectory according to the relative importance. We adopt the scale- dot product attention proposed in (Vaswani et al.; (2017)) as AM. Our AM consists of three components: query, key, and values. The output of AM is the weighted sum of values, where the weight of values is determined by the dot product of the query and the corresponding key. In our model, the query consists of the received messages, and the key and value consist of the imagined trajectory. For Agent at time step f. the query. key and value are defined as
di Wam, W/ [m - [mm .ll -.. lIm
N-1 | E Rd
(4)
A = WKTT.
W W Wk E "xxx
(5)
w = [W W Wi E xdd.
(6)
Although the fixed received messages cause bias, it is observed that the prediction of received messages generates more critical bias in simulation. Hence, we use m for all H prediction steps.
4

---

Published as a conference paper at ICLR 2021
where W R W R and W Rdmxxd xdr are learnable parameters and operation
|l denotes the concatenation of vectors. The output mú of the attention model, which used for message, is the weighted sum of the values:
m
h=1
where the weight vector at (af:
(44 computed as
L k
a' softmax
(8)
vdk
ddl
=00
The weight of each value is computed by the dot product of the corresponding key and query. Since the projections of the imagined trajectory and the received messages are used for key and query, respectively, the weight can be interpreted as the relative importance of imagined step given the received messages. Note that Wo. WK and Wv are updated through the gradients from the other agents.
4.3 TRAINING
We implement the proposed IS scheme on the top of MADDPG (Lowe et al. (2017)), )) but it can be applied to other MARL algorithms. MADDPG a well-known MARL algorithm and is briefly explained in Appendix A. In order to handle continuous state-action spaces, the actor, critic, obser- vation predictor, and action predictor are parameterized by deep neural networks. For Agent i, let 0i. 0g 0g and 0% be the deep neural network parameters of actor, critic, observation predictor, and action predictor, respectively. Let W (W). W Wi-) be the trainable parameters in the atten- tion module of Agent i. The centralized critic for Agent i, Q', is updated to minimize the following loss:
LQ(0%) Ex, (w' (z, m)) y' = + (x', d')aa'' =p' (01,,))
where Q' and je= are the target Q-function and the target policy of Agent and parameterized by 8% 8 respectivvll. The policy updated to minimize the policy gradient loss:
Vos o J(O = Exa Vo je m) V. Q'(,, a))a"=a
(10)
Since the MGN is connected to the agent's policy and other agents' policies, the attention module parameters W are trained by gradient flow from all agents. The gradient Agent i's attention module parameters is given by V I(Wt)
]. (11)
N R (W: ( V) n V Q' a)w mi]
where o² and are the previous observation and received messages, respectively. The gradient of the attention module parameters are obtained by applying the chain rule to policy gradient.
Both the action predictor and the observation predictor trained based supervised learning and the loss functions for agent are given by
I(( Bo (So: (0') -a (00) 11-
(12)
L(O')) E aori [(0'' oO) -o') -So. (o , 1
(13)
5 EXPERIMENT
In order to evaluate the proposed algorithm and compare with other communication schemes fairly, implemented existing baselines the top of the same MADDPG used for the proposed scheme.
5

---

Published as a conference paper at ICLR 2021
The considered baselines follows. 1) MADDPG (Lowe et al. (017))
assess the gain
of introducing communication from this baseline. 2) DIAL (Foerster et al. (2016)): we modified DIAL, which is based Q-learning, to our setting by connecting the message- generratinnnnn network to other agents' policies and allowing the gradient flow through communication channel. 3) TarMAC (Das et al. ) ): we adopted the key concept of TarMAC in which the agent sends targeted messages using a signature-based attention model. 4) Comm-OA: the message consists of its own observation and action. 5) ATOC (Jiang & Lu (2018): an attentional communication model which learns when communication is needed and how to combine the information of agents. We considered three multi-agent environments: predator- cooperative navigation, and traffic junction, and we slightly modified the conventional environments to require more coordination among agents.
Predator
Agent
Landmark
Prey
(a) PP
(b) CN
(c) TJ
Figure 2: Considered environments: (a) predator-prey (PP), (b) cooperative-navigatio. (CN), and
(c) traffic-junction (TJ)
5.1 ENVIRONMENTS
Predator-prey (PP) The predator- eecaator-pryyaoooo environment is a standard task in multi-agent systems. We used a PP environment that consists of N predators and fixed M preys in a continuous state-action domain. We control the actions of predators and the goal is to capture as many preys possible in a given time, Each agent observes the positions of predators and preys. When C' predators catch prey simultaneously, the prey is captured and all predators get shared reward Ri. At every time when all the preys are captured, the preys are respawn and the shared reward value R₃ increases by one with initial value one to accelerate the capture speed for the given time. We simulated
three cases: (N = C.C = 1), (N 3, C 1). and (N 4, C 2) with all M = 9 preys, where the fixed positions of the preys are shown in Fig a). In the cases of (N = 2,
1) and (N = 3, C = 1). the initial positions of all predators are the same and randomly determined. Thus, the predators should learn not only how to capture preys but also how to spread out. In the case of (N 4, C' 2). the initial positions of all predators are randomly determined independently. Thus, the predators should learn to capture preys in group of two.
nooperative-navigatinn (CN) The goal of cooperative navigation introduced in (Lowe et al. (2017) is for N agents to cover L landmarks while avoiding collisions among the agents. We modified the original environment that collisions occur more easily. We set T, N, increased the size of agent, and assigned a specific landmark to cover to each agent (i.e., each agent should cover the landmark of the same color in big. Each agent observes the positions of predators and landmarks. The agent receives shared reward R; which is the sum of the distance between each agent and the corresponding landmark each time step and success reward N' x Rg where N' is the number of the covered landmark. Agents who collide with other agents receive negative reward R3. We simulated the environment with N = R1 1/3, R2 and R3 -5.
(TJ) We modified the traffic-junction introduced in Sukhbaatar et al. (0166) to continuous state-action domain. the beginning of an episode, each agent randomly located in predefined initial position and assigned one of three routes: left, right or straight, as seen in cc). The observation of each agent consists of the positions all agents (no route information of other agents) and 2 one-hot vectors which encodes the initial position and assigned route of the agent. The action of each agent is a real value in (0, 1), which indicates the distance to go along the assigned route from the current position. The goal is to go to the destination fast as possible while avoiding collision with other agents. To achieve the goal, we design reward with three components. Each agent receives success reward R2 if it arrives at the destination without any collision with

---

Published as a conference paper at ICLR 2021
(a) PP (N=2)
(b) PP (N=3)
(c) PP (N=4)
2014
(d) CN (N=3)
(c) TJ (N=3)
(f) TJ (N=4)
Figure 3: Performance for MADDPG (blue), DIAL (green), TarMAC (red), Comm -AA (purple),
ATOC (cyan) and the proposed IS method (black).
other agents, collision negative reward R2 if its position is overlapped with that of other agent, and time negative reward R3 avoid traffic jam. When an agent arrives at the destination, the agent is assigned new initial position and the route. An episode ends when T time steps elapse. We set Ri 20, R2 = 10, and Rg -001T, where r total time step after agent initialized.
5.2
RESULTS
Fig. 3 shows the performance of the proposed IS scheme and the considered baselines the PP, CN. and TJ environments. BigsBaa)(cd shows the learning curves of the algorithms on PP and CN and Figs. (e)-(f) show the average return using deterministic policy
100 episodes every 250000 time steps. All performance averaged over 10 different seeds. It is seen that Comm-OA performs similarly to MADDPG in the considered environments. Since the received messages come from other agents at the previous time step, Comm- OA in which the communication message con- sists of agent's is observation and action performs similarly MADDPG. Unlike Comm- -OA, DIAL, TarMAC, and ATOC outperform MADDPG and the performance gain comes from the benefit of learning communication protocol the considered environments except PP with N 4. In PP with N 4, four agents need to coordinate to spread out in group of two capture preys. In this complicated coordination requirement, simply learning communication protocol based on past and current information did not obtain benefit from communication. On the contrary. the proposed IS scheme sharing intention with other agents achieved the required coordination even this compli- cated environment.
5.3 ANALYSIS
Imagined trajectory The proposed IS scheme uses the encoded imagined trajectory as the message content. Each agent rolls out an imagined trajectory based on its own policy and trained models including action predictor and observation predictor. Since the access to other agents' policies is not available, the true trajectory and the imagined trajectory can mismatch. Especially, the mismatch is large in the beginning of an episode because each agent does not receive any messages from other agents (In this case, we inject zero vector instead of the received messages into the policy). We expect that the mismatch will gradually decrease as the episode progresses and this can be interpreted as the procedure of coordination among agents. Fig shows the positions of all agents and each agent's imagined trajectory over time step in episode for predator prey with N 3 predators, where the initial positions of the agents after the end of training (t = 0) is bottom right the map. Note that each agent estimates the future positions of other agents well as their
7

---

Published as a conference paper at ICLR 2021
MADOPG
700
1 (H-3)
(M-3)
15 (It#7)
Aver fetuum per holm:
(4E
200
Suhumpp
600-
01m31
700
(hib)
15 01-71
200
15 wil attontonn (H-3)
15 wit ottention (1
15 sio othenbon (H+
0-
0.5
(a) PP (N=4)
(b) TJ (N=3)
Figure 4: Performance for our proposed method with different length of imagined trajectory and without attention module.
own future position due to the assumption of full observability. The first, second, and third row of Fig show the imagined trajectories of all agents at Agent 1 (red), Agent 2 (green) and Agent (blue), respectively. Note that the imagined trajectory of each agent represents its future plan for the environment. As seen in F at / O the intention of both Agent and Agent 3 is to move to the left to catch preys. At 1, all agents receive the messages from other agents. It is observed that Agent 3 changes its future plan to catch preys around the center while Agent maintains its future plan. This procedure shows that coordination between Agent 1 and Agent 3 starts occur. is seen that as time goes, each agent roughly predicts other agents' future actions.
We conducted experiments examine the impact of the length of the imagined trajectory H. Fig shows the performances of the proposed method for different values of H. It is seen that the training speed reduced when H as compared to 3 or H However, the final performance all outperforms the baseline.
Attention In the proposed IS scheme, the imagined trajectory is encoded based on the attention module to capture the importance of components in the imagined trajectory. Recall that the message 7 where ai denotes the importance
of Agent i is expressed m Σ af ofth as seen in of h which the encoded imagined step. Note that the previously proposed communication schemes are the special case corresponding to a' 01, ... O). In Fig the brightness of each circle is proportional to the attention weight. At time step = K, where = 37, ag, which indicates when Agent 1 moves the prey in the bottom middle, is the highest. In addition, af, which indicates when Agent 3 moves to the prey in the left middle, is the highest. Hence, the agent tends send future information when it is near a prey. Similar attention weight tendency is also captured in the time step t = K + 1 and f = K + 2.
As aforementioned, the aim of the IS scheme is communicate with other agents based on their own future plans. How far future is important depends on the environment and on the tasks. In order analyze the tendency in the importance of future plans, we averaged the attention weight over the trajectories on the fully observable PP environment with 3 agents and a partially observable PP environment with 3 agents in which each agent knows the locations of other agents within a certain range from the agent. The result is summarized in Table T] It is observed that the current information (time k) and the farthest future information (time k 4) are mainly used as the message content in the fully observable case, whereas the current information and the information next to the present (time k and & + 1) are mainly used in the partially observable environment. This is because sharing observation information is more critical in the partially observable case than the fully observable case. A key aspect of the proposed IS scheme is that it adaptively selects most important steps as the message content depending on the environment by using the attention module.
We conducted an ablation study for the attention module and the result is shown in Fig. We compared the proposed IS scheme with and without the attention module. We replace the attention
8

---

Published as a conference paper at ICLR 2021
Agent1
pesperr
Agent2
Agent3
c=00
t=
K
Figure 5: Imagined trajectories and attention weights of each agent on PP (N=3): 1st row agentl (red), 2nd agent2 (green), and 3rd row agent3 (blue). Black squares, circle inside the times- icon, and other circles denote the prey, current position, and estimated future positions, respectively. The brightness of the circle proportional the attention weight. K 37.
module with an averaging layer, which is the special case corresponding to a' shows that the proposed IS scheme with the attention module yields better performance than the one without the attention module. This shows the necessity of the attention module. In the PP environment with agents, the imagined trajectory alone without the attention module improves the training speed while the final performance is similar to that of MADDPG. In the TJ environment with 3 agents, the imagined trajectory alone without the attention module improves both the final performance and the training speed.
77 (
Fig.
Imagined steps
k
k+1
k+2
k+3
k+4
Fully observable PP (N=3)
0.33
0.18
0.15
0.14
0.20
Partially observable PP (N=3)
0.32
0.22
0.17
0.15
0.14
Table I: Averaged attention weight over the trajectory at time step
6 CONCLUSION
In this paper, we proposed the IS scheme, new communication protocol, based on sharing intention among multiple agents for MARL. The cesssage-generation network in the proposed IS scheme consists of ITGM, which is used for producing predicted future trajectories, and AM, which learns the importance imagined steps based on the received messages. The message in the proposed scheme is encoded imagined trajectory capturing the agent's intention so that the communication message includes the future information as well as the current information, and their weights are adaptively determined depending the environment. We studied examples of imagined trajectories and attention weights. It is observed that the proposed scheme generates meaningful imagined trajectories and attention weights. Numerical results show that the proposed IS scheme outperforms other communication algorithms including state-of-the-an algorithms. Furthermore, we expect that the key idea of the proposed IS scheme combining with other communication algorithms such as ATOC and TarMAC would yield even better performance.
7 ACKNOWLEDGMENTS
This research supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry Science, ICT & Future Planning(NRF- A1

---

Published as a conference paper at ICLR 2021
REFERENCES
Abhishek Das. Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-1546, 2019.
Jakob Foerster, loannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in neural information processing systems, pp. 2137-21455 2016.
Jakob N Foerster, Gregory Farguhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. In Thirty-second AAAI conference on artificial intelligence, 2018.
Shixiang Gu, Ethan Holly. Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEFE, 2017.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017.
Shariq lqbal and Fei Sha. Actor-attention-crittc for multi-agent reinforcement learning. arXiv preprint 1.X0v00010009111 2018.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A Ortega, Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. arXiv preprint 1rXiv:/810.0800 180008644, 2018.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Advances in neural information processing systems, pp.
2018.
Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. A maximum mutual informa- tion framework for multi-agent reinforcement learning. arXiv preprint arXiv:2006.02732, 2020.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-oompetiive environments. In Advances in Neural Information Processing Systems, pp. 6379-6390, 6390, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Emanuele Pesce and Giovanni Montana. Improving coordination in small-scale multi-agent deep inforcement learning through memory-driven communication. arXiv preprint (eXXv:1901.03800 2019.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. arXiv preprint arXiv:1802.077400 2018.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. imagination-augmente agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
Roberta Railcanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. arXiv preprint arXiv: 092400 2018
10

---

Published as a conference paper at ICLR 2021
Mariacristina Roscia, Michela Longo, and George Cristian Lazaroiu. Smart city by multi-agent systems. 2013 International Conference on Renewable Energy Research and Applications (ICRERA), pp. 371-376. IEEE, 2013.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3335 2017.
DJ Strouse, Max Kleiman- Bleiman-Weiner, Josh Tenenbaum, Matt Botvinick, and David Schwab. Learning to share and hide intentions using information regularization. In Advances in Neural Information Processing Systems. pp. 10249-10259, 2018.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropaga- tion. In Advances neural information processing systems, pp. 2244-2252, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.

---

Published as a conference paper at ICLR 2021
A MULTI-AGENT DEEP DETERMINISTIC POLICY GRADIENT (MADDPG)
MADDPG is an extended version of DDPG to multi-agent systems under the framework of CTDE Lowe et al. (2017)) Each agent has deterministic policy a, pd. (or) conditioned on its own observation and a centralized critic Qe a) = E [R = o = a] conditioned on the joint actions o, and state information zy. Here, 24 can be state 8 or the set of observations (of, TM .0 ) The centralized critic is trained by minimizing the following loss:
(f() = ,z, (u' - Qbg (x. y'= Oo (x', , a')
(14)
where o5 is the parameter of target O-function and μ² is the target policy of Agent i. The policy trained by Deterministic Policy Gradient (DPG), and the gradient of the objective with respect to the policy parameter p is given by
Vg (2²) Ez,a Vo...''(o')V Ohe (s,
(1)
(15)
12

---

Published as a conference paper at ICLR 2021
B TRAINING DETAILS AND HYPERPARAMETERS
Table 2: Hyperparameters of all algorithms
MADDPG
TARMAC
DIAL
ATOC
IS
REPLAY BUFFER SIZE
10
2 10
2 10
2 10
10
DISCOUNT FACTOR
0.99
0.99
0.99
0.99
0.99
MINI-BATCH SIZE
128
128
128
128
128
OPTIMIZER
ADAM
ADAM
ADAM
ADAM
ADAM
LEARNING RATE
0.0005
0.0005
0.0005
0.0005
0.0005
NUMBER OF HIDDEN LAYERS (ALL NETWORKS)
2
2
NUMBER OF HIDDEN UNITS PER LAYER
128
128
128
128
128
ACTIVATION FUNCTION FOR HIDDEN LAYER
RELU
RELU
RELU
RELU
RELU
MESSAGE DIMENSION ON PP
5
5
5
5
MESSAGE DIMENSION CN
3
3
MESSAGE DIMENSION ON
3
3
ATTENTION DIMENSION ON PP
5
5
5
ATTENTION DIMENSION ON CJ
3
3
3
ATTENTION DIMENSION TJ
3
3
IMAGINED TRAJECTORY LENGTH H
13

---

Published as a conference paper at ICLR 2021
C ADDITIONAL ABLATION STUDY
We conducted an additional experiment to examine whether performance improvement gained from sharing intention or having prediction of the future. We compared the proposed IS scheme with MADPPG-p in which the agent does not use communication, but uses their own imagined trajectory as additional input. Fig. shows that the proposed IS scheme outperforms MADDPG-p. Thus, sharing intention, which is core idea of this paper, is more important than having a prediction of the future.
136
MACONOS
RACOMO
700
300
120
C8
0.2
04
06
0.4
34
13
02
64
06
100
(a) PP (N=2)
(b) PP (N=3)
Figure 6: Performance for MADDPG (bluc), MADDPG-p (orange), and the proposed IS method (black).
14

---

Published as a conference paper at ICLR 2021
D PSEUDO CODE
Algorithm Intention Sharing (IS) Communication Scheme
Initialize parameter 6i fQQ o/ .0 Gr. 0% W vi Vi (1, N for episode = 1, 2, ... do
Initialize state s messages mo (0 0) and each agent observes of
for t <= T and & * terminal do
Each agent receives the messages m4-1 (m,
N 1)
Each agent selects action aj (-1o,, m+-1) 1) for each agent Execute a4 and each agent receives ra and 0/ for h 2, .... Hl do
Predict other agents' actions a from the action predictor få
Generate t44h from observation predictor fo # 11 (14h-1)
Generate 4+ T' (-10 m,-1)
end for
Each agent generates the messages mi by injecting (Té: "(111" (++ into Atten tion Module (AM)
Store transitions in end for
for each gradient step do
Update 0/ and (0g 0%) by minimizing the loss (9) and the loss 12
Update of and W based on the gradient (10 and the gradient end for Update 0i- 0%= using the moving average method
end for
15


Reviews:
[{"review_id": "K0iZIto-phI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---- Summary ----\nThe paper proposes a new method for training communication in multiagent systems. The method involves agents producing imagined future trajectories using learned environment dynamics, and then communicating some parts of these trajectories to other agents. This communication scheme is evaluated on several cooperative communication domains, outperforming other methods.\n\n---- Reasons for score ----\nThe paper addresses an important problem, the method is sensible, and the empirical results suggest the method proposed works. My main concern is a lack of ablations, which makes it hard to tell which parts of the method are important.\n\n---- Pros ----\n1. The problem is well motivated. The existing literature on learned multi-agent communication primarily addresses the problem of communicating information about partially observed states. However, this is not all there is to communication, and the communication of agent intentions is an important topic to address.\n\n2. The method suggested - basing messages on predicted future trajectories - is sensible and well motivated.\n\n3. The experiments compare the method to a good range of baseline algorithms for multiagent communication, and it performs well.\n\n---- Cons/Questions ----\nThe method proposed has several moving parts, and without ablations it is hard to tell which are the most important pieces. There are a few experiments I would like to see added to work out what is important in the method:\n* A comparison to the uniform attention case. While the qualitative analysis suggests the attention mechanism is meaningfully used, it would be good to show how this affects performance.\n* An ablation where all the attention is put on the first timestep, to show the effect of the imagined trajectories. While this is similar to the other communication methods discussed in the paper, I am not sure whether it directly corresponds to any of them.\n* A comparison to a case where the agents receive only their own messages to condition their next actions. At the moment is unclear whether the advantage is gained from sharing intentions between agents, or by having a prediction of the future.\n\nAll the environments used are modifications of existing environments; was there a reason to change these? Particularly, the Cooperative-Navigation environment looks like it could have been used as-is, for an easier comparison to previous work?\n\n---- Post-rebuttal ----\nThe additional ablation studies on the attention and MADDPG-p are helpful, and address most of my concerns - though it is still not clear to me whether any of the baselines compared to is exactly equivalent to the method used with attention weights (1, 0, 0, ...). I share the concerns of Reviewer 4 about the significance of the results - this is still not in the paper, and not present for the new experiments. Overall, I have not changed my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HMLeBTQRP5e", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work proposes to use 'intention' of each agent to enhance the message sharing scheme of MARL. For training, MADDPG is used as a backbone MARL and implement ITGM and AM on top of it. The proposed method is compared with several baseline approaches from three different environments.\n\nStrengths:\n+ The paper is well-written and technically sounds. \n+ The motivation is clear.\n\nWeaknesses:\n- This work wants to see the effectiveness of the use of intention as a communication scheme under MARL. However, in general, this is not the first to use intention. I suggest to review some relevant works and show how 'intention' has been explored in a similar/different way in the literature. \n- Although visualization of imagined trajectory is given in Figure 4, it does not fully demonstrate the idea of this approach. More qualitative evaluation seems required to validate from various aspects. \n- It seems that attention is used to capture which waypoint is more important in an imagined trajectory rather than whose imagined trajectory is more important. If so, is it really important? Where is the ablative study of using attention?\n- Intuitively, when two agents set the same goal (catching same prey), what is the rationale of determining who maintain the goal or who change the plan? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "37PtXFsUTvG", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\n\nPaper proposed to generate the communication message in MARL with the predicted trajectories of all the agents (include the agent itself). An extra self-attention model is also stacked over the trajectories to trade off the length of prediction and the possible explaining away issue.  The whole model is trained via a canonical MARL objective while the trajectory prediction model utilizes direct supervision collected from the environments. Experiments on several toy MARL benchmark demonstrates the effectiveness of the proposed method.\n\n[Strengh]\n\n+) The idea of communication with imagined intention is motivated properly with rich psychological background and also technically sound.\n\n+) The paper is overall clear and well-written. I found there are enough technical details to reproduce the main results of the main approach.\n\n+) Still limited though in terms of the converted domains, the empirical evaluations deliver impressive results over a reasonable collection of counterparts.\n\n[Weakness]\n\nThe main concerns I have with this submission lies in the overall novelty and the evaluation of the proposed method. After reading this paper, indeed I find the authors failed to capture some important research in this narrow area and it's still not clear how does the proposed method really works and whether it is sensitive to some specific implementing factors.\n\n-) The idea of intention sharing based on prediction is essentially not novel esp. in the MARL domain. In the CogSci & AI community, theory-of-mind (ToM) and its application in multi-agent execution (either collaborative or zero-sum games) have been well studied for years [1, 2, 3]. Although I may agree that there are few prior works [4] on sending these predictions as messages to other agents, it does not really introduce that many new ideas to how collaborative or adversarial agents could benefit from such ToM-based intention prediction. However, the authors failed to capture these counterparts in their discussion and evaluations. Specifically, I would like to see how does the proposed method formulate its intention prediction differently than the prior work and whether it could enjoy advantages in performance with such differences.\n\n-) The proposed method is essentially quite complex (stacked prediction, transformer, etc) than its Bayesian counterparts, while the authors only provide an overall evaluation against several MARL baselines. It will be critical to also conduct a serious ablation study given the overall complexity of the proposed method, i.e. whether to use the transformer, the architectural choice of the transformer (num. of heads, etc), and the length of predicted trajectories (H). Also, the disagreement between the predicted trajectories and actual observation and its relation to the performance should also be investigated.\n\n-) Although the selected tasks are all canonical to MARL, given the fact a growing number of recent MARL learners have been moved on to more challenging tasks, I feel it would be necessary to include some mini MOBA games or other tasks with vision-based observations. \n\n[1] Bayesian models of human action understanding\n\n[2] Theory-based Social Goal Inference\n\n[3] Bayesian Theory of Mind: Modeling Joint Belief-Desire Attribution\n\n[4] Machine theory of mind\n\n[Suggestions&Questions]\n\n(1) Add ablation studies on the use of self-attention model and the length of prediction (H).\n\n(2) Visualize&compare the predicted and observed trajectories, add some discussion on how such disagreement would affect the performances.\n\n(3) (minor) Try the proposed method on mini MOBA games or tasks with vision-based observations.\n\n(4) Add citations to the related work on ToM and its application in MAS.\n\n[Post-rebuttal]\n\nI have read through all the other reviews and the rebuttal. Would like to thank the authors for their efforts in improving this submission. However, the main issue on the lack of novelty remains and I also find R4's concern on the significance of results is valid. Therefore, I will keep my initial justification as is. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1njA1TRHroG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarization and Strengths:\nThis paper studies how to learn coordinated behavior among multiple agents by learning a communication protocol. Specifically, compared to existing works, this paper proposes to generate messages based on not only current information but also future information (referred to as imagined trajectory) (Section 4.1). Additionally, the attention module in Section 4.2 is learned to weigh between current and future information dynamically. Overall, the paper is well written, clearly explains the proposed approach, and compares to appropriate state-of-the-art baseline methods.\n\nWeaknesses: \nWhile the motivation of this work is clear, the experimental results fail to support the claim. The results in Figure 3 show a small performance difference between IS (proposed approach) and baselines due to the large variance. Performing a statistical test, such as the t-test, to verify whether the proposed approach achieves statistically significant results than baselines will be helpful. \n\nFurther Questions:\n1. In general, learning the attention mechanism improves results, but is it necessary for the proposed approach? Adding an abbreviation analysis, such as the performance difference between with and without attention, will be helpful.\n2. The action predictor (Equation (2)) predicts the peer agents' actions based on the agent's own observation. While the peer action prediction based on the agent's own observation supports the decentralized execution, the local observation may not include sufficient information about the others depending on a multiagent domain and thus the action prediction can fail. Instead, wouldn't it be better to predict the peer agents' actions based on both the agent's own observation and received message m_{t-1} because the received message can contain useful information about the peers' behaviors? \n3. As noted in footnote 1, the message at m_{t-1} is used for all H prediction steps. How does the performance differ with respect to different H values?\n\nMinor comments: \n1. Typo in Page 1: ''How to harness ... partial observation.'' -> ``How to harness ... partial observation.''\n2. In Equation (2), the subscript \"t\" is missing (e.g., \\hat{a}^{i-1} -> \\hat{a}^{i-1}_{t})\n3. Adding pseudo algorithm in Appendix will further clarify the proposed approach\n4. For PP domain explanation (Section 5.1), it is unclear whether prey can run away from predators. In general, preys can run away in PP, but in this paper, preys seem to be fixed and cannot move around.\n5. For PP and CN domain explanation (Section 5.1), it is unclear what each agent observes.\n6. The intention prediction is also related to the theory of mind and opponent modeling (e.g., He et al., ICML-16, Raileanu et al., ICML-18, Rabinowitz et al., ICML-18), which can be added in the related work section (Section 2).\n\nHe He, Jordan Boyd-Graber, Kevin Kwok, Hal Daumé III. Opponent Modeling in Deep Reinforcement Learning. ICML-16\n\nRoberta Raileanu, Emily Denton, Arthur Szlam, Rob Fergus. Modeling Others using Oneself in Multi-Agent Reinforcement Learning. ICML-18\n\nNeil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S.M. Ali Eslami, Matthew Botvinick. Machine Theory of Mind. ICML-18\n\nI have read over the rebuttal and discussion and will keep my evaluation score as it was since the concerns about the weak performance result still remain.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study co-ordination in multi-agent systems. Specifically they propose a scheme where agents model future trajectories through the environment dynamics and other agents' actions, they then use this to form a plan which forms the agents' intention which is then communicated to the other agents.

The major concerns raised by the reviewers were around novelty, lack of ablations and significance of results as improvements were modest. During the rebuttal, the authors have extended their work with ablations and have conducted a statistical test. While it is true the current results present a small improvement, i think this is an interesting contribution in the field of emergent communication
Decision: Accept (Poster)