Learning to communicate through imagination with model-based deep multi-agent reinforcement learning
Authors: Arnu Pretorius
Scott Cameron
Andries Petrus Smit
Elan van Biljon
Lawrence Francis
Femi Azeez
Alexandre Laterre
Karim Beguir
Venue:  (2021)

Abstract: The human imagination is an integral component of our intelligence. Furthermore, the core utility of our imagination is deeply coupled with communication. Language, argued to have been developed through complex interaction within growing collective societies serves as an instruction to the imagination, giving us the ability to share abstract mental representations and perform joint spatiotemporal planning. In this paper, we explore communication through imagination with multi-agent reinforcement learning. Specifically, we develop a model-based approach where agents jointly plan through recurrent communication of their respective predictions of the future. Each agent has access to a learned world model capable of producing model rollouts of future states and predicted rewards, conditioned on the actions sampled from the agent's policy. These rollouts are then encoded into messages and used to learn a communication protocol during training via differentiable message passing. We highlight the benefits of our model-based approach, compared to a set of strong baselines, by developing a set of specialised experiments using novel as well as well-known multi-agent environments.

Under review as a conference paper at ICLR 2021
LEARNING TO COMMUNICATE THROUGH IMAGINA- TION WITH MODEL-BASED DEEP MULTI-AGENT REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The human imagination is integral component of our intelligenne. Furthermore, the core utility of our imagination is deeply coupled with communication. Lan- guage, argued to have been developed through complex interaction within growing collective societies serves as an instruction to the imagination, giving us the ability to share abstract mental representations and perform joint spatiotemporal planning. In this paper, we explore communication through imagination with multi-agent reinforcement learning. Specifically, we develop a model- -ased approach where agents jointly plan through recurrent communication of their respective predictions the future. Each agent has access to a learned world model capable of producing model rollouts of future states and predicted rewards, conditioned on the actions sampled from the agent's 's policy. These rollouts are then encoded into messages and used to learn communication protocol during training via differentiable message passing. We highlight the benefits of our model-basee approach, compared to a set of strong baselines, by developing set of specialised experiments using novel well as well-known multi-agent environments.
INTRODUCTION
"We use imagination our ordinary perception of the world. This perception cannot be separated from interpretation." (Warnock, 1976). The human brain, and the mind that emerges from its working, currently our best example of general purpose intelligent learning system. And our ability to imagine, is an integral part of it (Abraham, 2020). The imagination is furthermore intimately connected to other parts our cognition such as our use language (Shulman, 2012). In fact, Dor (2015) argues that:
"The functional specificity of language lies in the very particular functional strategy it employs. It is dedicated to the systematic instruction of imagination: we use communicate directly with our interlocutors' imaginations."
However, the origin of language resides not only in individual cognition, but in society (Von Humboldt, 1999), grounded in part through interpersonal experience (Bisk al., 2020). The complexity of the world necessitates our use of individual mental models (Forrester, 1971), to store abstract representations of the information we perceive through the direct experiences of our senses (Chang and Tsao, 2017). As society expanded, the sharing of direct experiences within groups reached its limit. Growing societies could only continue function through the invention of language, unique and effective communication protocol where sender's coded message of abstract mental representations delivered through speech, could serve as a direct instruction to the receiver's s imagination (Dor, 2015). Therefore, the combination of language and imagination gave the ability to solve complex tasks by performing abstract reasoning (Perkins, 1985) and joint spatiotemporal planning (Reuland, 2010).
In this work, we explore plausible learning system architecture for the development of an artificial multi-agent communication protocol of the imagination. Based on the above discussion, the minimum set of required features of such system include: (1) that it be constructed from multiple individual agents where, (2) each agent possesses an abstract model of the world that can serve as an imagination, (3) has access communication medium, channel, and (4) jointly leams and interacts in

---

Under review as a conference paper at ICLR 2021
collective society. Consequently, these features map most directly onto the learning framework of model-based deep multi-agent reinforcement learning.
Reinforcement learning (RL) has demonstrated close connections with neuroscientific models of learning (Barto, 1995; Schultz et al., 1997). However, beside this connection, RL has proven to be an extremely useful computational framework for building effective artificial learning systems (Sutton and Barto, 2018). This is true, not only in simulated environments and games (Mnih et al., 2015; Silver et al., 2017). but also in real-world applications (Gregurié et al., 2020). Futhermore, RL approaches are being considered for some of humanities most pressing problems, such as the need to build sustainable food supply (Binas et al., 2019) and energy forecasting systems (Jeong and Kim, 2020), brought about through global climate change (Manabe and Wetherald, 1967; Hays et al., 1976; Hansen et al., 2012; Rolnick et al., 2019).
Our system. We develop our system specifically in the context of cooperative mutli-agent RL (OroojlooyJadid and Hajinezhad, 2019), where multiple agents jointly attempt to learn how to act in a partially observable environment by maximising a shared global reward. Our agents make use of model-based reinforcement learning (Langlois et al., 2019; Moerland et al., 2020). To learn an artificial language of the imagination, each individual agent in our system is given access to recurrent world model capable of learning rich abstract representations of real and imagined future states. We combine this world model with encoder function to encode world model rollouts as messages and use a recurrent differentiable message passing channel for communication. To show the benefits of our system, we develop set of ablation tests and specialised experiments using novel as well as well-known multi-agent environments and compare the performance of our system to a set of strong model-free deep MARL baselines.
Our findings and contributions. We find that joint planning using learned communication through imagination can significantly improve MARL system performance when compared to set of state- of-the-art baselines. We demonstrate this advantage of planning in a set of specialised environments specifically designed to test for the use of communication combined with imagined future prediction.
Our present work is not at scale and we only consider situations containing two agents. However, to the best of our knowledge, this is the first demonstration of a model-based deep MARL system that combines world models with differentiable communication for joint planning, able to solve tasks successfully, where state-of-the-an model-free deep MARL methods fail. We see this work as preliminary step towards building larger-scale joint planning systems using model-based deep multi-agent RL.
2 BACKGROUND AND RELATED WORK
Reinforcement learning is concerned with optimal sequential decision making within particular environment. In single agent RL, the problem is modeled as a Markov decision process (MDP) defined by the following tuple (S, Ar,,, r, p. Po, (Andreae, 1969; Watkins, 1989). At time step t, in a state s', which is a member of the state space S, the agent can select an action a' from a set of actions A. The environment state transition function p( p( |s", a²) provides distribution over next states s1+1 and a reward function r(s", ; g(+1) returns a scalar reward, given the current state, action and next state. The initial state distribution given by Po. with s" ~ Po. and y E (0, 1] is discount factor controlling the influence of future reward. The goal of RL is to find an optimal policy n², where the policy is a mapping from states to distribution over actions, that maximises long-term discounted future reward such that nT" = argmax, ,, E 7'r(s" a², s²+ If the environment state is partially observed by the agent, an observation function o(s') ) is assumed and the agent has access only to the observation o² = o(s²) ) at each time step, with the full observation space defined as O {o{s)]s E S). In this work, we focus only on the case of partial observability.
(==
Deep RL. Popular algorithms for solving the RL problem include value-based methods such as Q- learning (Watkins and Dayan, 1992) and policy gradient methods such as the REINFORCE algorithm (Williams, 1992). Q-learning learns a value function Q(s, a) for state-action pairs and obtains policy by selecting actions according to these learned values using a specific action selector, e.g. e-greedy (Watkins, 1989) or UCB (Auer et al., 2002). In contrast, policy gradient methods leam parameterised policy Tg. with parameters 8, directly by following a performance gradient signal with respect 0. The above approaches are combined in actor-critic methods (Sutton et al., 2000), where
2

---

Under review as a conference paper at ICLR 2021
the actor refers the policy being learned and the critic the value function. In deep RL, the policy and value functions use deep neural networks as high-capacity function approximators capable of learning distributed abstract representations from raw input signals that are useful for downstream decision making. Recent state-of-the-an deep RL methods include Deep Q-Networks (DQN) (Mnih al., 2013) and related variants (Hessel et al., 2017), as well as advanced actor-critic methods such as PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018). See (Arulkumaran et al., 2017; Li, 2017) for an in-depth review of deep RL.
Model-based RL. In RL, the environment tran- sition function p is typically unknown. As result, so-called model-free RL methods. such as DQN and PPO, rely solely on data gathered from the environment, i.e. real experience, to learn an optimal policy. However, if given ac- cess to a transition function, an agent can gen- crate useful simulated, or imagined experience, and use it to plan. Therefore, in model-based RL, a model o oo","' a²) with parameters is learned using stored transitions gathered from either a random, heuristic or learned policy to simulate transitions from the true (unknown) transition function p. The model can then be used for model-based planning, which can ei- ther happen in the background, or at decision- time. We briefly highlight the differences be- tween these two types of planning and discuss work related to each and how this relates to our own work.
Real exparience
tnag ted experience
Background plarring
Dedicon-ime plarning
Figure 1: Experience and planning in model-based reinforcement learning
-Background planning. In background planning, the model is primarily used to generate additional experience and assist learning, i.c. for updating the parameters of the policy and/or value functions. An early version of this approach is DYÑA-Q (Sutton, 1990) which uses the additional experience to help learn a value function. However, the usefulness of a model degrades over long time horizons as model rollout error starts to compound (Gu et al., 2016). This has lead to different approaches that either use fixed depth rollouts based on model uncertainty (Feinberg et al., 2018), dynamic rollout schedules (Buckman al., 2018), or short rollouts starting from intermediate states sampled from a buffer (Janner et al., 2019). A promising alternative approach is to update gradients directly via imagined rollouts in a dimee-dimeniional latent space (Hafner et al., 2019; 2020; Byravan al., 2020).
Decision-time planning. In decision-time planning, the model is used to generate imagined rollouts from given state for the purpose of selecting the optimal action or sequence of actions. Decision- time planning methods for discrete action spaces often rely on search methods such as Monte Carlo tree search (MCTS) (Coulom, 2006) and have been used successfully in several works (Silver et al., 2017: Anthony et al., 2017; Schrittwieser et al., 2019). In continuous action spaces, methods include trajectory optimisation approaches using trajectory sampling (Todorov and Li, 2005; Theodorou et al., 2010; Nagabandi et al., 2018; Chua et al., 2018) or collocation (Posa et al., 2014) (optimising reward while forcing the model's predictions to be close to already visited states).
The model in our system utilised for decision-time planning and follows the approach of Ha and Schmidhuber (2018), who used recurrent neural world models way give agent's the ability to learn how to think (Schmidhuber, 2015). Specifically, we make use of a recurrent world model that takes the form of a mixture density network LSTM (MDN-LSTM)) as used in (Ha and Eck, 2017). The model is therefore a form of a recurrent Gaussian mixture model and allows us to sample probabilistic predictions of imagined next states.
An illustration of the core features of model-based RL and the different types of planning is given in Figure 1. Also see (Janner, 2019) and (Mordatch and Hamrick, 2020) for useful overviews.
Multi-agent RL (MARL). In the multi-agent case with N agents, we use the formalism of partially observable Markov games (Littman, 1994), defined as the tuple given above for the single agent case. but with observation and action spaces given by the following cartesian products: O -I.. O, S
3

---

Under review as a conference paper at ICLR 2021
and A - - A,, for agents i = = L..,, N. The goal in this setting is to find an optimal joint policy T (a1...... AN 01 ON) that maximises a shared long-term discounted future reward for all agent as T = argmmxx E Di= E 7/r(o), of. of
Early work in MARL simply trained multiple independent Q-learning algorithms (Tan, 1993), which has since been extended to include deep neural networks, or more specifically independent DQNs (Tampuu et al., 2017). However, from the perspective of an individual agent, these approaches treat all other learning agents part of the environment, resulting in the optimal policy distribution to become non stationary. Furthermore, if the environment is only partially observable, the learning task can become even more difficult, where agents may struggle with credit assignment due to spurious rewards received from unobserved actions of other agents (Claus and Boutilier, 1998).
To mitigate the issue of non-stationarity, MARL systems are often designed within the paradigm of centralised training with decentralised execution (CTDE) (Olichoek et al., 2008; Lowe et al., 2017; Foerster et al., 2017). In CTDE, a centralised value function, or critic, is used during training, which conditions on the global state and joint actions from all the agents to make the learning problem stationary, but is then later removed once the individual agent's policies have been learned, making it possible to use each policy independently during system execution. However, individual agent policies extracted in this way may still perform poorly because training is not specifically aligned with the goal of performing well under decentralised execution. Therefore, state-of faate-of-the-iee value- based MARL approaches such as Q-mix (Rashid al., 2018) and QTRAN (Son et al., 2019) make use of value function decomposition strategies (Sunehag et al., 2017) to more closely resemble decentralised training, where each agent a recurrent DQN (Hausknecht and Stone, 2015) that has memory to also deal with partial observability. Another clear way to help with the issue of partial observability is for agents to be able to communicate.
Learned multi-agent communication has been key innovation in helping MARL systems to scale to more complex environments and solve more challenging tasks (Foerster et al., 2016; Sukhbaatar et al., 2016; Singh et al., 2018; Chu et al., 2020). To facilitate communication in our work, formally extend the Markov game M by having agents connected to each other via communication channels according a pre-defined neighbourhood graph G(V, E). The graph G is defined by set of nodes (vertices) V along with a set of edge connections & = ((i,.) j E V, & j}. where each agent is a node in the graph, locally connected other agent nodes. We define the connected neighbourhood surrounding agent i as N₁ = {j E (i.j) E E}. This networked Markov game Mg is then defined by the following tuple (G, S,S,A,,, r, Pa, 7). Our communication channels are recurrent and end-to-end differentiable allowing for agent-to-agent communication protocols to be learned during training. Unlike work studying the emergence of language through communication in MARL, e.g. (Lazaridou et al., 2016; Mordatch and Abbeel, 2017; Kajić et al., 2020) our work is more focused on communication through imagination as a useful system design for task solving, as apposed to uncovering new insights into emergent phenomena related to the human imagination.
Model-based MARL. To the best of our knowledge, the literature on model-based deep MARL is quite sparse and very little work has been done in this area. A notable exception is the recent work by Krupnik et al. (2020) on multi-agent model-based latent space trajectory optimisation. Here a multi-step generative model, specifically temporal segment model (Mishra et al., 2017), is used to generate rollouts in a disentangled latent space and optimisation is performed directly over agent latent variables. Our work is the first we are aware of in the area of modell-based deep MARL that combines communication with decision-time planning using recurrent neural world models.
METHOD
In this section, we provide the full details of our approach to model-based deep MARL and outline our system architecture, which we refer to MACI: Multi-Agent Communication through Imagination. We explain the details of the system by way of a walk-through from the perspective of a single agent i, from time step to L 1. At time step I, the agent receives the observation of and initiates an imagined rollout of possible future observations and predicted rewards.
Rollout. For A 1, 1,..., K rollout steps, the agent produces an action:
-1 I 1 af AgentConttooleermppp (o n/-1 m
(1)
4

---

Under review as a conference paper at ICLR 2021
(A)
Rollout (egent 1)
Agent imagi nation
Agent
I
178]
Agant Contrnla
oi 4+1
LIGT!!
Neighoou graph
Encocer
Ougging
Vooring mosseges
for 1
Eggat Tecsmnng
Communication 21
Rollout
Real action at C
Agent 2
0
mg
O m
aj ~ T( (o Mg.
hrightour 11SET
Figure 2: MACI: Multi-Agent Communication through Imagination system architecture. (A) Agent trajectories, real and imagined. (B) Rollout for agent 1. (C) Communication for agent 2.
using neural network (MLP) controller (we provide specifies, e.g. layer sizes and number of layers in the appendix), where of is imagined observation, h is the world model hidden state, and m. /.___1 is an aggregated incoming message to agent i, from connected agents in agent i's neighbourhood, i.e. agents given an observation of and an action af:
N₁. In turn, an imagined rollout is produced by the world model
k+1 k+1
WorldModelvnsssswweee k-1 ..
(2)
where the world model output includes the imagined next observation of+1 reward + and an updated hidden state h. To initialise the rollout procedure, set o === 0', 1)=0 1
and m, [_(e-1)=0 = m m 1 The final output of the rollout after K steps is the agent's imagination of the future summarised as follows: ! = CONCAT((o). o ..., r, .4...... 1). where z are the logits associated with a to maintain differentiability and we concatenate along depth dimension to maintain sequence of length K. Once the agent's rollout is complete, the agent starts to communicate its imagination to its neighbours.
1..
Communication. For 1.., C' communication rounds the agent encodes it's imagination into summarised abstract representation to serve as an outgoing message:
= ENCODER
(3)
and sends this message to its connected neighbours. To encode the imagined sequence, use a ID convolutional neural network (CNN). return, the agent receives all outgoing messages m; O.c from its neighbours E Wi, which is then turned into an incoming message using simple aggregation function (e.g. concatenation, average or summation):
m AGGREGATOR O.0i E (N))
(4)
Note that for the next round of communication begin, another rollout inner loop of K steps must first take place to produce an updated imagination I After C' communication rounds the agent takes real action at AgentController, (o,, ht, (.C.) conditioned the final message m.! I.C and receives from the environment the next real observation T and reward ,++, Finally, for our agents, employ weight sharing and make of single world model shared between all agents. An illustration of the MACI system with two agents is provided in Figure
Training.
5

---

Under review as a conference paper at ICLR 2021
Block 1: MACI Methods
Block 3: Python code outline
AgeniControlle:
WorldModel:: y
Bef
ppaydde env.rosac
Fnzoder: z
Aggregator:
states zerost) t) [id: borrddattae,, commutaate, message), ...]
Function playEpisode (G. &, e=None): o -... ~V (o e/i E V) for
Lime_tte 1....r T nos done:
actions, scates meatt_attion Lona (obs, scates] obe, reward, done erv.atepiact Fone)
# Plan ... T. emvrroment steps do
1,..., communicution rteps
# Imagine
def next act ona tobs, statesl:
aulion siales and volionn valuse (ubs, stalwal
agenti
for = 1,.... K. rollous steps do
actione argnax id,
act:
f(o 1/-11 m.
world model in with update_ wor doddddd Jaba,
imagine:
alo h .a
return actions, states
end consolidate:
def act values tobs, statest:
CONCAT((O)
)
Founde:
update aggregat=_neeseeee (obs, states])
encode outgoing message: O.5 = =(7))
id: cennt..cntoooiorrrs... jab, statel for 10, agent, ob, state siptagents, ots, statesi
end
Communicate
for agent E V do
aggregate incoming messages:
m. h((m) o. for j A.)).
dct aeorsgate_nessaagee (Obs, statesl:
end # Step
id: cecntt.ecode_paan.. stacel
for 16, agent, ob, state in ziptagents, ots, states!
act:
fl NS m..0)
observe:
Pema (ol, ela{)
neseage 20
id: nean (message_ (meraag__trentttterrr
(rom[othor] for other in adj)
ead
return o..oF o pt.... p
End Function
retutn
# Loss functions
id: @aste.uddat__neaaee (nessaze)
(2()) = (o' No, o 1))= cfr 9h-
for stato, message zip (states, message_to)
c..5(0) PPO_lsss loss (9, A,)))
Block 2: MACI-Training
def agent, cnccdoo.iaanncurrrottt statal:
obs, action_volues, revards [Ir U)
Initialize f.g and 2;
step 1... rolieat_ptaps: abc, is
for 1, ..., E, training steps do Initialize experience buffer (ex)
iasiioo_vllss..ppp spt.. wwtp://......ooooeeeeeeeeeeeeeee tebs [-1), state!)
for n 1,... N, episode steps do
ex += playEpisode(ff, g, z);
custent_ob, remard, hidden werf.weri
obs[-ll, (ommxxoottoo__vvuuee-lll state. sucld_state,
end
# Update world model parameters for e in do playEpisode(f, g, e);
state state, apdate (hidden)) areand
secten self encoder tobs, accien_values, zwnards)
argmax, BT EreB E (09)
end
# Update policy and encoder parameters for e in do
playEpisode(f, g, e);
argmax, 87 EtEB E 2/
end end

---

Under review as a conference paper at ICLR 2021
(A)
(0)
3
5
(B)
Agent
Agens
Tranning
Agant
Apert agent
of
(0)
4
2
0
Figure 3: Specialised experimental environments. (A) Digit game. (B) Invisible spread.
EXPERIMENTS
We test the feasibility of our system on a set of specialised environments. Each environment is conceptually simple. However, both environments still prove to be formidable challenge, even for state-of-the-ee MARL approaches, due their extreme partial observability and the need for agents to be able to think ahead and communicate to solve the task. Our first environment, Digit game, is inspired by a similar challenge in (Foerster et al., 2016) and our second environment, invisible spread, is based off of the popular multi-agent particle environment (Mordatch and Abbeel, 2017: Lowe et al., 2017). Similar to previous work on model-based MARL (Krupnik et al., 2020), we only consider the case of two agents. In all our experiments, compare system against basic basclines namely, independent PPO learners (IPPO) (a surprisingly strong baseline) and independent Q-learners (IQL), as well state-of-the-at art systems, Q-mix and QTRAN. Futhermore, in each experiment use short two-step rollouts to guard against compounding model error.
Digit game. At each time step, each agent receives random one-hot encoded digit (0-9), where the environment transition dynamics obeys the following rule: of+1 = (o' + of-1) mod 10, with o = O. The challenge is for each agent to produce as action the digit the other agent likely receive the next time step. If the agent action is correct, the agent gets a reward of 1, if the action is incorrect, it gets The digit game therefore specifically tests if agents can predict future observations based on present and past experience (i.e. use their imagination), as well as learn an effective communication protocol (as the other agent's observation can only be determined through communication). Figure (A) shows an example of the digit game. In this example, if agent selects as action the digit 8 at time step = 2, i.e. of = 8, it will receive reward of
Results. We provide our experimental results on the digit game environment in Figure 4 (A). Learning curves are mean rewards averaged over runs and include shaded standard deviation bounds. Our system, MACI, is shown to have clear advantage, significantly outperforming all the other baselines. Due to the fact that we make use of a world model to perform rollouts that cost compute time, also show (in the inset plot) the wall-clock time for each system over real environment steps. Although MACI scales less well than the baselines in this specific instance, we note that in complex scenarios, real environment steps may prove expensive than imagined rollouts and the sample efficiency of decision-tinn time planning could outweigh the extra model compute time.
Invisible spread. The goal in this environment is for each agent to occupy a separate landmark starting from random location. Agents receive shared reward based on how close both agents are to their respective landmarks. The observation space consists of values for the agent's 's position and velocity as well as the positions of the landmarks. To make the task more difficult, we make each agent invisible to the other. Therefore, agents must coordinate through communication and use their world models to show their intent (in terms of the landmark they are currently planning to occupy). Figure 3 (B) shows an example of the invisible spread environment, where the agents are represented as large purple circles and the landmarks are shown as small black circles.
Results. We provide our results on the invisible spread environment in Figure 4 (B). Learning curves are again mean rewards averaged over runs and include shaded standard deviation bounds. MACI is shown to again have a clear advantage, outperforming all the other baselines. Interestingly, in this environment, MACI scales well in terms of compute time and is shown to perform close to the most efficient baseline, IPPO.
7

---

Under review as a conference paper at ICLR 2021
(A)
(B)
Digit game
invilible spread
140
ppp
Wall-clock time
120
100
Hour
maci tours)
gtran
Houre
siran
(ours)
Moan reward
Ste ps
Moan roward
-400
-400
-600
0
-1000
2,0000
40000
60000
80000
100000
220000
140000
25000
60000
75000
100000
129000 160000
175000 200000
Steps
Shape
Figure 4: Experimental results (A) Digit game. (B) Invisible spread.
(B)
Digit game
Invisble spread
Full
-900
commus
400
900
20000
80003
100000
40000
50003
Steps
Figure 5: Ablation study (A) Digit game. (B) Invisible spread.
Ablation study. To disentangle the roles of communication and imagination we perform an ablation study both environments, digit game and invisible spread. In each case, we train the MACI system with and without using a world model. We perform 5 independent runs for each case, showing the average curves with standard deviations bounds. The results of this study is shown in Figure 5. For the digit game, shown in panel (A), the world model is a crucial component determining final performance. This is to be expected given the strong requirement for future prediction. In the invisible spread environment, the benefit is less significant and the system seems to rely more heavily on communication of past and present information.
CONCLUSION
Our ability to imagine plays an integral role in our intelligence. Inspired by the idea of language as the systematic instruction of imagination, we developed system for multi-agent communication through imagination (MACT) in the context of model-based deep mutli-agent RL. In our system, each agent has access recurrent world model for decision-time planning and uses a differentiable message passing channel for learned communication.
In our experiments on two specialised environments, digit game and invisible spread, we showed that using learned communication through imagination can significantly improve MARL system perfor- mance when compared to state-of-the-ar baselines. Although our environments are conceptually simple, both environments still proved to be formidable challenge, even for tate--of-hh-as methods. Furthermore, the sample efficiency decision-time planning was shown to outweigh the extra model compute time in the invisible spread environment.
8

---

Under review as a conference paper at ICLR 2021
Our work demonstrates the feasibility of a model- deep MARL system that combines world models with differentiable communication for joint planning. Specifically, it highlights the potential benefits of decision-tiin time planning in multi-agent setting as means to improve agent cooperation.
An interesting future line work could explore the combination of background and decision-tin time planning in the multi-agent setting. In addition, many interesting innovations from the single agent model-based RL literature could potentially find fruitful application in MARL. However, scaling MARL larger numbers of agents remains difficult task, and even more so for model-based methods. We see this work as first step towards building larger-scale joint planning systems, which we plan undertake in future work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. Warnock, Imagination.
Univ of California Press, 1976.
A. Abraham, The Cambridge Handbook of the Imagination. Cambridge University Press, 2020.
D. Shulman, More than Real. Harvard University Press, 2012.
D. Dor, The instruction of imagination: Language as a social communication technology. tions of Human Interacti, 2015.
Founda-
W. Von Humboldt, On Language: On the Diversity of Human Language Construction and Its Influence on the Mental Development of the Human Species. Cambridge University Press, 1999.
Y. Bisk, A. Holtzman, J. Thomason. J. Andreas, Bengio, J. Chai, M. Lapata, A. Lazaridou, May.
A. Nisnevich et al., "Experience grounds language," arXiv preprint arXiv:2004.10151, 2020.
W. Forrester, "Counterintuitive behavior of social systems," Theory and decision, vol. 2, no. 2, pp. 109-140, 1971.
L. Chang and D. Y. Tsao, "The code for facial identity the primate brain," Cell, vol. 169, no. 6, pp. 1013-1028, 2017.
D. N. Perkins, "Reasoning as imagination," Interchange, vol. 16, no. 1. pp. 14-26, 1985.
E. Reuland, "Imagination, planning, and working memory: the emergence of language," Current Anthropology, vol. 51, no. S1, pp. S99- $110, 2010.
A. G. Barto, "Adaptive critics and the basal ganglia," Models of information processing in the basal ganglia, vol. 215, 1995.
W. Schultz, P. Dayan, and P. R. Montague, "A neural substrate of prediction and reward," Science, vol. 275, no. 5306, pp. 1593-1599, 1997.
R. S. Sutton and A. G. Barto, Introduction to reinforcement learning. Second edition. MIT press Cambridge, 2018
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski et al., "Human-level control through deep reinforcement learning," nature, vol. 518, no. 7540, pp. 529-533, 2015.
D. Silver, J. Schrittieser,, K. Simonyan, Antonoglou, A. Huang, A. Guez, Hubert, L. Baker, M. Lai, A. Bolton el al., "Mastering the game of go without human knowledge," nature, vol. 550, no. 7676, pp. 354-359, 2017.
M. Greguric, M. Vujić, C. Alexopoulos, and M. Miletić, "Application of deep reinforcement learning in traffic signal control: An overview and impact of open traffic data," " Applied Sciences, vol. 10, 11, p. 4011, 2020.
Binas, L. Luginbuehl, and Y. Bengio, "Reinforcement learning for sustainable agriculture," " in ICML 2019 Workshop Climate Change: How Can Al Help, 2019.
Jeong and H. Kim, "Deep reinforcement learning based renew-able energy error compensable forecasting." 2020.
Manabe and R. T. Wetherald, "Thermal equilibrium of the atmosphere with a given distribution of relative humidity," Journal of the Atmospheric Sciences, vol. 24, no. 3, pp. 241-259, 1967.
D. Hays, J. Imbrie, N. Shackleton et al., "Variations the earth's orbit: pacemaker the ice ages," science, vol. 194, 4270, pp. 1121-1132, 1976.
Hansen, M. Sato, and R. Ruedy, "Perception of climate change," Proceedings of the National Academy of Sciences, vol. 109, no. 37, pp. E2415-E2423, 2012.
10

---

Under review as a conference paper at ICLR 2021
D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sankaran, A. S. Ross, N. Milojevic- Dupont, N. Jaques, A. Waldman-Bront et al., "Tackling climate change with machine learning," " arXiv preprint arXiv: 1906v.9900000000 00630.553, 2019.
A. OrodjlooyJadid and D. Hajinezhad, "A review of cooperative multi-agent deep reinforcement 2019.
learning," arXiv preprini
E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and Ba, "Benchmarking model- reinforcement learning," arXiv preprint arXiv: 1907002577, 2019.
T. M. Moerland, J. Broekens, and C. M. Jonker, "Model-based reinforcement learning: A survey," arXiv preprint arXiv:2006. 16712, 2020.
J. Andreae, "Learning machines: A unified view," Encyclopaedia of Linguistics, Information and Control, Pergamon Press, pp. 261-270, 01 1969.
C. J. Watkins, "Learning from delayed rewards," 1989.
C. J. Watkins and P. Dayan, "Q-learning," Machine learming, vol. 8, no. 3-4, pp. 279-292, 1992.
R. Williams, "Simple statistical
algorithms for connectionist reinforcement learn- ing." Machine Learning, pp. 229-256, 1992.
P. Auer, N. Cesa-Bianchi, and P. Fischer, "Finite-time analysis of the multiarmed bandit problem," Machine learning, vol. 47, no. 2-3, pp. 235-256, 2002.
R. S. Sutton, D. A. McAllester, P. Singh, and Mansour, "Policy gradient methods for reinforce- ment learning with function approximation." in Advances in neural information processing systems, 2000, pp. 1057-1063.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning," arXiv preprint arXiv: 1312.6602, 2013.
M. Hessel, Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, "Rainbow: Combining improvements in deep reinforcement learning." arXiv preprint arXiv: 1710027270002290 2017.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv: 1707.70077347 2017.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor," arXiv preprint arXiv: 101.v21801.0/290. 2018.
K. Arulkumaran, M. Deisenroth, M. Brundage, and A. Bharath, "A brief survey of deep reinforcement learning," arXiv preprint urXiv: 1708.0//0000866 2017.
Y. Li, "Deep reinforcement learning: An overview." arXiv preprint arXiv:1701.07274, 2017.
R. Sutton, "Integrated architectures for learning, planning, and reacting based on approximating dynamic programming." in Machine learning proceedings 1990. Elsevier, 1990, pp. 216-224.
S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, "Continuous deep q-learning with model-based acceleration," in International Conference on Machine Learning, 2016, pp. 2829-2838.
V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine, "Model-based value estimation for efficient model-free reinforcement learning," " arXiv preprint MXXi:1803300111 2018.
Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee, "Sample-efficient reinforcement learning with stochastic ensemble value expansion," in Advances in Neural Information Processing Systems, 2018, pp. 8234.
M. Janner, J. Fu, M. Zhang, and S. Levine, "When to trust your model: Model-based policy optimization," in Advances in Neural Information Processing Systems, 2019, pp. 12519-12530.
11

---

Under review as a conference paper at ICLR 2021
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, "Learning latent dynamics for planning from pixels," in International Conference on Machine Learning. PMLR, 2019, pp. 2555-2565.
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, "Dream control: imagination," " in International Conference on Learning Representations, 2020.
Learning behaviors by latent
A. Byravan, J. T. Springenberg, A. Abdolmalcki, R. Hafner, M. Neunert, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller, "Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models," in Conference on Robot Learning, 2020, pp. 566-589.
R. Coulom, "Efficient selectivity and backup operators in monte-carlo tree search," in International conference on computers and games. Springer, 2006, pp. 72-83.
T. Anthony, Z. Tian, and D. Barber, "Thinking fast and slow with deep learning and tree search," in Advances in Neural Information Processing Systems, 2017, pp. 5360-5370.
J. Schrittwieser, Antonoglou, T. Hubert, K. Simonyan, L. Sifrr,, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel et al., "Mastering atari, go, chess and shogi by planning with learned model," arXiv preprint arXiv: 2019.
E. Todorov and W. Li, "A generalized iterative lgg method for locally-optimal feedback control of constrained nonlinear stochastic systems," in Proceedings of the 2005, American Control Conference, 2005. IEEE, 2005, pp. 300-306.
E. Theodorou, Buchli, and Schaal, "Learning policy improvements with path integrals." in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010, pp. 825-835.
A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine, "Neural network dynamics for model- -aaed deep reinforcement learning with model-free fine-tuning," in 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 7559-7566.
K. Chua, R. Calandra, R. McAllister, and S. Levine, "Deep reinforcement learning a handful of trials using probabilistic dynamics models," in Advances in Neural Information Processing Systems, 2018, pp. 4754-4765.
M. Posa, C. Cantu, and R. Tedrake, "A direct method for trajectory optimization of rigid bodies through contact," The International Journal of Robotics Research, vol. 33, no. pp. 69-81, 2014.
D. Ha and J. Schmidhuber, "Recurrent world models facilitate policy evolution," in Advances in Neural Information Processing Systems, 2018, pp. 2450-2462.
Schmidhuber, "On learning to think: Algorithmic information theory for novel combina- tions of reinforcement learning controllers and recurrent neural world models," arXiv preprint ANXi::511..02444 2015.
D. Ha and D. Eck, "A neural representation of sketch drawings," arXiv preprint arXiv:1704.03477 2017.
M. Janner, "Model-based reinforcement learning: Theory and practice," Berkeley Artificial Intelli- gence Research (BAIR) blog., 2019.
I. Mordatch and J. Hamrick, "Tutorial on model-basee based methods in reinforcement learning," Interna- fional Conference on Machine Learning, 2020.
M. L. Littman, "Markov games framework for multi-agent reinforcement learning," in Machine learning proceedings 1994. Elsevier, 1994, pp. 157-
M. Tan, "Multi-agen reinforcement learning: Independent vs. cooperative agents," in Proceedings of the tenth international conference machine learning, 1993, pp. 330-337.
A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus. J. Aru, J. Aru, and R. Vicente, " "Multiagent cooperation and competition with deep reinforcement learning," PloS one, vol. 12, 4, P. e0172395, 2017.
12

---

Under review as a conference paper at ICLR 2021
C. Claus and C. Boutilier, "The dynamies of reinforcement learning in cooperative multiagent systems," AAAI/AАА,, vol. 1998, no. 746-752, p. 2, 1998.
A. Oliehock, M. T. Spaan, and N. Vlassis, "Optimal and approximate q-value functions for decentralized pomdps," Journal of Artificial Intelligence Research, vol. 32, pp. 289-353, 2008
R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, "Multi-agent actor-critic for mixed cooperative-competiiie environments," in Advances in neural information processing systems, 2017, pp. 6379-6390.
J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, "Counterfactual multi-agent policy gradients," arXiv preprint A 08926, 2017.
Rashid, M. Samvelyan, S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, "Qmix: Mono- tonic value function factorisation for deep multi-agent reinforcement learning," arXiv preprint arXiv: 11485 2018.
K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi, "Qtran: Learning factorize with trans- formation for cooperative multi-agent reinforcement learning," arXiv preprint 190.000005.05408, 2019.
Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Son- nerat, J. Z. Leibo, K. Tuyls et al., "Value-decomposition networks for cooperative multi-agent learning," arXiv preprint arXiv:1706.052900 2017.
M. Hausknecht and P. Stone, "Deep recurrent q-learming for partially observable mdps," arXiv preprint 1507.065 2015.
Foerster, I. A. Assael, N. Freitas, and Whiteson, "Learning to communicate with deep multi-agent reinforcement learning," in Advances in neural information processing systems, 2016, pp. 2137-2145.
Sukhbaatar, R. Fergus et al., "Learning multiagent communication with backpropagation." in Advances in neural information processing systems, 2016, pp. 2244-2252.
A. Singh, T. Jain, and S. Sukhbaatar, "Learning when to communicate at scale in multiagent coopera- tive and competitive tasks," arXiv preprint arXiv:1812.09755, 2018.
Chu, Chinchali, and S. Katti, "Multi-agent reinforcement learning for networked system control," arXiv preprint arXiv:2004.01339, 2020.
A. Lazaridou, A. Peysakhovich, and M. Baroni, "Multi-agent cooperation and the emergence of (natural) language." arXiv preprint 4rX1v:161..07182, 2016.
I. Mordatch and P. Abbeel, "Emergence of grounded compositional language in multi-agent popula- tions," arXiv preprint
2017.
I. Kajić, E. Aygün, and D. Precup, "Learning to cooperate: Emergent communication multi-agent navigation," arXiv preprint arXiv:2004.01097, 2020.
O. Krupnik, I. Mordatch, and A. Tamar, "Multi-agent reinforcement learning with multi-step genera- tive models," Conference on Robot Learning, 2020, pp. 776-790.
N. Mishra, P. Abbeel, and Mordatch, "Prediction and control with temporal segment models," arXiv preprint arXiv: 10X.vv:1703000000000 2017.
13

---

Under review as a conference paper at ICLR 2021
APPENDIX
Block 4: MACI Settings
Favironment:
obs dim, Dobs
action_dim, Dact
dyperparameters:
messuge_ dim, D... 16
agent controller, hiddon, dim, Da
agent, www.e/oooorerrheeeee hadden_layers, Hg
world model hidden, dim, Drom 16
world model modee_hidden_ hiddennnayyeae layers, Hum
rollout steps, K
communication steps,
Architectures:
Agent controller, f(-]:
Type: Feedlorward MLP
Input din: Dm. + Dabs + Dum
Hidden dim:
Hidden layers: Ho
Ourpud Dact
World model, gt-)=
Type: LSTM
Input dim:: Doha Dact
Hidden dim: Dwm
Hidden layers:
Outpu dim: Dona + (reward)
Encoder, 2(-):
Type: ID convact
Input width: Dobe + Dact + 1 (reward)
Input length: K
Output dime. Dmc
Aggregator, k(-):
Type: Concatenation
Neighbour graph:
Type: Fully connected
14


Reviews:
[{"review_id": "hD-pR4tcjv1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank you very much for sharing these cool ideas. I enjoyed the clear writing and excellent related work sections, and I genuinely believe this paper presents interesting concepts that warrant further investigation. Unfortunately, in its current state, this manuscript is not ready for to be shared with the wider community at ICLR.\n\nI will leave here a few suggestions for improvement and ideas on how to strengthen your argument. I sincerely hope you will find these useful as you continue your research on this topic.\n\nThe manuscript describes Multi-Agent Communication through Imagination (MACI). MACI is an imagination-inspired communication protocol that allows two sub-modules to exchange information about their non-overlapping observations.\n\nThe manuscript is well written and easy to follow, and the authors properly place their contributions in the context of existing ideas.\n\nWhile the experiments presented are clear and the results are encouraging, I think the experimental section could benefit from additional experiments, here is why:\n\n- The tasks presented here are extremely simple. I understand the need of didactic environments, but in a purely methods paper, the reader is left to wonder if this method scales to more complex environments, if it can work with more than two agents, and if it can handle non-cooperative settings. This is especially acute here, given that Fig. 6 suggests MACI only helps in 1 out of 2 environments, as the performance gains in Invisible Spread are obviously attributable to partial observability in the baselines.\n\n- The tasks presented are purely cooperative, and the communication system is differentiable. This means that by setting WorldModel, Encoder and Aggregator to the identity function, one would recover exactly a single-agent architecture that has access to the combined observations and operates in the product of the actions spaces. The only difference might lie in how the the system is supervised (it is unclear from the manuscript how WorldModel is trained). This is similar to what is presented in Fig. 6 in the ablation study, but would add including a shared world-model to produce an \"ideal\" agent. How does this perform? The baselines provided are at an obvious disadvantage as the environments are partially observable. This performance ceiling would guide the reader in understanding how much of the gap is recovered by MACI.\n\nAdditional minor remarks: \n\n- I cannot find in the methods section how WorldModel is trained. Could this be made clearer in the text?\n- How accurate is WorldModel? How important is this accuracy? What happens if we replace our learned WorldModel module with an ideal oracle?\n- There is a bunch of work in modeling MARL (see, e.g. Hierarchical Policy Models [Zheng 2016], VAIN [Hoshen 2017], NRI [Kipf 2018] and RFM [Tacchetti 2018]). In particular RFM introduces on-board imagination models that influence the decisions of each agent. It might be good to add these to your references.\n\nThank you again for sharing these cool ideas, I hope to see more of this soon and that you'll find some of this feedback useful.\nAll the best.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bAzXVsu0aHt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n \nThis paper proposes to combine model-based and multi-agent reinforcement learning. The authors follow the typical recurrent neural world models setting to generate imagined rollouts for decision-time planning. To tackle the non-stationarity of a multi-agent environment, they build end-to-end differentiable communication channels between agents within a pre-defined neighborhood. The communication message is defined as abstract information encoded from the imagined rollout. Agents then make decisions based on the message they received and the output of recurrent neural world models. Empirical studies are performed to show the superiority of proposed methods over SOTA model-free MARL approaches. Results are shown in two simple environments, which are designed to require communication between agents to solve the task.\n\n\n##########################################################################\n\npros: \n\n+ The motivation of doing model-based MARL is very clear and challenging.\n\n+ Overall, the paper is well written.\n\n+ The ablation study on the roles of world models and communication channels is interesting.\n\n##########################################################################\n \ncons: \n\n\n- Although the paper claims as a combination of model-based and multi-agent RL, my major concern is that the proposed model still deals with these two problems separately. In particular, the world model doesn't consider the dynamics of other agents, thus being an independent model only. The paper proposed to tackle the multi-agent part of the problem by building an explicit communication channel, which lacks enough novelty.\n\n- I'm also concerned about the lack of rigorous experimentation to support the paper's claim. \nThe two proposed environments are extremely tailored for algorithms with explicit communication channels and are limited in the number of agents. \n\t- For the digit game, the non-stationarity is not quite clear when there are only two agents. I'd like to see what would happen if the agent number in the digit game increases.\n\t- For the invisible spread, the ablation study shows that the role of world models is not important. I'd like to see the performance of other baseline algorithms that use explicit communication channels, which is not compared and seems to work well as the paper reported. If so, I don't see why this experiment supports the claim of combining model-based and multi-agent RL.\n\n##########################################################################\n\nPost rebuttal\n\nThe author's response does not address my primary concern and I'd like to keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TM1J4kPTxnU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper talks about developing a model-based method for cooperative multi-agent reinforcement learning. The proposed approach utilizes communication as a tool for mitigating the partial observability induced by the non-stationary task while also helping agents reason about other agents' behaviors. The authors present their motivation for using language as a medium in model-based RL stemming from early literature in psychology and linguistics.\n\nThe setup consists of decentralized agents each of which is equipped with a world model similar to Ha et al. 2018. Further, each agent also has a separate message input that is received from the other players. Each agent does a form of decision-time planning where it produces rollouts for K steps before taking a real action. The message is then the encoding produced by the concatenation of the observations, rewards, and the actions taken during the rollouts.\n\nThe approach is novel and one of the first works that combine model-based RL in a dec-POMDP. The paper does a good job of explaining prior work in related domains. The schematic diagram also depicts the setup in an efficient and standalone manner.\n\nStill, I have some qualms related to the experimental setup that arguably makes the contribution of the proposed imagination framework inconclusive.\n\n- In the digits game, the agents need to produce actions that represent the next observation of the other agent. The transition dynamics are defined in a way such that the next observation for an agent i is independent of the action taken at the current timestep. I find this formulation to be incoherent with the way MACI works. Specifically, \n     a) The AgentController that produces the action doesn't need to depend on the current observation since it has no effect on the action. \n    b) The WorldModel produces the next observations, next hidden states, and the rewards given the current observation, current action, and current hidden state. Similar to the above, the information about the current action is not needed to produce the next observation. Moreover, the rewards, in this case, are only tied to the action. So it would make sense to produce it along with the action in the AgentController with a recurrent network.\nOverall I believe this game is not aligned with the objectives of MACI, although I would love to have the authors clarify this.\n\n- There is no information about the objective functions used for optimization or any detail about the learning process without which it makes it hard to reproduce.\n\n- The choice of baselines doesn't seem to be appropriate for the task. Since all the baseline methods used do not use explicit communication in their original forms, the comparison thus becomes unfair. I would like the authors to reference if the baselines were modified in a way to accommodate this. This is important specifically in the two tasks chosen since I believe just adding communication should yield sufficient improvement.\n\n- The current approach is only applicable for a two-agent cooperative game narrowing down the scalability of the method. I believe the approach has the potential to extend to multiple agents either by having a confluence of messages or explicit grouping of agents. \n\n- An important missing ablation experiment is comparing comm+world model with only world model. This is crucial since it will determine whether the performance gain is due to the abstract planning or the communication.\n\n- The overall compute required is more than running a real-time experiment since the planning uses K-step rollouts. Some ablation of the choice of K would be interesting to look at especially in terms of wall time.\n\ntypo: Fig 6-A title\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MpVs55kCNJr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper claims to present an algorithm which enables a population of (two) agents\nto learn to communicate and coordinate to solve a task, and thus positions itself\nin the field of multi-agent Deep RL. After a long but rather vague and unspecific introduction\nand related work (see below), it describes the algorithm, then presents experiments where\nthe introduced algorithm is compared with model-free MARL baselines.\n\nWhile the algorithm presented is interesting and has potentially some novelties compared to\nthe state-of-the-art (e.g. differentiability of message passing in model-based MARL), it has\nalso a number of weaknesses:\n\n1) Globally, I had a lot of difficulty understanding clearly what are the aims of this paper:\nWhat are the problems it aims to solve? What are the scientific questions adressed?\nNeither the abstract nor the text provide sharp explanations of these aims.\n\n2) The paper uses very loaded but undefined vocabulary like \"imagination\", \"language\" and \"communication\".\nWhile in general I think it can be sometimes useful to use concepts and terms from human cognitive sciences\nto describe AI systems, in this particular case I found it very far fetched to speak of \"imagination\" and \"language\",\neven \"communication\". It seems in practice authors might simply mean something like \"prediction of future states\"\nwhen they use the term \"imagination\". \"Language\" and \"communication\" are also far-fetched because in cognitive\nscience and linguistics it refers to systems that enable different individuals, with different world views, to\ncommunicate an intent to each other. Here, the \"agents\" share the same world model, so they are not really\ndifferent individuals with their own world representations, and their communication is rather like \nmessage passing in GNNs, which is pretty far from \"language\" or \"human-like communication\".\n\n3) It is not even clear whether it is meaningful to call the presented system as \"multi-agent\", since in addition\nto a centralized shared reward, there is also a shared world model. To me, the system looks rather like an RL\nsystem that controls a multi-component body with local controllers that synchronize through message passing,\nquite similary to graph neural network controllers (also including message passing) used for e.g. in Pathak et al. 2019.\nA discussion of the similarities and differences with work such as Pathak et al. is needed.\n\n4) the authors are right to say that there is little research on model-based MARL, and cite one exception:\nKrupnik et al. However, it is not justified why this closely related work is not included in the baselines,\nor at least compared in discussion more thoroughly. Authors might also want to discuss another model-based MARL\npaper: Zhang et al. 2020.\n\n5) A large part of the related work section is not relevant to this paper, in particular about Deep RL and model-based RL,\nwhich are much broader topics than the one addressed in this paper\n\n6) The description of the method lacks sufficient technical details for reproducibility, in particular it lacks detailed\npseudo-code (some refs are said to be in an appendix, but I did not find an appendix), and no links to code is provided.\nFurthermore, there is no sufficient information on how hyperparameters selection for baselines was made.\n\n7) The two environments in the experiments are not sufficiently well motivated: why did you need to introduce them rather\nthan reuse existing test environments? E.g. which particular problems did you want to address that was not possible with\nexisting environments ?\n\n8) Since the claimed topic of the paper is about the emergence of a \"communication system\", one would expect a detailed\nanalysis of the emergent communication code (currently only figure 5 gives a quite superficial qualitative analysis).\n\n9) The quantitative comparison of algorithms is not made using a sufficiently strong statistical method (only 5 seeds,\nno tests such as Welch t-tests)\n\nFor these reasons, while the particular algorithms studied is in itself interesting, I think the paper would need a major\nconceptual reframing and a better experimental methodology and justification before publication.\n\nReferences:\n\nPathak et al. (2019) Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity\nhttps://arxiv.org/pdf/1902.05546.pdf\n\nZhang, K., Kakade, S. M., Başar, T., & Yang, L. F. (2020). Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. arXiv preprint arXiv:2007.07461.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a model-based method for cooperative multi-agent reinforcement learning and propose to use communication of future predictions (as given by a learned world model) as a way to overcome partial observability.

Overall, all reviewers found this work to be of great interest and the combination of planning + communication novel. However, all reviewers pointed that the claims that the papers makes are not fully supported by the experimental framing of the paper pointing to several shortcomings around experimental design in general and better control of appropriate baselines. The authors have since clarified several aspects in their paper and also included a new RL environment. 

However, as the paper still stands does not fully provide convincing evidence of their proposal, which is however very intriguing. I would like to echo though reviewers' suggestions that the authors work a bit more on the experimental design and I really hope this work will appear at a later venue.
Decision: Reject