Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning
Authors: Zhiwei Xu
Dapeng Li
Bin Zhang
Yuan Zhan
Yunpeng Baiia
Guoliang Fan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Agent Systems
Model-Based Reinforcement Learning

Abstract: Recently, model-based agents have achieved better performance than model-free ones using the same computational budget and training time in single-agent environments. However, due to the complexity of multi-agent systems, it is tough to learn the model of the environment. The significant compounding error may hinder the learning process when model-based methods are applied to multi-agent tasks. This paper proposes an implicit model-based multi-agent reinforcement learning method based on value decomposition methods. Under this method, agents can interact with the learned virtual environment and evaluate the current state value according to imagined future states in the latent space, making agents have the foresight. Our approach can be applied to any multi-agent value decomposition method. The experimental results show that our method improves the sample efficiency in different partially observable Markov decision process domains.

Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning
Zhiwei Xu, Dapeng Li, Bin Zhang, Yuan Zhan, Yunpeng Bai, Guoliang Fan Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences fxuzhiwei2019, 1idapeng2020, zhangbin2020, zhanyuan2020, baiyuanpeng2020, guoliang. fan Lann@ia... ai...c.cc. cn
Abstract
Recently, model-based agents have achieved better performance than model-free ones using the same computational budget and training time in single-agent envi- ronments. However, due to the complexity of multi-agent systems, it is tough to learn the model of the environment. The significant compounding error may hinder the learning process when model-based methods are applied to multi-agent tasks. This paper proposes implicit model-based multi-agent reinforcement learning method based on value decomposition methods. Under this method, agents can interact with the learned virtual environment and evaluate the current state value according to imagined future states in the latent space, making agents have the foresight. Our approach can be applied to any multi-agent value decomposition method. The experimental results show that our method improves the sample efficiency different partially observable Markov decision process domains.
Introduction
In recent years, reinforcement learning has made remarkable achievements in game AI 47 12].
robots (33), and autonomous driving [24). It is primarily due to the progress of model-free reinforce- ment learning (MFRL) research. Unlike model-based reinforcement learning (MBRL), MFRL does not require dynamics of the environment and learns from the data generated by directly interacting with the environment. Therefore, although most environment models are unavailable in the real world, MFRL can master complex tasks through large-capacity value function approximators. However, MFRL requires large amounts of training data that can be costly and risky in reality, which means that the sample efficiency is the main bottleneck of the current MFRL algorithms. So in the problem that the dynamics model is known, we should prioritize using MBRL, which can even obtain an analytical solution to optimal control problems.
Multi-agent systems are more complex than single-agent tasks. The strategy of each agent changes during the learning process in multi-agent systems, which makes the environment unstable from the is for agents to use channels to transmit information to each other, in this way to reach a consensus between agents. In addition, many current studies on multi- aueti- reinforcement learning have focused on the paradigm of centralized training with decentralized execution (CTDE) (29]. The agent only obtains global information during training can significantly alleviate the instability problem. The first branch (29 11 [6] that emerged is using the Actor-Critic structure's 's advantages to build the centralized critic and decentralized actor 53) is also the most popular CTDE approach lately, and it has obtained excellent performance in the decentralized partially observable Markov decision process IDec-POMDP domains [32]. Nevertheless, most of the above methods are model-free and require access an impractically large number of trajectories.
perspective of each agent. One method 44 T0
38
19
22
framework. The value decomposition method [45] 39
43
30
50
49
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Some work combines model- based and model-free freee-ffee methods. They usually learn environment models and solve control problems simultaneously, and have achieved good performance in some single-agent scenarios, such as MuJoCo [3] and Atari 20 However, in multi-agent problems, the training of the world model faces great challenges because of the complexity of the environments. All current work on model-based multi-agent reinforcement learning can only be applicable for simple scenarios such as matrix games. In this paper, we propose Model-Based Value Decomposition (MBVD), a method that introduces the idea of model learning into value decomposition. When humans make decisions, they not only rely on the current state but also consider the future state obtained after several interactions with the environment following their current strategies. The phenomenon is called "long-term vision" or foresight. Enlightened by this ability of humans, we make agents have the foresight to cooperate by obtaining the aggregated latent states in the future. Finally, we evaluated the performance of MBVD in several different domains, including StarCraft II [41], Google Research Foothall (26) and Multi-Agent MuJoCo [7]. We clarified that MBVD has high sample efficiency, which exceeds the performance of other baselines in most scenarios. To the best of our knowledge, our study is the first attempt apply the model-based ideas to Dec-POMDS problems.
2 Related Work
2.1 Single-Agent Model-based RL
The model-based approaches are divided into three different categories. The first branch is represented by Dyna-Q [46] 27 20 6 52), the method of alternating the two processes, including learning enviromment models and improving policies. The focus of this idea is that the world model generates trajectories used by the training of reinforcement learning. analytic-gradien
algorithms can calculate the analytic gradient related to the reinforcement learning optimization goal through the parameterized world model, thereby directly improving the policy. This method is also called Policy Search with Backpropagation through Time 14
g 15 [3]. The last method rolls out the learned models over multiple time steps to predict the value of states. VPN (31] plans and calculates the path with the largest accumulated reward in rollouts and goes back to the current moment to improve targets. MVE [9] uses an update method similar to n-step q-learning to calculate the target state value. However, a fundamental limitation is that the above algorithms rely heavily on the world model. So if the learned environment model is inaccurate, it will lead to the collapse of the entire learning process. Especially the last method suffers from the limitation because, in addition to learning the dynamics model of the environment, model mogel--ugmented value expansion algorithms [2 [1 [8] [56,6 3 also need to learn the reward function of the environment to calculate the accurate value function. is impractical in sparse reward environments.
2.2 Multi-Agent Model-based RL
So far, there is relatively little work on multi-agent model-based RL. [54] obtained the sample complexity of multi-agent model-based RL through theoretical proof when the dynamics model was available. Nevertheless, it only applies to simple infinite-horizon zero-sum discounted Markov games and has not been implemented. 3 -CRUL 3d combines model--bsed RL with mean-field game theory, which can be used in cooperation problems like stylized swarm motion. However, due to the limitation of the mean-fiell theory, -CCULL can only be suitable for the scenarios of large population of agents. Both 355 and (55) construct the environment model that includes transition function and a prediction model for the opponents' actions, and then train their policies with the opponent-vv wise rollouts. These two methods have been evaluated in the multi-particle environment, and the results demonstrate that they can reduce sample complexity. However, it is not feasible to build the opponent models without access to opponents' information. CPS [2] approximates factored sparse Q-function, which is similar to the value decomposition method. Besides, CPS Iearns the environment model and maintains replay buffer with priority, but it cannot solve the partially observable problems. Similarly, in the multi-agent model-based RL. field, we also face compounding errors caused by imperfect learned models.
MBVD focuses on learning a standard forward dynamics model and aggregating the information contained in the environment model rollouts. Then the current state value, which is with respect to the imagined rollouts, can be obtained. We believe the imagined information can enable all agents to understand the current situation better.
2

---

Preliminaries
3.1 Dec-POMDP
Partially observable stochastic games (POSGs) are one of the most general games, and the Dec- POMDP [32] is an important subelass of POSGs. Formally, the Dec-POMDP defined as tuple G U, Prr, Z, O, n,,)) 7). Each agent a E A := [I.....n) selects the appropriate action u at each time step. with only access to the local observation 2" E Z provided by the observation function O(s.a) S x A + Z, where E S is the global state of the environment. U = U denotes the combined action of all agents. The state transition function, commonly known the environmental dynamies, is written as P(s' u) S U S > [0, 1]. Noted that all agents in Dec-POMDPS have the same reward function: r(s, u) S U IR. And y the discount factor.
3.2 Value Decomposition
When there are multiple agents in the system, is impossible to train each agent separately treat all agents as one entity for joint training in cooperative multi-agen reinforcement learning. The appearance of value decomposition methods promotes the collaboration between agents and solves the credit assignment problem. In the Dec-POMDP, we make the following assumption:
arg max.3, Q1 (T1,21)
arg max Qtor u) &
arg max On (Tn unp)
where T E 7 represents the joint action-observation histories of all agents, Qtor is the global action- value function, and Qn is the individual ones. The assumption is known as the andivduual-Gloud Max (IGM) (3 principle, which states that a task may only be decentralized if there is consistency between the local greedy actions and global ones. Many multi-agent reinforcement learning algorithms have performed well by observing the IGM principle, like VDN 45 and QMIX (39.
3.3 Environment Models
The most notable feature of MBRL is that while training the policy, it also learns the world model, a parametric model used to simulate the environment. Two different models can be learned unsupervised from local observations and actions, namely auto-regressive and state-space models.
Auto-regressive models [21 intuitive and straightforward. However, there are two reasons for the high computational complexity of the auto-regressive models: calculated items of the generation process cannot be reused, and auto-regressive models need to render -ig-diimessionl observations first abstract the environment to find a compact latent state space S containing all vital information. In the multi-agent system, each E S is abstract representation of the local observations z, of all agents. So the observation function can be expressed by p(z,] $0:1, u0=-11 p(z, 8t). Using imagined rollouts obtained by the transition function on the low-dimensional latent state space, state-space models can significantly reduce the amount of calculation. The factorization of the predictive distribution follows:
explicitly. On the contrary, state-space models 34 25
P(Z1T, Wo.T) Pinit (50) /
/II (p (p(()) (Se 8;-1, 4--1, p(uc, 2.)p(z; 8c)) d51.T,
where Pani (mni(Su) denotes the prior distribution of the initial state Sg. which usually constant. p(& $;-1, u1 z1) is the true posterior of the latent state, and p( (u zx) means the joint policy π of all agents.
Model-Based Value Decomposition
This section will elaborate MBVD, a novel model-based multi-agent reinforcement learning algorithm based on value decomposition. We will introduce the framework and flowchart of MBVD first and then describe the detailed implementation of MBVD.

---

Traygueio Riiken
Prior Encoder
Decoder
Decoder
MLP
Agent Network
Posterior Decader
Fine
Agent Network
Minimbze
KL
Encoder
Encoder
Encoder
Mixng
Rollst Encoder
Posteriar Decader
Agent Network
Network
th
(a)
(b)
Figure 1: Illustration of MBVD implementation. (a) The imagination module in MBVD. (b) The workflow of MBVD. The rollout horizon in the figure is 3.
4.1 The MBVD Framework
To maintain the applicability of algorithms, the framework of the reinforcement learning part in MBVD is consistent with other value decomposition methods, including the agent network and the mixing network. We focus on the model learning part of MBVD, called the imagination module.
Inspired by amortized variational inference, we employ neural networks to approximate the intractable posterior p(,, $;-1, -r zt). The approximate posterior is defined by ( $ u;-1, where / the parameters. So we can derive the evidence lower bound (ELBO) as follows:
logp (21:T (
p ($1.T, Uo:T 11:7)
log E (i
40 ($1.
Mo:T, #1:7)
p (Si:r; Mo:ry 21:1)
BB log
90 (81.T
Uo:T' #1:T)
and the ELBO can broken down as:
L (21:T, up:r)
T
Σ flog (p( 2x)] + log [p(z, [&)]] DxI. [go (8s 88-1, 11-1, 2x) l1p 14-1,
The first term log jp(u; xt)] is the joint policy and we can ignore it. Furthermore, the term log [p(z; {,]] can be viewed as the observation model. In addition, propose new parameterized Prior (3t 8c-1. uz-1) to approximate the dynamics model p(š, 8 u;-1), which is
function p unavailable.
MBVD follows the idea of planning in latent spaces, so we use the Variational Autoencoder (VAE) 23 maximize the ELBO. The reason that generative models find low-dimensional space by reconstructing the original data, which is consistent with abstracting the decision- related state from the real observations. For the last term Equation we regard the two items Kullback- Leibler (KL) divergence as the posterior and the prior. Intuitively, according to Equation I we need minimize the KL divergence between the posterior (o(-), which incorporates information about the current observations with the prior Prior (.) that tries to predict the posterior without access to the current observations. Then we use two VAEs as the prior and the posterior, respectively, as shown in Figure I(a) In this way, obtain the prior latent state and the posterior latent state:
8 Prior
Prior (- 8 u1-1)
Po
8 ~ qo (- 3 u;
However, we use modified version of the posterior model the implementation process. Since the hidden output hy of the recurrent neural network in the agent network can be regarded as the

---

integration of all past information of the individual agent, the posterior (ā, $t-1, u1-1, zz) can be rewritten as (3, h2). The advantages of minimizing the KL loss are that we can not only guide the prior to predicting the latent posterior state distribution without the information of the current step but also control how much information the posterior abstracts from the original observation. is worth noting that the learned prior model can predict forward in the latent space based on the actions of all agents, even if the actions are different from reality, So the imagination module in MBVD we proposed is an action-conditional environment model. From another perspective, we can view the prior model as the transition function, and the decoder qo (2x 8e) of the posterior can be regarded as the observation function. Through the above framework, we have built the imagination module that is general and model-agnostic for the multi-agent value decomposition methods.
In addition to learning the transition function and observation function, in some complex environments such as StarCraft II, the imagination module also needs to predict the feasible action set A to choose imagined actions more reasonably. Since the posterior can access the actual information of the current step, we can perform multiple auxiliary tasks by using the latent state & ~ qo (& h4)) as the intermediate variable to predict the additional environmental signals. The reward function r(-) is crucial in reinforcement learning because it steers the agents' behavior. However, predicting the reward function in noisy, sparse-reward, or complex environments is often difficult. So the imagination module in MBVD does not utilize the reward signal in the environment. MB VD adopts the implicit model-based idea without reward prediction.
The imagination model we offered has all the essential components of the enviromment, except the action-value model, which can evaluate the expected returns. We leave the implementation of the action-value model to the reinforcement learning process. With the help of the imagined states generated from the simulated model, the agents seem to have the foresight, which means that the agents estimate the action value based on long-term consequences rather than direct results. Thus, we can discern the role of the imagination module in MBVD.
4.2 The MBVD Flow Diagram
The generation process of imagined rollouts can be found in Figure 1(b) At time 4, T, refers to the trajectory history of all agents. Since the posterior encoder can be regarded as the inverse function of the observation function, we can infer the latent state & from Tr. Furthermore, according to the current joint policy π of all agents (implemented by the agent network), we can get the actions â ~ T.. Tt) that all agents should perform to maximize the estimate of the state-action value. Then the learned prior model is used to derive the next latent state st+1 based on & and ú, To get the joint action ú at the next step in the imagined rollouts, we need to obtain the trajectory history T qo (- &Prior) corresponding to the latent state $141 through the observation function. We perform one step forward in the imagined rollout in the above way. Then we can roll out the environment model over multiple time steps by feeding the following imagined variables into the model. As mentioned above, we use all individual hidden outputs h instead of the trajectories T to accelerate the training speed in the implementation.
We use the latest policies of all agents as the rollout policies. Here is a reasonable and intuitive explanation: when people make strategic decision, they always use their current policies as the imagined policies for their decision-making, rather than previous strategies that have led to poor performance. Besides, for the stability of reinforeement learning, turn the random process involved in the imagination module during reinforcement learning into a deterministic form. For example, the imagined actions i are selected from greedy policies. Moreover, we directly take the mean of the approximating distribution as the inferred latent state & and trajectory history 4. However, we still use the reparameterization trick when learning the imagination module.
One k-step rollout starting with & can be written as ((S., Te, ū,), (sPrier Th ue+1))
(sping T' }. Since the latent state $ is inferred from all agents' historical trajectories and actions, 8 contains the information of T and u. Furthermore, to ensure that the parameters of the imagination module do not increase with the rollout horizon k; the set of imagined latent states (Se, .Prior prior) is fed into the GRU (5) get the aggregated rollout state & goollout, Finally, s .., t+*
concatenate the aggregated rollout state & RRollout with the real global state 8, and input them into the
mixing network of the value decomposition framework. We believe that the imagined states contain information about the possible states of the future, which can help all agents evaluate the current state more accurately.
5

---

MBVD does not require pre-trained environment model, which means the simulated model and the action-value model all trained from scratch simultaneously. As an end-to-end efficient MBRL algorithm, MBVD can be extended to any value decomposition method with the mixing network.
4.3 Overall Learning Objective
Next, will elaborate the training objectives of MBVD. MBVD involves processes: the reinforcement learning process, whose objective is to minimize the td-error; the other is learning the imagination module, which includes the optimization of the prior and the posterior. These two processes are carried out simultaneously. For the convenience of description, we define the parameters of the value decomposition framework vh.
MBVD can be seen as the value decomposition method with an imagination module, so the loss function for reinforcement learning consistent with the original value decomposition method. The most obvious difference is that the global action-value function Qtat is calculated with respect to the actual state and the latent rollout state gRollout It is important the aggregated rollout state gRollout conditioned on the past state 8 in the replay buffer, so we do not change the off-policy update paradigm. The loss function for reinforcement learning can be obtained:
that MBVD generates
LRL = (y QTo ut St, st
&Rollout, v²))
where yd8t = Tt + max.4+1 Qtot (Te+1: u1+11 St41, @Rollaut, and 1 represents the parame- ters of the target network.
The loss function of the posterior can be divided into reconstruction loss and KI. divergence loss. The posterior model infers the current latent state after the observations of all agents are given, which requires that the posterior model extract helpful information from the original input. It can be achieved by narrowing the difference between the model output r' and the model input The reconstruction loss function the prior model is similar, and both of them can be computed by:
LRC MSE (Tr; O);
MSE (8:-1,0h-1) (5%-1 :
RC
where MSE means the mean square error. Furthermore, in addition to the KL term Equation I throw in the KL divergence between the prior distribution and the standard Gaussian distribution ((O,, 1) as a regular term to avoid the overly complex prior distribution of the latent state PPoor So the KL loss is follows:
CKL DKL [p Prior (&c $ u1 ||W
+ DRI. [go (&e, 8 U 2+) I (&s 8t- ut-1))
We use the KL balancing mechanism to optimize the KL term in the actual implementation. KL balancing enables the prior to minimize KL loss at a faster learning rate than the posterior, which can be expressed as:
D KLLaalancing [go (-) Iiprr (.)
DxL [40 (.) || Detach Prior (.)] + (1 a.)DKI. [Detach (go (-)) Ip Price (-)]
where a e [0, 1] is configurable parameter, Detach(-) can stop the backpropagation from gradients of certain variables or functions.
In addition, have other optional training objectives for auxiliary tasks in complex scenarios. In this paper, we make predictions of the feasible action set only in StarCraft II:
LFA BCE Al;
where BCE denotes the binary cross-entropy error. Thus, the total loss function can be written as:
L = LRL Lrc +
Prior LXL. + Lrx- LRC
(2)
By minimizing the total loss function L, we can guide MBVD to accelerate reinforcement learning with the help of the imagination module.

---

Experiments
This section will consider whether the value decomposition method can benefit from the implicit model-based method. We compare the performance of MBVD with other popular baselines, including VDN, QMIX, MAVEN [30], Weighted QMIX [40), ROMA 499, and RODE [50]. Then by conducting the ablation experiments, we can clarify that every component in the learned model is indispensable, and different horizons of the rollout have a significant impact on performance. Finally, we will visualize the imagined rollouts, which will more intuitively and powerfully show the foresight of MBVD. Note that the implementation of agents in MBVD in all experiments is based on QMIX. The details of all experiments can be found in Appendix ??
(a) StarCraft 11
(b) Google Research Football
(c) Multi-Agent Discrete MuJoCo
Figure 2: Examples of the three experimental platforms.
5.1 Performance StarCraft II
SMAC multi-agent miirre-management experimental platform based on the real-time strategy game StarCraft II, which contains wealth of scenarios corresponding to different challenges. To verify the performance improvement brought by the imagination module, we pay more attention to the performance comparison between MBVD and QMIX. We select some representative scenarios.
The median performance and the 25-75% percentiles are illustrated in Figure 3
RODE
2532
1c0edi
82
0.5
Tond
5m, 6m
bane
vs. bane
2o, vS. 54zg
Vodua 0414000000
21
0.4
0.
12
24
0.5
Trne
122
2.0
00
0.4
0.
1.2
1.5
2.0
Vidae Te4
100
Visian Tear 780
8 18888
Uoda T0410002011
R 2 R
Figure Performance comparison with baselines in different SMAC scenarios.
MBVD reaches state-of of-te-of-theaat performance in most scenarios and exceeds the basic algorithm QMIX. Especially hard or super hard scenarios such 2c ys 642g, MMM2, and 3s52_vs vs ys_366, the superiority of MBVD is more significant. In 5m_vss 6m_ due to the uncertainty of the environment, other baselines will encounter performance bottleneck. However, MBVD can predict the future state to achieve the best performance in 5m_vs vy_6m. Even in some easy scenarios, MB VD still has high sampling efficiency. impossible for some complex QMIX-style variants. MBVD can robustly reduce the sampling complexity in both casy and hard scenarios.
7

---

5.2 Performance on Google Research Football
The Google Research Football environment provides novel reinforeement learning environment where multiple agents can be trained to play football. To verify the effectiveness of our proposed method, we carry out MBVD and other baselines on the Football Academy, which is a diverse set of mini scenarios of varying difficulty. We select three representative official scenarios, and the experimental results are delivered in Figure
RODE
ROMA acadaryy pass, and ahoot, in Jasper
OW--MMIX
CV-QMI
academyy. with Jaspar
academy JUL JMBX,
shoot, wiTh kampar
Figure 4: Performance comparison with baselines in different Google Research Football scenarios.
Compared with QMIX, which has a relatively simple structure, algorithms such as RODE and ROMA do not perform well. possible reason that the number agents in these scenarios not large, and the role assignments may hinder the learning. Conversely, MBVD based on QMIX achieves the highest sample efficiency in three scenarios with different difficulty levels. The experimental results obtained in Google Research Football also fully demonstrate the generalization capability of MBVD
5.3 Performance on Multi-Agent Discrete MuJoCo
Many model-based reinforcement learning algorithms, including the work mentioned above, tend to be compared on MuJoCo. (37] recently proposed a novel benchmark for continuous cooperative multi-agent robotic control in the multi-agent field, called Multi-Agent MuJoCo. A given single robotic agent is viewed as a body graph containing many disjoint sub-graphs, and each sub-graph contains one or more joints that can controlled. this paper, each joint regarded as an agent that makes local decisions conditioned on partial observations. Since this paper focuses on the impacts of the world model, we propose a discrete variant of Multi-Agent MuJoCo. Detailed environment settings can be found in Appendix ??. We selected four representative scenarios, and Figure S describes the episode return of each algorithm. Since most of the relatively complex algorithms cannot converge in this environment, we only choose MBVD, QMIX, and VDN for comparison.
MBVD
QMIX
VDN
HalfCheetah
Hopper
Walker
Humanoid
Eprcdd Ratum
2
Eppco
(100
1160
210)
010)
Figure 5: Median episode return on different Multi-Agent Discrete MuloCo tasks.
From the results, we can intuitively see the performance improvement brought by the imagination module because we see MBVD as a variant of QMIX that has an additional implicit foresight module. MBVD outperformed QMIX in all given scenarios, which also means MBVD proposed can be applicd to different environments.
5.4 Ablation Studies
We carry out ablation studies to test the contribution of the components of the imagination module and investigate how the rollout horizon & affects the performance of MBVD. For the first study, we proposed two variants of QMIX that input the additional information to the mixing network, QMIX-RS and QMIX-LS. Both of them aggregate the information of the next steps from the actual trajectories rather than imagined rollouts. However, the difference is that QMIX-RS uses the real states and QMIX-LS the latent states. answer the second problem, we run MBVD with different rollout horizons the 2c 642g scenario of SMAC and compare their performance.

---

MBVD
QMIX-RS
CMIX-US
QMIX
642g
(a) Win rates for MBVD and the ablation.
(b) Influence of the le for MBVD.
Figure 6: Results for ablation studies on 2c ____v__6444 64zg map.
As shown in Figure 6[a] both QMIX-RS and QMIX-LS failed to solve the task, which means that the transition function that can generate the imagined rollouts is critical. Besides,
also infer the contribution of the observation function from that the better performance of QMIX-LS than QMIX- RS. For the horizon of the rollouts, we can conclude from Figure 6(b) When using short-horizon rollouts, the agents cannot fully interact with the imagined model, resulting in the same performance vanilla QMIX. However, as k increases significantly, MBVD loses the monotonic improvements because of the compounding error. The effect of k holds the same for the other scenarios, but the
optimal choice of & in each task is different. If not explicitly stated, we will set & to 3 in this paper for convenience.
5.5 Visualization
To intuitively explain agents' foresight ability in MBVD, we visualized the latent state sequence generated by the interaction between agents and the imagined model. For the trajectories of an episode in the 2c_vs_6426 scenario, visualized the t-sne embeddings of the latent states predicted at each step in the rollout and compared them with the real latent state embedding. From Figure 7 we find that the difference between the embedding of the imagined latent states and real ones gradually increases as the horizon length grows, but there are similarities between them. It also reveals that our proposed imagination module can capture and learn the dynamics of the multi-agent environment.
1sl Imagined Lakend States
2nd Imagined Laient States
3rd Inagired L Letant States
Figure 7: The 2D t-SNE embedding of real latent states (red) and imagined latent states (blue).
Discussion on the Rollout Horizon
As describe in ablation studies, the choice of horizon length & affects the performance of MBVD. Short horizons are easier to be predicted but do not provide enough information to be useful for decision-mand makknng. Long horizons can carry more valuable information, but the generated imagined rollouts are inaccurate because of compounding errors. There has been some work on model- -ased reinforcement learning that attempts break this trade-off. For bias that might exist in sampling and environmental models, MBPO [17] suggests a branched rollout. And MBPO can determine the longest tolerable rollout length according the lower bound of return. BMPO 28 uses the newly introduced bidirectional models to significantly reduce model compounding error. In addition, most of the model-based value expansion (MVE) methods 9 involve adaptive selection of horizons.
They must use dynamics model to simulate the short-term horizon and Q-learning to estimate the long-term value beyond the simulation horizon. The automatic selection of would be made possible by explicitly estimating uncertainty in the dynamics model or ensemble models. STEVE [3] suggests

---

interpolating the estimated values of various rollout steps, and the weight for each rollout step is chosen by considering the ensemble predictions' variance. AdaMVE [5i] mitigates the detrimental effects of the compounding error by selecting the rollout horizon for any state based on the learned model error function. RAVE [56] uses probabilistic models to capture uncertainty (including aleatoric and epistemic uncertainty) and uses the lower confidence bound for value estimation to avoid optimistic estimation. DMVE [48] exploits the fact that the uncertainty of the model and the novelty of data are highly relevant in deep learning. So DMVE selects horizons based on the top k minimum reconstruction errors of the auto-encoder. To sum up, in order to further solve the problem of adaptive horizon selection, state as a reference.
try select an appropriate i value with the uncertainty of the imagined
Conclusion
Due to the high complexity of multi-agent systems, MBRL algorithms are difficult to be applied to them. This paper proposes MBVD, a novel and implicit model- based cooperative multi-agent reinforcement learning method. By learning the world model and imagining the future latent states after making several decisions under the current policy to estimate the current state value, agents in MBVD obtain the foresight ability. Through experiments and visualization, we have proved the efficiency and generalization of MBVD. This is the first study on Dec-POMDPs from the perspective of model-based reinforcement learning.
The defect this study is that the rollout horizon is manually chosen. In our future research, we intend to concentrate how to choose the appropriate rollout horizon. The issue is intriguing one which could be usefully explored in further research.
Acknowledgment and Disclosure of Funding
The work is supported by the National Defence Foundation Reinforcement Fund.
References
[1] Zaheer Abbas, Samuel Sokota, Erin Talvitie, and Martha White. Selective dyna-style planning under limited model capacity. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1-10. PMLR, 2020.
[2] Eugenio Bargiacchi, Timothy Verstraeten. Diederik M. Roijers, and Ann Now'e. Model- multi-agent reinforcement learning with cooperative prioritized sweeping. ArXiv, abs/2001.072 107527, 2020.
[3] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lec. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Information P Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 8234 8244, 2018
and Roman Garnett, editors, Advances in Neural
[4] Lars Buesing, Théophane Weber, Sébastien Racanière, S. M. Ali Eslami, Danilo Jimenez Rezende, David P. Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, and Daan Wierstra. Learning and querying fast generative models for reinforcement learning. ArXiv, abs/1802.00 802.03006. 2018
[5] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar, 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14- 1179.
[6] Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and P. Abheel. Model- based reinforcement lcarning via meta-policy optimization. ArXiv, abs/1 00511, 2018.
[7] Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: model-basee and data-efficient approach to policy search. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 July 2, 2011, pages 465 472. Omnipress, 2011.
10

---

[8] Michael Fairbank. Reinforcement learning by value gradients. ArXiv, abs/0803. 2008.
[9] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning, ArXiv, abs/ 803.0000, 2018.
[10] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson, Learning to communicate with deep multi-agent reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Tsabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2137-2145, 2016.
[11] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Chirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposiuu on Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 2974-2982. AAAI Press, 2018.
[12] Chen Gong, Qiang He, Yunpeng Bai, Xinwen Hou, Guoliang Fan, and Yu Liu. Wide-senss stationary policy optimization with bellman residual on video games. 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1 6, 2021.
[13] Danijar Hafner, Timothy P. I Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[14] Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[15] Nicolas Heess, Gregory Wayne, David Silver, Timothy P. Lillicrap, Tom Erez, and Yuval Tassa, Learning continuous control policies by stochastic value gradients. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2944 2952, 2015.
[16] Shariq lqbal and Fei Sha. Actor- attenaionnnniiiiic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2961-2970. PMLR, 2019.
[17] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alcch-BBue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8- 8-14, 2019, Vancouver. BC, Canada, pages 12498 12509, 2019.
[18] Michael Janner, Igor Mordatch, and Sergey Levine. Gamma- models: Generative temporal difference learning for infinite-horizon prediction. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tiee -ainn Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[19] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bac Bianchi, and Roman Garnett, editors, Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 7265-7275, 2018
[20] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Camphell, Konrad Czechowski, Dumitru Erhan. Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020.
11

---

[21] Mehdi Khashei, Mehdi Bijari, and Gholam Ali Raissi Ardali. Improvement of aulo-regressive integrated moving average models using fuzzy logic and artificial neural networks (anns). Neurocomputingg 72: 956-967, 2009.
[22] Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning schedule communication in multi-agent reinforcement learning. In 7th International Con- ference on Learning Representatons. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. 2019.
[23] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014.
[24] Bangalore Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Kumar Yogamani, and Patrick P'erez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23:4909 2022.
[25] Jayant Kulkarni and Liam Paninski. State-space decoding of goal-directed movements. IEEE Signal Processing Magazine, 25:78-86, 2008
[26] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lassc Espcholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: novel reinforcement learning environment. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, [AAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 4501-4510. AAAI Press, 2020.
(27] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Picter Abbeel. Model-ensemble trust-region policy optimization. In 6th International Conference Leurning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Truck Proceedings. OpenReview.net, 2018
[28] Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In Proceedings of the 37th International Conference on Machine Learning, ICMI, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5618-5627. PMLR, 2020.
[29] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic environments. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6379-6390, 2017.
for mixed cooperativecompetiiie
[30] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi-agent, varia- tional exploration. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'AchééBuc, Emily B. Fox, and Roman Garnett, editors. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pr Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, 2019.
Vancouver. BC, Canada, pages
[31] Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, Advances
Advaors.Avvaness in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 6118-6128, 2017.
[32] Frans A. Oliehoek and Chris Amato. A concise introduction to decentralized pomdps. In SpringerBriefs in Intelligent Systems, 2016
[33] OpenAL, Ilge Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas A. Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei M. Zhang, Solving rubik's 's cube with robot hand. ArXiv, abx/1910.071 103, 2019.
[34] Liam Paninski, Yashar Ahmadian, Daniel Gil Ferreira, Shinsuke Koyama, Kamiar Rahnama Rad, Michael Vidne, Joshua T. Vogelstein, and Wei Wu. A new look state-space models for neural data. Journal of Computational Neuroscience, 19:107-126, 2009.
[35] Young Joon Park, Yoon Sang Cho, and Scoung Bum Kim. Multi-agent reinforcement learning with approximate model learning for competitive games. PLoS ONE, 14, 2019.
[36] Barna Pasztor. Ilija Bogunovic, and Andreas Krause. Efficient model-based multi- mean-field reinforcement leamiin. ArXiv, abs/2107.04050, 2021.
12

---

[37] Bei Peng, Tabish Rashid. Christian Schroeder de Witt. Plerre-Alexandre Kamienny, Philip Torr. Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems, 2021.
[38] Peng Peng. Ying Wen, Yaodong Yang. Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent cotoriiillly- nets: Emergence of numan-leved coordination in learning to play starcraft combat games. arXiv: Artificial Intelligence, 2017.
[39] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages
PMLR, 2018
[40] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[41] Mikayel Samvelyan. Tabish Rashid, C. S. D. Witl, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner. Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ArXiv, aws//902..// 1902.04024,4 2019.
[42] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillierap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 7839:604-609, 2020.
[43] Kyunghwan Son, Daewoo Kim, Wan Ju Kang. David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5887-5896. PMLR, 2019.
[44] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with back- propagation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2244 22522, 2016.
[45] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech M. Czameeki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- dccomposition networks for cooperative multi-agent learning. ArXiv, abs/1706.05296, 2018
[46] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-286, 2005.
[47] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring. Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pages 1-5, 2019.
[48] Junjie Wang. Qichao Zhang, Dongbin Zhao, Mengehen Zhao, and Jianye Hao. Dynamic horizon value estimation for model-based reinforcement learning. arXiv preprint arXiv:2009.05993 2020.
[49] Tonghan Wang, Heng Dong, Victor R. Lesser, and Chongjie Zhang. ROMA: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9876-9886. PMLR, 2020.
[50] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: learning roles to decompose multi-agent tasks. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
13

---

[51] Chenjun Xiao, Yifan Wu, Chen Ma. Dale Schuurmans, and Martin Müller. Learning to combat compounding-erroo in model-based reinforcement learning. ArXiv, abs/1912. 11266 2019.
[52] Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based reinforcement learning with theoretical guarantees. ArXiv, abs/1807 0000800 2019.
[53] Zhiwei Xu, Yunpeng Bai, Bin Zhang, Dapeng Li, and Guoliang Fan. Haven: Hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism, ArXiv, abs/21 10.07246, 2021.
[54] Kaiqing Zhang, Sham M. Kakade, Tamer Basar, and Lin F. Yang. Model-based multi-agent RL. in zero-sum markov games with near-optimal sample complexity. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, Hsuan- Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[55] Weinan Zhang, Xihuai Wang, Jian Shen, and Ming Zhou. Model-based multi-agent policy optimization with adaptive opponent-wise rollouts. IJCAI, 2021.
[56] Bo Zhou, Hongsheng Zeng, Fan Wang. Yunxiang Li, and Hao Tian. Efficient and robust reinforcement learning with uncertainty based value expansion. ArXiv, abs/1912.05328, 2019. 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofss of all theoretical results? [Yes] See Appendix ??.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix ??.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix ??.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'rere
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Gdt6spgrmxD", "reviewer": "Reviewer_UNCB", "summary": "This paper introduces the model-based value decomposition (MBVD) framework, which extends a value decomposition method (e.g., QMIX) with the imagination module. Specifically, the imagination module consists of two VAEs and outputs the rollout of imagined future states in the latent space ($\\hat{s}^{\\text{Rollout}}_{t}$). Then, this rollout information is concatenated with the current state ($s_t$) and becomes an input to the mixing network of QMIX. MBVD assumes that these imagined states contain helpful information for agents to evaluate the current state's value more accurately. Empirical results show that MBVD outperforms baselines in various domains of SMAC, Football, and multi-agent MuJoCo. ", "questions": "1. In Section 4.2, the paper notes that the aggregated rollout state ($\\hat{s}^{\\text{Rollout}}_{t}$) is concatenated with the real global state ($s_t$). However, this paper assumes Dec-POMDP settings, so it is not possible to observe the real global state information. As such, it is confusing how the real global state information ($s_t$) can be obtained in MBVD.\n2. After reading Section 5.4 (ablation studies), it is confusing to interpret the results for MBVD vs QMIX-RS. Because QMIX-RS receives the actual trajectories instead of predictions (which can be noisy approximations of true trajectories), I thought QMIX-RS would be the upper-bar performance of MBVD, but this is not true in Figure 6(a). Would it be possible to ask for a clarification?\n3. Figure 7 is not straightforward to compare the real and imagined latent states. Possibly, it might be better to have 3 subfigures instead of 4 subfigures by overlaying the real latent state subfigure to the imagined latent state subfigures.\n4. It is not specified how $p(u_t|z_t)$ is represented (e.g., $p(u_t|z_t)=\\prod_a p(u^a_t|z^a_t)$) \n5. There is a typo in Section 2.2: \"MBVD we proposed\".\n6. A citation is missing for Dec-POMDP in Section 3.1.\n7. In Section 4.2, the (fixed) latest policies of all agents are used for the rollout. However, in practice in the future, they will have non-stationary policies due to their policy updates, so I wonder whether it will be helpful to consider their non-stationary policies in generating the rollout as a future direction (related works: Foerster et al., 2018; Kim et al., 2021). \n\n**References:**  \nFoerster et al., DiCE: The Infinitely Differentiable Monte-Carlo Estimator, 2018.  \nKim et al., A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning, 2021.  ", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper introduces a new framework of how learned models can be integrated into an existing value decomposition method. \n2. The evaluation is performed across complex domains, demonstrating the scalability of MBVD.\n\n**Weaknesses:**\n1. Because this paper's main contribution is related to the model-based aspects, it would be desirable to have comparisons against existing model-based multi-agent baselines, but there are no comparisons against model-based multi-agent baselines in the evaluation. \n2. MBVD uses the existing methods in Section 4.3 (e.g., VAE loss, QMIX loss), so there might not be a new contribution from the optimization perspective. However, as noted in Strength #1, how learned models can be combined in QMIX is new, and if this paper studied deeper on various ways to integrate learned models (other than concatenating the imagined states to the current state), this contribution could have been more significant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F9_W-R4o6C-", "reviewer": "Reviewer_LYEL", "summary": "This paper proposes Model-Based Value Decomposition (MBVD), an implicit model-based multi-agent reinforcement learning (MARL) method that integrates the idea of model learning into value decomposition. The authors are enlightened by the ability of long-term vision of humans, which refers to humans’ consideration of future states after several interactions with the environment following their current strategies. Thus the authors let the agents cooperate by obtaining the aggregated latent states in the future. They elaborate on MBVD by discussing the framework, the flowchart, and the learning objectives of the algorithm. The MBVD method is tested in different domains, such as StarCraft II, Google Research Football, and Multi-Agent MuJoCo. The paper clarifies that MBVD exceeds the performance of several baselines in terms of sample efficiency. A derivation of the evidence lower bound (ELBO) and the settings of the environments are included in the appendix.", "questions": "In lines 33-34, should the “decentralized partial observable Markov decision process” be “decentralized partially observable Markov decision process” instead?\n\nIn line 237, what is the definition of Detach(·)? I can only find the description of what this function can do. References can be added here, but are not necessary.\n\nIn lines 142 and 149, the authors mentioned that they used two VAEs as the prior and the posterior for the MBVD framework. Because there have been many variants of the VAE model since it was proposed by Kingma and Welling in 2014, I don’t know which one was used by the authors or if they made a new variant.   \n\nFrom what I understand, when we apply Equation 2 to the experiments in Google Research Football and Multi-Agent MuJoCo, the last error (L_{FA}) is 0 because predictions of the feasible action set are only made in StarCraft II?\n", "limitations": "The authors specified the limitation of the paper is that the value of the rollout horizon was chosen manually. It would be nice if they can elaborate on the limitations, for example, by discussing how other researchers deal with the rollout horizon in their works. It does not seem likely that this paper will cause any potential negative social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis work has good originality. The authors state that the study is the first attempt to apply the model-based reinforcement learning ideas to decentralized partially observable Markov decision process (Dec-POMDP) problems, which is true to the best of my knowledge.\nThe manuscript is clearly written. Important contents, such as the method itself and the results of the experiments, are presented in a well-organized way. Some more details like the experimental setup are placed in the appendix. \n\nThe proof of breaking down the ELBO (Eq. 1) is given in the appendix, which is correct to me and improves the soundness of the submission. The manuscript proves itself to be a complete piece of work rather than a work in progress.\n\nWeaknesses:\n\nIn the Related Work section, the authors used a long paragraph to talk about Single-Agent Model-based RL. However, it would be better if the majority of the discussion is on Multi-Agent Model-based RL because the authors’ research is exactly about MARL. There may be relatively fewer papers on multi-agent model-based RL, but the authors can discuss each of them in greater detail.\n\nIn the Experiments section, the performance of MBVD is not very impressive in relatively simpler scenarios of StarCraft II. It is hard to decide whether MBVD outperforms other baselines in every hard scenario because only representative scenarios are selected. Also, the ablation studies only show the results from the 2c_vs_64zg map. I am curious to see results from other scenarios for a better generalization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept