Efficient Multi-agent Reinforcement Learning by Planning
Authors: Qihan Liu
Jianing Ye
Xiaoteng Ma
Jun Yang
Bin Liang
Chongjie Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Model-based RL
MARL
Planning
MCTS
Primary area: reinforcement learning

Abstract: Multi-agent reinforcement learning (MARL) algorithms have accomplished remarkable breakthroughs in solving large-scale decision-making tasks. Nonetheless, most existing MARL algorithms are model-free, limiting sample efficiency and hindering their applicability in more challenging scenarios. In contrast, model-based reinforcement learning (MBRL), particularly algorithms integrating planning, such as MuZero, has demonstrated superhuman performance with limited data in many tasks. Hence, we aim to boost the sample efficiency of MARL by adopting model-based approaches. However, incorporating planning and search methods into multi-agent systems poses significant challenges. The expansive action space of multi-agent systems often necessitates leveraging the nearly-independent property of agents to accelerate learning. To tackle this issue, we propose the MAZero algorithm, which combines a centralized model with Monte Carlo Tree Search (MCTS) for policy search. We design an ingenious network structure to facilitate distributed execution and parameter sharing. To enhance search efficiency in deterministic environments with sizable action spaces, we introduce two novel techniques: Optimistic Search Lambda (OS($\lambda$)) and Advantage-Weighted Policy Optimization (AWPO). Extensive experiments on the SMAC benchmark demonstrate that MAZero outperforms model-free approaches in terms of sample efficiency and provides comparable or better performance than existing model-based methods in terms of both sample and computational efficiency.

Published as a conference paper at ICLR 2024
EFFICIENT MULTI-AGENT REINFORCEMENT LEARN- ING BY PLANNING
Qihan Liu Jianinn Xiaoteng Ma'", Jun Yang"t, Bin Liang', Chongjie Zhang DDepartment of Automation, Tsinghua University Institute for Interdisciplinary Information Sciences, Tsinghua University 'Department of Computer Science & Engineering, Washington University in St. Louis [1qh20, yein21, ma-xt17)eman yyangjun60 bliang] 2 inghua edu. en chongj wewustl..c.m edu
tsinghua. edu. cn
ABSTRACT
Multi-agent reinforcement learning (MARL) algorithms have accomplished re- markable breakthroughs in solving large-scale decision-making tasks. Nonethe- less, most existing MARL algorithms are model-free, limiting sample efficiency and hindering their applicability in more challenging scenarios. In contrast, model-based reinforcement learning (MBRL), particularly algorithms integrating planning, such as MuZero, has demonstrated superhuman performance with lim- ited data in many tasks. Hence, we aim to boost the sample efficiency of MARL by adopting model-based approaches. However, incorporating planning and search methods into multi-agent systems poses significant challenges. The expansive action space of multi-agent systems often necessitates leveraging the nearly- independent property of agents to accelerate learning. To tackle this issue. we propose the MAZero algorithm, which combines centralized model with Monte Carlo Tree Search (MCTS) for policy search. We design a novel network structure to facilitate distributed execution and parameter sharing. To enhance search effi- ciency in deterministic environments with sizable action spaces, we introduce two novel techniques: Optimistic Search Lambda (OS(A) and Advantage-Weigeee Policy Optimization (AWPO). Extensive experiments the SMAC benchmark demonstrate that MAZero outperforms model-fre -free approaches in terms of sample efficiency and provides comparable better performance than existing model- based methods in terms of both sample and computational efficiency. Our code is available https: /ttp:/ggtttaa com/ http:/Lhgghhhh//MA222
INTRODUCTION
Multi-Agent Reinforcement Learning (MARL) has significant success in recent years, with ap- plications in real-time strategy games AAulkumaran et al. 2019 Ye et al. 2020), card games (Bard et al. 2020), sports games (Kurach al. 2020), autonomous driving (Zhou al. 2020), and multi- robot navigation (Long et al. 22018). Nonetheless, challenges within multi-agent environments have led the problem of sample inefficiency. One key issue is the non-stationarity of multi-agent set- tings, where agents continuously update their policies based on observations and rewards, resulting in changing environment for individual agents (Nguyen et al. 2020). Additionally, the joint action space's 's dimension can exponentially increase with the number of agents, leading to an immense policy search space (Hernandez- as partial observation, coordination, and credit assignment, necessitate considerable demand for samples in MARL for effective training (Gronauer Diepold (002)).
et al., 2020). These challenges, combined with issues such
Conversely, Model-Based Reinforcement Learning (MBRL) has demonstrated its worth in terms of sample efficiency within single-agent RL scenarios, both practice (Wang et al. 2019) and
"quual contribution. Corresponning authors.

---

Published as a conference paper at ICLR 2024
theory (Sun et al. 2019). Unlike model-free methods, MBRL approaches typically focus learn- ing a parameterized model that characterizes the transition or reward functions of the real envi-
ronment (Sutton & Barto
2018 Corneil al., 2018 Ha & Schmidhuber, 2018). Based on the usage of the learned model, MBRL methods can be roughly divided into two lines: planning with
model (Hewing et al.] 2020 Nagabandi et al.] 2018: Wang & Baa 2019 (Schrittieesrr et al. 2020
Hansen et al. 2022) and data augmentation with model (Kurutach et al. 2018) Janner et al. 2019
Hafner al. 2019 2020 2023). Due the foresight inherent in planning methods and their theo-
retically guaranteed convergence properties, MBRL with planning often demonstrates significantly higher sample efficiency and converges more rapidly (Zhang al.) 2020p. A well-known planning- based MBRL method is MuZero, which conducts Monte Carlo Tree Search (MCTS) with a value- equivalent learned model and achieves superhuman performance video games like Atari and board games including Go, Chess and Shogi (Schrittwieser et al.) 2020).
Despite the success of MBRL in single-agent settings, planning with the multi-agent environment model remains in its early stages of development. Recent efforts have emerged to bridge the gap by combining single-agent MBRL algorithms with MARL frameworks (Willemsen et al. 2021 Bar-
giacchi et al. 2021 Mahajan et al., 2021: Egorov & Shpilman 2022). However, the extension of single-agent MBRL methods to multi-agent settings presents formidable challenges. On one hand, existing model designs for single-agent algorithms do not account for the unique biases inherent to multi-agent environments, such as the nearly-independent property of agents. Consequently, di- rectly employing models from single-agent MBRL algorithms typically falls short in supporting efficient learning within real-world multi-agent environments, underscoring the paramount impor- tance of model redesign. On the other hand, multi-agent environments possess action spaces sig- nificantly more intricate than their single-agent counterparts, thereby exponentially escalating the search complexity within multi-agent settings. This necessitates exploration into specialized plan- ning algorithms tailored to these complex action spaces.
In this paper, we propose MAZero, the first empirically effective approach that extends the MuZero paradigm into multi-agent cooperative environments. In particular, our contributions are fourfold:
1. Inspired by the Centralized Training with Decentralized Execution (CTDE) concept, we develop a centralized-value, individual-dynamic model with shared parameters among all agents. We incorporate an additional communication block using the attention mechanism to promote coop- eration during the model unrolling (see Section 4.1).
Given the deterministic characteristics of the learned model, we have devised the Optimistic Search Lambda (OS(A) approach (see Section 4.2 It optimistically estimates the sampled re- turns while mitigating unrolling errors at larger depths using the parameter X.
3. We propose novel policy loss named Advantage- Addantage-Weighted Policy Optimization (AWPO) (see Section 4.3) utilizing the value information calculated by OS(A) to improve the sampled actions.
4. We conduct extensive experiments on the SMAC benchmark, showing that MAZero achieves superior sample efficiency compared to model-free approaches and provides better or compara- ble performance than existing model-basee methods in terms of both sample and computation efficiency (see Section
2 BACKGROUND
This section introduces essential notations, and related works will be discussed in Appendix A
POMDP Reinforcement Learning (RL) addresses the problem of an agent learning to act in an en- viromment in order to maximize a scalar reward signal, which is characterized as partially observ- able Markov decision process (POMDP) (Kaelbling et al. 1998) deñned by (S, A, T, U2 O,7), where S is set of states, A is set of possible actions, T: S.Sx x A x S + [0, 1] is a transition func- tion over next states given the actions at current states, S A + R is the reward function. 82 is the set of observations for the agent and observing function O : S + S maps states to observations. y E [0, 1) is the discounted factor. At each timestep f, the agent acquire an observation O = (ss) based current state st. choose action a, upon the history of observations Oct and receive cor- responding reward u, = Uss a,) from the environment. The objective of the agent is to learn a policy æ that maximizes the expected discounted return J(T) E, EEe 2 la, at (-(0g))))
2

---

Published as a conference paper at ICLR 2024
MuZero MuZero (Schrittwieser al., 2020 is an MBRL algorithm for single-agent settings that amalgamates learned model of environmental dynamics with MCTS planning algorithm. MuZero's learned model consists of three functions: representation function h, a dynamics func- tion and a prediction function f. The model is conditioned on the observation history Oc: and a sequence of future actions 0ta++K and is trained to predict rewards rr,0:K, values 0r,0:K and policies Pr where kt Vt Pt.k are model predictions based on imaginary future state St.k unrolling & steps from current time Specifically, the representation function maps the current observation history ogg into hidden state St,u, which is used as the root node of the MCTS tree. The dynamic function g inputs the previous hidden state $t,k with an action Oc+* and outputs the next hidden state $1,k+1 and the predicted reward ri,k- The prediction function I computes the value V2,k and policy Pt.k at each hidden state St.k- To perform MCTS, MuZero runs N simulation steps where each consists of 3 stages: Selection, Expansion and Backup. During the Selection stage, MuZero starts traversing from the root node and selects action by employing the probabilistic Upper Confidence Tree (pUCT) rule (Kocsis & Szepesvári 2006; Silver et al. (016) until reaching leaf node:
(s.b)
arg max Q(s.a) + P(s,a) 1 1+ N(s,a)
c(s)
(1)
where Q(s, a) denotes the estimation for Q-value, (s, the visit count, P(s, a) the prior prob- ability of selecting action a received from the prediction function, and c(s) is used to control the influence of the prior relative to the Q-value. The target search policy <(-(5t,0) is the normalized distribution of visit counts at the root node $1,0- The model is trained minimize the following 3 loss: reward loss L,,, value loss ly and policy loss Ip between true values and network predictions.
K
L, MuZero
[1 Te,kk + (t,E)))) + (o(TT+k.PE.)))
where u is the real reward from environment, 24t- Ei= u ty ve+ton is n-step return consists of reward and MCTS search value, Ti+kk is the MCTS search policy.
EfficientZero Ye et al. 2021) proposes an asynchronous parallel workflow to alleviate the computa- tional demands of the reanalysis MCTS. EfficientZero incorporates the self supervised consistency loss 1, = || 8t+,,0ll lI using the SimSiam network to ensure consistency between the hidden state from the kth step's s dynamic 8t,k and the direct representation of the future observation 8
Sampled MuZero Since the efficiency of MCTS planning is intricately tied to the number of simulations performed, the application of MuZero has traditionally been limited to domains with relatively small action spaces, which can be fully enumerated on each node during the tree search. To tackle larger action spaces, Sampled MuZero (Hubert et al. 202T) introduces sampling baased based framework where policy improvement and evaluation are computed over small subsets of sampled actions. Specifically, every time in the Expansion stage, only a subset ((s) of the full action space A is expanded, where (ss) is on-time resampled from a policy 3 base on the prior policy n. After that, in the Selection stage, an action is picked according to the modified pUCT formula.
N2,
Q(ssaa 3 P(s.a)
arg max (
1+ I+N(s,a) c(s)
(3)
where 3 is the empirical distribution of sampled actions, which means its support is T(s).
In this way, when making actual decisions at some root node state st,0, Sampled MCTS will yield a policy w-))st,)) after N simulations, which is a stochastic policy supported in T(..,)). The actual im- proved policy is denoted as π. (-Is,,) = E[w((-|st,)),, which is the expectation of (- The randomness that the expectation taken over is from all sampling operations in Sampled MCTS. As demonstrated in the original paper, Sampled MuZero can be seamlessly extended to multi-agent settings by considering each agent's policy as a distinct dimension for a single agent's 's multi-discrent action space during centralized planning. The improved policy is denoted as π MCTS (+|st.))
For the sake of simplicity, we will not emphasize the differences between search policy x MCTS (-1s.,) in multi-agent settings and n MMCTS (-1s..) in single-agent settings apart from the mathematical expressions.
3

---

Published as a conference paper at ICLR 2024
3 CHALLENGES IN PLANNING-BASED MULTI-AGENT MODEL-BASED RL
Extending single-agent Planning- Waaseing-bssed MBRL methods to multi-agen agget environments is highly non- trivial. On one hand, existing model designs for single-agent algorithms do not account for the unique biases inherent to multi-agent environments, such the nearly-independent property of agents. This renders the direct employing a flattened model from single-agent MBRL algorithms typically falls short in supporting efficient learning within real-world multi-agent environments. On the other hand, multi-agent environments possess state-action spaces significantly more intricate than their single-agent counterparts, thereby exponentially escalating the search complexity within multi-agent settings. This, in turn, compels us to explore specialized search algorithms tailored to complex action spaces. Moreover, the form of the model, in tandem with its generalization ability, collectively constrains the form and efficiency of the search algorithm, and vice versa, making the design of the model and the design of the search algorithm highly interrelated.
In this section, we shall discuss the challenges encountered in the current design of planning-base MBRL algorithms, encompassing both model design and searching algorithm aspects. We will elucidate how our MAZero algorithm systematically addresses these two issues in the Section 4
3.1 MODEL DESIGN
Within the paradigm of centralized training with decentralized execution (CTDE), one straightfor- ward approach is to learn a joint model that enables agents do centralized planning within the joint policy space. However, given the large size of the state-action space in multi-agent environments, the direct application of a flattened model from single-agent MBRL algorithms tends to render the learning process inefficient (Figure (6) This inefficiency stems from the inadequacy of flattened model accommodating the unique biases inherent multi-agent environments.
In numerous real-world scenarios featuring multi-agent environments, agents typically engage in independent mecisinn-makkng for the majority of instances, with collaboration reserved for excep- tional circumstances. Furthermore, akin agents often exhibit homogeneous behavior (e.g.. focusing fire in the SMAC environment). For the former, a series of algorithms, including IPPO YYu et al., 2022). have demonstrated the efficacy of independent learning in multitude of MARL settings. Regarding the latter, previous research in multi ageet model-free approaches has underscored the huge success of parameter-sharing (Rashid et al. 2020 (Sunhhag al. (017) within MARL, which can be regarded as an exploitation of agents' homogenous biases. Hence, encoding these biases effectively within the design of the model becomes one main challenge in multi-agent MBRL.
3.2 PLANNING IN EXPONENTIAL SIZED ACTION SPACE
The action space in multi-agent environments grows exponentially with the number of agents, rendering it immensely challenging for vanilla MCTS algorithms to expand all actions. While Sampled MuZero, in prior single-agent MBRL work, attempted to address scenarios with large action spaces by utilizing action sampling, the size of the environments it tackled (e.g., Go, ~ 300 actions) still exhibits substantial disparity compared to typical multi-agent environments (e.g., SWSMAAAAA2mmsS3333 Om, ~ 10 actions).
Bandit
100
value
80
AWPO (Ours)
60
BC
0
200
400
600
OptimSteps
Figure 1: Bandit Experiment We compare the Behavior Cloning (BC) loss and our Advantage- Weighted Policy Optimization (AWPO) loss bandit with action space |A| = 100 and sampling time B 2. It is evident that AWPO converges much faster than BC, owing to the effective uti- lization of values.
In practical applications like MARL, due to the significantly fewer samples taken compared to the original action space size, the underestima- tion issue of Sampled MCTS becomes more pronounced, making it less than ideal in terms of searching efficiency. This motivates us to de- sign more optimistic search process.
4

---

Published as a conference paper at ICLR 2024
Moreover, the behavior cloning (BC) loss that Sampled MCTS algorithms adopt disregards the value information. This because the target policy w(1(st,) only encapsulates the relative magnitude of sampled action values while disregarding the information of the absolute action values. The issue is not particularly severe when dealing with smaller action space. However, in multi-agent environments, the disparity between the sampling time B and |A| becomes significant, making it impossible to overlook the repercussions of disregarding value information [Figure[]
4 MAZERO ALGORITHM
4.1 MODEL STRUCTURE
The MAZero model comprises 6 key functions: representation function ha for mapping the cur- rent observation history ozg, of agent to an individual latent state $i communication func- tion Eg which generates additional cooperative features ef, for each agent through the attention mechanism, dynamic function go tasked with deriving the subsequent local latent state si,411 based on the agent's s individual state 8i future action a/_x and communication feature ei, a re- ward prediction function forecasting the cooperative team reward ri from the global hidden state ,1 (s si,k) and joint action ar+* (a) kk.., x. value prediction function Va aimed at predicting value U/ for each global hidden state st,k. and a policy prediction function Pa
which given an individual state s/,x and generates the corresponding policy Pi, for agent i. The model equations are shown in Equation
Representation:
$1,0 ho(okz)
Communication:
e.,,...., eip = =c((s s a a o84
Dynamie:
. go at af k,,)
Reward Prediction:
Tt,k Ro(st , a - aFtx)
N
Value Prediction:
Ut Vo(s)
N
Policy Prediction:
PÊ, Po(st.u)
Notably, the representation function he. the dynamic function go. and the policy prediction func- tion Po all operate using local information, enabling distributed execution. Conversely, the other functions deal with value information and team cooperation, necessitating centralized training for effective learning.
Fin (pé.
Pinn
T1.1
(pL
Shared ed Individual Dynamic Network
Reward model
(ffa-4f))
$1,0
(s).0
a2
(s
Communication Network e
Centralized Dynamic
Block
(og,
Oc41
Figure 2: MAZero model structure Given the current observations of for each agent, the model separately maps them into local hidden states si,o using a shared representation network h. Value
prediction fr, is computed based on the global hidden state $t.0 while policy priors pi, are in- dividually calculated for each agent using their corresponding local hidden states. Agents use the communication network to access team information ét,o and generate next local hidden states sf the shared individual dynamic network g, subsequently deriving reward Ft, value Ut, and poli- cies pi.1. During the training stage, real future observations 0.+1 can be obtained generate the target for the next hidden state, denoted 814100-
5

---

Published as a conference paper at ICLR 2024
4.2 OPTIMISTIC SEARCH LAMBDA
Having discerned that MAZero has acquired deterministic world model, we devise a Optimistic Search Lambda (OS(A)) approach to better harness this characteristic of the model.
In previous endeavors, the selection stage of MCTS employed two metrics (value score and explo- ration bonus) gauge our interest in particular action. In this context, the value score utilized the mean estimate of values obtained from all simulations within that node's subtree. However, within the deterministic model, the mean estimate appears excessively conservative. Contemplating sce- consistently yields
nario where the tree degenerates into multi- armed bandit, if pulling deterministic outcome, there arises no necessity, akin to the Upper Confidence Bound (UCB) algo- rithm, to repeatedly sample the same arm and calculate its average. Similarly, within the tree-based version of the UCT (Upper Confidence Trees) algorithm, averaging values across all simulations in subtree may be substituted with more optimistic estimation when the environment is deterministic.
Hence, courtesy of the deterministic model, our focus narrows to managing model generalization errors rather than contending with errors introduced by environmental stochasticity. Built upon this conceptual foundation, we have devised methodology for calculating the value score that places a heightened emphasis on optimistic values.
Specifically, for each node we define:
dep(s) Ua(s) En ree + (u((ss) dep(s') dep(s))
(5)
kied
(s(s) Top (1 (1-p) p) values in Ud
(s) Σ Σ Xx/ [s))((s)
(E)
AR(s, r(s.a) (Dynamicc as) v(s)
(8)
where dep(u) denotes the depth of node in the MCTS tree, p. [0, 1] are hyperparameters, r, v, and Dynamic are all model predictions calculated according to Equation (4).
To offer brief elucidation this matter, Equation (6) is the of optimistic values depth d, the degree of optimism is controlled by the quantile parameter Since the model errors magnify with increasing depth in practice, Equation (7) calculates a weighted mean of all optimistic values in the subtree as the value estimation of s, where the weight of different estimations are discounted by factor À over the depth.
Finally, we use the optimistic advantage (Equation (8) to replace the value score in Equation for optimistic search, that is:
B As(s, a) P(s, a)
VE, N(s b)
arg max aET(s)CA
1 N(, c(s)
(9)
B
4.3 ADVANTAGE- LDVANTAAE-WWIIGHTED POLICY OPTIMIZATION
In order utilize the value information calculated by OS(A), propose Sowantage-Weigheee Policy Optimization (AWPO), novel policy loss function that incorporates the optimistic advantage (Equation 8 into the behavior cloning loss
In terms of outcome, AWPO loss weights the per-action behavior cloning loss by exp Ah(s,a)
( (AR(s, a
1, (10; -E
Σ (E)(s)
w(als) exp
log (als;;
(10)
II this section, the expectation operator is taken over the randomness of OS(A). For the sake of simplicity, will omit this in the formula.

---

Published as a conference paper at ICLR 2024
where 9 denotes parameters of the learned model, [[=]s;)) is the network predict policy to be im- proved, () is the search policy supported in action subset T(s), A&, is the optimistic advantage 0 is hyperparameter controlling the degree of optimism. Theoret-
derived from OS(A) and ically, AWPO can be regarded as a cross-entropy loss between vy and π [3] where 1)" is the non-
parametric solution of the following constrained optimization problem:
maximize
Ba [A] as)
(II)
s.t. KL KL (n(-ss)) MCTS (-1s)) S
To be more specific, by Lagrangian method, we have:
17" (als) (CTT((as)) exp (f(( -E w(als) exp
Af((s
(12)
a
Thereby, minimizing the KL divergence between 17" and x gives the form of AWPO loss:
arg min KL (-1s)]1
MCTS (als) A log lals(als;;) H (+1s))
arg min [(s)
a
arg min (0; w. AD)
where 2(8)= = MCTS (a(s) ( exp (A](,,a)
is a normalizing factor, H(y") stands for the entropy
of of" which a constant.
Equation 11 shows that AWPO aims to optimize the value improvement of π in proximity to MCTS which effectively combines the improved policy obtained from OS(A) with the optimistic value, thereby compensating for the shortcomings of BC loss and enhancing the efficiency of policy optimization.
Details of the derivation can be found in Appendix G
4.4 MODEL TRAINING
The MAZero model is unrolled and trained in an end-to-end schema similar to MuZero. Specifically, given trajectory sequence of length K + for observation Ora+K joint actions Gu.+K. rewards Wet+K, value targets žEt+K, policy targets MCTS and optimistic advantages Ag. the model is unrolled for K steps is shown in Figure 2 trained minimize the following loss:
ttt+
K
K
C.
Σ 1=1
L, + L. + 1.)
(13)
where 1, = ||rr, wr+kl and I, |1ut - Z are reward and value losses similar to MuZero,
| 814k.oll is the consistency loss akin to EfficientZero and I, is the AWPO policy loss
(Equation (0)) under the realm of multi-agent joint action space:
[A (A a) (ogT((())
1p
Σ
w(alst+k.o) exp
(14)
a
where OS(A) is performed under the hidden state 824k,0 directly represented from the actual future observation Octk, deriving corresponding action subset T($++k,)) search policy w(a)st+k.a) and optimistic advantage (8 a). æ (a)st,,; 0) = II Pa(a" (a(a"ls].) denotes the joint policy to be improved at the kth step's ' hidden state $t,. unrolling from dynamic function.
It is equivalent minimizing the KI. divergence of n7 and

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
5.1 STARCRAFT MULTI-AGENT CHALLENGE
Baselines We compare MAZero with both model-based and model-free baseline methods on StarCraft Multi-Agent Challenge (SMAC) environments. The model-based baseline is MAMBA (Egorov & Shpilman 2022), , a recently introduced multi-agent MARL algorithm based on DreamerV2 (Hafner et 2020) known for its state-of-the-ar sample efficiency in various SMAC tasks. The model-free MARL methods includes QMIX (Rashid et al. 22220, QPLEX (Wang et al,
2020a), RODE <Wang et al. 2020b), CDS (Li al. 2021). 1) and MAPPO YYu et al. 2022).
WiGenDDuri
OPLEX
GMX
2t.v, Gdap
16m, oe. 1ir
2
3
0.2
64
5.6
0.8
1.0
66
0.2
E4
6.6
0.
10
0.2
544
54
10
EwiStas
plary,y
8. 412
164
Figure 3: Comparisons against baselines in SMAC. Y axis denotes the win rate and axis denotes the number of steps taken in the environment. Each algorithm is executed with 10 random seeds.
Figure illustrates the results in the SMAC environments. Among the 23 available scenarios, have chosen 8 for presentation this paper. Specifically, we have selected four random scenarios categorized Easy tasks and 4 Hard tasks. It evident that MAZero outperforms all baseline algo- rithms across eight scenarios with a given number of steps taken in the environment. Notably, both MAZero and MAMBA, which are categorized as model-based methods, exhibit markedly superior sample efficiency in easy tasks when compared to model-free baselines. Furthermore, MAZero dis- plays a more stable training curve and smoother win rate during evaluation than MAMBA. In the realm of hard tasks, MAZero surpasses MAMBA in terms of overall performance, with a notewor- thy emphasis on the 2c_vs_64: scenario. In this particular scenario, MAZero attains higher win rate with a significantly reduced sample size when contrasted with other methods. This scenario involves a unique property, featuring only two agents and an expansive action space of up to 70 for each agent, in contrast to other scenarios where the agent's action space is generally fewer than 20. This distinctive characteristic enhances the role of planning MAZero components, similar to MuZero's 's outstanding performance in the domain of Go.
MAZero builds upon the MuZero framework, we perform end-to-end training directly using planning results from the replay buffer. Consequently, this approach circumvents the time overhead for data augmentation, as seen Dreamer-based methods. Figure g illustrates the superior per- formance of MAZero with respect to the temporal cost in SMAC environments when compared to MAMBA.
5.2 ABLATION
We perform several ablation experiments on the two proposed techniques: OS(A) and AWPO. The results with algorithms executed with three random seeds are reported in Figure 5 In particular, examine whether disabling OS(A), AWPO, both of them (i.e., using original Sampled MCTS and BC loss function) impairs final performance. Significantly, both techniques greatly impact final performance and learning efficiency.
8

---

Published as a conference paper at ICLR 2024
(uuZero(Orrt
MANBA
2m_ye_1c
soumeny so_rrnyybereeing
Winfato
WnFata
Winiatt
0.5
0.5
0.0
0
0.0
2.5
5.0
7.5
10.0
12.5
5
10
15
20
Timecosthours) 24_ va__a__ee 1ac
Timacost(hours)
Timacosthhuur)
2e_ va_ _o__6
_s__
1.0
0.6
WinRato
Whiiatt
Wensatt
0.4
0.5
0.0
0
10
15
20
10
15
20
25
10
15
20
25
30
Timesosithours)
Timecos((hours)
Timecosthours)
Figure 4: Comparisons against MBRL baselines in SMAC. The y-axis denotes the win rate, and the X-axis denotes the cumulative run time of algorithms in the same platform.
Figure 5: Ablation study on proposed approaches for planning.
In our ablation of the MAZero network structure (Figure 6). we have discerned the substantial impact of two components, "communication" and "sharing", on algorithmic performance. In stark contrast, the flattened model employed in single-agent MBRL, due to its failure to encapsulate the unique biases of multi-agent environments, can only learn victorious strategies in the most rudimentary of scenarios, such as the 2m vs lz map.
2m, MR
2e, Mt 64rg
Sin
16rr 1tr
10
5.2
8.5
US
0.4
3.0
0.2
54
656
2.
1.0
3.5
0.2
54
566
0.3
1.0
8.
0.2
54
5.6
0.8
1.0
8.0
0.2
544
0.5
1.0
Figure 6: Ablation study on network structure.
CONCLUSION
In this paper, we introduce a model-based multi-agent algorithm, MAZero, which utilizes the CTDE framework and MCTS planning. This approach boasts superior sample efficiency compared to state- of-the-art model-free methods and provides comparable or better performance than existing model- based methods in terms of both sample and computational efficiency. We also develop two novel approaches, OS(A) and AWPO, to improve search efficiency in vast action spaces based on sampled MCTS. In the future, we aim to address this issue through reducing the dimensionality of the action space in search, such action representation.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY
We ensure the reproducibility of our research by providing comprchensive information and resources that allow others replicate findings. All experimental settings in this paper available in Appendix B including details on environmental setup, network structure, training procedures, hy- perparameters, and more. The source code utilized in our experiments along with clear instructions on how to reproduce our results will be made available upon finalization of the camera-ready ver- sion. The benchmark employed in this investigation is either publicly accessible or can be obtained by contacting the appropriate data providers. Additionally, detailed derivations for our theoretical claims can be found in Appendix G We are dedicated to addressing any concerns or inquíries re- lated to reproducibility and are open to collaborating with others to further validate and verify our findings.
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Innovation 2030- Major Project (Grant No. 2022ZD020880)).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- lion processing systems, 34:29304-29935 293200, 2021.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Plan- ning in stochastic environments with a learned model. In International Conference on Learning Representations, 2021.
Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pp. 314-315, 2019.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Eugenio Bargiacchi, Timothy Verstracten, and Diederik M Roijers. Cooperative prioritized sweep- ing. In AAMAS, pp. 160-168, 2021.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-informatin games. Advances in Neural Information Processing Systems, 13:17757-1706, 2020.
Dane Coreill, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforcement learning with variational state tabulation. In International Conference Machine Learning, pp- 1049-1058. PMLR, 2018
Ivo Danihelka, Arthur Guez, Julian Schritwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2021.
Vladimir Egorov and Aleksei Shpilman. Scalable multi-agent model-based reinforcement learning. arXiv preprint 220iv:2205. /5023, 2022.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: Intelligence Review, pp. 1-49, 2022.
survey, Artificial
David Ha and Jürgen Schmidhuber. World models. arXiv preprint
2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 2019.
10

---

Published as a conference paper at ICLR 2024
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 10044,, 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0499 04955, 2022.
Jinke He, Thomas M Moerland, and Frans A Olichock. What model does muzero learn? arXiv preprint arXiv:2306.000 00840, 2023.
Pablo Hermandez-Lear Bilal Kartal, and Matthew E. Taylor. very condensed survey and critique of multiagent deep reinforcement learning. In Proceedings of the 19th international conference on autonomous agents and multiagent systems, pp. 2146-2148, 2020.
Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy opti- mization. In International conference machine learning, pp. 4214 4226. PMLR, 2021.
Lukas Hewing, Kim P Wabersich, Marcel Menner, and Melanie N Zeilinger. Learning- based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3:269-296, 2020.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pp. 1476-4486 PMLR, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 69800 2014.
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
Karol Kurach, Anton Raichuk, Piotr Stanezyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4501-4510, 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 1802. 10592, 2018.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang, Cel- ebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991-4002. 2021.
Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7211-7218, 2020.
Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6252-6259, IEEE, 2018.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitee environments. Advances in neural informa- fion processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2024
Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 7301- 7312. PMLR, 2021.
Yixuan Mei, Jiaxuan Gao, Weirui Ye, Shaohuai Liu, Yang Gao, and Yi Wu. Speedyzero: Mas- tering atari with limited data and time. In The Eleventh International Conference on Learning Representations, 2022.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine. Neural network dynamics for model- deep reinforcement learning with model-free fine-tuning. In 2018 IEEE interna- tional conference on robotics and automation (ICRA), pp. 7559-7566. IEEE, 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets, 2021.
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, (009))3826-3833 2020.
Frans A Olichock, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Fierre-Alexande Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gra- dients. Advances in Neural Information Processing Systems, 34:12208-12221, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farguhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234 7284, 2020.
Heechang Ryu, Hayong Shin, and Jinkyoo Park. Multi-agent actor-critic with hierarchical graph attention network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7236-7243, 2020.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 0rXiv:1511.059555 2015.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604 609, 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through play. Science, 162 1144, 2018.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional conference on machine learning, pp. 5887-5896. PMLR, 2019.
Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference learning theory, pp. 2898-2933. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2024
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. scoopsscconootttion networks for cooperative multi-agent learning. arXiv preprint arXiv: 1e..111066..969 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Gerald Tesauro.
Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-219 1994.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Jianhao Wang. Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020a.
Tingwu Wang and Jimmy Ba. Exploring model- planning with policy networks. arXiv preprint arXiv: 1906: 08649, 2019.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint 1907:000002 2019.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020b.
Daniël Willemsen, Mario Coppola, and Guido CHE de Croon. Mambpo: Sample-efficient multi- robot reinforcement learning using learned world models. In 202/ IEEE/RSJ International Con-
ference on Intelligent Robots and Systems (IROS), pp. 5635-5640. IEEE, 2021.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint ar arXiv:2002..333 03939, 2020.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo. et al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2020.
Jianing Ye, Chenghao Li, Jianhao Wang, and Chongjie Zhang- Towards global optimality in coop- crative marl with the transformation and distillation framework, 2023.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances Neural Information Processing Systems, 35:2461 24624, 2022.
Kaiqing Zhang, Sham Kakade, Tamer Basar, and Lin Yang. Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. Advances in Neural Information Processing Systems, 33:1 1178, 2020.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang. David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, et al. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776, 2020.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORKS
Dec-POMDP In this work, we focus on the fully cooperative multi-agent systems that can be for- malized as Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Olichoek] et al. 2016), which are defined by (N, W, A T, U,N.NN O ()) where N is the number of agents, S is the global state space, T a global transition function, a shared reward function and A 22', O are the action space, observation space and observing function for agent i. Given state 8t at timestep 1, agent can only acquire local observation of = =O*(s) and choose action a E A' according its policy x' based on local observation history d's.- The environment then shifts the next state 1 T(sss, az) and returns a scalar reward ur Ust, at). The objec- tive for all agents is to learn joint policy т that maximizes the expectation of discounted return ((T) En Ee= 7 u la} T' (-10g)). IN....
Combining reinforcement learning and planning algorithms The integration of reinforcement learning and planning within common paradigm has yielded superhuman performance in diverse domains (Tesauro 1994, Silver et al. 2017, Anthony et al. 2017, Silver et al. 2018 Schrittwieser jet al. 2020 Brown et al. 2020). In this approach, RL acquires knowledge through the learning of value and policy networks, which in turn guide the planning process. Simultancously, planning generates expert targets that facilitate RL training. For example, MuZero-based algorithms (Schrit- twieser et al., 22200 Hubert et al.) 2021 Antonoglou et af., [202] Ye et al. 2021] Mei et al. (002) combine Monte-Card Carlo Tree Search (MCTS), TD-MPC (Hansen et al.. (022)) integrates Model Pre-
dictive Control (MPC), and ReBeL (Brown et al. 2020) incorporate a fusion of Counterfactual
Regret Minimization (CFR). Nevertheless, the prevailing approach among these algorithms typi- cally involves treating the planning results as targets and subsequently employing behavior cloning to enable the RL network to emulate these targets. Muesli (Hessel al., (221)) first combines regu- larized policy optimization with model learning as an auxiliary loss within the MuZero framework. However, it focuses solely on the policy gradient loss while neglecting the potential advantages of planning. Consequently, it attains at most performance level equivalent to that of MuZero. To the best of our knowledge, we are the first to combined policy gradient and MCTS planning to accurate policy learning in model-based reinforcement learning.
Variants of MuZero-based algorithms While the original MuZero algorithm (Schrittwieser al. 2020) has achieved superhuman performance levels in Go and Atari, it is important to note that harbors several limitations. To address and overcome these limitations, subsequent advancements have been made in the field. EfficientZero (Ye et al., 2021). for instance, introduces self-supervised consistency loss, value prefix prediction. and off- off-policy correction mechanisms expedite and sta- bilize the training process. Sampled MuZero (Hubert et al. (022] extends its capabilities to complex action spaces through the incorporation of sample-based policy iteration. Gumbel MuZero (Dani- helka et al. (021) leverages the Gumbel-Top-k trick and modifies the planning process using Se- quential Halving, thereby reducing the demands of MCTS simulations. SpeedyZero (Mei et al.] 2022) integrates a distributed RL framework to facilitate parallel training, further enhancing train- ing efficiency. He et al.] (2023) demonstrate that the model acquired by MuZero typically lacks accuracy when employed for policy evaluation, rendering it ineffective in generalizing to assess previously unseen policies.
Multi-agent reinforcement learning The typical approach for MARL in cooperative settings in- volves centralized training and decentralized execution (CTDB). During the training phase, this ap- proach leverages global or communication information, while during the execution phase, it restricts itself to the observation information relevant to the current agent. This paradigm encompasses both value-based (Sunchag et al. Rashid
Rashid et al.) (2020) Son et al.] 2019] Yang et al. 20,00 Wang et al.) 2020a) and policy-based (Lowee et 2017] 7] Liu al.. 2020 Peng et al.) [021] et al. [2200] Ye et al. (023) MARL methods within the context of model-free scenarios. Model-basee MARL algo-
rithms remains relatively underexplored with only a few methods as follows: MAMBPO (Willemsen)
et al. 2 pioneers the fusion of the CTDE framework with Dyan-style model-based techniques, emphasizing the utilization of generated data closely resembling real-world data. CPS (Bargiac- chi et al. 2021) introduces dynamic and reward models determine data priorities based on the MAMBPO approach. Tesseract (Mahajan et al. 202T) dissects states and actions into low-rank
tensors and evaluates the Q function using Dynamic Programming (DP) within an approximate
14

---

Published as a conference paper at ICLR 2024
vironment model. MAMBA (Egorov & Shpilman 2022) incorporates the word model proposed in DreamerV2 (Hafner et al. 2020 and introduces an attention mechanism to facilitate communica- tion. This integration leads to noteworthy performance improvements and substantial enhancement in sample efficiency when compared to previous model-free methods. To the best of our knowledge, are the first to expand the MuZero framework and incorporate MCTS planning into the context of model-based MARL settings.
B IMPLEMENTATION DETAILS
B. NETWORK STRUCTREE
For the SMAC scenarios where the input observations are dimensional vectors (as opposed to 2 dimensional images for board games or Atari used by MuZero), we use variation of the MuZero model architecture in which all convolutions consists of 6 modules: representation function, communication function, dynamic function, reward prediction function, value prediction function, and policy prediction function, which all repre- sented as neural networks. All the modules except the communication block are implemented as Multi-Layer Perception (MLP) networks, where each liner layer MLP is followed by Rectified Linear Unit (ReLU) actívation and a Layer Normalisation (LN) layer. Specifically, we use hid- den state size of 128 for all SMAC scenarios, and the hidden layers for each MLP module are as follows: Representation Network [128, 128], Dynamic Network [128, 128]. Reward Network (32). Value Network [32] and Policy Network = [32]. We use Transformer architecture (Vaswani et al., 2017) with three stacked layers for encoding state-action pairs. These encodings are then used by the agents to process local dynamic and make predictions. We use a dropout technique with probability p = 0.1 to prevent the model from over-fitting and use positional encoding to distinguish agents in homogeneous settings.
replaced by fully connected layers. The model
For the representation network, stack the last four local observations together as input for each agent to deal with partial observability. Additionally, the concatenated observations are processed by an extra LN to normalize observed features before representation.
The dynamic function first concatenates the current local state, the individual action, and the com- munication encoding based current state-action pairs as input features. To alleviate the problem that gradients tend to zero during the continuous unroll of the model, the dynamic function employs residual connection between the next hidden state and the current hidden state.
For value and reward prediction, we follow in scaling targets using an invertible transform h(x) sign(z)
V I++ x 0.001 x and use the categorical representation introduced in MuZero (Schrit- twieser et al. 2020p. We use ten bins for both the value and the reward predictions, with the predic- tions being able to represent values between [-5, 5]. We used n- step bootstrapping with n = 5 and discount of 0.99.
B.2 TRAINING DETAILS
MAZero employs a similar pipeline to EfficientZero (Ye et al.. 2021) while asynchronizing the parallel stages of data collection (Self-play workers), reanalysis (Reanalyze workers), and training (Main Thread) to maintain the reproducibility the same random seed.
We use the Adam optimizer (Kingma & Ba 2014) for training, with batch size of 256 and constant learning rate of 10-4 Samples are drawn from the replay buffer according to prioritized replay (Schaul et al. 2015) using the same priority and hyperparameters in MuZero.
In practice, re-execute OS(A) using delayed target model ê in the reanalysis stage reduce off off-policy error. Consequently, the AWPO loss function actually used is the following equation:
ÀR (8(
lp
Σ (ET()
@@((8r+k,)) exp
log ग (ā)st,n;0)
For other details, we provide hyper-parameters in Table
15

---

Published as a conference paper at ICLR 2024
Parameter
Setting
Observations stacked
4
Discount factor
0.99
Minibatch size
256
Optimizer
Adam
Optimizer: learning rate
10
Optimizer: RMSprop epsilon
5
Optimizer: weight decay
0
Max gradient norm
5
Priority exponent(c,)
0.6
Priority correction(ca)
0.4 -
Evaluation episodes
32
Min replay size for sampling
300
Target network updating interval
200
Unroll steps
5
TD steps(n)
5
Number of MCTS sampled actions(K)
10
Number of MCTS simulations(LV (N)
100
Quantile in MCTS value estimation(p)
0.75
Decay lambda in MCTS value estimation(A)
0.8
Exponential factor in Weighted-AAvvantageaa))
3
Table 1: Hyper-parameters for MAZero in SMAC environments
B.3 DETAILS BASELINE ALGORIHHS IMPLEMENTATION
MAMBA Egorrv Shpilman 2022) executed based on open-source implementation:
https: gi thub. com/ jor ai labs mamba with the hyper- paper-parameters in Table 2
Parameter
Setting
Batch size
256
GAE X
0.95
Entropy coefficient
0.001
Entropy annealing
0.99998
Number of updates
4
Epochs per update
5
Update clipping parameter
0.2
Actor Learning rate
5 X 10
Critic Learning rate
x 10-4
Discount factor
0.99
Model Learning rate
2 x 10
Number of epochs
60
Number of sampled rollouts
40
Sequence length
20
Rollout horizon H
15
Buffer size
2.5 x 105
Number of categoricals
32
Number of classes
32
KL balancing entropy weight
0.2
KL balancing cross entropy weight
0.8
Gradient clipping
100
Trajectories between updates
1
Hidden size
256
Table 2: Hyper Sypeee-prrmmeters for MAMBA in SMAC environments
16

---

Published as a conference paper at ICLR 2024
QMIX (Rashid al. 2020) is executed based on the open-source implementation: qi thub. com/ wwwwiriiii...o pymarl with the haper-parameters in Table 3
https://
Parameter
Setting
Batch size
32
Buffer size
5000
Discount factor
0.99
Actor Learning rate
5 X 10
Critic Learning rate
10-1
Optimizer
RMSProp
RMSProp a
0.99
RMSProp
10-5
Gradient clipping
10
e-greedy
1.0 0.05
6 aannealing time
50000
Table Syyper-parameten for QMIX in SMAC environments
QPLEX (Wang al. 2020a is executed based the open-source implementation:
https://
gi vithub. com w jh7 20/ QP OPEEX with the hyper-parameters in Table 4
Parameter
Setting
Batch size
32 5000
Buffer size
Discount factor
0.99
Actor Learning rate
5 x 10
Critic Learning rate
10
Optimizer
RMSProp
RMSProp a
0.99
RMSProp
10-5
Gradient clipping
10
é-greedy c aannealing time
1.0 - 0.05
50000
Table 4: Hyper-parameters for QPLEX in SMAC environments
RODE WWng et al. 2020b is executed based on the open-source implementation:
https://
qi github.
RRODE with the hyper- parameters in Table 5
Parameter
Setting
Batch size
32 5000
Buffer size
Discount factor
0.99
Actor Learning rate
5
10-4
Critic Learning rate
10
Optimizer
RMSProp
RMSProp &
0.99
RMSProp e
10-5
Gradient clipping
10 1.0 - 0.05
e-greedy annealing time number of clusters
50K 500K
2 ~ 5
Table 5: Hyper- parameerrs for RODE in SMAC environments
17

---

Published as a conference paper at ICLR 2024
CDS Li al. 2021
executed based the open-source implementation: with the hyper- prper-parameters in Table 6
https://github:
com/ 1ichl CDS
Parameter
Setting
Batch size
32
Buffer size
5000
Discount factor
0.99
Actor Learning rate
5 10 10-4
Critic Learning rate
5 x 10 -4
Optimizer
RMSProp
RMSProp a
0.99
RMSProp e
10-5
Gradient clipping
10
<-greedy
1.0 - 0.05
0.05
31
0.5
82
0.5
A
0.1
attention regulation coefficient
10-3
Table 6: Hyper-parameters for CDS in SMAC environments
MAPPO Yu
2022) is executed based on the open-source implementation: https://
gi thub.. com / mar www.hhaarkeeeeeee..eeaaa on policcy with the hyper-parameters in Table
Parameter
Setting
Recurrent data chunk length
10
Gradient clipping
10
GAE
0.95
Discount factor
0.99
Value loss
huber loss
Huber delta
10.0
Batch size
num envs x buffer length x num agents batch size mini-batch
Mini batch size
Optimizer
Adam
Optimizer: learning rate
5 x 10-4
Optimizer: RMSprop epsilon
10-5
Optimizer: weight decay
0
Table parrm--ars for MAPPO in SMAC environments
B.4 DETAILS OF THE BANDIT EXPERIMENT
The bandit experiment showed Figure compares the behavior cloning loss and the AWPO loss on a 100-armed bandit, with action values 0, ... .99 and sampling time B = 2. The policy π BCC and o AWPO are parameterized by Sofumax policy, which means
exp(B
n BC (a; @CC)
E, (Ot))
exp exp( (OAWPO)
AWPO π
(a; (AWPO)-
Σn exp exp(0,,PO)
where JBC, .OAPO
1100 randomly initialized and identical in the beginning.
18

---

Published as a conference paper at ICLR 2024
To exclude the influence of stochasticity in the search algorithm and facilitate precise and fair comparison of the differences in loss functions, we made three targeted adjustments.
1. Let the subset of sampled action be T, we denote the target policy w(a) = I(a = arg maxter value(b)).
2. We calculate the expectation of the loss over the randomness of all possible 7's.
3. We normalize the advantage in AWPO into 0-mean-1-std and choose a = 1.
Formally, we have
(0) Σ log na (a)tt(a)
(BC Σ log no (a) (u) exp (adv(a))
where t(a)= = EEiza Tb( K (Eiza me(b)) is the expectation of w(a), the over line stands for stop-gradi ent. adv(a) (a 09
is the normalized advantage.
C STANDARDISED PERFORMANCE EVALUATION PROTOCOL
We report our experiments based on the standardised performance evaluation protocol
Agarwal
ct al. 202I).
Median
IQM
Mean
Optimality Gap
MAZero(Ours)
QMIX
RODE
OPLEX MAPPO MAMBA
Normalized Absolute Retur
Figure 7: Aggregate metrics on the SMAC benchmark with 95% stratified bootstrap confidence intervals. Higher median, interquartile mean (IQM), and mean, but lower optimality gap indicate better performance.
Algorithm Y
Algorithm x
MAZero(Ours)
QMIX
MAZero(Ours)
MAZero(Ours)
RODE
MAZero(Ours)
QPLEX
MAZero(Ours)
MAPPO
MAZero(Ours)
MAMBA
0.75
0.80
0.85
0.90
0.95
PIX
Figure 8: Probabilities of improvement, i.e. how likely it for MAZero outperform baselines the SMAC benchmark.
D MORE ABLATIONS
In the experiment section, we list some ablation studies prove the effectiveness of each component in MAZero. In this section, we will display more results for the ablation study about hyperparame- ters.
19

---

Published as a conference paper at ICLR 2024
First, perform an ablation study the training stage optimizers, contrasting SGD and Adam. The SGD optimizer prevalent in most MuZero- basro- algorithms (Schrittwieser et al. 2020 Ye
2021 while Adam is frequently used in MARL environments (Rashid et al. 2020 Wang et al.
et al. 2020a Yu et al. 2022). Figure / indicates that Adam demonstrates superior performance and consistent stability compared SGD.
Azom
800
5m
5mL vs_ M5_6m
0.30
0.8
0.25
0.6
0.4
0.10
0.59
0.00
0.0
0.5
1.5 EnwSteps
2.0
3.0
0.0
1.0
1.5
2.5
8.0
Em/Steps
Figure 9: Ablation optimizer
Sm_ys_0m
0.8
08
0.6
0.4
0.2
6.0
0.6
EnvtSwps
fed
EnvStepe
1e6
Figure 10: Ablation on sampled scale
p-0.75 75.75
p5
p-0.85
25
J-0.8
A-0.7
1-1.0
0.8
0.8
Welolee
WeRalee
0.4
0.2
0.0
0.0
EnvStaps
EnvSlaps
(a) Ablation
(b) Ablation À
Figure 11: Ablation for parameters OS(X)
In addition, we perform an ablation study on the sampled scale (the action sampling times K and simulation numbers N in MCTS), which has been shown be essential for final performance in Sampled MuZero (Hubert et al. 202T). Given the limitations of time and computational resources,
only compare two cases: K 5. N 50 and K 10, N 100 for easy map 8m and hard
20

---

Published as a conference paper at ICLR 2024
map Sm.vs.óm. Figure 10 reveals that our method yields better performance with larger scale of samples.
We also test the impact of and À in our Optimistic Search Lambda (OS(A)) algorithm map
8m. We test p = 0.65, 0.75, 0.85, 0.95 by fixing = 0.8, and test À = 0.7, 0.8, 0.9, 1.0 by fixing 0.75 for MAZero. Figure 11 shows that the optimistic approach stably improves the sample efficiency, and the term is useful when dealing with the model error.
We perform ablation study about MCTS planning in the evaluation stage. The MAZero algorithm is designed under the CTDE framework, which means the global reward allows the agents to learn and optimize their policies collectively during centralized training. The predicted global reward is used in MCTS planning to search for a better policy based on the network prior. Table 8 shows that agents maintain comparable performance without MCTS planning during evaluation, i.c., directly using the final model and local observations without global reward, communication and MCTS
planning.
Map
Env steps
w MCTS
w/o MCTS
performance ratio
3m
50k
0.985 = 0.015
0.936 0.107
95.0 = 10.8%
2m_vs_lz
50k
1.0 0.0
1.0 0.0
100 = 0.0%
so_many bbanelig
50k
0.959 0.023
0.938 + 0,045
97.8 + 4.7%
2s_
100k
0.623 0.185
65.7 19.5%
2c 64zg
400k
0,893 0.114
0.768 0.182
= 20.4%
5m vs 6m
1M
0.875 + 0.031
0.821 0.165
93.8 = 18.9%
8m_vs_9 _m_
1M
0.906 0.092
0.855 0.127
94.4 14.0%
10m_vs.1
IM
0.922 + 0.064
0.863 = 0.023
93.6 + 2.5%
average performance
100%
90.1 = 11.3%
Table 8: Ablation for using MCTS during evaluation in SMAC environments
E EXPERIMENTS ON SINGLE-AGENT ENVIRONMENTS
We have considered demonstrating the effectiveness OS(A) and AWPO in aagle- decision problems with large action spaces. This might help establish the general applicability of these tech- niques beyond the multi-agent SMAC benchmark.
We choose the classical LunarLander environment as the single-agent benchmark, but discretize the 2-dimensional continuous action space into 400 discrete actions. Additionally, we select the Walker2D scenario in MuloCo environment and discretize each dimension of continuous action space into 7 discrete actions, i.e., 67 As 280, 000 legal actions. Tables 9 a0 i0lllustrats the results where both techniques greatly impact learning efficiency and final performance.
Environment Env steps
MAZero
w/o OS(A) and AWPO
250k
184.6+22.8
104.0#87.5 227.7456.3
LunarL
500k
259.8+12.9
1M
276.9+2.9
274.1+3.5
Table 9: Ablation for using OS(A) and AWPO LunarLander environments
Environment
Eny steps
MAZero
w/o OS(A) and AWPO
TD3
SAC
300k
3424 246
2302 472
1101 + 386
1989 500
Walker2D
500k
4507 411
3859 + 424
2878 # 343
3381 + 329
IM
5189 382
4266 509
3946 + 292
4314 256
Table 10: Ablation for using OS(A) and AWPO in Walker2I) environments
21

---

Published as a conference paper at ICLR 2024
F EXPERIMENTS ON OTHER MULTI-AGENT ENVIRONMENTS
is beneficial to validate the performance of MAZero in other tasks beyond the SMAC bench- mark. We further benchmark MAZero on Google Research Footbal((RFF) (Kurach et al. 2020). academy-peee pasttayy-ppssaae. and_shoot wotootttth__ha with scenario and compare our methods with several model-free baseline algorithms. Table 11 shows that our method outperforms baselines terms of sample efficiency.
Environment
Env steps
MAZero
CDS
RODE
QMIX
500k
0.123 0.089
0.069 0.041
0
academy-pass.ane
1M
0.214 = 0.072
0.148 0.117
0
shoot_with.keperr
2M
0.619 0.114
0.426 0.083
0.290 0.104
Table 11: Comparisons against baselines in GRF.
G DERIVATION OF AWPO Loss
To prove that AWPO loss (Equation 10 is essentially solving the corresponding constrained op- timization problem (Equation we only need to prove the closed form of Equation is Equation (12).
We follow the derivation in Nair et al. 2021) Note that the following optimization problem
17" arg a [A(s, a)]
KL (n(: s)][[( s))
(15)
/ n1aa s)da = 1.
has Lagrangian
[(1, 0x) -Rann(()s)) [A(s,a)]
+ x(ee DKL (n(( s)][ (+))) s))
(16)
+ a 1 nia s)da
Applying KKT condition, we have
ac
A(s, a) + Alog s) - Alogn(as () + X - a =
(17)
On
Solving the above equation gives
7(s) n (a 8) exp A(s,aa)
17 (a 8)
(18)
where 7(x) a normalizing factor.
To make the KKT condition hold, can let 17 and use the LICQ (Linear independence straint qualification) condition.
Plugging the original problem (Equation |1 into Equation
15
completes our proof.
22


Reviews:
[{"review_id": "th69FZTPq1", "reviewer": "Reviewer_iouX", "summary": "The paper considers MARL in a game environment, using a model-based approach. The heart of the paper is to extend MuZero, a model-based single agent RL algorithm that incorporates planning, to the multi-agent case in a Dec-POMDP setup.  The new algorithm is called mullti-agent Zero (MAZero). The extension of MuZero is achieved by adding and modifying the 3 functions of MuZero to create 6 functions that incorporate communications and expands how predictions are made.  The paper then focuses on efficient Monte Carlo tree search (MCTS) so that the planning complexity is reasonable. Numerical studies are carried out in Starcraft multi-agent challenge (SMAC) environments.  The experiments compare to model-free and model-based methods as baseline.  The results generally show improved learning efficiency for CDTE execution.", "questions": "It seems that the method assumes that the agents have access to the global reward at each step during inference?  \n\nIn Figure 6, please clarify the difference between communication and sharing.  \n\nSection 3.1, the discussion about the degree to which a homogeneous solution is a good one is interesting and certainly depends on the particular scenario, but it isn’t clear how the MAZero fits into this.\n\nCould you say more about the Shared Individual Dynamic Network g ?  Perhaps it is just the terminology but the idea seems confusing.  \n\nSection B.1, could you say more about how to “use positional encoding to distinguish agents in homogeneous settings”?\n\nAppendix C, why do you think Adam is more effective than stochastic gradient descent?  Isn't this well known?\n\nSome small items:  Perhaps “ingenious” and “audacious” are terms better left for the reader to decide for themselves?  The paper refers to “real world” cases but ultimately this is about a game environment.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is a logical extension of MuZero to the multi-agent case, and builds on those ideas to create six neural network functions that underly the model.  The writing is clear and the various cost functions and parameters are well laid out.  T\n\nThe experiments push the MARL problem in terms of action space complexity, and show that the MCTS method is effective in providing good performance with reduced search.  The primary contribution of the paper is to use prediction along with the reduced search in the multi-agent setting. \n\nIt appears that the method is generally applicable to Dec-POMDP problems under the assumptions in the paper, in particular the CDTE assumption. \n\nThe advantage-weighted policy optimization (AWPO) is an interesting way to balance cloning loss and the reduced tree search (the optimistic value).", "weaknesses": "Ultimately, the method gains in learning efficiency for the game studied, which is an important contribution, although it isn’t clear that there is any performance gain compared to other CDTE methods. \n\nThe method seems to require global reward information at each agent during execution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7lf5rN7hW6", "reviewer": "Reviewer_6eaK", "summary": "The paper presents MAZero, a model-based multi-agent reinforcement learning (MARL) algorithm that combines a centralized model with Monte Carlo Tree Search (MCTS) for policy search. The authors propose an ingenious network structure to facilitate distributed execution and parameter sharing. They introduce two novel techniques, Optimistic Search Lambda (OS(λ)) and Advantage-Weighted Policy Optimization (AWPO), to enhance search efficiency in deterministic environments with sizable action spaces. Extensive experiments on the SMAC benchmark demonstrate that MAZero outperforms model-free approaches in terms of sample efficiency and provides comparable or better performance than existing model-based methods in terms of both sample and computational efficiency.", "questions": "* How does MAZero perform in stochastic environments compared to deterministic ones?\n* Can the proposed OS(λ) and AWPO techniques be applied to other model-based MARL algorithms?\n* Are there any potential drawbacks or limitations of the proposed network structure that the authors have not discussed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* MAZero is the first empirically effective approach that extends the MuZero paradigm into multi-agent cooperative environments.\n* The proposed OS(λ) and AWPO techniques improve search efficiency in large action spaces.\n* Extensive experiments on the SMAC benchmark demonstrate the effectiveness of MAZero in terms of sample efficiency and performance.", "weaknesses": "* The paper focuses on deterministic environments, and it is unclear how well MAZero would perform in stochastic environments.\n* The proposed techniques may not be applicable to all types of multi-agent environments, and further research is needed to generalize the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0eQMhcYIYq", "reviewer": "Reviewer_MHMy", "summary": "The authors introduce a novel algorithm, MAZero, based on the model-based ideas in MuZero, for the multi agent setting in order to improve problems with sample efficiency. Because the model based approach requires roll-outs to select policies, an efficient method of tree search is required, and so Optimistic Search Lambda (a method which weights optimistic values more highly) and Advantage-Weighted Policy Optimization (which uses a novel policy loss function based on the optimistic search lambda values).\n\nThe authors then compare this model based MARL algorithm to a number of others on the Starcraft Multi-Agent Challenge followed by a number of ablations of the different methods.", "questions": "The questions all relate to the weaknesses described in the previous section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is in general well-written, and clear, with thorough appendices for the details. \n\nThe results given show that this model-based approach is not only more sample efficient than model-free approaches in the MARL setting but also, importantly, tractable when the Optimistic Search Lambda Algorithm and Weighted Policy Optimization are used within the tree-search. While these results are not surprising, such an approach has not been taken before and so there is definitely originality and significance to these results within the domain explored.", "weaknesses": "The major issue comes down to the single, very specific domain that this has been tested on. While the results are, as stated above, impressive, they are only impressive in this single domain, and it would not seem difficult to show that they are just as significant in other domains with different types of action and state spaces (continuous, discrete, visual, tabular). \n\nIn addition, I believe that it should become standard within the community to utilise the evaluation protocol of Gorsanne et al (https://arxiv.org/abs/2209.10485) in the Multi-agent setting. These standard practices have not been followed and I believe weaken what could be a strong case for the effectiveness of these algorithms.\n\nNo discussion is given to hyper parameter tuning, or the choice of hyperparameters for the baselines that MAZero is being compared against.\n\nOn a stylistic note, the paper has a lot of grammatical typos and needs to be gone over thoroughly. An LLM should be able to pick up all of these mistakes.\n\nI would not use the word \"ingenious\" in the abstract to describe your own algorithm, or, as later used \"audacious\".\nCTDE is not spelled out the first time it is mentioned.\nThe different loss terms in equation 2 are not spelled out.\nIs Figure 1 a single seed? It looks remarkably smooth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vq3jbnESFy", "reviewer": "Reviewer_efig", "summary": "This work proposes a model-based multi-agent RL approach similar to MuZero, performing MCTS over the joint action space of all agents as imagined by learned models of the dynamics and reward. The authors propose to use optimism (Optimistic Search Lambda in conjunction with Advantage-Weighted Policy Optimization) to improve the performance over the behavioral cloning loss used in Sampled MuZero. Experiments on the SMAC benchmark compare the performance of this method with some previously proposed model-based and model-free MARL methods.", "questions": "Is MCTS being performed at evaluation time, or is only the learned policy used? If search is being performed at evaluation time, there should be a quantification of the search overhead compared with baselines.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates better or similar performance compared with several baselines. The paper is easy to follow (at least for someone familiar with MuZero and related works). Regarding novelty, I have not found any previous application of MuZero to multi-agent setting.", "weaknesses": "There are several major weaknesses of the paper.\n\n1. The empirical validation of the method is very weak.\n    1. Despite the authors proposing the method to tackle large action spaces and mentioning the 27m_vs_30m settings as a motivating example in Section 3.2, there is no experiment in these large action space settings.\n    2. There’s a lack of experimental validation beyond the SMAC benchmark, and the same benchmarks settings are used across all results and ablations. For example, Google Research Football and Multi-agent MuJoCo would be candidates for other tasks.\n    3. The SMAC benchmark is outdated and should be replaced by the SMACv2 benchmark, as many tasks have been shown to be trivially solvable due to the lack of stochasticity in the SMAC benchmark.\n    4. The baselines used are relatively few compared to other published MARL works. There should be comparison with other recent methods like MBVD, RODE, CDS, etc.\n    5. It’s not clear how many seeds the work used for the main Table 3. The work mentions 3 seeds for followup ablations, but this is too few to ensure that the performance is not due to luck. The results should be reported across at least 10 seeds.\n2. To my knowledge, the search-based component of methods like MuZero are very reduced and not significantly useful for many non-board game tasks. For example, while MuZero utilizes a large search depth for Go, it uses a very small search depth for Atari, which raises the questions of how much performance gain the search component contributes. This work mentions that it uses 5 unroll steps, which is still very small. For MCTS-based works like this one, there should be empirical investigations with respect to the number of unroll steps at both training and test time to see whether search is offering any benefits.\n4. As this work mostly treats multi-agent decision problem as a single-agent RL with a very large action space, the authors should demonstrate $OS(\\lambda)$ and AWPO more extensively on (single-agent) decision problems with large action space; indeed, the author could even solely consider these single-agent settings rather than bringing in the multi-agent SMAC benchmark at all. The single-agent bandit experiment in Figure 1 is too simplistic and insufficient to demonstrate the authors’ point. Also, in very large action spaces, MCTS with a limited budget (e.g. $N = 100$ MCTS simulations as considered in this work) seems very sparse and unlikely to simulate the same action multiple times from a given state. This would significantly weaken the motivation for $OS(\\lambda)$, which relies on simulating each action many times from a given state. In such large action spaces, a very relevant heuristics baseline is to sample $N$ simulations (each starting with a different action) with the learned policy and simply follow the action given by the best simulation. Overall, there's insufficient experiments demonstrating that $OS(\\lambda)$ and AWPO solve the stated large action space problem without suffering from unintended side-effects of the additional optimism, and there should be more trivial heuristics baselines which may also work well for searching large action spaces with a fixed budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)