Grounded Answers for Multi-agent Decision-making Problem through Generative World Model
Authors: Zeyang Liu
Xinrui Yang
Shiguang Sun
Long Qian
Lipeng Wan
Xingyu Chen
Xuguang Lan
Venue: NeurIPS 2024 poster (2024)
Keywords: World Model
Multi-agent Reinforcement Learning
Model-based Reinforcement Learning
Primary area: generative_models
TL;DR: We explore a paradigm that integrates a language-guided simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer for decision-making problems from generative models.

Abstract: Recent progress in generative models has stimulated significant innovations in many fields, such as image generation and chatbots. Despite their success, these models often produce sketchy and misleading solutions for complex multi-agent decision-making problems because they miss the trial-and-error experience and reasoning as humans. To address this limitation, we explore a paradigm that integrates a language-guided simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer. The simulator is a world model that separately learns dynamics and reward, where the dynamics model comprises an image tokenizer as well as a causal transformer to generate interaction transitions autoregressively, and the reward model is a bidirectional transformer learned by maximizing the likelihood of trajectories in the expert demonstrations under language guidance. Given an image of the current state and the task description, we use the world model to train the joint policy and produce the image sequence as the answer by running the converged policy on the dynamics model. The empirical results demonstrate that this framework can improve the answers for multi-agent decision-making problems by showing superior performance on the training and unseen tasks of the StarCraft Multi-Agent Challenge benchmark. In particular, it can generate consistent interaction sequences and explainable reward functions at interaction states, opening the path for training generative models of the future.

Grounded Answers for Multi-agent Decision-making Problem through Generative World Model
Zeyang Liu
Xinrui Yang
Shiguang Sun
zeyang liu@stu. xjtu. edu
xinrui perrui.ynndssu.xxji.. www...euu.uu........ edu.
asg20190stu.x xjtu. edu.
Long Qian
Lipeng Wan
Xingyu Chen"
glanlongym@stu. xjtu. edu. cn
wanl 1pen777xx7 edu, en
1henxinggy__1990x tu. edu. cn
Xuguang Lan"
xg xglan xglan@mail .xtt..euui edu.edu.c. .n
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence National Engineering Research Center for Visual Information and Application Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, China
Abstract
Recent progress in generative models has stimulated significant innovations in many fields, such as image generation and chatbots. Despite their success, these models often produce sketchy and misleading solutions for complex multi-agent decision- making problems because they miss the tral-and-error experience and reasoning explore paradigm that integrates
humans. To address this limitation, ainguageggidedd simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer. The simulator is world model that separately learns dynamics and reward, where the dynamics model comprises an image tokenizer well causal transformer generate interaction transitions autoregressively, and the reward model is a bidirectional transformer learned by maximizing the likelihood of trajectories the expert demonstrations under language guidance. Given an image of the current state and the task description, we use the world model by running
train the joint policy and produce the image sequence as the the converged policy on the dynamics model. The empirical results demonstrate that this framework can improve the answers for multi-agent decision-making problems by showing superior performance on the training and unseen tasks of the StarCraft Multi-Agent Challenge benchmark. particular, it can generate consistent interaction sequences and explainable reward functions at interaction states, opening the path for training generative models of the future.
Introduction
Recent progress in generative artificial intelligence with models capable of generating creative content has shown attractive prospects for real-world applications, such image generation TTkagi & Nishimoto 2023). embodied agents (Brohan et al. 2023b), and chatbots (Köpf et al.) (024). Most generative models attempt directly obtain the answer by training natural language image datasets and inserting decomposed reasoning steps in few-shot demonstrations. However, these methods do experience firsthand the situations described by the language and the image. They
"Corresponingg authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Vision-Language Model
Image-Texts tate Transformation
(Learned) World Model
LBI Augmented Policy
Environment
Update
LBI Output
Dutput
Question
Anstyce Compostoon
4. Usiize Abiltee Statogically
1. Focus fire
Positorinng Formaton
marnes the lett What
strllggy should wi use defiaal
Adepi Fremy Nicomeess
ene marthes on the
Figure 1: Complex decision problems that require a good understanding of the environment's
dynamics and the objective are still challenging for current vision-language models, e-g., the answer elicited by GPT-4 is sketchy and misleading. Instead, Learning before Interaction (LBI) enables grounded reasoning by simulating the task the given question. LBI utilizes the simulator train MARL policy and generate the answer by running the converged policy on the simulator.
cannot find the correct answers through trial and error as humans, which is necessary to ground reasoning on complicated problems and transfer learned knowledge to unfamiliar domains. For example, as shown in Figure [ when asked complex multi-agent decision problem, one of the most widely-used large language models, GPT4 though achieving superhuman performance in many reasoning tasks will generate sketchy and misleading answers.
To tackle this problem, we can utilize the generative models to understand the properties of the task that the user describes and simulate the effects of the actions. We can derive the answer with a highly realistic simulator by pnperrment-reasoniin or training any machine intelligence from simulated experience. The origin of this idea can be traced back to Dyna architecture (Sutton) 1990) and has spawned a series model-based reinforcement learning (MBRL) theories and methods (Janner et al. 2019: Kaiser et al.) 2019 Lai et al., 2020). Inspired by this, Mind's Eye (Liu et al. 2022) enables language models to perform reasoning conditioned on the simulation results by running the corresponding experiment on computational physics engine named MuJoCo (Todorov et al. 2012). Mind's ' Eye can boost reasoning performance zero-shot and few-shot settings by infusing such physical knowledge into language models. However, it is particularly designed for physical reasoning rather than decision-making problems.
In contrast, UniSim YYnng et al. (2224) formulates the action siree-iii-viioouuue framework as an observation prediction diffusion model conditioned on finite history. shows that the simulator learned from broad data can generalize to the real world and bridge the sim-to-real gap. Genie (Bruce al.) 2024) enables users to act the generated environments on a frame-by-frande basis, opening the path for training generalist agents of the future. Notably, most of the existing breakthroughs the imagined experience have been focusing on single-agent scenarios and leave the world model largely unstudied for multi-agent reinforcement learning (MARL) tasks it is common in real-world applications that multiple agents are required to solve a task a coordinated fashion.
Icarning in
The roadblocks to building a simulator for MARL tasks are twofold. First, MARL tasks involve multiple entities' attributes, e.g. positions and roles, making it difficult to describe a state using only text. The text and image information can be brought together to enrich the inputs for the simulator, such a dataset is limited quantity. Second, the dynamics and reward models of the MARL environment are more intricate than the single-agent setting. Current approaches assume the reward known in the dataset Meng et al. 2023) or can be easily deduced by the frame information (Yang al., 2024), which could be challenging for MARL methods due to the abundance of agents' tactics and the compositional nature of their functionalities.
This work explores paradigm that adds language annuage--uiide guided simulation to the MARL pipeline to make policy learning grounded within the learned world model. First, we propose new offline MARL datasets to provide paired state-image information for the StarCraft Multi-Agent Challenge (SMAC) environment by transforming the state the trajectory to corresponding image. We also designed parser to convert each trajectory to task description using natural language. Then, pre-train vector quantized variational autoencoder (VQ-VAE) (Van Den Oord et al., (017) to generate discrete representations for each frame. The world model is formulated as interactive simulator that consists

---

of dynamics and reward model. The dynamics model comprises an image tokenizer and a causal transformer to generate interaction transitions autoregressively. The reward model is bidirectional transformer learned by maximizing the likelihood of trajectories in the expert demonstrations under the task description.
Given a decision- masiing problem by the user and an image from the environment, we store the simulated interaction trajectories into replay buffer by running an off-poliey MARL algorithm on the generated dynamics model. Then, we utilize the generated reward model to label the reward for each acte-action pair based on the whole trajectory. We update the policy network according to the reward with a behavior behavior-regularizze regularization term, which serves as the conservatism for out-of di-ssiitiuution State-action pairs. We use the image sequence generated by the interaction of the dynamics model and the converged policy model as the answer to the decision-making problem.
We summarize the main contributions of this paper in three folds: (1) It proposes novel MARL datasets for SMAC, where a parser automatically generates the ground-truth image of a given state and task description. (2) It introduces Learning before Interaction (LBI), an interactive simulator that generates trial-and-error experiences and improves the answers for multi-agent decision-making problems. (3) The empirical results show that LBI outperforms various offline learning methods on training and unseen MARL tasks. The visualization also indicates that LBI can produce consistent imagined trajectories and explainable rewards for interaction states.
2 Background
Decentralized Partially Observable Markov Decision Process. A fully cooperative multi-agent task in the partially observable setting can be formulated as Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato 2016), consisting of a tuple G = (A, S, O,U, p.r, r),, where a E A = fl..... n is a set of agents, S is a set of states, and f2 is set of joint observations. At each time step, each agent obtains its observation o E Q based on the observation function O (8,a) S x A + Ω and an action-observation history Ta E T = (52 x U)*. Each agent a chooses an action u. E U by a stochastic policy Ta (Ua(Ta) : T x U - [0, 1], which forms joint action It results in joint reward r(s, u) and a transit the next state s' ~ P(-|s, u). The formal objective function is to find the joint policy π that maximizes a joint action-value function Q (st uz) r(s,, u,) + 2 [V* (s'), where V a discounted factor.
E IE "rr, |s0 s, π]. and y [0, 1) is
Inverse Reinforcement Learning. Suppose we do not have access to the ground truth reward function but have demonstrations D provided by an expert policy wg. Imitation learning aims to directly learn policies that behave similarly to these demonstrations, whereas inverse reinforcement learning (IRL) seeks to infer the underlying reward functions which induce the expert policies. The MaxEnt IRL framework (Ziebart et al.] 2008) aims to recover reward function that rationalizes the expert behaviors with the least commitment, denoted as (RL()
(RL(TE) = arg max B [r(s, w)] RL(
RL max H(TT) + E (r(, u]]
where H((x) = 1- log x [u]ss] is the entropy of current policy π. It looks for a reward function that assigns high reward to the expert policy and a low reward to the current policy ι while searching for the best policy for the reward function in the inner loop.
3 Methodology
We formulate an interaction simulator as a transition prediction model that, given some state of the world and descriptions of the task, can take some actions as input and produce the consequence of the actions in the form of images, states, and rewards. In this paper, we consider building such simulators for a multi-agent decision-making environment named StarCraft Multi-Agent Challenge (SMAC) (Samvelyan al. 2019), known for its rich environments and high control complexity. More information about SMAC can be found in Appendix C
3

---

State-based
Trajectories
Mis Dataset
Images
2 VQ-VAE Leaming
Parser
Image Tokenizer Encoder
Task Description
Codebook
O ffline B Buffr
We have Marines, Marauders, Mecivac. What the best strategy for us to completely eliminate the enemy's 8 Marines, Marauders, M Medivac?
Image Tokenizer Decoder
Materials
Figure 2: Datasets construction and VQ-VAE training.
3.1 VisionSMAC
The SMAC benchmark saves replay of episode as SC2REPLAY file rather than providing the image feature during exploration. It is computationally expensive to construct datasets of images by watching such replay within the StarCraft II client and then subsampling frame that captures meaningful actions. To solve this problem, we introduce VisionSMAC to convert the state into images and languages through a parser f. decoupled from StarCraft, making it easy to create
content.
First, we collect offline datasets across ten training maps in the SMAC benchmark by running multi- agent exploration methods named EMC (Zheng al., 2021) and IIE (Liu et al. 2024). Each dataset contains a large number of interaction trajectories:
7 -= (st, {o]]a=> (ut) [47] >==
(2)
where 8; denotes the state, {of)== is the observation of each agent, {uā (u]]==1 is the joint action, and the done signal dē = 1 when the agent is killed at timestep I, n and T denote the number of agents and the length of the episode, respectively. We further collect the element images that appear the game and affect the state, such as the units and the background terrain of training maps.
Given multi-agent system and its interaction trajectory. the parser f reads predefined map informa- tion, such as the number and races of agents and enemies. Then, the parser converts the original state information into structured information, reading agents' and enemies' positions and health points. It will generate the corresponding interaction scenes by placing each unit image and its health bar their positions with the corresponding background terrain. The image generated by the parser can resemble the frame subsampled from the original replay by running a SC2REPLAY file within the StarCraft client, where the comparisons can be found Appendix B We also perform data augmentation to enable better feature abstraction by changing the background to different terrains.
Finally, we define a task description can be the terminated state, a slice of trajectory, or any other representation of the episode. In this paper, we use the terrain information, the number and unit types of agents and enemies, and the sum of enemies' remaining health points at the terminated state as the task description. To this end, the parser reads the last state of the trajectory and extracts the remaining health points of both sides. We can obtain the practical task description by filling in predefined description templates (e.g., "Consider that we control (number of agents) (agent races) on the left.") and adding connecting words (e.g., "What plan should we use"). The detailed description of the datasets and the parser ʃ can be found in Appendix B
specify the environment and the task. The task description
3.2 Training An Interactive Simulator
With trajectories with corresponding images and language guidance from different scenarios, we can formulate the interactions with StarCraft as interacting with an interactive simulator. The simulator contains three key components: (1) an image tokenizer that converts raw video frame into discrete tokens, (2) dynamics model that predicts the next frame and state given past frame and state tokens, (3) reward model that infers the reward of state-action pair given trajectory. The idea behind decomposing the world model into a dynamics model and a reward model reuse the dynamics model for different tasks by combining it with any reward function.

---

<ask descriptin>
<magg>>
Reward-free free Data Collection
Language Takenizer
Image Tokenizer
Supervised Learning
Dataset
Updltte
Interaction
Dynamics Model (+)
Policy Model (t)
Reward- free Trajectory
Sanguage-guiden Reward Labeling
<assi>>
Behaavar-regullarized RL
Reward Madel (p)
Inner-loop Policy (8)
Interaction
Environment
Inverse RLL
Figure The overview of Learning before Interaction.
Image Tokenizer
We compress images into discrete tokens to reduce dimensionality and enable higher-quality image generation. We make of vector quantized variational autoencoder (VQ- VAE) (Van Den Oord et al.) 2017). which takes every single image of the state as input, generating discrete representations. The tokenizer is trained using standard VQ-VAE objective.
Dynamics Model The dynamics model a causal transformer parameterized by vi where the target sequence has the following form {..., L, zt, 8t, of -,.;0000 of, " ut", Ze+1, 8t++1, where 2, is the image representation generated by the fixed image tokenizer. We utilize the task description specify the dynamics of the environment, remaining consistent in one sequence. An embedding for each timestep is learned and added to each token. The dynamics model processes all past tokens and predicts future tokens via autoregressive modeling.
Then, we use the prediction heads to decode the predicted tokens to the corresponding element the sequence and train them by minimizing the cross-entropy loss for actions and mean-squared error for others. The actions would serve as the reference policy for the learning with the simulated trajectories described in Section 3.3 In particular, and the stability of the generation by changing the target from Sc+1 to As++1 = 8t+11 8, for the state prediction head. We also apply this term predict image representations. In addition, since the observation is only related to the current state and the vision range of the agents, we filter out the historical memories and use S; as the input for the observation prediction.
dynamics residual term to improve the accuracy
Reward Model We resemble the training pipeline of inverse reinforcement learning maximizing the likelihood of trajectories in the expert demonstrations while minimizing the likelihood of trajecto- ries collected from an inner-loop policy. We introduce a reward function f, which receives the entire trajectory as inputs to perform credit assignment under deterministic dynamics within the trajectory. We remake this formulation as generalized version of the conventional reward design; if the reward function is Markovian, the temporal dependencies on other sate-action pairs should always be zero.
To this end, we model the reward function as a bidirectional transformer model parameterized by ø, where the sequence is f {.... St, L, ..., uf,' ft, st.1....), and {Fr) is set of individual rewards for the agents. Again, we utilize the task description to perform hindsight relabeling, which converts imperfect trajectories into possible solutions for reaching the last state sT of the episode, generating the expert demonstration D. We optimize the reward function by minimizing the following loss:
VoL -E ETED
ΣΣ TVo
+ E ETazz
(3)
where ~ = (1 0) the inner-loop MA-SAC policy parameterized by and we use the reward constraint by imposing an 1.2 penalization of the predicted rewards over all possible actions, which can be viewed as conservative update for disti--iitti state-action pairs. particular, alternate between k-step of policy update and k-step of reward update to avoid completely solving
5

---

the policy optimization subproblem before updating the reward parameters, where k is a given update iteration.
3.3 Inference: Learning Policy in the Simulator
We now describe how to generate grounded answers for multi-agent decision-making making problems via LBI. Given image of the initial state and task description from the user, the agents interact with the dynamics model using a randomly initialized off-policy MARL algorithm to collect reward-free trajectories in autoregressive manner. Then, the reward model predicts the immediate reward for each transition pair in the simulation trajectories. These relabeled trajectories are added to the replay buffer, serving as the training data for the policy network.
In practice, we construct the MARL problem in the simulator as a behavior-regulariized MDP by imposing a Dehavior-regularizatio term:
max π
Σ
7
n Σ
((((:))) a. log
(uf g( Teayf
(4)
where मे = ( ()):
)== ))); is the joint policy, and g( [ v)) is the reference policy provided by the dynamics model. The last term enables in- sample learning and further mitigate the impact of exploring OOD regions in the state-action space. We use independent Q-learning to train the
parameter-sharing agent network.
Since it is possible for specific agents to become inactive before the game terminates, we mark the terminated timestep for each agent and enemy once its predicted health less than zero and then use zero vectors as the subsequent actions and observations. It can mitigate the hallucinating unrealistic outcomes dead agent performs a "moving" action. We also mask the predicted reward after the terminated timestep for the inactive agent to get more accurate value estimate.
4 Related Work
World Models. There is a long-standing history of learning predictive models of the world. We list three categories of model-based reinforcement learning (MBRL) according to the type of model usage. The first category applies planning methods with world model simulation. AlphaGo (Silver et al., 2016) and MuZero (Schrittwieser et al. (2220) learn a transition model and apply Monte Carlo Tree Search to search for an action sequence with the highest accumulated rewards. By contrast, MBMF (Nagabandi et al. 2018). PETS (Chua et al.) 2018). and PlaNet (Hafner et al.) (019b) integrate model predictive control (MPC) into the learned world model and sample high-reward action sequences. TD-MPC (Hansen et al. 2022) and TD-MPC2 (Hansen et al. 2024) utilize value functions to bootstrap the trajectories used for MPC planning.
The second category models differenniable world model and utilizes the internal structure of the model to facilitate policy learning. GPS (Levine & Koltun| 2013) and GDP (Srinivas et al.) 2018 perform differential planning and obtain the analytic form of the optimal policy. SVG (Heess et al. 2015) re- p-aaametterizes the policy and the world model, then computes the policy gradient estimate by backpropagation via the world model. MAAC (Clavera et al.) 2019). Dreamer (Hafner et al.) 2019a) and its subsequent variants (Hafner et al. 2020 2023) use the recurrent state-space model PlaNet to learn the world model in a compact latent space and learn the policy entirely within this space.
The last one utilizes the learned world model to generate more experiences and then trains a policy on the dataset augmented by the model, also known as Dyna-style methods (Sutton) 1990). MVE (Fein- berg al. 2018) and STEVE (Buckman et al., 2018)) depict a learned world model calculate multi-step temporal-dificee difference prediction for better value estimation. In contrast, SLBO (Luo et al.) 2018 MBPO (Janner et al.) 2019). and BMPO (Lai al. 2020) theoretically analyze this learning framework and prove that the policy performance will improve monotonically in world model with certain model bias and rollout length. To further increase the rollout length and avoid compounding errors, M2AC (Pan et al., (020) and COPlanner (Wang et al., 2023) compute the uncertainty of each rollout step and perform conservative model rollouts by discarding the samples with high uncertainty or adding a penalty term into total reward. In practice, GAIA-1 (Hu et al. 2023). UniSim (Yang jet al., 2024), and Genie (Bruce et al. 2024) show that the learned world model can enable the control policy to generalize to the real world when trained purely in simulation and bridge the sim-to-real

---

gap. These methods have impressive performance and theoretical bounds, attracting much research interest in the MBRL community. However, they focus on generating videos or trajectories that only involve one single agent instead of building a multi-agent simulator that can be used to further improve decision-making performance on coordination tasks in our work.
Imitation Learning. Imitation Learning (Bain & Sammut, 1995) formulates imitating an expert
as a supervised learning problem, which has been widely adopted in various domains due to its simplicity and effecttivnness (Silver et al. 2016; Swamy et al., 2020). GAIL (Ho & Ermon, (201)) and its extensions (Song et al. 2018] Ghasemipour et al., (020) stand as a comeestone approach, which trains a generator policy to imitate expert behaviors and discriminator to distinguish between the expert and the learner's 's state- aate-action pair distributions. In light of the recent interest in foundational models, the conditional diffusion model is used to represent and learn an imitation learning policy, which produces a predicted action conditioning on state and a sampled noise vector |Pearce et al. (2022); Chi et al. (2023). These methods achieve encouraging results in modeling stochastic and multimodal behaviors from human experts or play data. DT-style methods (Chen et al., [022]] Wu al. 2024) formulate the trajectory generation
vector|Pearce
sequence modeling problem, which generates states, actions, and rewards by conditioning on a return-to-go token in an autoregressive manner.
In contrast, inverse reinforcement learning (IRL) is designed to infer the reward function that underlies the expert demonstrations, taking into account the temporal structure and showing better generalization than direct Behavioral Cloning (Ng & Russell 2000; Ross et al. [2011] Barde et al.
2020). A main class of algorithms, Maximum entropy (MaxEnt) IRL (Haarnoja et al. (017) and its extensions (Liu et al.) 2021; Rolland et al., 2022). learn stationary reward by minimizing divergence between the agent and expert distribution. Since the learned reward function can solve downstream tasks and transfer behavior across different dynamics, IRL. is also helpful in several broader applications, e.g., IRL with natural language goals (Fu et al. 2018a| Zhou & Small, 2021 Xu et al. 2022), and RL with human feedback (Ziegler et al. 2019; Zhu et al. 2023). and dynamics learning (Luo et al.) 2023). Furthermore, a series of sample-efticient a algorithms are proposed to solve the MaxEnt IRL formulation (Fu et al. 201 Zeng et al. 2022 2024).
2023; Wu et al.
To side-step the expensive online environmental interactions in classic IRL, some work ams to learn reward function from static dataset by a variational Bayesian framework (Chan & van der
Schaar 202I), representing reward function via a learned soft Q-function (Garg et al. [2021). or incorporating conservatism into the estimated reward like offline Q-learning ( (Yue al. (222)). The
major bottleneck for these methods includes a lack of knowledge of the dynamics information and the reward overestimation for out-o--distributtin state-action pairs. We formulate the reward model as a bidirectional transformer to receive the whole trajectory as the input, making it possible to solve non-Markovian rewards. In addition, we leverage the reward constraint and the behavior regularization to perform in-sample learning to avoid reward overestimation. The amount of expert demonstrations in these existing studies is also limited, as they do not treat hindsight relabeling via the textual description as an expert trajectory like in our work.
Offline Q-Learning. Offline Q-learning learns policy from fixed dataset where the reward is provided for each transition sample. Most off-policy reinforcement learning (RL) algorithms are applicable in offline Q-learning. However, they typically suffer from the overestimation problem of diitr---ditiitttonn (OOD) actions due to the distribution shift between the action distribution
in the training dataset and that induced by the learned policy (Fujimoto et al.] 2019). Several constraint methods are proposed to restrict the learned policy from producing OOD actions by leveraging importance sampling (Suttoon et al., 2016 Nachum et al. 2019), incorporating explicit policy constraints (Kostrikov et al., 2021 | |akoor et al. 2022 Fujimoto & Gu 202T; Tarasov et al. 2024). penalizing value estimates (Kumar et al. 2020: An et al. 2021; Shao et al. (024), and uncertainty quantification (Wu et al. 2021; Zanette et al.
2021). Another branch resorts to learning without querying OOD actions and thus constrain the learning process within the support of the
dataset (Bai et al. 2021
Lyu et al. 2022)
Experiments
In this section, we conduct empirical experiments to answer the following questions: (1) Is Learning before Interaction (LBI) better than the existing multi-agent reinforcement learning (MARL) methods in complex cooperative scenarios? (2) Can LBI generate long-horizon trajectories and reasonable reward functions at critical states? (3) Does LBI have the zero-shot ability to generalize to unseen
7

---

Table 1: Test win rates (9) and standard deviations compared with imitation learning methods.
Map
MA-AIRL
MAPT
MA-TREX
LB1
1c3s52
16.441 1.35
7,88 2.49
61.351 7.26
74,77 5.15
64,7611
11.62
94.5911 3.41
10m, Hm
26.19:1 4.42
41.69: 7.12
82.76:: 4.41
66.85 9.28
48.78 11.28
90.45:
2e_ vs_ 642g
17.37 10.12
24.75= 10.83
61.90 5.74
58.28 7.84
22.45 7.74
71 8.83
36_va_
0.00:
0.05=
72.33+ 3,93
55.38
Smyys, 6an
13.78.1: 2.15
11.59:1 6.75
79,78.1 4.98
56.01 3.17
50.01 1: 14.87
87.98.1 5.10
vs
9.28:1:
16.47:: 8.08
30.94:1:
37.16:: 6.27
66.61:
0.001: 0,00
0.00:: 0.00
27.44:1: 9.49
34.901 6.84
36.16:1 3.68
83.34:1 4.27
corridor
0.0011 0.00
0.76.1: 0.15
69.85.1 1.54
45.91 15.47
30.59 9.86
87.45 2.94
MMM2
0,00: 0.00
0.00 0.00
54.34: 12.83
19.21 5.59
21.52
95.96: 4.65
Table 2: Test return and standard deviations compared with offline reinforcement learning methods.
Map Name
BCQ-MA
CQL-MA
ICQ
OMAR
OMIGA
5m_ys, 6m
9:13 0.21
10.154 0.15
9.47 0.45
8.76+t: 0.52
10.38 0.50
18.96: 0.56
2o_vs, 64zg
18.86 0.35
19.20= 1.25
18:47 0.25
17.10= 0.94
19.25 0.38
20.45: 0.25
6h_ vs 8r
11.91 0.44
9.95 0.32
11.55= 0.15
9.744 0.28
12.744 0.21
18.97 0.28
16.42 1.55 corridor
6.64# 0.90
16.74= 1.78
8.15+ 0.89
17,10 1.33
19,50: 0.73
tasks? Then, we investigate the contribution of each component in the dynamics and the reward model. We provide the information of training datasets and experimental settings Appendix [Band and D We also discuss this paper's ' broader impacts and limitations in Appendix A.] and A.2
5.1 Performance Comparison
Reward-free Offline Learning We compare LBI with the following imitation learning baselines:
(1) BC: behavior cloning that imitates the whole datasets. (2) MA-AIRL (Yu al., 2019): using
adversarial learning to perform policy imitation, (3) MADT (Meng et al. utilizing the Decision Transformer (Chen et al., (02T) to perform sequence modeling, (4) MA-TREX: infering the reward according to ranked demonstrations, the multi-agent version of TREX (Brown et al. 2019), (5)
MAPT Zhu et al.) 2024): infering the team rewards according to the preference return from well-trained scripted teacher.
As shown in Table I LBI outperforms the baselines by a significant margin on various maps with different difficulty levels, indicating the importance and effectiveness of learning reward functions via the proposed world model. In contrast, BC and MA-AIRL fail achieve success rates in most tasks because they imitate all past interaction sequences and cannot generalize and avoid sub-optima solutions. MA-TREX and MAPT have plateaued in performance because they use the accumulated rewards and the preference deduced by the scripted teacher to specify the quality of the training data, respectively. MADT performs better than other baselines because Decision Transformer can be thought of as performing imitation learning on a subset of the data with a certain return.
We also compare LBI with the existing offline MARL methods with ground-truth rewards from the StarCraft Multi-Agent Challenge (SMAC), including the multi-agent version of BCQ (Fujimoto et al. 2019) and CQL (Kumar et al., 2020) (namely BCQ-MA and CQL-MA), ICQ (Yang al., 2021). OMAR (Pan et al. [d.22))), and OMIGA (Wang et 2024). Table 2 shows that the performance of these offline MARL methods degrades dramatically y with an increasing number of agents and much lower than that of LBI. We hypothesize that the reasons for this gap
Offline MARL
are: (1) It is challenging and unnecessary to recover the Q-value based on the reward functions provided by SMAC (the hit-point damage dealt) because such reward design is inefficient for learning optimal policy. (2) These methods may introduce too much conservatism and affect the learning of the optimal policy, as the conservative update of the out-of-distribution (OOD) suboptimal policy that consists of some agents taking non-optimal actions and others taking optimal will inhibit the learning of the agents that take the optimal actions.
5.2 Generalization on Unseen Tasks
Since zero-shot generalization ability is crucial for generating grounded answers for multi-agent decision-making problems, we also test LBI's ability to generalize to extensive unseen scenarios
8

---

Table 3: Test win rates (%) and standard deviations unseen tasks.
Unseen Task
MADT
MA-TREX
LBI
Unseen Task
MADT
MA-TREX
LBI
1c3s
16.21 5.38
23.53 + 8.83
56.47= 5.63
1c2s72
6.16+ 3,09
5.69=381
28.26 6.41
6m
49.28: 4.06
37.12+22.59
97.85= 2.15
6m_ys_77m
73.45-4-..2 7.22
32.88:44.47
81,07 5.17
32ng
2.08 1.51
11.41:141
58.33 6.44
364z
90.2111 1.82
79.7113.56
87,55 1.76
362,
23_6x
0.00= 0.00
9.16455.2
18.22 2.46
3652 __367
10.21:: 3.66
15.8841.344
22,084:: 7.63
1c3s6z
16.41 6.44
58.09#33.41
65,38= 5.12
76.44: 4.17
70.91
75.05=: 2.16
Table 4: The ablation results for the dynamics model without residual term (wo-RT), image reference (wo-IR), and using ground-trun truth im- age (GTI) as the reference for state prediction.
Table 5: The ablation results for the reward model
without reward constraint (wo-RC), behavior regu- larization (wo-BR), and using ground rewards (w-GTR) provided by the SMAC benchmark.
Algorithm
Prediction Error
Return (All)
Algorithm
Return (Training)
Return (unseen)
LBI
0.016 + 0.025
18.91 = 1.33
LBI
19.47 0.77
18,54 = 1.49
LBI-GTI
0.014 0.018
18.98 0.89
LBI-GTR
16.68 = 1.55
14.07
LBI-wo-RT'
0.434 + 0.351
14.25 1.84
LBI-wo-RC
17.85 0.59
14.75
1.67
LBI-wo-IR
0.029 0.041
18.63 :: 1.01
LBI-wo- BR
18.82 1 1.28
17.46
= 2.01
L.B--o-RTTal
0.744 1,164
12.13 = 2.33
L.B.-woRR&&BB
12.35 2.38
9.83 1.46
without retraining. Specifically, we evaluate our LBI and MADT on the ten unseen testing maps, varying agent numbers, action spaces, and levels of environment complexity. 3 shows that LBI consistently outperforms MADT in unseen scenarios by a large margin, successfully transferring knowledge new tasks without requiring additional fine-tuning. It highlights that learning reward function has better sero-shot generalization performance than simple policy adaptation.
5.3 Ablation Studies
In this section, we conduct ablation studies to analyze the contributions of each component in the dynamics model and the reward model across five evaluation runs on four training maps (6h_vs_82 vz._ 3s5z_vs vs 3s6z, corridor, and MMM2) and four unseen maps B35z_vs_3ss77 Ic3s7z, 3s4z, 1c_ 32zg). We show the results of the dynamics model in Table 4 Using the dynamics residual term is necessary to reduce the prediction error of the subsequent states and obtain good performance across all training and unseen tasks. The image reference is not effective, even if we use ground- truth images as the reference. However, since images are more powerful in representing some situations than language state information, we believe that the image serves as another modality to correct the prediction of the state. We would leave for future work.
We demonstrate the ablation results of the reward model in Table 5 Compared with LBI-wO-RC&BR, the reward constraint and behavior regularization term can improve the overall performance on the training tasks. However, LBI-wo-BR performs better than LBI-wo-RC on unseen tasks, suggesting that the conservatism for reward is more important than the policy when OOD state-action pairs exist. The poor performance of LBI-w-GTH --RR indicates that learning rewards from conditioned demonstrations may be more accessible and valuable for policy updates than reconstructing the pre-defined rewards by human experts.
5.4 Visualization
This section evaluates the dynamics model as long-horizon condiy-conditooned predictive model. Figure 4 showcases examples of length-40 image trajectories generated by the dynamics model, including MMM2, 3s ___vs___. 5z, and 5m vs_ 6s_ We do not observe conspicuous compounding errors as the single-step prediction model does, highlighting that LBI has consistency and long-horizon generation ability. In the case of 5m vs_6m, we present the following frames after taking one of the possible actions, showing that LBI can also perform action-conttrllable generation.
We also investigate the reward prediction at critical junction in the state-action space that can transit to various states and significantly influence the success rate on the 5m_vs_6m task. At the moment, the agents have to learn to micromanage leapfrogging to achieve good coordination. Specifically, Agent has a low health point and must move backward avoid the enemies focusing fire on it; otherwise, the enemies will eliminate Agent immediately and weaken our scarce forces. In Figure 4,

---

Generated (MMM2)
t=0
t=5
t=10
t=15
t=20
t=25
t=30
t=35
h=40
Generated (3s_ vs_ [5)
Generated vs 6m)
Agent ①
Agent low-health Expert policy: leapfrog
Agents 1-0 attack 1
Figure 4: Visualization of the prediction from dynamics and reward model, where "np-op" and "s" denote no-operation and stopping, respectively.
we visualize the learned reward function of Agent I, where the action space is no-operation, stopping. moving cardinal directions, and selecting an enemy's s identity to attack. The learned reward for moving to the left is much higher than the other actions, allowing quickly. The rewards provided by the SMAC benchmark do not show this property, where multiple Monte Carlo samples required to find the correct policy by estimating the expected return.
to learn the optimal joint policy
6 Conclusion and Future Work
We proposed Learning before Interaction (LBI), novel paradigm that enables generative models to ground their answers for multi-agent decision- making problems with simulations between the world and the multi-agent system. We formulate an interactive simulator consisting of dynamics and reward models, given some states of the world and the task descriptions, generating the consequence of the actions in the form of images, states, and rewards. We hope the idea of including simulations in the reasoning will instigate broad interest applying generative models aid machine intelligence and decision-making.
7
Acknowledgements
This work was supported in part by NSFC under grant No.62125305, No. 20.62088102, 102, No. U23A20339, No. 62203348.
10

---

References
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- baaeee offline reinforcement learning with diversified q-ensemble. Advances in neural information prucessing systems, 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhi- Hong Deng. Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Paul Barde, Julien Roy. Wonseok Jeon, Joelle Pineau, Chris Pal, and Derek Nowrouzezahrai. Adversarial soft advantage fitting: Imitation learning without policy optimization. Advances Neural Information Processing Systems, 33:1 1234--1334, 2020.
Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. International Conference on Machine Learning, pp. 980-991. PMLR, 2020.
Anthony Brohan, Noah Brown, Justice Carbajal. Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: lasion-languag--ann models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023a.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, al. Do as can, not say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287-318. PMLR, 2023b.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint arXiv:2402. 15391, 2024.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforce- ment learning with stochastic ensemble value expansion. Advances in neural information processing systems. 31, 2018.
Alex J Chan and M van der Schaar. Scalable bayesian inverse reinforcement learning. In International Conference on Learning Representations, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeol, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 34:15084-15097, 2021.
Cheng Chi. Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffussio policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-augmented actor-critic: Backpropagating through paths. In International Conference on Learning Representations, 2019.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pincau. Tarmac: Targeted multi-agent communication. In International Conference Machine Learning, pp. 1538 1546. PMLR, 2019.
Danny Driess, Fei Xia, Mchdi SM Sajjadi, Corcy Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pp. 8469-8488. PMLR, 2023.
Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander Smola. Continuous doubly constrained batch reinforcement learning. Advances in Newral Information Processing Systems, 34: 11260-11277 2021.

---

Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model- based value estimation for efficient model-free reinforcement learning, arXiv preprint vrXiv:1803.0001,, 2018.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. International Conference on Learning Representations, 2018a.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018b.
Scout Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning. pp. 2052-20000 2062. PMLR, 2019.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-40323 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on robot learning, pp. 1259-1277. PMLR, 2020.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy- based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Leamning behaviors by latent imagination. In International Conference on Learning Representations, 2019a.
Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak I Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp- 2555-2565. PMIR, 2019b.
Danijar Hafner, Timothy P Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04/00 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In The Twelfth International Conference Learning Representations, 2024.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pp. 8387-8406. PMLR, 2022.
Nicolas Heess, Gregory Wayne, David Silver. Timothy Lillicrap. Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.122 17080, 2023.
Michael Janner, Justin Fu, Marvin Zhang. and Sergey Levine. When trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Lukasz Kaiser, Mohammad Babacizadch, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2019.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In International Conference on Learning Representations, 2020.
Andreas Köpf, Yannic Kilcher. Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversation--ddmmcraiiiiin large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-99 94, 2016.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-laaniing for offline reinforcement learning. Advances in Neural Information Processing Systems, 119911999,, 2020.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model- based policy optimization. In International conference on machine learning, pp. 5618 5627. PMLR, 2020.
Martin Lauer. An algorithm for distributed reinforcement learning in cooperative multiagent systems. In Proc. 17th International Conf. on Machine Learning, 2000.
Kuang-Hust Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, al. Multi-game decision transformers. Advances Neural Information Processing Systems, 129-79911-7796, 2022.
Sergey Levine and Vladlen Koltun. Guided policy search. In International conference machine learning, pp- 1-9. PMLR, 2013.
Minghuan Liu, Tairan Hc, Minkai Xu, and Weinan Zhang, Energy-based imitation learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 809-817, 2021.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. Mind's eye: Grounded language model reasoning through simulation. The Eleventh International Conference on Learning Representations, 2022.
Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Zeyang Liu. Lipeng Wan, Xue Sui, Zhuoran Chen, Kewu Sun, and Xuguang Lan. Deep hierarchical communica- tion graph in multi-agent reinforcement learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 208 226, 2023.
Zeyang Liu, Lipeng Wan, Xiarui Yang. Zhuoran Chen, Xingyu Chen, and Xuguang Lan. Imagine, initialize, and explore: An effective exploration method in multi-agent reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16) 17487-17495, Mar. 2024. doi: 10. 1..1669/aaii.v33 v38i16.29699
Fan-Ming Luo, Tian Xu, Xingchen Cao. and Yang Yu. Reward- consistent dynamics models are strongly generalizable for offfine reinforcement learning. In The Twelfth International Conference on Learming Representations, 2023.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrel, and Tengyu Ma. Algorithmic framework for bodel-based deep reinforcement learning with thcoretical guarantees. In International Conference on Learning Representations, 2018
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative lcamming for offline reinforcement learning. Advances in Neural Information Prucessing Systems, 35:1711-1724, 2022.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances Neural Information Processing Systems, 32, 2019.
Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon White- son, Yuke Zhu, and Animashree Anandkumar. Tesseract: Tensorised actors for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 7301-7312. PMLR, 2021.
Laëtitia Matignon, Guillaume Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In IEEE/SSJ International Conference on Inielligent Robots and Systems, pp. 64-69, IEEE, 2007.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang. et al. Offline pre-trained multi-agen agent decision transformer. Machine Intelligence Research, 20(2):33 228222282 2023.
13

---

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances neural information processing systems, 32, 2019.
Anusha Nagahandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. 2018 IEEE international conference on robotics and automation (ICRA), pp. 7559-7566. IEEE, 2018.
Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference Machine Learning, pp. 663-670 2000.
Frans A Olichock and Christopher Amato. concise introduction decentralized POMDPs. Springer, 2016.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-352. -22289933,,200 2008
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog. Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint 10.08664,, 2023.
Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when it is confident: Masked model-based actor-critic. Advances in neural information processing systems, 33:10537-100566 2020.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforeement learning with actor rectification. In International conference on machine learning, pp. 17221- 17237. PMLR, 2022.
Emilio Parisotto and Russ Salakhutdinov. Efficient transformers in reinforcement learning using actor-learner distillation. In International Conference on Learning Representations, 2021.
Emilio Parisotto. Francis Song, Jack Rae. Razvan Pascanu, Caglar Gulcehre. Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487 7487-7498. PMLR, 2020.
Tim Pearce. Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann. et al. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4295-4304. PMLR, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 13:10199--1222 2020.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. generalist agent. arXiv preprini TrXiv:2205.061777 2022.
Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolow, and Volkan Cevher. Identifiability and generaliz- ability from multiple experts in inverse reinforcement learning. Advances in Neural Information Processing Systems, 55:500-564, 2022.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakoh Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint /rxiiv//902 042.04 2019.
Julian Schritwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604660) 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, arXiv preprint 17070777770
14

---

Jianzhun Shao, Yun Qu Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative learning for offline multi-agent reinforcement learning. Advances in Neural Information P Processing Systems, 36, 2024.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panncershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-499, 2016.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtram: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 1887-5896, PMLR, 2019.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks: Learning generalizable representations for visuomotor control. In International conference on machine learning, pp. 4732 4741. PMLR, 2018.
Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czamecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-dccompositioon networks for cooper- ative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085-2087, 2018.
Richard S Sutton. Integrated architectures for learning, planning. and reacting based on approximating dynamic programming. Machine learning proceedings 1990, pp. 216-224, Elsevier, 1990.
Richard S Sutton, Rupam Mahmood, and Martha White. An emphatic approach the problem of off off-policy temporal-difeee difference learning. Journal of Machine Learning Research, 17(73):1-29, 2016.
Gokul Swamy, Siddharth Reddy, Sergey Levine, and Anca D Dragan. Scaled autonomy: Enabling human operators to control rohot flects. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 19942-594... IEEE, 2020.
Yu Takagi and Shinji Nishimoto. High- High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14453 14463, 2023.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for bosel-based control. 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026 5033. IEEE, 2012.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-learning. In International Conference on Learning Representations, 2020a.
Tonghan Wang. Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. In International Conference on Learning Representations, 2019.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.0888 200399 2020h.
Tonghan Wang. Tarun Gupta, Anuj Mahajan. Bei Peng. Shimon Whiteson, and Chongjie Zhang, Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523. 2020c.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offfine multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Syxtems, 36, 2024.
Xiyao Wang. Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. Coplanner: Plan to roll out conservatively but to explore optimistically for model-based rl. In The Twelfth International Conference on Learning Representations, 2023.
Ermo Wei and Sean Luke. Lenient learning in independent- leanner stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2914-295 2016.
15

---

Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning sequence modeling problem. arXiv preprint urXiv:2205. 14953, 2022.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin
Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 11319-11322 PMLR, 2021.
Yuch-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In A. Oh, T. Neumann, A. Globerson. K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 59008 59033.
Curran Associates, Inc., 2023.
Shusheng Xu, Huaijie Wang, and Yi Wu. Grounded reinforcement learning: Learning to win the game under human commands. Advances in Neural Information Processing Systems, 35:7504-7511 7519, 2022.
Sherry Yang. Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson. Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024.
Yaodong Yang. Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen. Yujing Hu, Changjie Fan, and Zhongyu Wei. Q-value path decomposition for deep multiagent reinforcement learning. In International Conference on Muchine Learning, pp. 10706-10715, PMLR, 2020a.
Yaodong Yang. Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03923 2020b.
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for oftiine multi-agent reinforcement learning. Advances in Newral Information Processing Systems, 34: (0299-10911) 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo in cooperative multi agent games, Advances Neural Information Processing Systems, 35:24611 12-22411-24624, 2022.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, pp. 7194-729100 PMLR, 2019.
Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin. Ju Ren, and Junshan Zhang. Clare: Conservative model-based reward learning offline inverse reinforcement learning. The Eleventh International Conference on Learning Representations, 2022.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626 13640, 2021.
Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum likellhood inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 35:10122 10135, 2022.
Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He. Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34, 2021.
Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35-12, pp. 11116-11124, 2021.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43000 PMLR, 2023.
Tianchen Zhu, Yue Qiu, Haoyi Zhou, and Jianxin Li. Decoding global preferences: Temporal and cooperative dependency modeling in multi-agent preference preference-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38-15 15, pp. 17202-17210, 2024.
16

---

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8. pp. 1438. Chicago, IL. USA, 2008.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 2019.
17

---

A Broader Impacts and Limitations
A.1 Broader Impacts
Learning before Interaction provides grounded answers to complex multi-agent rreision-makingg problems through the generation of simulators and trial-and-erro learning. This can benefit those seeking to make decisions through long-term planning. With significant technological advancements, exploring the use of this technology may be crucial for enhancing existing human decision-making capabilities. For instance, negotiators could describe the opponent's 's personality traits and their decision-making limits to generate better negotiation strategies.
At the same time, we recognize that current generative simulators still cannot reliably generate state transitions across multiple domains, and learning joint multi-agent strategies still faces convergence difficulties. Therefore, Learning before Interaction may lead to incorrect decisions in specific fields. If humans intentionally follow the generated answers instead of using them as references, it could lead to unsafe or worse consequences. On the other hand, could also have negative impacts when Learning before Interaction is misused in harmful applications if the generated environments and answers are sufficiently accurate.
A.2 Limitations
Although we have already seen significant improvements in reasoning capabilities for complex multi- agent tasks with Learning before Interaction, performance may be affected by the simulator's 's accuracy and the multi-agent policy learning performance. Unqualified simulators and difficult-to-converge multi-agent policies may lead erroneous simulation results, which could be more misleading than the vague answers generated by existing visual language models. For example, the world model has limited out-of-domain generalization for domains that are not represented in the training data, e.g., unseen unit types. Further scaling up training data could help, as the parser can quickly and automatically generate images based on given state.
While the learned reward functions can enhance the speed of multi-agent policy learning compared to other inverse reinforcement learning and online interaction learning methods, it still requires considerable waiting time to obtain a converged policy and the final answer. Such long waiting time unacceptable in applications requiring real-time feedback, such as chatbots. One possible solution is to replace multi-agent reinforcement learning with planning methods based on the learned rewards and dynamics models, thereby accelerating the reasoning process. We will leave this issue in future work.
In addition, this paper is confined to scenarios within the game StarCraft II. This is an environment that, while complex, cannot represent the dynamics of multi-agent tasks. Evaluation of multi-agent reinforcement learning algorithms, therefore, should not be limited to one benchmark but should target a variety with a range tasks.
Map Name
Return Distribution
Map Name
Return Distribution
3s5z Ic3s5z
19.43 + 1.86
5m_vs vs__mm 6m
19.83 = 2.16
19.66 +\: 1.25
6h_ vs 8z
18.84 = 2.09
10m_vs mim_vs_lim 11
19.75 + 1.03
3s5z_vs vs__vs__362
19.76 = 1.26
2c _______6e 64zg
19.98 + 0.71
corridor MMM2
19.69 = 1.48
3s vs _s_5z
19.88 + 1.40
19.63 = 2.07
Table 6: Return distribution on training maps.
B Dataset Preparation
The training maps include 3s5z, 1c3s5z, 10m_vs_ 1mm 2c_vs_647g_ 3s_vs__5 5z, 5m vs_6m, 6h_vs__ vs_ 8z, 3s57_ vs vs_3s67, corridor, MMM2 in StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019). We use EMC (Zheng et al. (02T) and IE (Liu al., 2024) to collect 50000 trajectories for cach map and save these data as NPY files. The data includes the states, the observations, the
18

---

terminated signals, the actions, the available actions, and the rewards. The return distribution on training maps is shown in 6 The average return 19.64 1.63 across ten training maps.
As shown Figure 5] we collect the element images that appear in the game and affect the state, including units and background terrains of training maps. In Figure we have presented the whole procedure of converting a state vector into an image for simulation and parsing a trajectory to produce textual task description.
Space Square
Jungle
Space Octagon
Redstone
Space Bridge
Space Terrain
Unit name
Ally
Enemy
Unit name
Ally
Enemy
Clolossus)
(talker))
H(ydralisk)
Z(ealot)
M(aine)
Zerglling)
Medliaac)
Marfauder))
Figure 5: Images of units and terrains.
and
unile siatt aco dbeiti and t.. dfilti mevitio
scall),
3 Place agents at corresponding positions
also 2w1. Heede ailu Meell alce Mer.eetiie ovet)
mostte indutteent, Sadiu an Lecket atoti ettar sutt wiur eitert EuPti) nilt hedtrooii atllle staelio witeate 1000,
agents' health bars
Load the map unit
Image in the original replay (SC2REPLAY)
Generated image by the parser
Task description
walth thenteer otfh
(Win) Cons ider that we control fnumber of agents) legent races on the left. What plan should completaly destroy (number enemies) (enemy racejon on right?
aigur state www.ea......... (owuuk))
neny davuie the openni unen all
(Win) Let's Let's assume we are managing (humber agents) (agent races) on the What method should we take to fully eiminate the (number of enemies] Jenemy race) on the right?
alla heatth sait on ewny with saao
leading juumber agents) fagent rn frumber of enemies) tenemy race]. What approach should we take to achieve total points enemies
Imagine
against
remaining enemy health of (he sum the remaining the sum the remaining hea points agents }
2 Read the state information
@ Generate the task description
Positions and remaining health points
The remaining health points at the last state
Figure 6: The whole pipeline of how the parser generates the image and the task description for given state. Here, we only show three task descriptions the parser produces for demo purposes.
19

---

C StarCraft Multi-agent Challenge
StarCraft real-time strategy game featuring three different races, Protoss, Terran, and Zerg. with different properties and associated strategies. The objective is to build an army powerful enough to destroy the enemy's ' base. When battling two armies, players must ensure army units are acting optimally. StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et 2019) is partially observable reinforcement learning benchmark built in StarCraft II. An individual agent with parameter sharing controls each allied unit, and a hand-coded built-in StarCraft II Al controls enemy units. The difficulty of the game Al is set to the "very difficult" level.
On the SMAC benchmark, agents can access their local observations within the field of view at each time step. The feature vector contains attributes of both allied and enemy units: distance, relative x, relative y, health, shield, and unit_type. In addition, agents can observe the last actions of allied units and the terrain features surrounding them. The global state vector includes the coordinates of all agents relative to the center of the map and other features present in the local observation of agents. The state stores the energy of Medivaes, the cooldown of the rest of the allied units, and the last actions of all agents. Note that the global state information is only available to agents during centralized training. All features in state and local observations are normalized by their maximum values. After receiving the observations, each agent is allowed to take action from discrete set which consists of move [direction], attack [enemy_ _id], stop and no-op- Move direction includes north, south, east, and west. Note that the dead agents can only take no-op action while live agents cannot. For health units, Medivacs use heal [agent_id] actions instead of attack [enemy_ise _id]
Depending on different scenarios, the maximum number of actions varies between 7 and 70. Note that agents can only perform the attack [enemy_ _i]] action when the enemy is within its shooting range. At each time step, agents take joint action and receive a positive global reward based on the total damage dealt to the enemy units. In addition, they can receive an extra reward of 10 points after killing each enemy unit and 200 points after killing all enemy units. The rewards are scaled to around 20, so the maximum cumulative reward is achievable in each scenario.
D Experiment Setting
In this section, we describe the ground-truth environment that agents interact, the implementa- tion details of online learning methods, offline learning methods, and our model Learning before Interaction.
D.1
Online Learning
We adopt the same architectures for QMIX. QPLEX, CW-QMIX RODE 2 MAVEN 3
EMC their official implementations (Samvelyan et al., 2019 Wang et al. 2020a Rashid et al. 2020: Wang et al.
2020c Mahajan et al. 2019: Zheng al. (2021). Each agent independently learns policy with fully shared parameters between all policies. We used RMSProp with a learning rate of 5e-4 and y = 0.99, buffer size 5000, and mini-batch size 32 for all algorithms. The dimension of each agent's 's GRU
hidden state is set to 64.
For our experiments, we employ an e-greedy exploration scheme for the joint policy, where e decreases from to 0.05 over million timesteps in 6h_ ft__s__2z _s_8z, 3s5z_vs vs _as_______6__ 3s62 and corridor, and over 50 thousand timesteps in other maps. The implementation of MAPPO is consistent with their official repositories Yu et al. 2022). As shown in Table 7 all hyperparameters are left unchanged at the origin best-perlorming status. For CW-QMIX, the weight for negative samples is set to 0.5 for all scenarios.
https:/gjithb.commovwwwtt///p//
3 http::/gggttuucccom/uuuuuaaaaant
20

---

Hyperparameter
Value
Hyperparameter
Value
critic Ir
5c-4
actor Ir
5e-4
ppo epoch
5
ppo-clip
0.2
optimizer
Adam
batch size
3200
optim eps
le-5
hidden layer
gain
0.01
training threads
32
rollout threads
8
0.99
hidden layer dim
64
activation
ReLU
Table 7: Hyper-parameters in MAPPO.
All figures in online learning experiments plotted using mean and standard deviation with confidence internal 95%. We conduct five independent runs with different random seeds for each learning curve.
D.2 Offline Learning
We adopt the same architectures for MA-AIRI
MADT MAPT ICC OMAR
and OMIGA
as their official implementations (Yu et al. 2019 Meng et al. 2023 Zhu ct al.
2024
Fujimoto et al.
2019 Kumar et al., 2020. Yang al.) 2021 an et al. 2022; Wang et al.
2024). We implement
MA- MA-TREX, BCQ-MA and CQL-MA based on TREX (Brown et al. 2019). BCQ (Fujimoto et al.
2019), and CQL (Kumar et al. 2020), respectively. In particular, add the task description into
MADT's target sequence because it deprecates the reward-to-go term.
D.3
Learning before Interaction
We train image tokenizer for 100k steps using the AdamW optimizer, with cosine decay, using the hyperparameters Table The batch size 32, and the learning rate le-4.
Component
Hyperparameter
Value
Encoder
num Jayers num_res res_layers
5e-4
num num_chamm channels
(256,256)
num res channels
(256,256)
downsample
(2,4,1,1)
Decoder
num layers
5e-4
num _en__res____ layers
num num_chame channels
(256,256)
num_ num_res_ _ss_nnensels
(256,256)
upsample
(2,4,1,1,0)
Codebook
num num_codes latent_dii
256
32
commitment_com _ost
0.25
Table 8: Hyper-parameters in VQ-VAE.
We build our dynamics model implementation based Decision Transforme Chen et al. 2021)
The complete list of hyperparameters can be found in Table 9 The dynamics models were trained using the AdamW W optimizer.
Prapeeerrrrrr-r----eere--e-- trained-Multe Maae-MMMMliitttttt Multi-Aggen-Deeisiion Decision- Transformer
htttss//gtthhb.ccm/eeee////AA
21

---

Hyperparameter
Value
Hyperparameter
Value
number of layers
grad norm clip
1.0
attention heads
8
weight decay
0.1
embedding dims
64
Adam betas
(0.9.0.95)
Table 9: Hyperparameters in the transformer model.
The reward shares the same architecture the dynamics model, but the attention mask in the transformer model is modified in order to receive the whole trajectory as input rather than the tokens that have come before the current one. Here are some tricks for reward learning: (1) we control the gap between the rewards of the expert behavior and the policy action we stop the gradient for the reward of the expert behavior a given state if it is greater than the one of the policy action, where beta is the margin and set to 2: (2) we also set the target of unavailable actions' rewards to O; (3) we alternate between k-step of policy update and reward update to avoid completely solving the policy optimization subproblem before updating the reward parameters, where k = 5.
In this paper, all experiments are implemented with Pytorch and executed eight NVIDIA A800 GPUs.
E
Additional Results
Using Text-to-Code Converter can generate scenarios with the original game engine and then learn
the joint policy. Therefore, we also consider the comparison with online MARL methods including
CW-QMIX (Rashid et al., 2020), QPLEX (Wang et al.) 2020a). MAVEN (Mahajan et al.) 2019)).
Zheng et al. 2021), J. RODE (Wang et al. (020c), QMIX (Rashid et al. 2018), MAPPO (Yu ct al. 2022). Figure 7 demonstrates significant improvement in the sample efficiency of LBI
EMC
compared to the online MARL methods, suggesting that a pre-trained world model necessary to
reduce the waiting time for generating grounded answers for multi-agent decision decision-making making problems.
LBI
CW-QMIX
OPLEX
EMC
MAVEN
RODE
QMIX
MAPPO
(b) 8h_
(c) 2g 642g
(d) va,
100
100
lest Win Kate Kate
Text Win Rate Rate %
80
8 8
Teat Winn
80
80
Win
40
Teat Win
80
40
20
20
20
20
0
0
0.5M
1.5M
2M
1M
2M Timeshapa
3M
4M
SM
0.5M 1M
1.5M
2M
1M
2M 3M Timestepe
4M
SM
Timestaps
Timestepe
(e) 5m, ve. 6m
(0) 3a, vs 52
(g) 10m, va_ 11m
(h) contido
lest Win Kate %
1000
Test Win Rate %
100
Test Whh Rate %
100
Test Wh Rate %
100
80
8 8
8 &
80
60
80
40
8 &
&
8
20
3
8
0
0.5M
1M
1.5M
2M
0.5M
1M Timestaps
1.5M
2M
0.5M
1M Timestaps
1.5M
2M
2M 3M
4M
5M
Timestaps
Timestops
Figure 7: Performance comparisons between online learning methods using ground-truth rewards on the SMAC benchmark and LBI using the learned reward functions on the imagined world model.
In addition, we also show the qualitative comparison between the target and the generated sequences in Figure Both trajectories are collected by running the same policy. We can see that the generated sequence can resemble the target one in most frames, but some differences exist in positions and health bars. However, compounding errors in the single-step model, which lead to physically implausible predictions, are not observed in the dynamics model generated by the causal transformer. For example, at the timestep of 10 in the MMM2 scenario, the generated frame does not contain the ally's Medivac, but we can see it in the following frames.
22

---

Target (3s, vs. 5z)
t=0
t=5
t=10
t=15
t=20
1=25
t=30
t=35
t=
Generated (3s,
5z)
t=0
t=5
t-10
b-15
t-20
t-
t-30
b-35
t=40
Target (MMM2)
t=10
t=15
t=20
t-25
t=0
Generated (MMM2)
t=0
h=5
t=10
t=15
1=20
1-25
t=30
h=35
h=40
Target (5m vs_ 6m)
t=6
[=9
t=12
[-21
1-24
Generated (5m 6m)
t=-
t=3
b=
t=12
1=15
t=21
t-24
Figure 8: Comparisons of the target and the generated sequences across three different maps.
Additional Related Work
Transformer Model Several works have explored the integration of transformer models into reinforcement learning (RI.) settings. We classify them into two major categories depending on the usage pattern. The first category focuses on representing components in RL algorithms, such as policies and value functions (Parisotto et al., 2020)] Parisotto & Salakhutdinoy 2021). These methods rely on standard RL algorithms update policy, where the transtormer only provides large representation capacity and improves feature extraction. Conversely, the second category aims to replace the RL pipeline with sequence modeling. They autoregressively generate states, actions, and rewards by conditioning on the desired return-to- during inference (Chen et al., [222] Lee et al. 2022, Reed al. 2022). Due its simplicity and potential generalization ability, this category is widely used in various domains, such as robotics control (Brohan et al. 2023a Padalkar et al. 2023
Driess al. 2023) and multi-agent reinforcement learning (Meng al. 2023 Liu et al. 2024).
Multi-agent Reinforcement Learning This section briefly introduces recent related work on
cooperative multi-age agent reinforcement learning (MARL). In the paradigm centralized training with decentralized execution (CTDE), agents' policies are trained with access to global information in centralized way and executed only based on local histories in a decentralized way (Oliehoek et al.) 2008, Kraemer& & Banerjee 2016). One of the most significant challenges in CTDE is ensure correspondence between the individual Q-value functions and the joint Q-value function Qtot- i... the Individual-Globa. Max (IGM) principle (Son et al. 2019). VDN (Sunehag et al., 2018
23

---

and QMIX (Rashid et al. 2018) learn the joint Q-values and factorize them into individual Q-vauue functions an additive and monotomic fashion, respectively. Several works (Yang et al. 2020b aj Wang al. 2020b have been proposed to improve the performance of QMIX, but as many previous studies pointed out, monotonic value function factorization limits the representational capacity Qto and fails to learn the optimal policy when the target Q-vauue functions are non-monotonic (Mahajan ict al. 2019; Son et al., 2019; Rashid et al. 2020). To solve this problem, some recent works (Wang et al. 2020a; Mahajan et al. 2021 try to achieve the full representational capacity of Qtot, while others prioritize the potential optimal joint action and learn a biased Qtot.
Some independent learning algorithms have also proven robust in solving multi-agen cooperative tasks. Distributed Q-learning (Lauer (0000) and Hysteretic Q-learning (Matignon al. (007) place more importance on positive updates that increase a Q-value estimate, which is similar to the weighting function in WQMIX. However, Wei Luke (20T6) prove that these methods are vulnerable towards misleading stochasticity and propose LMRL2, where agents forgive the other's s miscoordination the initial exploration phase but become less lenient when the visitation of state- action pair increases. MAPPO (Yu et al., 2022) applies PPO (Schulman et al.) 2017) into MARL and shows strong empirical performance. However, Kuba al. (222)) points out MAPPO suffers from instability arising from the non-stationarity induced by simultaneously learning and exploring agents. Therefore, they introduce the sequential policy update scheme to achieve monotonic improvement on the joint policy.
Learning communication protocols solve cooperative tasks is one the desired emergent behaviors of agent interactions. It has recently become active area in MARL,, such as learning to share Liu al. 2020) and intentions (Kim al. 2020
observations (Das et al. 2019 Wang et al. 2019
Böhmer et al. 2020 Wen et al. 2022 Liu et al. 2023).
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction include the contributions made in the paper. See Section for more information.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Appendix A.2 such limited
out-of-domain generalization and considerable cost time.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Ye]]
Justification: We describe the steps taken to construct the dataset in Appendix B and the implementation details of our model and baselines in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We choose not release the data and code present. We would like have the opportunity to further engage with the research community and to ensure that any future such releases respectful, safe, and responsible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..om
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
y /nips. cttplssccpppbbccguuccieeecoo httppp/lll/giiiieeddddeeeaaaaaia www.oeebaiieeiioooool..oooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification:
We provide the training and test details in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct five independent runs with different random seeds for each result, The results are accompanied by standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yss]
Justification:
We provide the information on the computer resources in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:::////eurttt newrips....../. wwwturisscccpuulcc//eeec cc/puublc/Ethiccsuidedlllll
Answer: [Yes]
Justification:
We conduct the research with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts in Appendix A.I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We does not use a pre-trained model (which may generate unsafe images), and we construct the image dataset through parser. See Appendix for more information.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original paper and provide the URLs for the assets in Appendix D
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We describe the steps taken to construct the dataset in Appendix [ and the implementation details of our model and baselines in Appendix [ However, We choose not release the data and code present. We would like have the opportunity to further engage with the research community and to ensure that any future such releases are respectful, safe and responsible.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "KPFJ2rWTjM", "reviewer": "Reviewer_Frvn", "summary": "The paper presents a novel approach called Learning before Interaction (LBI) that integrates a language-guided simulator into the multi-agent reinforcement learning process to enhance the quality of solutions for complex decision-making problems. The LBI paradigm uses a world model comprising a dynamics model and a reward model to simulate interactions and improve policy learning. The authors introduce new offline MARL datasets for the StarCraft Multi-Agent Challenge and demonstrate LBI's effectiveness through notable performance on training and unseen tasks.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Endeavors in building a feasible task-guided multi-agent world model. By introducing task description containing thorough information and the image of states of the task, this work empirically manage to build a world model being adaptive to various tasks in the training data distribution. \n\n2. Designs for separate reward model and dynamics model. By decoupling the prediction of future observations and the prediction of rewards, this work can learn an efficient reward function involving reasonable credit assignments by adopting the inverse RL.", "weaknesses": "1. Trivial framework designed for multi-agent world model. Overall, the method proposed in this work seems not to capture the distinct characteristics in the context of multi-agent scenarios as compared to single-agent scenarios. The built world model appears to be based on reduce multi-agent problems into a single-agent problem with joint observations and joint actions. What are the benefits of this approach? Why is it necessary to adopt this simplification? The article does not provide specific evidence or thorough analysis to support these architecture design decisions.\n\n2. Lack of the investigation and citation of related multi-agent world model literature. Given that the method proposed in this work derives the optimal policy via learning in the imaginations of the world model, it is essential to elucidate the differences between this approach and those presented in [1, 2, 3, 4] and to conduct performance comparisons. Notably, the architecture closely resembles that of [1] in the absence of reward prediction. However, the entire paper lacks citations and clarifications regarding these similarities.\n\n3. Unfair comparisons. According to the contents in the appendix, the average returns are almost 20 in all offline dataset of ten maps, which are nearly optimal in SMAC environments since the maximum return of an episode is 20 in almost all SMAC maps. Thus with a pretrained world model which has full access to the optimal demonstrations, the comparison with online methods is definitely not meaningful and convincing at all.\n\n4. Heavily handcrafted processing. “Since it is possible for specific agents to become inactive before the game terminates, we mark the terminated timestep for each agent and enemy once its predicted health is less than zero …” To identify the predicted health, the algorithm has implicitly already known the specific meaning of the value in some certain dimensions in the observation, which is obviously induced from the prior knowledge or predefined rules from human beings. Unfortunately, such an operation makes this work only narrowed in the domain of SMAC.\n\n[1] Micheli, Vincent, Eloi Alonso, and François Fleuret. \"Transformers are Sample-Efficient World Models.\" The Eleventh International Conference on Learning Representations.\n\n[2] Egorov, Vladimir, and Alexei Shpilman. \"Scalable Multi-Agent Model-Based Reinforcement Learning.\" Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.\n\n[3] Zhang, Yang, et al. \"Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models.\" arXiv preprint arXiv:2406.15836 (2024).\n\n[4] Liu, Qihan, et al. \"Efficient Multi-agent Reinforcement Learning by Planning.\" The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1&2. Trivial framework and lack of investigation of related multi-agent world model literature.**\n\nIf I understand correctly, the distinct characteristic of multi-agent scenarios you mentioned should be decentralized execution, which is a common way to simplify computational demands during the execution and thus enhance the scalability in practical applications. However, centralized training is generally required during the training process because agents learn the joint policy in a non-stationary environment. For example, QMIX [1] uses monotonic value decomposition to learn the joint Q-value function, and HAPPO [2] proposes sequential learning and multi-agent advantage decomposition. Otherwise, using individual Q-learning could lead the algorithm into local optima (unless we use some independent learning methods like lenient learning or design individual reward functions). \n\nThe most significant difference is that our aim is to pre-train a world model and utilize it to train the policy model rather than directly using it as a part of the policy model. \n\n* [3] is an image-based single-agent world model, while ours is a multi-modal multi-agent world model. This dynamics model structure is general and is also used in GAIA, DT, TT, and Genie, which we mentioned in the related work on model-based and Transformer methods. \n\n* [4][5][6] are online learning methods that use the world model to generate imagined future trajectories at a given state for planning or better decision-making. Our goal is to learn an interactive simulator from offline datasets.\n\n* [3][4][5][6] suppose the reward is given, and [4][5][6] enable communication during execution to add necessary inputs for the agents. We suppose the reward is unknown and use inverse RL to learn the reward, and one can make use of different MARL methods (e.g., model-based or model-free, value-based or policy-based) to learn the policy on the proposed world model.\n\nFor the world model, since we do not use it in the policy execution but treat it as a simulator – the learned policy can still be decentralized. To provide enough information for the input, we train directly using joint behaviors and observations. \n\nThanks for pointing out these works. We would like to add them to the World Model and Multi-agent Reinforcement Learning subsection.\n\n**3. Unfair comparisons.**\n\nFirst, the pre-trained world model is an interactive simulator, and we compare it with the real environment, not the online methods themselves. This experiment aims to demonstrate that training a joint policy on the learned world model is faster than training it in the real environment - the learned world model (interactive simulator), specifically the learned transition and reward functions, is accurate. Designing a multi-agent environment, precisely the reward function, is complex and requires much prior. Our framework shows that utilizing the world model (a causal transformer for dynamics and a bidirectional transformer for reward) can successfully perform inverse reinforcement learning and learn meaningful reward functions.\n\nSecond, this paper mainly compares offline Q-learning and inverse learning methods, demonstrating their generalization ability to unseen tasks. The comparison with online MARL is only briefly mentioned in the main text, with detailed results provided in the appendix. \n\n**4. Heavily handcrafted processing - the agent death masks.**\n\nThis design aims to simplify the inference difficulty of algorithms, and it seems task-specific. However, it could be a general trick for multi-agent environments with attrition, and it can also be applied to scenarios like MOBA or RTS games. For example, MAPPO [7] also uses such death masking for better value estimation.\n\nIn addition, developing such tricks might be a promising direction in generative models. For example, the outputs of large models are not always reliable, such as generating movements that do not satisfy physical constraints. Introducing human priors into scene prediction results could be an essential way to solve these issues. Methods like PINNs and Lagrangian conservation can impose constraints on certain predictions, thereby enhancing the reliability of the generated results. The tricks specific to the SMAC game in this paper can also be categorized as one of these methods.\n\n[1] Tabish Rashid, et al. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning.\n\n[2] Jakub Grudzien Kuba, et al. Trust region policy optimisation in multi-agent reinforcement learning. \n\n[3] Micheli, Vincent, Eloi Alonso, and François Fleuret. Transformers are Sample-Efficient World Models.\n\n[4] Egorov, Vladimir, and Alexei Shpilman. Scalable Multi-Agent Model-Based Reinforcement Learning.\n\n[5] Zhang, Yang, et al. Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models.\n\n[6] Liu, Qihan, et al. Efficient Multi-agent Reinforcement Learning by Planning.\n\n[7] Chao Yu, et al. The surprising effectiveness of ppo in cooperative multi-agent games.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7oXcn1e1cQ", "reviewer": "Reviewer_sAvh", "summary": "This paper introduces a pipeline called Learning before Interaction, a approach to improve generative models' answers for multi-agent decision-making problems. The key idea is to integrate a language-guided simulator into the multi-agent reinforcement learning pipeline.\n\nThe main contributions are \n- Creating VisionSMAC, a dataset that pairs states with images and task descriptions for the StarCraft Multi-Agent Challenge\n- Training an interactive simulator consisting of a dynamics model and a reward model\n\nThe pipeline use the simulator to train a multi-agent policy through simulated interactions and generate answers by running the trained policy on the simulator.", "questions": "- What is the world model architecture and how do you train it?\n- Could you please write an algorithmic block for both inference and training? I found there are still many processes unclear after reading the paper. It will be very helpful if you can upload a sample code (please be careful about the anonymity in the code).", "limitations": "The authors do not mention a limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- LBI outperforms existing offline and imitation learning methods on unseen tasks in the SMAC. (good zero-shot generalization ability)\n- The approach produces explainable reward functions at critical states, which can help in understanding the decision-making process", "weaknesses": "- (Not mentioned in the paper) Training an interactive simulator with both dynamics and reward models likely requires significant computational resources\n- It would be great if there are environments other than SMAC tested. Otherwise, the scope might be too narrow.\n- The approach requires a substantial amount of offline data to train the world model and compared with online MARL methods which is not fair comparison. The offline data is also not always available.\n- The ablation study shows that using image reference doesn't significantly improve performance, suggesting that this aspect of the model might not be fully utilized or optimized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. It requires significant computational resources**\n\nWe listed the hyperparameters in the interactive simulator in Tables 8 and 9, which requires computational resources. However, we believe this framework is meaningful. We utilize the generative model as an interactive environment for RL to enhance the quality of large models' responses to decision-making problems. RL requires significant interaction with the environment. Since the world model is implemented entirely through neural networks, distributed data collection is easier than with the original game engine and does not require communication with the game engine (which is the main reason for the slow interaction in the SMAC environment). When transferred to a real-world task, distributed data collection without interacting with the real-world environment offers higher speed and safety than online learning. \n\n**2. The scope might be too narrow.**\n\nDue to the substantial computational resources required to train MARL methods, especially in most cooperative MARL environments that lack image inputs and have extremely low sampling efficiency, we have conducted experiments only on SMAC, which we have mentioned in the limitations section. \n\nAlthough SMAC cannot represent all possible multi-agent tasks, as a highly complex multi-agent reinforcement learning environment, SMAC involves heterogeneous agents, varying numbers and tactics, and different maps, making the learning process inherently challenging. This complexity can reflect the proposed algorithm's potential applications.\n\n**3. Unfair comparisons.**\n\nFirst, the pre-trained world model is an interactive simulator, and we compare it with the real environment, not the online methods themselves. This experiment aims to demonstrate that training a joint policy on the learned world model is faster than training it in the real environment - the learned world model (interactive simulator), specifically the learned transition and reward functions, is accurate. \n\nSecond, this paper mainly compares offline Q-learning and inverse learning methods, demonstrating their generalization ability to unseen tasks. \n\n**4. The effectiveness of image reference.**\n\nUsing images as a reference can improve performance (LBI and LBI-wo-IR), although less significant than the residual term (LBI and LBI-wo-RT). However, we demonstrated that the error and return when using real images (LBI-GTI) are similar to LBI, indicating that the benefit of image reference is limited rather than suggesting that the model is not optimized. The visual results show that the generated images are similar to the real ones. \n\nIt is important to note that the model uses images and text as initial inputs and then performs autoregressive prediction, making images necessary for this task.\n\n**5. The world model architecture.**\n\nThe world model consists of a VQ-VAE model for the image tokenizer, a casual transformer model for dynamics prediction, and a bi-directional transformer model for reward prediction. We build the VQ-VAE and the transformer model implementation based on (https://github.com/Project-MONAI/GenerativeModels/blob/main/generative/networks/nets/vqvae.py) and Decision Transformer (which borrows the code from minGPT https://github.com/karpathy/minGPT), the publicly available re-implementation of VQ-VAE and GPT2, respectively. The dynamics and reward models share the same transformer architecture, where the reward model does not use causal masks. We mentioned it in Appendix D.3.\n\n**6. Traning and inference.**\n\nThe model is trained in two phases following a standard autoregressive video generation pipeline: We train the image tokenizer first, which is used for the dynamics and reward models. We then train the dynamics model and reward model separately.\n\n*VQ-VAE*: (1) sample a batch of images; (2) calculate the reconstruction and the quantization error; (3) optimize the VQ-VAE model.\n\n*Dynamics model*: (1) sample a batch of trajectories from the offline buffer; (2) calculate the output for each token; (3) optimize the dynamics model by minimizing the cross-entropy loss for actions and mean-squared error for others.\n\n*Reward model*: \n\n* Randomly initialize a MA-SAC policy network\n\n* While True do:\n\n    + Use MA-SAC to collect a batch of trajectories $B$ from the ground-truth environment\n\n    + Sample a batch of trajectories $D$ from the offline buffer\n\n    + Mask out rewards of the trajectories $B$ and $D$\n\n    + Generate the reward via the reward model\n\n    + Update the reward model by maximizing the predicted reward for $D$, and minimizing the predicted rewards for $B$; \n\n    + Update MA-SAC policy network for k-step (k=5).\n\n*Inference*:\n\n* Randomly initialize a policy network\n\n    + While True do:\n\n        * Given an initialized image and a task description, calculate the image embedding via the VQ-VAE model and the language embedding via a pre-trained language tokenizer named bert-base-uncased\n\n        * While all agents or all enemies are alive do\n\n            + Generate state via dynamics model\n\n            + Generate joint observations via dynamics model\n\n            + Generate actions (only for the behavior-regularization) via the dynamics model\n\n            + Obtain joint actions (for dynamics prediction) via current policy network (using $\\epsilon$-greedy)\n\n            + Generate the next image embedding\n\n            + Concatenate the language embedding\n \n        * Add masks into the reward-free trajectory for the reward prediction\n\n        * The reward model takes the entire trajectory as input and generates corresponding rewards\n\n        * Store the whole trajectory with actions for the behavior regularization in replay buffer\n\n        * If the trajectories in replay buffer > batch size then\n\n            + Sample a random batch of trajectories\n\n            + Calculate the TD loss with the behavior regularization for the policy network\n\n            + Update policy network", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5LohXD89T", "reviewer": "Reviewer_2fu7", "summary": "This paper proposes a method of improving the sample efficiency and performance of multi-agent reinforcement learning (RL) on the StarCraft Multi-Agent Challenge (SMAC) benchmark by learning to simulate the environment. The simulator consists of an image tokenizer, a dynamics model, and a reward function, all trained on data collected from interactions with the SMAC environment. A multi-agent policy then interacts with the simulator to obtain trajectories. These trajectories are then used to optimise the policy, but with a behaviour regularisation term towards the simulated trajectories. Evaluations on the SMAC benchmark show improved win rates and returns over existing online and offline multi-agent RL baselines. The paper also ablates the various components of the proposed method and provides visualisation of the simulated trajectories and their close match with actual trajectories.", "questions": "In the input notation of the dynamics model why is L_t time-varying across a trajectory? From the text L is constant across an entire trajectory, could the input contain more than 1 trajectory? The notation could be improved.\n\nCould you explain in more detail how hindsight-relabelling is done and why it is necessary for reward function training? Is it the same or different from the text label used on the same set of collected trajectories as in dynamics model training? Is it only for data collected from the simulator?\n\nClarifications for equation (1): What is D? Why does the f term have \\tau as an input and an index i? Where do they come from? Is there a missing sum(s)? In the description it says that term is “over all possible actions”, what does this mean? Its inputs are trajectories and aren’t guaranteed to contain all actions?\n\nWhy did the authors choose to only focus on a single multi-agent domain and not evaluate the method on any other, even toy, multi-agent benchmarks?\n\nWhy do some tables report win rate and some rewards? \n\nFigure 8, does LBI timesteps account for the number of time steps in the data collected to train the simulator components? Would it be more accurate to shift the LBI curve to the right by that amount?", "limitations": "Limitations have been adequately addressed.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Proposes a method of training multi-agent RL using a learned simulator. It carefully considers issues such as data collection (by creating a method of generating images from stored game state), image tokenization, using inverse RL to learn the reward function, and regularising the policy update. As a result they achieve quite impressive performance gains in SMAC when compared with prior work. \n\nThe paper thoroughly ablates the different components of their method and shows that while somewhat complex, each component is necessary for the best performance.", "weaknesses": "The paper only studies SMAC and no other multi-agent domain. It is hard to evaluate the generality of the method without other evaluations on standard benchmarks. Adding evaluation results from one or two more domains would help strengthen the generality of this method. \n\nThe write-up is a bit difficult to follow and contains a number of confusing parts (see also questions). The main figure, Figure 3, is never discussed in the main text. The part of Figure 1 on GPT-4 is not necessary for the paper. \n\nA significant amount of detail is located in the Appendix, for example Appendix B on how the training data for the simulation components are collected, and Appendix E on the experimental details. Consider moving these sections into the main text.\n\nThe paper does not explain the methods used to collect the training data (EMC and IIE). To be better self contained, please give short explanations of these methods in the main text. Are they particularly tuned to the SMAC environment? What would have to change in domains where an effective exploration strategy is unknown?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The exploration methods (EMC and IIE) for offline data collection**\n\nEMC is curiosity-based to boost multi-agent exploration for novel states, i.e., reduce out-of-distribution regions in the offline datasets. IIE belongs to go-explore algorithms that perform imagination without exploration, initialize agents at the end state of the imagination, and then start to explore. This method can reduce the multi-agent exploration space in the time dimension. We chose them to collect data because they have tuned in to the SMAC benchmark and have performed well in super hard maps.\n\nWhen collecting offline datasets in other MARL tasks, we must choose appropriate exploration methods based on the task properties. If the task involves many decision steps (long-horizon task), go-explore methods can be more effective; if the action space of each agent is huge, we can perform role-based learning (RODE) to decompose the action space; if we want to cover the entire state space, we can use curiosity-based exploration methods (EMC) or hierarchical exploration (MAVEN). Since the joint action and state space are huge in SMAC, we chose EMC to reduce the impact from OOD regions and used IIE to reduce exploration space.\n\n**2. $L_t$ in the dynamics model**\n\nThanks for pointing it out. The input only contains one target sequence. In general, $L_t$ should reflect the task and the environment, which can be time-varying. The language description for the trajectory from $s_t$ and the terminated (goal) state $s_T$, e.g., return-to-go). However, in practice, we simplify this language description to be time-invariant, which is always the language description of terminated (goal) state $s_T$. We have corrected this for better understanding.\n\n**3. Hindsight-relabelling in the reward model**\n\nWe use the same text label for the reward and dynamics models to specify the trajectory and the reward generation. We use hindsight learning to provide a specific condition for reward generation to avoid the influence of suboptimal data in the offline dataset. Without this condition, the reward model would learn to approximate all possible trajectories in the dataset. Due to the presence of suboptimal trajectories, this would negatively impact the reward model's learning. We train the reward model using conditional generation to learn the reward functions corresponding to different outcomes. During inference, we set a condition, such as asking the agents to completely eliminate the enemy, to extract the reward function corresponding to a successful trajectory from the reward model.\n\nIn policy models, this type of condition is usually set as return-to-go (RTG) to achieve desired effects. In contrast, we use language descriptions to set the conditions, which allows for the generation of an interactive simulator and enables the model to have multimodal processing capabilities.\n\n**4. Only focus on the SMAC benchmark**\n\nDue to the substantial computational resources required to train MARL methods, especially in most cooperative MARL environments that lack image inputs and have extremely low sampling efficiency, we have conducted experiments only on SMAC, which we have mentioned in the limitations section.\n\nAlthough SMAC cannot represent all possible multi-agent tasks, as a highly complex multi-agent reinforcement learning environment, SMAC involves heterogeneous agents, varying numbers and tactics, and different maps, making the learning process inherently challenging. This complexity can reflect the proposed algorithm's potential applications.\n\n**5. Why do some tables report win rates and some rewards?**\n\nIn SMAC, a slight difference in return (19 vs. 20) can result in a significant difference in win rate (0% vs. 90%). If we only show the win rate, the results for most current offline methods can always be 0, making comparisons less meaningful. Using cumulative rewards as a metric is also reasonable because reinforcement learning inherently optimizes cumulative rewards.\n\n**6. Consider the number of time steps in the data collection.**\n\nFirst, the pre-trained world model is an interactive simulator, and we compare it with the real environment, not the online methods themselves. This experiment aims to demonstrate that training a randomly initialized joint policy on the *learned world model* is faster than training it in the *real environment* - the learned world model (interactive simulator), specifically the learned transition and reward functions, is accurate. So, we do not shift the LBI curve to the right by the number of time steps in the data collection.\n\n**7. Clarifications for equation (1).**\n\n$D$: the demonstrations provided by an expert policy.\n\nThe index $i$ of $\\tau$ in the $f$ term: Thanks for pointing this out. We perform this regularization on the predicted rewards from the expert demonstrations and the data collected by the MA-SAC policy. Namely, in the $f$ term, $\\tau\\sim \\\\{\\boldsymbol{\\pi}^\\theta,D\\\\}$, $i$ and $t$ should be added in the agent and the timestep dimensions.\n\nOver all possible actions: This is an implenmentation trick. In practice, the reward model $RM$ takes the action-free trajectory and outputs the rewards for all actions of each agent. Namely, the individual reward head outputs $|U|$ rewards for each agent, where $|U|$ is the action space. Then, we choose the reward corresponding to the action using the torch.gather function. We add the L2 penalization for these $|U|$-dim predicted rewards to avoid overestimation on out-of-distribution actions. For example, if the offline dataset does not contain the action $u_t^i$, then the predicted reward of $u_t^i$ does not have a target, introducing potential risks for policy learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1MEPLpssHn", "reviewer": "Reviewer_yaXk", "summary": "The paper proposes a new approach to multi-agent decision-making problems by integrating a language-guided simulator into the reinforcement learning pipeline. \n\nLearning before Interaction (LBI) uses a world model to generate trial-and-error experiences and improve the answers to complex decision-making problems. \n\nThe world model consists of a dynamics model and a reward model, which are learned from expert demonstrations and task descriptions. \n\nThe LBI framework is evaluated on the StarCraft Multi-Agent Challenge benchmark and shows superior performance on both training and unseen tasks, generating consistent and explainable interaction sequences and rewards.", "questions": "See weaknesses", "limitations": "no negative societal impact", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed VisionSMAC dataset is a significant contribution, as it provides a large-scale, diverse, and realistic dataset for multi-agent decision-making problems in the StarCraft II environment.\nThe dataset is constructed by collecting offline trajectories using multi-agent exploration methods and then converting the state into images and languages through a parser, making it easy to create new content.\n\nLBI is a novel approach that integrates a language-guided simulator into the reinforcement learning pipeline, enabling the generation of trial-and-error experiences and improving the answers to complex decision-making problems.\n\n the LBI framework and the VisionSMAC dataset provide a powerful tool for learning and reasoning in complex multi-agent decision-making problems.", "weaknesses": "The motivation for mentioning and comparing the answer of GPT-4, at the beginning, seems misleading. I thought that the proposed LBI would benefit the general response of LLMs on complex decision-making. However, it seems that the proposed method heavily relied on the gaming environment, and and result is limited to SMAC benchmark.\n\nThe dataset collection is a crucial part of developing a successful LBI, especially for the reward prediction part. However, it must be constructed via a world model (1) where the dynamic is explicit (2) simulation speed is fast enough to perform learning and interaction. How the generalizability of the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Motivation seems misleading, and results are limited to the SMAC benchmark.**\n\nWe aim to improve the output of large language models by constructing a differentiable interactive environment and then learning the policy on this interactive environment. The paper focuses on constructing such an environment, including the dynamics and reward models, and demonstrates the potential application of this framework.\n\nDue to the substantial computational resources required to train MARL methods, especially in most cooperative MARL environments that lack image inputs and have extremely low sampling efficiency, we have conducted experiments only on SMAC, which we have mentioned in the limitations section.\n\nAlthough SMAC cannot represent all possible multi-agent tasks, as a highly complex multi-agent reinforcement learning environment, SMAC involves heterogeneous agents, varying numbers and tactics, and different maps, making the learning process inherently challenging. This complexity can reflect the proposed algorithm's potential applications.\n\nThe trick of masking out dead agents aims to simplify the inference difficulty of algorithms, and it seems task-specific. However, it could be a general trick for multi-agent environments with attrition, and it can also be applied to scenarios like MOBA or RTS games. In addition, developing such tricks might be a promising direction in generative models. For example, the outputs of large models are not always reliable, such as generating movements that do not satisfy physical constraints. Introducing human priors into scene prediction results could be an essential way to solve these issues. Methods like PINNs and Lagrangian conservation can impose constraints on certain predictions, thereby enhancing the reliability of the generated results. The tricks specific to the SMAC game in this paper can also be categorized as one of these methods.\n\n**2.  The generalizability of the proposed method.**\n\nThe proposed method has a certain generalization ability in multi-agent environments.\n\n* Regarding the *dynamic is explicit* issue, if we understand correctly, you mean that the input for the reward is the entire trajectory rather than (s, a), as in most previous methods. This is necessary because we aim to achieve temporal utility distribution across the entire trajectory, which cannot be done by relying solely on (s, a), thus failing to address issues like delayed reward. During inference, we first run the dynamics model to obtain the entire reward-free trajectory and then run the reward model to label the rewards so the entire trajectory is known.  \nIf you mean that dynamics might not be deterministic, such as taking the same action in a given state leading to different following states and rewards, this is not considered in this paper. This is because most multi-agent reinforcement learning environments, including SMAC, Google Research Football, and Multi-agent Particle World, are deterministic. To address this, we may modify the loss function by introducing heteroscedastic aleatoric uncertainty estimation - we can model the predicted state and reward as distributions and estimate the mean and variance of both state and reward.\n\n* Concerning the issue of *simulation speed being fast enough for learning and interaction*, in our world model, dynamics is an auto-regressive generation process corresponding to interacting with the real environment (e.g., the function of env.step()). This is time-consuming and a common issue in reinforcement learning algorithms. The reward is predicted based on a bidirectional Transformer by inputting the entire trajectory, which is less time-consuming than the auto-regressive dynamics model. Additionally, since the world model is implemented entirely through neural networks, distributed data collection is easier than with the original game engine and does not require communication with the game engine (which is the main reason for the slow interaction in the SMAC environment). When transferred to a real environment, distributed data collection without interacting with the real environment offers higher speed and safety than online learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the LBI framework, which employs a world model consisting of dynamics and reward components to simulate interactions and enhance policy learning. A multi-agent policy interacts with this simulator to generate trajectories. These trajectories are subsequently utilized to optimize the policy.

**strengths**

* Novel idea of integrating a language-guided simulator into RL pipeline

* VisionSMAC: a large-scale, diverse, and realistic dataset for multi-agent decision-making problems in the StarCraft II environment

* Extensive experiments and analysis on different components

**weaknesses/suggestions**

* Discussions on computational cost

* Limited domain (only SMAC) 

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version.
Decision: Accept (poster)