Grounded Answers for Multi-agent Decision-making Problem through Generative World Model

Authors: zeyang  liu
xinrui  yang
long  qian
xingyu  chen
xuguang  lan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:54:46

Abstract: Recent progress in generative models has stimulated significant innovations in many fields, such as image generation and chatbots. Despite their success, these models often produce sketchy and misleading solutions for complex multi-agent decision-making problems because they miss the trial-and-error experience and reasoning as humans. To address this limitation, we explore a paradigm that integrates a language-guided simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer. The simulator is a world model that separately learns dynamics and reward, where the dynamics model comprises an image tokenizer as well as a causal transformer to generate interaction transitions autoregressively, and the reward model is a bidirectional transformer learned by maximizing the likelihood of trajectories in the expert demonstrations under language guidance. Given an image of the current state and the task description, we use the world model to train the joint policy and produce the image sequence as the answer by running the converged policy on the dynamics model. The empirical results demonstrate that this framework can improve the answers for multi-agent decision-making problems by showing superior performance on the training and unseen tasks of the StarCraft Multi-Agent Challenge benchmark. In particular, it can generate consistent interaction sequences and explainable reward functions at interaction states, opening the path for training generative models of the future.

Main paper: 
 introduction: 
 Recent progress in generative artificial intelligence with models capable of generating creative content has shown attractive prospects for real-world applications, such as image generation (Takagi & Nishimoto, 2023) , embodied agents (Brohan et al., 2023b) , and chatbots (Köpf et al., 2024) . Most generative models attempt to directly obtain the answer by training on natural language or image datasets and inserting decomposed reasoning steps in few-shot demonstrations. However, these methods do not experience firsthand the situations described by the language and the image. They Figure 1 : Complex decision problems that require a good understanding of the environment's dynamics and the objective are still challenging for current vision-language models, e.g., the answer elicited by GPT-4 is sketchy and misleading. Instead, Learning before Interaction (LBI) enables grounded reasoning by simulating the task in the given question. LBI utilizes the simulator to train a MARL policy and generate the answer by running the converged policy on the simulator. cannot find the correct answers through trial and error as humans, which is necessary to ground reasoning on complicated problems and transfer learned knowledge to unfamiliar domains. For example, as shown in Figure 1 , when asked a complex multi-agent decision problem, one of the most widely-used large language models, GPT4 -though achieving superhuman performance in many reasoning tasks -will generate sketchy and misleading answers.To tackle this problem, we can utilize the generative models to understand the properties of the task that the user describes and simulate the effects of the actions. We can derive the answer with a highly realistic simulator by experiment-reasoning or training any machine intelligence from simulated experience. The origin of this idea can be traced back to Dyna architecture (Sutton, 1990) and has spawned a series of model-based reinforcement learning (MBRL) theories and methods (Janner et al., 2019; Kaiser et al., 2019; Lai et al., 2020) . Inspired by this, Mind's Eye (Liu et al., 2022) enables language models to perform reasoning conditioned on the simulation results by running the corresponding experiment on a computational physics engine named MuJoCo (Todorov et al., 2012) . Mind's Eye can boost reasoning performance in zero-shot and few-shot settings by infusing such physical knowledge into language models. However, it is particularly designed for physical reasoning rather than decision-making problems.In contrast, UniSim (Yang et al., 2024) formulates the action-in-video-out framework as an observation prediction diffusion model conditioned on finite history. It shows that the simulator learned from broad data can generalize to the real world and bridge the sim-to-real gap. Genie (Bruce et al., 2024) enables users to act in the generated environments on a frame-by-frame basis, opening the path for training generalist agents of the future. Notably, most of the existing breakthroughs on learning in the imagined experience have been focusing on single-agent scenarios and leave the world model largely unstudied for multi-agent reinforcement learning (MARL) tasks -it is common in real-world applications that multiple agents are required to solve a task in a coordinated fashion.The roadblocks to building a simulator for MARL tasks are twofold. First, MARL tasks involve multiple entities' attributes, e.g., positions and roles, making it difficult to describe a state using only text. The text and image information can be brought together to enrich the inputs for the simulator, but such a dataset is limited in quantity. Second, the dynamics and reward models of the MARL environment are more intricate than the single-agent setting. Current approaches assume the reward is known in the dataset (Meng et al., 2023) or can be easily deduced by the frame information (Yang et al., 2024) , which could be challenging for MARL methods due to the abundance of agents' tactics and the compositional nature of their functionalities. This work explores a paradigm that adds language-guided simulation to the MARL pipeline to make policy learning grounded within the learned world model. First, we propose new offline MARL datasets to provide paired state-image information for the StarCraft Multi-Agent Challenge (SMAC) environment by transforming the state in the trajectory to the corresponding image. We also designed a parser to convert each trajectory to a task description using natural language. Then, we pre-train a vector quantized variational autoencoder (VQ-VAE) (Van Den Oord et al., 2017) to generate discrete representations for each frame. The world model is formulated as an interactive simulator that consists of a dynamics and a reward model. The dynamics model comprises an image tokenizer and a causal transformer to generate interaction transitions autoregressively. The reward model is a bidirectional transformer learned by maximizing the likelihood of trajectories in the expert demonstrations under the task description.Given a decision-making problem by the user and an image from the environment, we store the simulated interaction trajectories into a replay buffer by running an off-policy MARL algorithm on the generated dynamics model. Then, we utilize the generated reward model to label the reward for each state-action pair based on the whole trajectory. We update the policy network according to the reward with a behavior-regularization term, which serves as the conservatism for out-of-distribution state-action pairs. We use the image sequence generated by the interaction of the dynamics model and the converged policy model as the answer to the decision-making problem.We summarize the main contributions of this paper in three folds: (1) It proposes novel MARL datasets for SMAC, where a parser automatically generates the ground-truth image of a given state and task description. (2) It introduces Learning before Interaction (LBI), an interactive simulator that generates trial-and-error experiences and improves the answers for multi-agent decision-making problems.(3) The empirical results show that LBI outperforms various offline learning methods on training and unseen MARL tasks. The visualization also indicates that LBI can produce consistent imagined trajectories and explainable rewards for interaction states.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction include the contributions made in the paper. See Section 1 for more information.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 Decentralized Partially Observable Markov Decision Process. A fully cooperative multi-agent task in the partially observable setting can be formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016), consisting of a tuple G = A, S, Ω, O, U, P, r, γ , where a ∈ A ≡ {1, . . . , n} is a set of agents, S is a set of states, and Ω is a set of joint observations. At each time step, each agent obtains its observation o ∈ Ω based on the observation function O (s, a) : S × A → Ω, and an action-observation history τ a ∈ T ≡ (Ω × U ) * . Each agent a chooses an action u a ∈ U by a stochastic policy π a (u a |τ a ) : T × U → [0, 1], which forms a joint action u. It results in a joint reward r(s, u) and a transit to the next state s ∼ P (•|s, u). The formal objective function is to find the joint policy π that maximizes a joint action-value functionQ π (s t , u t ) = r(s t , u t ) + γE s [V π (s )], where V π (s) = E [ ∞ t=0 γ t r t |s 0 = s, π], and γ ∈ [0, 1) is a discounted factor.Inverse Reinforcement Learning. Suppose we do not have access to the ground truth reward function but have demonstrations D provided by an expert policy π E . Imitation learning aims to directly learn policies that behave similarly to these demonstrations, whereas inverse reinforcement learning (IRL) seeks to infer the underlying reward functions which induce the expert policies. The MaxEnt IRL framework (Ziebart et al., 2008) aims to recover a reward function that rationalizes the expert behaviors with the least commitment, denoted as IRL(π E ):EQUATIONwhereH(π) = E π [-log π(u|s)]is the entropy of current policy π. It looks for a reward function that assigns a high reward to the expert policy and a low reward to the current policy π while searching for the best policy for the reward function in the inner loop.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of this work in Appendix A.2, such as limited out-of-domain generalization and considerable cost time.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 We formulate an interaction simulator as a transition prediction model that, given some state of the world and descriptions of the task, can take some actions as input and produce the consequence of the actions in the form of images, states, and rewards. In this paper, we consider building such simulators for a multi-agent decision-making environment named StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) , known for its rich environments and high control complexity. More information about SMAC can be found in Appendix C. The SMAC benchmark saves a replay of an episode as a SC2REPLAY file rather than providing the image feature during exploration. It is computationally expensive to construct datasets of images by watching such replay within the StarCraft II client and then subsampling a frame that captures meaningful actions. To solve this problem, we introduce VisionSMAC to convert the state into images and languages through a parser f , decoupled from StarCraft, making it easy to create new content.First, we collect offline datasets across ten training maps in the SMAC benchmark by running multiagent exploration methods named EMC (Zheng et al., 2021) and IIE (Liu et al., 2024) . Each dataset contains a large number of interaction trajectories:EQUATIONwhere s t denotes the state, {o a t } n a=1 is the observation of each agent, {u a t } n a=1 is the joint action, and the done signal d a t = 1 when the agent a is killed at timestep t, n and T denote the number of agents and the length of the episode, respectively. We further collect the element images that appear in the game and affect the state, such as the units and the background terrain of training maps.Given a multi-agent system and its interaction trajectory, the parser f reads predefined map information, such as the number and races of agents and enemies. Then, the parser converts the original state information into structured information, reading agents' and enemies' positions and health points. It will generate the corresponding interaction scenes by placing each unit image and its health bar at their positions with the corresponding background terrain. The image generated by the parser can resemble the frame subsampled from the original replay by running a SC2REPLAY file within the StarCraft II client, where the comparisons can be found in Appendix B. We also perform data augmentation to enable better feature abstraction by changing the background to different terrains.Finally, we define a task description L to specify the environment and the task. The task description can be the terminated state, a slice of a trajectory, or any other representation of the episode. In this paper, we use the terrain information, the number and unit types of agents and enemies, and the sum of enemies' remaining health points at the terminated state as the task description. To this end, the parser reads the last state of the trajectory and extracts the remaining health points of both sides. We can obtain the practical task description by filling in predefined description templates (e.g., "Consider that we control {number of agents} {agent races} on the left.") and adding connecting words (e.g., "What plan should we use"). The detailed description of the datasets and the parser f can be found in Appendix B.With trajectories with corresponding images and language guidance from different scenarios, we can formulate the interactions with StarCraft II as interacting with an interactive simulator. The simulator contains three key components: (1) an image tokenizer that converts raw video frame into discrete tokens, (2) a dynamics model that predicts the next frame and state given past frame and state tokens, (3) a reward model that infers the reward of a state-action pair given a trajectory. The idea behind decomposing the world model into a dynamics model and a reward model is to reuse the dynamics model for different tasks by combining it with any reward function. Image Tokenizer We compress images into discrete tokens to reduce dimensionality and enable higher-quality image generation. We make use of vector quantized variational autoencoder (VQ-VAE) (Van Den Oord et al., 2017) , which takes every single image of the state as input, generating discrete representations. The tokenizer is trained using a standard VQ-VAE objective.The dynamics model is a causal transformer q parameterized by ψ, where the target sequence has the following form x = {..., L, z t , s t , o 1 t , ..., o n t , u 1 t , ..., u n t , z t+1 , s t+1 , ...}, where z t is the image representation generated by the fixed image tokenizer. We utilize the task description L to specify the dynamics of the environment, remaining consistent in one sequence. An embedding for each timestep is learned and added to each token. The dynamics model processes all past tokens and predicts future tokens via autoregressive modeling.Then, we use the prediction heads to decode the predicted tokens to the corresponding element in the sequence and train them by minimizing the cross-entropy loss for actions and mean-squared error for others. The actions would serve as the reference policy for the learning with the simulated trajectories described in Section 3.3. In particular, we use a dynamics residual term to improve the accuracy and the stability of the generation by changing the target from s t+1 to ∆s t+1 = s t+1 -s t for the state prediction head. We also apply this term to predict image representations. In addition, since the observation is only related to the current state and the vision range of the agents, we filter out the historical memories and use s t as the input for the observation prediction.We resemble the training pipeline of inverse reinforcement learning -maximizing the likelihood of trajectories in the expert demonstrations while minimizing the likelihood of trajectories collected from an inner-loop policy. We introduce a reward function r, which receives the entire trajectory as inputs to perform credit assignment under deterministic dynamics within the trajectory. We remake this formulation as a generalized version of the conventional reward design; if the reward function is Markovian, the temporal dependencies on other state-action pairs should always be zero.To this end, we model the reward function as a bidirectional transformer model parameterized by φ, where the sequence is x = {..., s t , L, u 1 t , ..., u n t , rt , s t+1 , ...}, and rt = {r a t } N a=1 is a set of individual rewards for the agents. Again, we utilize the task description L to perform hindsight relabeling, which converts imperfect trajectories into possible solutions for reaching the last state s T of the episode, generating the expert demonstration D. We optimize the reward function by minimizing the following loss:EQUATIONwhere π θ = {π a (u a |s; θ)} N a=1 is the inner-loop MA-SAC policy parameterized by θ, and we use the reward constraint by imposing an L2 penalization of the predicted rewards over all possible actions, which can be viewed as a conservative update for out-of-distribution state-action pairs. In particular, we alternate between k-step of policy update and k-step of reward update to avoid completely solving the policy optimization subproblem before updating the reward parameters, where k is a given update iteration.We now describe how to generate grounded answers for multi-agent decision-making problems via LBI. Given an image of the initial state and a task description from the user, the agents interact with the dynamics model using a randomly initialized off-policy MARL algorithm to collect reward-free trajectories in an autoregressive manner. Then, the reward model predicts the immediate reward for each transition pair in the simulation trajectories. These relabeled trajectories are added to the replay buffer, serving as the training data for the policy network.In practice, we construct the MARL problem in the simulator as a behavior-regularized MDP by imposing a behavior-regularization term:EQUATIONwhere π = {π i (u a t |o a t ; η)} n a=1 is the joint policy, and q(u a t |x <u a t ; ψ) is the reference policy provided by the dynamics model. The last term enables in-sample learning and further mitigate the impact of exploring OOD regions in the state-action space. We use independent Q-learning to train the parameter-sharing agent network.Since it is possible for specific agents to become inactive before the game terminates, we mark the terminated timestep for each agent and enemy once its predicted health is less than zero and then use zero vectors as the subsequent actions and observations. It can mitigate the hallucinating unrealistic outcomes -a dead agent performs a "moving" action. We also mask the predicted reward after the terminated timestep for the inactive agent to get a more accurate value estimate.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We choose not to release the data and code at present. We would like to have the opportunity to further engage with the research community and to ensure that any future such releases are respectful, safe, and responsible.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 World Models. There is a long-standing history of learning predictive models of the world. We list three categories of model-based reinforcement learning (MBRL) according to the type of model usage. The first category applies planning methods with world model simulation. AlphaGo (Silver et al., 2016) and MuZero (Schrittwieser et al., 2020 ) learn a transition model and apply Monte Carlo Tree Search to search for an action sequence with the highest accumulated rewards. By contrast, MBMF (Nagabandi et al., 2018) , PETS (Chua et al., 2018) , and PlaNet (Hafner et al., 2019b) integrate model predictive control (MPC) into the learned world model and sample high-reward action sequences. TD-MPC (Hansen et al., 2022) and TD-MPC2 (Hansen et al., 2024) utilize value functions to bootstrap the trajectories used for MPC planning.The second category models a differentiable world model and utilizes the internal structure of the model to facilitate policy learning. GPS (Levine & Koltun, 2013) and GDP (Srinivas et al., 2018) perform differential planning and obtain the analytic form of the optimal policy. SVG (Heess et al., 2015) re-parameterizes the policy and the world model, then computes the policy gradient estimate by backpropagation via the world model. MAAC (Clavera et al., 2019) , Dreamer (Hafner et al., 2019a) and its subsequent variants (Hafner et al., 2020 (Hafner et al., , 2023) ) use the recurrent state-space model in PlaNet to learn the world model in a compact latent space and learn the policy entirely within this space.The last one utilizes the learned world model to generate more experiences and then trains a policy on the dataset augmented by the model, also known as Dyna-style methods (Sutton, 1990) . MVE (Feinberg et al., 2018) and STEVE (Buckman et al., 2018) depict a learned world model to calculate multi-step temporal-difference prediction for better value estimation. In contrast, SLBO (Luo et al., 2018) , MBPO (Janner et al., 2019) , and BMPO (Lai et al., 2020) theoretically analyze this learning framework and prove that the policy performance will improve monotonically in a world model with certain model bias and rollout length. To further increase the rollout length and avoid compounding errors, M2AC (Pan et al., 2020) and COPlanner (Wang et al., 2023) compute the uncertainty of each rollout step and perform conservative model rollouts by discarding the samples with high uncertainty or adding a penalty term into total reward. In practice, GAIA-1 (Hu et al., 2023) , UniSim (Yang et al., 2024) , and Genie (Bruce et al., 2024) show that the learned world model can enable the control policy to generalize to the real world when trained purely in simulation and bridge the sim-to-real gap. These methods have impressive performance and theoretical bounds, attracting much research interest in the MBRL community. However, they focus on generating videos or trajectories that only involve one single agent instead of building a multi-agent simulator that can be used to further improve decision-making performance on coordination tasks in our work.Imitation Learning. Imitation Learning (Bain & Sammut, 1995) formulates imitating an expert as a supervised learning problem, which has been widely adopted in various domains due to its simplicity and effectiveness (Silver et al., 2016; Swamy et al., 2020) . GAIL (Ho & Ermon, 2016) and its extensions (Song et al., 2018; Ghasemipour et al., 2020) stand as a cornerstone approach, which trains a generator policy to imitate expert behaviors and a discriminator to distinguish between the expert and the learner's state-action pair distributions. In light of the recent interest in foundational models, the conditional diffusion model is used to represent and learn an imitation learning policy, which produces a predicted action conditioning on a state and a sampled noise vector Pearce et al. (2022) ; Chi et al. (2023) . These methods achieve encouraging results in modeling stochastic and multimodal behaviors from human experts or play data. DT-style methods (Chen et al., 2021; Wu et al., 2024) formulate the trajectory generation as a sequence modeling problem, which generates states, actions, and rewards by conditioning on a return-to-go token in an autoregressive manner.In contrast, inverse reinforcement learning (IRL) is designed to infer the reward function that underlies the expert demonstrations, taking into account the temporal structure and showing better generalization than direct Behavioral Cloning (Ng & Russell, 2000; Ross et al., 2011; Barde et al., 2020) . A main class of algorithms, Maximum entropy (MaxEnt) IRL (Haarnoja et al., 2017) and its extensions (Liu et al., 2021; Rolland et al., 2022) , learn a stationary reward by minimizing divergence between the agent and expert distribution. Since the learned reward function can solve downstream tasks and transfer behavior across different dynamics, IRL is also helpful in several broader applications, e.g., IRL with natural language goals (Fu et al., 2018a; Zhou & Small, 2021; Xu et al., 2022) , and RL with human feedback (Ziegler et al., 2019; Zhu et al., 2023; Wu et al., 2023) , and dynamics learning (Luo et al., 2023) . Furthermore, a series of sample-efficient algorithms are proposed to solve the MaxEnt IRL formulation (Fu et al., 2018b; Zeng et al., 2022 Zeng et al., , 2024)) . To side-step the expensive online environmental interactions in classic IRL, some work aims to learn a reward function from a static dataset by a variational Bayesian framework (Chan & van der Schaar, 2021), representing reward function via a learned soft Q-function (Garg et al., 2021) , or incorporating conservatism into the estimated reward like offline Q-learning (Yue et al., 2022) . The major bottleneck for these methods includes a lack of knowledge of the dynamics information and the reward overestimation for out-of-distribution state-action pairs. We formulate the reward model as a bidirectional transformer to receive the whole trajectory as the input, making it possible to solve non-Markovian rewards. In addition, we leverage the reward constraint and the behavior regularization to perform in-sample learning to avoid reward overestimation. The amount of expert demonstrations in these existing studies is also limited, as they do not treat hindsight relabeling via the textual description as an expert trajectory like in our work.Offline Q-Learning. Offline Q-learning learns a policy from a fixed dataset where the reward is provided for each transition sample. Most off-policy reinforcement learning (RL) algorithms are applicable in offline Q-learning. However, they typically suffer from the overestimation problem of out-of-distribution (OOD) actions due to the distribution shift between the action distribution in the training dataset and that induced by the learned policy (Fujimoto et al., 2019) . Several constraint methods are proposed to restrict the learned policy from producing OOD actions by leveraging importance sampling (Sutton et al., 2016; Nachum et al., 2019) , incorporating explicit policy constraints (Kostrikov et al., 2021; Fakoor et al., 2021; Fujimoto & Gu, 2021; Tarasov et al., 2024) , penalizing value estimates (Kumar et al., 2020; An et al., 2021; Shao et al., 2024) , and uncertainty quantification (Wu et al., 2021; Zanette et al., 2021) . Another branch resorts to learning without querying OOD actions and thus constrain the learning process within the support of the dataset (Bai et al., 2021; Lyu et al., 2022) . 
experiments: 
 In this section, we conduct empirical experiments to answer the following questions: (1) Is Learning before Interaction (LBI) better than the existing multi-agent reinforcement learning (MARL) methods in complex cooperative scenarios? (2) Can LBI generate long-horizon trajectories and reasonable reward functions at critical states? (3) Does LBI have the zero-shot ability to generalize to unseen tasks? Then, we investigate the contribution of each component in the dynamics and the reward model. We provide the information of training datasets and experimental settings in Appendix B and D. We also discuss this paper's broader impacts and limitations in Appendix A.1 and A.2.Reward-free Offline Learning We compare LBI with the following imitation learning baselines:(1) BC: behavior cloning that imitates the whole datasets, (2) MA-AIRL (Yu et al., 2019) : using adversarial learning to perform policy imitation, (3) MADT (Meng et al., 2023) : utilizing the Decision Transformer (Chen et al., 2021) to perform sequence modeling, (4) MA-TREX: infering the reward according to ranked demonstrations, the multi-agent version of TREX (Brown et al., 2019) , ( 5) MAPT (Zhu et al., 2024) : infering the team rewards according to the preference return from a well-trained scripted teacher.As shown in Table 1 , LBI outperforms the baselines by a significant margin on various maps with different difficulty levels, indicating the importance and effectiveness of learning reward functions via the proposed world model. In contrast, BC and MA-AIRL fail to achieve success rates in most tasks because they imitate all past interaction sequences and cannot generalize and avoid sub-optimal solutions. MA-TREX and MAPT have plateaued in performance because they use the accumulated rewards and the preference deduced by the scripted teacher to specify the quality of the training data, respectively. MADT performs better than other baselines because Decision Transformer can be thought of as performing imitation learning on a subset of the data with a certain return.Offline MARL We also compare LBI with the existing offline MARL methods with ground-truth rewards from the StarCraft Multi-Agent Challenge (SMAC), including the multi-agent version of BCQ (Fujimoto et al., 2019) and CQL (Kumar et al., 2020) (namely BCQ-MA and CQL-MA), ICQ (Yang et al., 2021) , OMAR (Pan et al., 2022) , and OMIGA (Wang et al., 2024) . Table 2 shows that the performance of these offline MARL methods degrades dramatically with an increasing number of agents and is much lower than that of LBI. We hypothesize that the reasons for this gap are: (1) It is challenging and unnecessary to recover the Q-value based on the reward functions provided by SMAC (the hit-point damage dealt) because such reward design is inefficient for learning optimal policy. (2) These methods may introduce too much conservatism and affect the learning of the optimal policy, as the conservative update of the out-of-distribution (OOD) suboptimal policy that consists of some agents taking non-optimal actions and others taking optimal will inhibit the learning of the agents that take the optimal actions.Since zero-shot generalization ability is crucial for generating grounded answers for multi-agent decision-making problems, we also test LBI's ability to generalize to extensive unseen scenarios without retraining. Specifically, we evaluate our LBI and MADT on the ten unseen testing maps, varying agent numbers, action spaces, and levels of environment complexity. Table 3 shows that LBI consistently outperforms MADT in unseen scenarios by a large margin, successfully transferring knowledge to new tasks without requiring additional fine-tuning. It highlights that learning a reward function has better zero-shot generalization performance than simple policy adaptation.In this section, we conduct ablation studies to analyze the contributions of each component in the dynamics model and the reward model across five evaluation runs on four training maps (6h_vs_8z, 3s5z_vs_3s6z, corridor, and MMM2) and four unseen maps (3s5z_vs_3s7z, 1c3s7z, 3s4z, 1c_vs_32zg). We show the results of the dynamics model in Table 4 . Using the dynamics residual term is necessary to reduce the prediction error of the subsequent states and obtain good performance across all training and unseen tasks. The image reference is not so effective, even if we use ground-truth images as the reference. However, since images are more powerful in representing some situations than language or state information, we believe that the image serves as another modality to correct the prediction of the state. We would leave it for future work.We demonstrate the ablation results of the reward model in Table 5 . Compared with LBI-wo-RC&BR, the reward constraint and behavior regularization term can improve the overall performance on the training tasks. However, LBI-wo-BR performs better than LBI-wo-RC on unseen tasks, suggesting that the conservatism for reward is more important than the policy when OOD state-action pairs exist. The poor performance of LBI-w-GTR indicates that learning rewards from conditioned demonstrations may be more accessible and valuable for policy updates than reconstructing the pre-defined rewards by human experts.This section evaluates the dynamics model as a long-horizon policy-conditioned predictive model. Figure 4 showcases examples of length-40 image trajectories generated by the dynamics model, including MMM2, 3s_vs_5z, and 5m_vs_6m. We do not observe conspicuous compounding errors as the single-step prediction model does, highlighting that LBI has consistency and long-horizon generation ability. In the case of 5m_vs_6m, we present the following frames after taking one of the possible actions, showing that LBI can also perform action-controllable generation.We also investigate the reward prediction at a critical junction in the state-action space that can transit to various states and significantly influence the success rate on the 5m_vs_6m task. At the moment, the agents have to learn to micromanage leapfrogging to achieve good coordination. Specifically, Agent 1 has a low health point and must move backward to avoid the enemies focusing fire on it; otherwise, the enemies will eliminate Agent 1 immediately and weaken our scarce forces. In Figure 4 , we visualize the learned reward function of Agent 1, where the action space is no-operation, stopping, moving in cardinal directions, and selecting an enemy's identity to attack. The learned reward for moving to the left is much higher than the other actions, allowing one to learn the optimal joint policy quickly. The rewards provided by the SMAC benchmark do not show this property, where multiple Monte Carlo samples are required to find the correct policy by estimating the expected return. 
conclusion and future work: 
 We proposed Learning before Interaction (LBI), a novel paradigm that enables generative models to ground their answers for multi-agent decision-making problems with simulations between the world and the multi-agent system. We formulate an interactive simulator consisting of dynamics and reward models, given some states of the world and the task descriptions, generating the consequence of the actions in the form of images, states, and rewards. We hope the idea of including simulations in the reasoning will instigate broad interest in applying generative models to aid machine intelligence and decision-making.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide the training and test details in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We conduct five independent runs with different random seeds for each result.The results are accompanied by standard deviations.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide the information on the computer resources in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We conduct the research with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss both potential positive societal impacts and negative societal impacts in Appendix A.1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We does not use a pre-trained model (which may generate unsafe images), and we construct the image dataset through a parser. See Appendix B for more information.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper and provide the URLs for the assets in Appendix D.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We describe the steps taken to construct the dataset in Appendix B, and the implementation details of our model and baselines in Appendix D. However, We choose not to release the data and code at present. We would like to have the opportunity to further engage with the research community and to ensure that any future such releases are respectful, safe and responsible. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Broader ImpactsLearning before Interaction provides grounded answers to complex multi-agent decision-making problems through the generation of simulators and trial-and-error learning. This can benefit those seeking to make decisions through long-term planning. With significant technological advancements, exploring the use of this technology may be crucial for enhancing existing human decision-making capabilities. For instance, negotiators could describe the opponent's personality traits and their decision-making limits to generate better negotiation strategies.At the same time, we recognize that current generative simulators still cannot reliably generate state transitions across multiple domains, and learning joint multi-agent strategies still faces convergence difficulties. Therefore, Learning before Interaction may lead to incorrect decisions in specific fields.If humans intentionally follow the generated answers instead of using them as references, it could lead to unsafe or worse consequences. On the other hand, it could also have negative impacts when Learning before Interaction is misused in harmful applications if the generated environments and answers are sufficiently accurate.Although we have already seen significant improvements in reasoning capabilities for complex multiagent tasks with Learning before Interaction, performance may be affected by the simulator's accuracy and the multi-agent policy learning performance. Unqualified simulators and difficult-to-converge multi-agent policies may lead to erroneous simulation results, which could be more misleading than the vague answers generated by existing visual language models. For example, the world model has limited out-of-domain generalization for domains that are not represented in the training data, e.g., unseen unit types. Further scaling up training data could help, as the parser can quickly and automatically generate images based on a given state.While the learned reward functions can enhance the speed of multi-agent policy learning compared to other inverse reinforcement learning and online interaction learning methods, it still requires considerable waiting time to obtain a converged policy and the final answer. Such long waiting time is unacceptable in applications requiring real-time feedback, such as chatbots. One possible solution is to replace multi-agent reinforcement learning with planning methods based on the learned rewards and dynamics models, thereby accelerating the reasoning process. We will leave this issue in future work.In addition, this paper is confined to scenarios within the game StarCraft II. This is an environment that, while complex, cannot represent the dynamics of all multi-agent tasks. Evaluation of multi-agent reinforcement learning algorithms, therefore, should not be limited to one benchmark but should target a variety with a range of tasks. The training maps include 3s5z, 1c3s5z, 10m_vs_11m, 2c_vs_64zg, 3s_vs_5z, 5m_vs_6m, 6h_vs_8z, 3s5z_vs_3s6z, corridor, MMM2 in StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) . We use EMC (Zheng et al., 2021) and IIE (Liu et al., 2024) to collect 50000 trajectories for each map and save these data as NPY files. The data includes the states, the observations, the terminated signals, the actions, the available actions, and the rewards. The return distribution on training maps is shown in Table 6 . The average return is 19.64 ± 1.63 across ten training maps.As shown in Figure 5 , we collect the element images that appear in the game and affect the state, including units and background terrains of training maps. In Figure 6 , we have presented the whole procedure of converting a state vector into an image for simulation and parsing a trajectory to produce a textual task description. (Loss) Imagine we are leading {number of agents} {agent races} on the left to against {number of enemies} {enemy race}. What approach should we take to achieve a total remaining enemy health of {the sum of the remaining health points of enemies} and ours is {the sum of the remaining health points of agents}?⑤ Generate the task description The remaining health points at the last state StarCraft II is a real-time strategy game featuring three different races, Protoss, Terran, and Zerg, with different properties and associated strategies. The objective is to build an army powerful enough to destroy the enemy's base. When battling two armies, players must ensure army units are acting optimally. StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) is a partially observable reinforcement learning benchmark built in StarCraft II. An individual agent with parameter sharing controls each allied unit, and a hand-coded built-in StarCraft II AI controls enemy units. The difficulty of the game AI is set to the "very difficult" level.On the SMAC benchmark, agents can access their local observations within the field of view at each time step. The feature vector contains attributes of both allied and enemy units: distance, relative x, relative y, health, shield, and unit_type. In addition, agents can observe the last actions of allied units and the terrain features surrounding them. The global state vector includes the coordinates of all agents relative to the center of the map and other features present in the local observation of agents. The state stores the energy of Medivacs, the cooldown of the rest of the allied units, and the last actions of all agents. Note that the global state information is only available to agents during centralized training. All features in state and local observations are normalized by their maximum values. After receiving the observations, each agent is allowed to take action from a discrete set which consists of move[direction], attack[enemy_id], stop and no-op. Move direction includes north, south, east, and west. Note that the dead agents can only take no-op action while live agents cannot. For health units, Medivacs use heal[agent_id] actions instead of attack [enemy_id] .Depending on different scenarios, the maximum number of actions varies between 7 and 70. Note that agents can only perform the attack[enemy_id] action when the enemy is within its shooting range. At each time step, agents take joint action and receive a positive global reward based on the total damage dealt to the enemy units. In addition, they can receive an extra reward of 10 points after killing each enemy unit and 200 points after killing all enemy units. The rewards are scaled to around 20, so the maximum cumulative reward is achievable in each scenario.In this section, we describe the ground-truth environment that agents interact, the implementation details of online learning methods, offline learning methods, and our model Learning before Interaction.We adopt the same architectures for QMIX, QPLEX, CW-QMIX1 , RODE2 , MAVEN3 , EMC4 as their official implementations (Samvelyan et al., 2019; Wang et al., 2020a; Rashid et al., 2020; Wang et al., 2020c; Mahajan et al., 2019; Zheng et al., 2021) . Each agent independently learns a policy with fully shared parameters between all policies. We used RMSProp with a learning rate of 5e-4 and γ = 0.99, buffer size 5000, and mini-batch size 32 for all algorithms. The dimension of each agent's GRU hidden state is set to 64.For our experiments, we employ an -greedy exploration scheme for the joint policy, where decreases from 1 to 0.05 over 1 million timesteps in 6h_vs_8z, 3s5z_vs_3s6z and corridor, and over 50 thousand timesteps in other maps. The implementation of MAPPO is consistent with their official repositories5 (Yu et al., 2022) . As shown in Table 7 , all hyperparameters are left unchanged at the origin best-performing status. For CW-QMIX, the weight for negative samples is set to α = 0.5 for all scenarios. All figures in online learning experiments are plotted using mean and standard deviation with confidence internal 95%. We conduct five independent runs with different random seeds for each learning curve.We adopt the same architectures for MA-AIRL6 , MADT7 , MAPT8 , ICQ9 , OMAR10 , and OMIGA11 as their official implementations (Yu et al., 2019; Meng et al., 2023; Zhu et al., 2024; Fujimoto et al., 2019; Kumar et al., 2020; Yang et al., 2021; Pan et al., 2022; Wang et al., 2024) . We implement MA-TREX, BCQ-MA and CQL-MA based on TREX (Brown et al., 2019) , BCQ (Fujimoto et al., 2019) , and CQL (Kumar et al., 2020) , respectively. In particular, we add the task description into MADT's target sequence because it deprecates the reward-to-go term.We train our image tokenizer for 100k steps using the AdamW optimizer, with cosine decay, using the hyperparameters in Table 8 . The batch size is 32, and the learning rate is 1e-4.Hyperparameter We build our dynamics model implementation based on Decision Transformer12 (Chen et al., 2021) . The complete list of hyperparameters can be found in Table 9 . The dynamics models were trained using the AdamW optimizer.Value number of layers 6 grad norm clip 1.0 attention heads 8 weight decay 0.1 embedding dims 64 Adam betas (0.9,0.95) Table 9 : Hyperparameters in the transformer model.The reward shares the same architecture as the dynamics model, but the attention mask in the transformer model is modified in order to receive the whole trajectory as input rather than the tokens that have come before the current one. Here are some tricks for reward learning: (1) we control the gap between the rewards of the expert behavior and the policy action -we stop the gradient for the reward of the expert behavior at a given state if it is greater than the one of the policy action, where beta is the margin and set to 2; (2) we also set the target of unavailable actions' rewards to 0; (3) we alternate between k-step of policy update and reward update to avoid completely solving the policy optimization subproblem before updating the reward parameters, where k = 5.In this paper, all experiments are implemented with Pytorch and executed on eight NVIDIA A800 GPUs.Using a Text-to-Code Converter can generate scenarios with the original game engine and then learn the joint policy. Therefore, we also consider the comparison with online MARL methods including CW-QMIX (Rashid et al., 2020) , QPLEX (Wang et al., 2020a) , MAVEN (Mahajan et al., 2019) , EMC (Zheng et al., 2021) , RODE (Wang et al., 2020c) , QMIX (Rashid et al., 2018) , MAPPO (Yu et al., 2022) . Figure 7 demonstrates a significant improvement in the sample efficiency of LBI compared to the online MARL methods, suggesting that a pre-trained world model is necessary to reduce the waiting time for generating grounded answers for multi-agent decision-making problems. : Performance comparisons between online learning methods using ground-truth rewards on the SMAC benchmark and LBI using the learned reward functions on the imagined world model.0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH D000 0 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH EKBYVB] 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH FFBYVB]J 0 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH GV]BYVBV] 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH HPBYVBP 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH IVBYVB] 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH JPBYVBP 0 0 0 0 0 7LPHVWHSV 7HVW:LQ5DWH KFRUULGRU /%In addition, we also show the qualitative comparison between the target and the generated sequences in Figure 8 . Both trajectories are collected by running the same policy. We can see that the generated sequence can resemble the target one in most frames, but some differences exist in positions and health bars. However, compounding errors in the single-step model, which lead to physically implausible predictions, are not observed in the dynamics model generated by the causal transformer. For example, at the timestep of 10 in the MMM2 scenario, the generated frame does not contain the ally's Medivac, but we can see it in the following frames. Transformer Model Several works have explored the integration of transformer models into reinforcement learning (RL) settings. We classify them into two major categories depending on the usage pattern. The first category focuses on representing components in RL algorithms, such as policies and value functions (Parisotto et al., 2020; Parisotto & Salakhutdinov, 2021) . These methods rely on standard RL algorithms to update policy, where the transformer only provides a large representation capacity and improves feature extraction. Conversely, the second category aims to replace the RL pipeline with sequence modeling. They autoregressively generate states, actions, and rewards by conditioning on the desired return-to-go during inference (Chen et al., 2021; Lee et al., 2022; Reed et al., 2022) . Due to its simplicity and potential generalization ability, this category is widely used in various domains, such as robotics control (Brohan et al., 2023a; Padalkar et al., 2023; Driess et al., 2023) and multi-agent reinforcement learning (Meng et al., 2023; Liu et al., 2024) .Multi-agent Reinforcement Learning This section briefly introduces recent related work on cooperative multi-agent reinforcement learning (MARL). In the paradigm of centralized training with decentralized execution (CTDE), agents' policies are trained with access to global information in a centralized way and executed only based on local histories in a decentralized way (Oliehoek et al., 2008; Kraemer & Banerjee, 2016) . One of the most significant challenges in CTDE is to ensure the correspondence between the individual Q-value functions and the joint Q-value function Q tot , i.e., the Individual-Global Max (IGM) principle (Son et al., 2019) . VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) learn the joint Q-values and factorize them into individual Q-value functions in an additive and a monotonic fashion, respectively. Several works (Yang et al., 2020b,a; Wang et al., 2020b,c) have been proposed to improve the performance of QMIX, but as many previous studies pointed out, monotonic value function factorization limits the representational capacity of Q tot and fails to learn the optimal policy when the target Q-value functions are non-monotonic (Mahajan et al., 2019; Son et al., 2019; Rashid et al., 2020) . To solve this problem, some recent works (Wang et al., 2020a; Mahajan et al., 2021) try to achieve the full representational capacity of Q tot , while others prioritize the potential optimal joint action and learn a biased Q tot .Some independent learning algorithms have also proven robust in solving multi-agent cooperative tasks. Distributed Q-learning (Lauer, 2000) and Hysteretic Q-learning (Matignon et al., 2007) place more importance on positive updates that increase a Q-value estimate, which is similar to the weighting function in WQMIX. However, Wei & Luke (2016) prove that these methods are vulnerable towards misleading stochasticity and propose LMRL2, where agents forgive the other's miscoordination in the initial exploration phase but become less lenient when the visitation of stateaction pair increases. MAPPO (Yu et al., 2022) applies PPO (Schulman et al., 2017) into MARL and shows strong empirical performance. However, Kuba et al. ( 2021) points out MAPPO suffers from instability arising from the non-stationarity induced by simultaneously learning and exploring agents. Therefore, they introduce the sequential policy update scheme to achieve monotonic improvement on the joint policy.Learning communication protocols to solve cooperative tasks is one of the desired emergent behaviors of agent interactions. It has recently become an active area in MARL, such as learning to share observations (Das et al., 2019; Wang et al., 2019; Liu et al., 2020) and intentions (Kim et al., 2020; Böhmer et al., 2020; Wen et al., 2022; Liu et al., 2023) .NeurIPS Paper Checklisthttps://github.com/oxwhirl/wqmixhttps://github.com/TonghanWang/RODEhttps://github.com/AnujMahajanOxf/MAVENhttps://github.com/kikojay/EMChttps://github.com/zoeyuchao/mappohttps://github.com/ermongroup/MA-AIRLhttps://github.com/ReinholdM/Offline-Pre-trained-Multi-Agent-Decision-Transformerhttps://github.com/catezi/MAPThttps://github.com/YiqinYang/ICQhttps://github.com/ling-pan/OMARhttps://github.com/ZhengYinan-AIR/OMIGAhttps://github.com/kzl/decision-transformer 


Decision: This paper introduces the LBI framework, which employs a world model consisting of dynamics and reward components to simulate interactions and enhance policy learning. A multi-agent policy interacts with this simulator to generate trajectories. These trajectories are subsequently utilized to optimize the policy.

**strengths**

* Novel idea of integrating a language-guided simulator into RL pipeline

* VisionSMAC: a large-scale, diverse, and realistic dataset for multi-agent decision-making problems in the StarCraft II environment

* Extensive experiments and analysis on different components

**weaknesses/suggestions**

* Discussions on computational cost

* Limited domain (only SMAC) 

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version.

Reviews: review: paper_summary: paper_summary: The paper presents a novel approach called Learning before Interaction (LBI) that integrates a language-guided simulator into the multi-agent reinforcement learning process to enhance the quality of solutions for complex decision-making problems. The LBI paradigm uses a world model comprising a dynamics model and a reward model to simulate interactions and improve policy learning. The authors introduce new offline MARL datasets for the StarCraft Multi-Agent Challenge and demonstrate LBI's effectiveness through notable performance on training and unseen tasks.
strength_weakness: strength_weakness: 1. Trivial framework designed for multi-agent world model. Overall, the method proposed in this work seems not to capture the distinct characteristics in the context of multi-agent scenarios as compared to single-agent scenarios. The built world model appears to be based on reduce multi-agent problems into a single-agent problem with joint observations and joint actions. What are the benefits of this approach? Why is it necessary to adopt this simplification? The article does not provide specific evidence or thorough analysis to support these architecture design decisions.

2. Lack of the investigation and citation of related multi-agent world model literature. Given that the method proposed in this work derives the optimal policy via learning in the imaginations of the world model, it is essential to elucidate the differences between this approach and those presented in [1, 2, 3, 4] and to conduct performance comparisons. Notably, the architecture closely resembles that of [1] in the absence of reward prediction. However, the entire paper lacks citations and clarifications regarding these similarities.

3. Unfair comparisons. According to the contents in the appendix, the average returns are almost 20 in all offline dataset of ten maps, which are nearly optimal in SMAC environments since the maximum return of an episode is 20 in almost all SMAC maps. Thus with a pretrained world model which has full access to the optimal demonstrations, the comparison with online methods is definitely not meaningful and convincing at all.

4. Heavily handcrafted processing. “Since it is possible for specific agents to become inactive before the game terminates, we mark the terminated timestep for each agent and enemy once its predicted health is less than zero …” To identify the predicted health, the algorithm has implicitly already known the specific meaning of the value in some certain dimensions in the observation, which is obviously induced from the prior knowledge or predefined rules from human beings. Unfortunately, such an operation makes this work only narrowed in the domain of SMAC.

[1] Micheli, Vincent, Eloi Alonso, and François Fleuret. "Transformers are Sample-Efficient World Models." The Eleventh International Conference on Learning Representations.

[2] Egorov, Vladimir, and Alexei Shpilman. "Scalable Multi-Agent Model-Based Reinforcement Learning." Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022.

[3] Zhang, Yang, et al. "Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models." arXiv preprint arXiv:2406.15836 (2024).

[4] Liu, Qihan, et al. "Efficient Multi-agent Reinforcement Learning by Planning." The Twelfth International Conference on Learning Representations.
questions: questions: See weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces a pipeline called Learning before Interaction, a approach to improve generative models' answers for multi-agent decision-making problems. The key idea is to integrate a language-guided simulator into the multi-agent reinforcement learning pipeline.

The main contributions are 
- Creating VisionSMAC, a dataset that pairs states with images and task descriptions for the StarCraft Multi-Agent Challenge
- Training an interactive simulator consisting of a dynamics model and a reward model

The pipeline use the simulator to train a multi-agent policy through simulated interactions and generate answers by running the trained policy on the simulator.
strength_weakness: strength_weakness: - (Not mentioned in the paper) Training an interactive simulator with both dynamics and reward models likely requires significant computational resources
- It would be great if there are environments other than SMAC tested. Otherwise, the scope might be too narrow.
- The approach requires a substantial amount of offline data to train the world model and compared with online MARL methods which is not fair comparison. The offline data is also not always available.
- The ablation study shows that using image reference doesn't significantly improve performance, suggesting that this aspect of the model might not be fully utilized or optimized.
questions: questions: - What is the world model architecture and how do you train it?
- Could you please write an algorithmic block for both inference and training? I found there are still many processes unclear after reading the paper. It will be very helpful if you can upload a sample code (please be careful about the anonymity in the code).
limitations: limitations: The authors do not mention a limitation.
paper_summary: paper_summary: This paper proposes a method of improving the sample efficiency and performance of multi-agent reinforcement learning (RL) on the StarCraft Multi-Agent Challenge (SMAC) benchmark by learning to simulate the environment. The simulator consists of an image tokenizer, a dynamics model, and a reward function, all trained on data collected from interactions with the SMAC environment. A multi-agent policy then interacts with the simulator to obtain trajectories. These trajectories are then used to optimise the policy, but with a behaviour regularisation term towards the simulated trajectories. Evaluations on the SMAC benchmark show improved win rates and returns over existing online and offline multi-agent RL baselines. The paper also ablates the various components of the proposed method and provides visualisation of the simulated trajectories and their close match with actual trajectories.
strength_weakness: strength_weakness: The paper only studies SMAC and no other multi-agent domain. It is hard to evaluate the generality of the method without other evaluations on standard benchmarks. Adding evaluation results from one or two more domains would help strengthen the generality of this method. 

The write-up is a bit difficult to follow and contains a number of confusing parts (see also questions). The main figure, Figure 3, is never discussed in the main text. The part of Figure 1 on GPT-4 is not necessary for the paper. 

A significant amount of detail is located in the Appendix, for example Appendix B on how the training data for the simulation components are collected, and Appendix E on the experimental details. Consider moving these sections into the main text.

The paper does not explain the methods used to collect the training data (EMC and IIE). To be better self contained, please give short explanations of these methods in the main text. Are they particularly tuned to the SMAC environment? What would have to change in domains where an effective exploration strategy is unknown?
questions: questions: In the input notation of the dynamics model why is L_t time-varying across a trajectory? From the text L is constant across an entire trajectory, could the input contain more than 1 trajectory? The notation could be improved.

Could you explain in more detail how hindsight-relabelling is done and why it is necessary for reward function training? Is it the same or different from the text label used on the same set of collected trajectories as in dynamics model training? Is it only for data collected from the simulator?

Clarifications for equation (1): What is D? Why does the f term have \tau as an input and an index i? Where do they come from? Is there a missing sum(s)? In the description it says that term is “over all possible actions”, what does this mean? Its inputs are trajectories and aren’t guaranteed to contain all actions?

Why did the authors choose to only focus on a single multi-agent domain and not evaluate the method on any other, even toy, multi-agent benchmarks?

Why do some tables report win rate and some rewards? 

Figure 8, does LBI timesteps account for the number of time steps in the data collected to train the simulator components? Would it be more accurate to shift the LBI curve to the right by that amount?
limitations: limitations: Limitations have been adequately addressed.
paper_summary: paper_summary: The paper proposes a new approach to multi-agent decision-making problems by integrating a language-guided simulator into the reinforcement learning pipeline. 

Learning before Interaction (LBI) uses a world model to generate trial-and-error experiences and improve the answers to complex decision-making problems. 

The world model consists of a dynamics model and a reward model, which are learned from expert demonstrations and task descriptions. 

The LBI framework is evaluated on the StarCraft Multi-Agent Challenge benchmark and shows superior performance on both training and unseen tasks, generating consistent and explainable interaction sequences and rewards.
strength_weakness: strength_weakness: The motivation for mentioning and comparing the answer of GPT-4, at the beginning, seems misleading. I thought that the proposed LBI would benefit the general response of LLMs on complex decision-making. However, it seems that the proposed method heavily relied on the gaming environment, and and result is limited to SMAC benchmark.

The dataset collection is a crucial part of developing a successful LBI, especially for the reward prediction part. However, it must be constructed via a world model (1) where the dynamic is explicit (2) simulation speed is fast enough to perform learning and interaction. How the generalizability of the proposed method?
questions: questions: See weaknesses
limitations: limitations: no negative societal impact

Comments: title: Reply
Rebuttal during the discussion
Response
comment: I thank the authors for the detailed rebuttal. Most of my concerns has been addressed. However, one key concern of the limiting scope of the work only on the SMAC benchmark (which it seems is also shared with other reviewers) has not been addressed. I will maintain my original rating.
We appreciate the reviewer’s feedback!

**1. The non-stationarity and the scalability issue.**

We provide the state and joint actions as the input to train the world model (avoid the non-stationarity) and utilize the Transformer model (GPT2-like dynamics and Bert-like reward models) to provide enough representational capacity, where their attention mechanism performs feature selection and aggregation (for the scalability).

To further elucidate our approach, we list the following paradigms:

*(1) Decentralized training with decentralized execution*: While this paradigm excels in scalability during execution, it often leads to non-stationarity as agents treat others as part of the environment.

*(2) Centralized training with decentralized execution*: It maintains execution scalability while introducing a centralized mechanism (e.g., a mixing network to aggregate the individual Q-values) to mitigate the non-stationarity problem.

*(3) Communication-enabled training and execution*: Each agent learns and executes its policy based on their observations and (weighted) messages from others (e.g., actions, observations). This paradigm creates a trade-off between interaction complexity and representational capacity, a great compromise for non-stationarity and scalability issues.

*(4) Centralized training with centralized execution*: While this paradigm eliminates the non-stationarity issue, it suffers from limited scalability during execution.

Our world model serves as the environment, which leverages the Transformer model’s attention mechanism to capture interdependencies among agents and timesteps, similar to communication-enabled paradigms (e.g., MAT [1]). The concurrent work [2] also proposes a Perceiver Transformer as a communication module, which utilizes the querying and self-attention mechanisms to perform agent-wise aggregation, transforming the joint representation sequence (joint observations and actions) into lower-dimensional features. The difference is that our world model integrates these processes into the large transformer model. 

**2&3. MAMBA and MARIE, and the low data regime.**

We acknowledge that MAMBA [3] and MARIE [2] are indeed Dyna-style algorithms that learn policy in the imagination, and their induced policies are decentralized. We would like to clarify our stance:

* MAMBA and MARIE assume that the reward function is presented in the replay buffer. They train the reward predictor using log-likelihood and cross-entropy loss, respectively. **Conversely**, we suppose the reward is unknown in the offline dataset. We construct the reward function using a bidirectional Transformer, which maximizes the likelihood of trajectories in hindsight relabeled expert demonstrations.

* MAMBA and MARIE are online model-based RL methods. They explore the environment using a policy, store the collected data in a replay buffer, update the world model with samples from this buffer, and use rollouts from the world model to refine the policy. They can benefit from interacting with the real environment, allowing them to update and rectify their world models continuously - e.g., if agents imagine trajectories in out-of-distribution regions that lead to suboptimal updates, they can correct this during exploration by gathering more data in these regions. Consequently, they conducted experiments in low-data regimes like Atari-100K to show their data efficiency.

  **In contrast**, our dynamics model is trained in an offline manner via a static, offline dataset. Unlike online methods, no exploration is available for the model to rectify itself, leading to the distributional shift issue - where the deployment scenario may differ significantly from the historical dataset regarding state and action space. To mitigate this, we employ a behavior regularization term to encourage exploitation, specifically by discouraging agents from executing out-of-distribution behaviors during inference.

**4. The agent death masks.**

Thank you for your insightful suggestion. We would like to use a binary gate function for each agent to replace the handcrafted death masks and learn it via binary cross-entropy loss in a supervised manner. We acknowledge the potential for further generalization.

However, we respectfully disagree with the notion that the current approach oversimplifies the learning process. As shown in Fig.4, the learned world model captures the dynamics and learns a meaningful reward function even in this complex scenario. This demonstrates that the model correctly understands the transitions within the game, validating its ability to grasp the underlying mechanics of SMAC.

[1] Wen, Muning, et al. Multi-agent reinforcement learning is a sequence modeling problem.

[2] Zhana, Yang, et al. Decentralized Transformers with centralized Aggregation are Sample-Efficient Multi-Agent World Models.

[3] Micheli. Vincent, Eloi Alonso, and Francois Fleuret, Transformers are Sample-Efficient World Models.
Thanks for the comments. I am happy to raise the score.
1. First, different from the single-agent scenario, the scalability issue stemming from increasing agents and the non-stationarity issue arising from the continuously changing environment of individual agents, are inherent in the multi-agent dynamics. However, the authors gave little explanation on whether and how this work can tackle these challenges.
2. The author may not carefully investigate the related works [1-2]. MAMBA and MARIE adopted a learning policy in the imagination scheme, enabling them to seamlessly incorporate different MARL methods. Their induced policies also remained decentralized.
3. I don't think the current experiment is aligned to demonstrate that training a policy in the world model is faster than training it in a real environment. In contrast, like how IRIS [3] delivered this similar insight via its experiments, it would be better to carry out experiments with a low data regime like Atari-100K to demonstrate the aim.
4. In terms of the handcrafted processing for obtaining agent death masks, it makes the work only applicable to the limited SMAC domain, weakening the contribution. Moreover, a prediction towards the agent death masks in the world model can better validate how accurate the world model, as the world model needs to understand the underlying game mechanics in SMAC. The handcrafted processing oversimplifies the learning.

[1] Egorov, Vladimir, and Alexei Shpilman. Scalable Multi-Agent Model-Based Reinforcement Learning.

[2] Zhana, Yang, et al. Decentralized Transformers with centralized Aggregation are Sample-Efficient Multi-Agent World Models

[3] Micheli. Vincent, Eloi Alonso, and Francois Fleuret, Transformers are Sample-Efficient World Models