MADiff: Offline Multi-agent Learning with Diffusion Models
Authors: Zhengbang Zhu
Minghuan Liu
Liyuan Mao
Bingyi Kang
Minkai Xu
Yong Yu
Stefano Ermon
Weinan Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-agent RL
Diffusion Models
Offline RL
Primary area: reinforcement_learning
TL;DR: We propose a multi-agent diffusion framework that unifies decentralized policy, centralized controller, teammate modeling, and trajectory prediction.

Abstract: Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task. Q-learning algorithms struggle with extrapolation errors in offline settings, while supervised learning methods are constrained by model expressiveness. Recently, diffusion models (DMs) have shown promise in overcoming these limitations in single-agent learning, but their application in multi-agent scenarios remains unclear. Generating trajectories for each agent with independent DMs may impede coordination, while concatenating all agents’ information can lead to low sample efficiency. Accordingly, we propose MADiff, which is realized with an attention-based diffusion model to model the complex coordination among behaviors of multiple agents. To our knowledge, MADiff is the first diffusion-based multi-agent learning framework, functioning as both a decentralized policy and a centralized controller. During decentralized executions, MADiff simultaneously performs teammate modeling, and the centralized controller can also be applied in multi-agent trajectory predictions. Our experiments demonstrate that MADiff outperforms baseline algorithms across various multi-agent learning tasks, highlighting its effectiveness in modeling complex multi-agent interactions.

MADIFF: Offline Multi-agent Learning with Diffusion Models
Zhengbang Zhu'
Minghuan Liu' Liyuan Mao' Bingyi Kang²
Minkai Xu²
Yong Yu' Stefano Ermon Weinan Zhang'1 if Shanghai Jiao Tong University, 2 ByteDance, 3 Stanford University
{zhengbangzhu, minghuanliu, maoliyuan, yyu, wnzhang> Cajtu. tu, edu. cn, bingykang@gmail. com, (minkai, ermon]@cs. stanford edu
Abstract
Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it challenging task. Q-learning algorithms struggle with extrapolation errors in offline settings, while supervised learning methods are constrained by model expressiveness. Recently, diffusion models (DMs) have shown promise in overcoming these limitations in single-agent learning, but their application in multi-agent scenarios remains unclear. Generat- ing trajectories for each agent with independent DMs may impede coordination, while concatenating all agents' information can lead to low sample efficiency. Accordingly, we propose MADIFF, which is realized with an attention-based dif- fusion model to model the complex coordination among behaviors of multiple agents. To our knowledge, MADIIF is the first diffusion iiffusion-based multi-agent learning framework. functioning as both decentralized policy and a centralized controller. During decentralized executions, MADIFF simultancously performs teammate modeling. and the centralized controller can also be applied in multi-agent trajec- tory predictions. Our experiments demonstrate that MIDIFF outperforms baseline algorithms across various multi-agent learning tasks, highlighting its effectiveness in modeling complex multi-agent interactions.
Introduction
Offline reinforcement learning (RL) [Fujimoto et al., 2019, Kumar et al., 2020] learns exclusively from static datasets without online interactions, enabling the effective use pre-collected large-scale data. However, applying temporal difference (TD) learning in offline settings causes extrapolation errors [Fujimoto et al., 2019], where target value functions are evaluated on out-of-distribution actions. Sequence modeling algorithms bypass lea-ninn by directly fitting the dataset distribution [Chen al., 2021, Janner et al., 2021]. Nevertheless, these methods are limited by the model's ss expressive- ness, making it difficult to handle diverse datasets. They also suffer from compounding errors [Xiao al., 2019] due to autoregressive generation. Recently, diffusion models (DMs) have achieved remarkable success in various generative modeling tasks [Song and Ermon, 2019, Ho et al., 2020, Xu et al., 2022], owing to their exceptional abilities at capturing complex, distributions. Their successes have also been introduced into offline RL, offering superior modeling choice for sequence modeling algorithms [Janner et al., 2022, Ajay et al., 2023].
sigh-dimensional data
Compared to agelt learning, offline multi-agent learning (MAL) has been less studied and is more challenging. Since the behaviors of all agents are interrelated, each agent required model interactions and coordination among agents, while making decisions in decentralized manner to achieve the goal. Current MAI. approaches typically train a centralized value function to update
I Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

individual agents' policies [Rashid et al., 2020] or use an autoregressive transformer to determine each agent's actions [Meng al., 2021, Wen et al., 2022]. However, without online interactions, an incorrect centralized value can lead to significant extrapolation errors, and the transformer can only serve as an independent model for each agent.
In this paper, we aim to study the potential of employing DMs to solve the above challenges in offline MAL problems. Merely adopting existing diffusion RL methods by using independent DMs model each agent can result in serious inconsistencies due to a lack of proper credit assignment among agents, Another possible solution is to concatenate all agents' information as the imput and output of the DM. However, treating the agents as a single unified agent neglects the important nature of multi-agent systems. One agent may have strong correlations with only a few other agents, which makes full feature interaction redundant. In many multi-agent systems, agents exhibit certain symmetry and can share model parameters for efficient learning [Arel et al., 2010]. However, concatenating them in a fixed order breaks this symmetry, forcing the model to treat each agent differently.
To address the aforementioned coordination challenges. we propose the first centralized-training decentralized-exxccuioo (CTDE) diffusion framework for MA problems, named MADIFF. MADIFF adopts novel attention-based DM to learn a return-conditionll trajectory generation model on reward-labeled multi-agent interaction dataset. In particular, the designed attention is computed in several latent layers of the model of each agent to fully interchange the information and integrate the global information of all agents. To model the coordination among agents, MADIFF applies the attention mechanism on latent embedding for information interaction across agents. The attention mechanism enables the dynamic modeling of agent interactions through learned weights, while also enabling the use of a shared backbone to model each agent's trajectory, significantly reducing the number of parameters. During training, MADIFF performs centralized training on the joint trajectory distributions of all agents from offline datasets, including different levels of expected returns. During inference, MADIFF adopts classifier-free guidance with low-temperature sampling to generate behaviors given the conditioned high expected returns, allowing for decentralized execution by predicting the behavior of other agents and generating its own behavior. Therefore, MADIFF can be regarded as a principled offline MAL solution that not only serves as a decentralized policy for each agent centralized controller for all agents, but also includes teammate modeling without additional cost. Comprchensive experiments demonstrated superior performances of MADIFF on various multi-agent learning tasks, including offline MARL and trajectory prediction.
In summary, our contributions are (1) the first diffusion- baated multi-agent learning framework that unifies decentralized policy, centralized controller, teammate modeling, and trajectory prediction; (2) a novel attention-based DM structure that is designed explicitly for MAL and enables coordination among agents in each denoising step; (3) achieving superior performances for various offline multi- agent problems.
2 Preliminaries
2.1 Multi-agent Offline Reinforcement Learning
We consider a partially observable and fully cooperative multi-agent learning (MAL) problem, where agents with local observations cooperate finish the task. Formally, it is defined as a Dec- POMDP [Oliehoek and Amato, 2016]: G AS, P, r. Ω, N,N,U, 7), where S and A denote state and action space separately, and 7 is the discounted factor. The system includes N agents (1,2..... N} act in discrete time steps, and starts with an initial global state 80 E S sampled from the distribution U. At each time step t, every agent only observes a local observation O² Ω produced by the function (ss, a) S x A + Ω and decides a E A. which forms the joint action a aEA A = AN leading the system transits to the next state s' according to the dynamics function P(s'(s, S x A + S. Normally, agents receive a shared reward r(s, a) at each step, and the optimization objective to learn a policy xO for each agent that maximizes the discounted cumulative reward E.,a, IE, r'r(st, a.)l. In offline settings, instead of collecting online data in environments, we only have access to a static dataset to learn the policies. The dataset D is generally composed of trajectories T. i.e., abservation-action sequences [On, ag. 01, a1 Or, aT) or observation sequences (oo, 01, or)). We use bold symbols to denote the joint vectors of all agents.
2

---

Returna
Inverse Dynamics Model
K
Loyer
K steps
Sea
o
Product
Enawder Lover 1
Layer
Freadar
Desidar
MAP
MADiff
and clossifie-free guidonce
sompling diffuston
Eesedor
Loyer
Middle
Deceder
Loyer1
Devesingle
Cawith Sevaphore Miis
Upsanghee
Figure 1: The architecture of MADIFF, which is an attention-basse diffusion network framework that performs attention across all agents every decoder layer of each agent.
2.2 Diffusion Probabilistic Models
Diffusion models (DMs) [Sohl-Dickstein al., 2015, Song and Ermon, 2019, Ho al., 2020], as powerful class of generative models, implement the data generation process as reversing forward noising process (denoising process). For each data point zip ~ Pdata (x) from the dataset DD the noising process is discrete Markov chain W0:K such that p(TA (4x-1) = (zall/og.re (1 ()))), where ((u, Σ) denotes Gaussian distribution with mean j and variance Σ, and do:K E IR are hyperparameters which control the variance schedule. The variational reverse Markov chain is parameterized with (p( = N
k), (1 - ax).). The data sampling process begins sampling initial noise IK (0, I), and follows the reverse process until 2g. The reverse process can be estimated by optimizing a simplified surrogate loss as in Ho et al. [2020]:
((0) 11o--11.K |||e- ())k)))11)
(1)
The estimated Gaussian mean can written as po(
L yor
@x 1-a VI-ac
co(axk hi)) where
0g = II 0g.
2.3 Diffusing Decision Making
Diffusing state trajectories and acting with inverse dynamics model. Among existing works in single-agent learning, Janner et al. [2022] chose to diffuse over state-action sequences, so that the generated actions for the current step can be directly used for executing. Another choice is diffusing over state trajectories only [Ajay et al., 2023], which is claimed to be easier to model and obtain better performance due the less smooth nature of action sequences:
== [8t, S 11),
(2)
where i is the sampled time step and H denotes the trajectory length (horizon) modeled by DMs. But the generated state sequences can not provide actions to be executed during online evaluation. Therefore, an inverse dynamics model is trained to predict the action à, that makes the transit from 8t to the generated next state 8141:
Lo(st, $c-1).
Therefore, at every environment step f, the agent first plans the state trajectories using an offline- trained DM, and infers the action with the inverse dynamics model.
Classifier-free guided generation. For targeted behavior synthesis, DMs should be able to generate future trajectories by conditioning the diffusion process on an observed state 8; and information y.
use classifier-free guidance [Ho and Salimans, 2022] which requires taking y(r) as additional inputs for the diffusion model. Formally, the sampling procedure starts with Gaussian noise Åx (0 aT), and diffuse fx into f--1 at each diffusion step k. Here [0, is the scaling factor used in

---

low-temperature sampling to scale down the variance of initial samples [Ajay et al., 2023]. We use Zg.t to denote the denoised state s; k's diffusion step. fR. denotes the denoised trajectory k's diffusion step for single agent: fx == [st, -,, .EE.t Hf - Note that for sampling during evaluations, the first state of the trajectory always set to the current observed state at all diffusion steps for conditioning, and every diffusion step proceeds with the perturbed noise:
é = eo (fx. 0, k) + (Fk: u(T), k) c 0, k))
(4)
where w is a scalar for extracting the distinct portions of data with characteristic U(T). By iterative diffusing the noisy samples, we can obtain clean state trajectory: (T( == (&t, St+1:" ,84 1++8-11 H-1
3 Methodology
We formulate the problem of MAL conditional generative modeling:
max ET~D [log pe (e
(5)
#
where Po is learned for estimating the conditional data distribution of joint trajectory given information y()), such as observations, rewards, and constraints. When all agents are managed by a centralized controller, i.e., the decisions of all agents are made jointly, model by conditioning the global information aggregated from all agents (T)); otherwise, if we consider each agent i separately and require each agent to make decisions in decentralized manner, we can only utilize the local information y' (r*) of each agent i, including the private information and the common information shared by all (e.g., team rewards).
learn the generative
3.1 Multi-Agent Diffusion with Attention
In order to handle MAL problems, agents must learn to coordinate. To solve the challenge of modeling the complex inter-agent coordination in the dataset, we propose novel attention-based diffusion architecture designed to interchange information among agents.
In Figure illustrate the architecture of MADIFF model. In detail, adopt U-Net as the base structure for modeling agents' individual trajectories, which consists repeated -me-iimnsioonal convolutional residual blocks. The convolution is performed over the time step dimension, and the observation feature dimension is treated as the channel dimension. To encourage information interchange and improve coordination ability, critical change is made by adopting attention [Vaswani et al., 2017] 7] layers before all decoder blocks the U-Nets of all agents. Since embedding vectors from different agents are aggregated by the attention operation rather than concatenations, MADIFF is index-free such that the input order of agents can be arbitrary and does not affect the results.
Formally, the input to 1-th decoder layer in the U-Net of each agent i is composed of two components: the skip-connected feature of from the symmetric 1-th encoder layer and the embedding ej from the previous decoder layer. The computation of attention MADIFF conducted on of rather than ef since in the U-Net structure the encoder layers are supposed to extract informative features from the input data. We use e') to denote the skip-connected feature after attention operations which aggregate information across agents. We adopt the multi-head attention mechanism to fuse the encoded feature of with other agents' information, which is important in effective multi-agent coordination.
3.2 Centralized Training Objectives
Given multi-agent offline dataset D, train MADIFF which is parameterized through the unified noise model eg for all agents and the inverse dynamics model 13 of each agent i with the reverse diffusion loss and the inverse dynamics loss:
C(0,0)
Σ at o
(6)
+ E. - ca co( (1 (1)((T))(T) + 30, 1)
where 3 sampled from Bernoulli distribution to balance the training effort on unconditioned and conditioned models. For training the DM, we sample noise (N I) and time step

---

k~ } construct noise corrupted joint state sequence тı from T and predict the noise Eg == (t. (T)), k). Note that the noisy array f& is applied with the same condition required by the sampling process, as we will discuss in Section 3.3 in detail. As for the inverse dynamies training. we sample the observation transitions of each agent to predict the action.
It is worth noting that the choice of whether agents should share their parameters of eb and lg. depends on the homogeneous nature and requirements of tasks. If agents choose to share their parameters, only one shared DM and inverse dynamics model are used for generating all agents' trajectories; otherwise, each agent i has extra parameters (i.e., the U-Net and inverse dynamic models) to generate their states and predict their actions. The attention modules are always shared to incorporate global information into generating each agent's 's trajectory.
3.3 Centralized Control Decentralized Execution
Centralized control. A direct and straightforward way to utilize MADIFF in online deeiiioonmakingg tasks is to have a centralized controller for all agents. The centralized controller has access to all agents' current local observations and generates all agents" trajectories along with predicting their actions, which are sent to every single agent for acting in the environment. This is applicable for multi-agent trajectory prediction problems and when interactive agents are permitted to be centralized controlled, such as in team games. During the generation process, we sample an initial noise trajectory टk. condition the current joint states of all agents and the global information to utilize (To);; following the diffusion step described in Equation (4) with Eg, we finally sample the joint observation sequence fo as below:
K steps
Oo:
IK4HH-1
Oo 10+H
(7)
PK
where every KKGt ~ (O 1) is a noise vector sampled from the normal Gaussian. After generation, each agent obtains the action through its own inverse dynamics model following Equation (3) using the current observation of and the predicted next observation of+ and takes step in the environment. We highlight that MADIFF provides an efficient way to generate joint actions and the attention module guarantees sufficient feature interactions and information interchange among agents.
Decentralized execution with teammate modeling. Compared with centralized control, a more popular and widely-adopted setting is that each agent only makes its own decision without any communication with other agents, which is what most current works [Lowe et al., 2017, Rashid et al., 2020, Wang et al., 2023] dealt with. In this case, we can only utilize the current local observation of each agent to plan its own trajectory. To this end, the initial noisy trajectory is conditioned on the current observation of the agent i. Similar to the centralized case, by iterative diffusion steps, we finally sample the joint state sequence based on the local observation of agent as:
i0
K.
20
--
6
60
H-1
o
H-1
K stops
o
(8)
i K,t
K-+
H
and we also obtain the action through the agent i's inverse dynamics model as mentioned above. An important observation is that, the decentralized execution of MADIFF includes teammate modeling such that the agent i infers all others' observation sequences based on its own local observation. We show in experiments that this achieves great performances in various tasks, indicating the effectiveness of teammate modeling and the great ability in coordination.
History- generation. We find DMs are good at modeling the long-term joint distributions, and as result MADIFF perform better in some cases when we choose to condition on the trajectory of the past history instead of only the current observation. This implies that we replace the joint observation
O;-1, 0.], and replace the independent observation of in Equation (8) as the history sequence h == (o [ of 1: of] of each agent i. Appendix Section illustrates how agents' history and future trajectories are modeled by MADIFF in both centralized control and decentralized execution.
o, Equation (7) as the C-length joint history sequence h == [o:-C,
5

---

4 Related Work
Multi-agent Offline RL. While offline RL has become an active research topic, only limited number of works studied offline MARL due to the challenge of offline coordination. Jiang and Lu [2021] extended BCQ [Fujimoto et al., 2019], a single-agent offline RL algorithm with policy regularization to multi-agent; Yang et al. [2021] developed an implicit constraint approach for offline Q learning, which was found to perform particularly well in MAL tasks; Pan et al. [2022] argued the actor update tends to be trapped in local minima when the number of agents increases, and correspondingly proposed an actor regularization method named OMAR. All of these Q-learning- based methods naturally have extrapolation error problem [Fujimoto et al., 2019] in offline settings, and their solution cannot get rid of it but only mitigate some. As an alternative, MADT [Meng et al., 2021] formulated offline MARL as return-condee couum-conditioee supervised learning. and use similar structure to previous transformer- bbased offline RL work [Chen et al., 2021]. However, offline MADT learns an independent model for each agent without modeling agent interactions; it relies on the gradient from centralized critics during online fine-tuning integrate global information into each agent's decentralized policy. MADIFF not only avoids the problem of extrapolation error, but also achieves the modeling of collaborative information while allowing CTDE in completely offline training manner.
Diffusion Models for Decision-Making There is recent line of work applying diffusion models (DMs) to reciiion-mmking problems such as RL and imitation learning. Janner et al. [2022] design a diffusion-based trajectory generation model and train value function to sample high-rewarded trajectories. A consequent work [Ajay et al., 2023] takes conditions as inputs to the DM, thus bringing more flexibility that generates behaviors that satisfy combinations of diverse conditions. Another line of work [Wang et al., 2022, Hansen-Fstruch et al., 2023, Kang et al., 2024] uses the DM as form of policy, i.c., generating actions conditioned on states, and the training objective behaves as a regularization under the framework of TD-based offline RL algorithms. Different from the above, SynthER [Lu et al., 2024] adopts the DM to upsample the rollout data to facilitate learning of any RL algorithms. All of these existing methods focus solving single-agent tasks. The proposed MADIFF is structurally similar to Ajay et al. [2023], but includes effective modules to model agent coordination in MAL tasks.
Opponent Modeling in MARL. Our modeling of teammates can be placed under the larger frame- work of opponent modeling, which refers the process by which an agent tries to infer the behaviors or intentions of other agents using its local information. There is rich literature on utilizing opponent modeling in online MARL. Rabinowitz al. [2018] used meta-learning to build three models, and can adapt to new agents after observing their behavior. SOM [Raileanu et al., 2018] uses the agent's 's own goal- conditiooe policy to infer other agents' goals from maximum likelihood perspective. LIAM [Papoudakis et al., 2021] extracts representations of other agents with variational auto-encoders conditioned on the controlled agent's local observations. Considering the impact of the ego agent's policy on other agents' policies, LOLA [Foerster et al., 2017] and following works [Willi et al., 2022, Zhao et al., 2022] instead model the parameter update of the opponents. Different from these methods, MADIFF use the same generative model to jointly output plans of its own trajectory and predictions of other agents' trajectories and is shown to be effective in offline settings.
5 Experiments
In experiments, we are aiming at excavating the ability of MADIFF in modeling the complex interactions among cooperative agents, particularly, whether MADIFF able to (i) generate high- quality multi-agent trajectories; (ii) appropriately infer teammates' behavior; (iii) learn effective, coordinated policies from offline data.
5.1 Task Descriptions
We conduct experiments on multiple commonly used multi-agent testbeds.
Multi-agent partiele environments (MPE) [Lowe et al., 2017]: multiple 2D particles cooperate to achieve a common goal. Spread, three agents start at some random locations and have to cover three landmarks without collisions; Tag, three predators try catch a pre-trained prey opponent

---

Table 1: The average score on offline MARL tasks. Shaded columns represent our methods. The mean and standard error are computed over 5 different seeds.
Testbed
Task
MA-ICQ
MA-TDA-BC
MA-CQI.
OMAR
MADIT
MADIFF-D
Expert
1040
106.3
53
Specad
Mé-Replay
38
29,3
34.1
189
58.2
0.5
9.8
24.0
Fxpert
1130+ 14.4
1152
116.2
Tug
Replay
34.5
9.2
132.9 15.0
1.2.0
2.6
5.7
5.0
3.2
10.7
Expert
28.1
110:4
16.8
World
Mi-Replay
Medium
2:
71.
11.2
11.5
0.3
0.5
1.0
0.6
3.5
Good
6845 574
7025 1: 439
1434 1903
8246 1: 342
8514 336
Zhalfcheetah
Medium
1627 1 187
2561
1892 220
2207 23
2203 1 66
Poer
465 1 59
736
384 1 420
759 18
760 1 15
MA
Good
2097
267
2922 194
464 1. 489
2946 1: 77
3009 60
Majsco
2ant
Medium
1145 1 136
1664 283
799 1: 196
1211 1 69
1243 1 37
Poot
954
80
1206 +: 122
857
946
1038 26
Good
2802
133
2628 971
344 1. 631
3080 1 38
3068 44
Aunt
Medium
1617
153
1843 494
929 :: 349
1669 100
1871 1
Poor
1083
122
1075 à: 96
518 a: 112
120G +: 57
1353
160
18.8 0.6
Medium
8.2
1.2
5.8
9.5
Glod
1822+04
193+002
19.0:2
12.3
17.6
SMAC
9.5
5mbom
17.3
10:4
Medium
10.5
4.6
4.7
5.1.1.011
that moves faster and needs cooperative containment; World, also requires three predators catch a pre-trained prey, whose goal is to eat the food on the map while not getting caught, and the map has forests that agents can hide and invisible from the outside.
Datasets: we use the offline datasets constructed by Pan et al. [2022], including four datasets collected by policies of different qualities trained by MATD3 [Ackermann et al., 2019], namely, Expert, Medium-Replay (Md-Replay), Medium and Random.
Multi-Agent Mujoco (MA Mujoco) [Peng et al., 2021]: independent agents control different subsets of robot's joints to run forward fast possible. We use three configurations: 2-agent halfcheetah (2halfcheetah), 2-agent ant (2ant), and 4-agent ant (4ant).
Datasets:
the off-the-gric thid offline dataset [Formanek al., 2023], including three datasets with different qualities for each robot control task, e.g., Good, Medium, and Poor.
StarCraft Multi-Agent Challenge (SMAC) [Samvelyan et al., 2019]: a team of either homoge- neous or heterogeneous units collaborates to fight against the enemy team that is controlled by the hand-coded built-in StarCraft II AI. We cover four maps: 3m, both teams control three Marines; 2s3z, both teams control two Stalkers and Zealots; 5m_vs_ón (5m6m), requires controlling five Marines and the enemy team has six Marines; 8m, both teams control eight Marines.
Datasets: we use the off-the-gnt offline dataset [Formanck et al.. 2023], including three datasets with different qualities for each map. e.g., Good, Medium, and Poor.
Multi-Agent Trajectory Prediction (MATP): different from the former offline MARL challenges which should learn the policy for each agent, the MATP problem only requires predicting the future behaviors of all agents, and no decentralized model is needed.
NBA dataset: the dataset consists various basketball players' recorded trajectories from 631 games in the 2015-16 season. Following Alcor and Nguyen [2021], we split 569/30/32 trining/validationnness games, with downsampling from 25 Hz to 5Hz. Different from MARL tasks, other information apart from agents' historical trajectories is available for making predictions, including the ball's historical trajectories, player ids, and a binary variable indicating the side of each player's frontcourt. Each term is encoded and concatenated with diffusion time embeddings as side inputs to each U-Net block.
7

---

5.2 Compared Baselines and Metrics
For offline MARL experiments, we use the episodic return obtained in online rollout as the perfor- mance measure. We include MA-ICQ [Yang et al., 2021] and MA-CQL [Kumar et al., 2020] as baselines on all offline RL tasks. On MPE, we also include OMAR and MA- -D-+TD+BBC [Fujimoto and Gu, 2021] in baseline algorithms and the results reported by Pan et al. [2022]. On MA Mujoco, baseline results are adopted from Formanek et al. [2023]. On SMAC, we include MADT [Meng et al., 2021] as sequence modeling baseline, while other baseline results reported by Formanek et al. [2023]. We implement independent behavior cloning (BC) as a naive supervised learning baseline.
We use settance-basel metrics including average displacement error (ADE) EN E Σ 110 N
(FDE) Σ | -0,, |l. where is the prediction length et 2020]. We also report minADF2o and Din as additional metrics to balance the stochasticity in sampling, which are the minimum ADE and FDE among 20 predicted trajectories. respectively. We compare MADIFF with Baller2 V [Alcorn and Nguyen, 202 I]. an autoregressive MATP algorithm based the transformer structure and specifically designed for the NBA dataset.
of and final displacement
5.3 Numerical Results
We reported the numerical results both for the CTDE version of MADIFF (denoted as MADIFF- D) and the centralized version MADIFF (MADIFF-C). For offline MARL, since baselines are tested decentralized style, iee,, all agents independently decide their actions with only local observations, MADIFF-C is not meant to be a fair comparison but to show if MADIFF-D fills the gap for coordination without global information. For MATP, due its centralized prediction nature, MADIFF-CI is the only variant involved.
Offline MARL. As listed in Table 1, MADIFF-D achieves the best result on most of the datasets. Similar to the single-agent case, direct supervised learning (BC) the dataset behaves poorly when datasets are mixed quality. Offline RL algorithms such as MA-CQL that compute conservative values have a relatively large drop in performance when the dataset quality is low. Part of the reason may come from the fact that those algorithms are more likely to fall into local optima in multi-agent scenarios [Pan al., 2022]. Thanks the distributional modeling ability of the DM, MADIFF-I) generally obtains better ог competitive performance compared with OMAR [Pan et al., 2022] without any design for avoiding bad local optima similar to Pan et al. [2022]. On SMAC tasks, MADIFEDD achieves comparable performances, although it is slightly degraded compared with MADIFF-C.
MATP on the NBA dataset. In Table when comparing ADE and FDE, MADIFF-C significantly outperforms the baseline; however, our algorithm only slightly beats baseline for Din and has higher minFDE20- We suspect the reason that Baller2vec++ has a large prediction variance. When Baller2 Vec++ only predicts one trajectory, a few players' trajectories deviate from the truth so far that deteriorate the overall ADE and FDE. When allowing to sample 20 times and calculating the minimum ADE/FDE according the ground truth, Baller2Vec++ for every single agent, which makes minADE 20 and minFDE20 significantly smaller than one-shot metrics. However, considering it may be not practical to select the best trajectories without access to the ground truth, MADIFF-C is much more stable than Baller2Vec++.. Predicted trajectories of MADIFF-C and Baller2Vec++
choose the best trajectory
provided in the Appendix Section H.4.
Table Multi-agent trajectory prediction re- sults on NBA dataset across 3 seeds, given the first step of all agents' positions.
Enconabiient
Cansiettee Batile
Emc
Convistest
Traj. Len.
Baller2Vec++
MADIFF-C
ADE
15.15 + 0.38
7.92 0.86
FDE
24.91
14,06 1.16
minADFas
5.62 0.05
5.20
0.04
minFDEso
5.60 0.12
7.61 = 0.19
Landmarks
Planning agent
Other agents
ADE
32.07 = 1.93
17.24 0.80
FDR
44.93 + 3.02
26.69 0.13
Figure 2: Visualization of episode in the Spread task. Solid lines are real rollouts, and dashed lines are DM-planned trajectories.
minADEgs
0.06
minFDE>p
10.41 1: 0.36
11.26 0.26
8

---

5.4 Qualitative Analysis on Teammate modeling
We discuss the quality of teammate modeling as mentioned in Section 3.3 and how it is related to the decentralized execution scenario. In Figure left, we visualize an episode generated by MADIFF-D trained on the Expert dataset of Spread task. The top and bottom rows are snapshots of entities' positions on the initial and intermediate time steps. The three rows from left to right in each column represent the perspectives of the three agents, red, purple, and green, respectively. Dashed lines are the planned trajectories for the controlled agent and other agents output by DMs, and solid lines are the real rollout trajectories. We observe that at the start, the red agent and the purple agent generate inconsistent plans, where both agents decide to move towards the middle landmark and assume the other agent is going to the upper landmark. At the intermediate time step, when the red agent is close to the middle landmark while far from the uppermost ones, the purple agent altered the planned trajectories of both itself and the red teammate, which makes all agents' plans consistent with each other. This particular case indicates that MADIFF is able to correct the prediction teammates' behaviors during rollout and modify each agent's own desired goal correspondingly.
In Figure 2 right, we demonstrate that such corrections of teammate modeling are common and can help agents make globally coherent behaviors. We sample 100 episodes with different initial states and define Consistent Ratio at some time step as the proportion episodes in which the three agents make consistent planned trajectories. We plot the curve up to step t = 9, which is approximately halfway through the episode length limit in MPE. The horizontal red line represents how many portions of the real rollout trajectories increasing curve reaches the red line before = 9, and ends up even higher. This indicates that the planned teammates' trajectories are guiding the multi-agent interactions beforehand, which is strong exemplar the benefits of MADIFF'S ' teammate modeling abilities. We also include visualizations of imagined teammate observation sequences in SMAC 3m task in the Appendix Section H.3.
consistent step 9. The interesting part is that the
5.5
Ablation Study
Our key argument is that the great coordination ability of MADIFF is brought by the attention modules among individual agents' diffusion networks. We validate this insight through set of ablation experiments on MPE. We compare MADIFF-D with independent DMs, i.e., each agent learns from corresponding offline data using independent U-Nets without attention. We denote this variant as MADIFF-D -nndd In addition, we also ablate the choice of whether each agent should share parameters of their basic U-Net, noted as Share NoShare. Without causing ambiguity, omit the name of MADIFF, and notate the different variants as D-Share, D-NoShare, Ind-Share, Ind-NoShare.
Spread
Tag
Iverage Normal leed Score
100
120
D-Shars (Delutt)
Sha-Share
Average Noormalized Score
120-
00.
100
100
60.
60.
60
Exsort
KoRippay
Modium
Rardoms
Exper:
MM Maplyy
Modkur
Random
Ho-lasaay
Medium
Rancom
Figure 3: The average normalized score of MADIFF ablation variants in MPE tasks. The mean and standard error are computed over 5 different seeds.
As is obviously observed in Figure 3, with attention modules, MADIFF-D significantly exceeds that of the independent version on most tasks, justifying the importance of inter-agent attentions. The advantage of MADIFF-D is more evident when the task becomes more challenging and the data becomes more confounded, results on World, where the gap between centralized and independent models is larger, indicating the difficulty of solving offline coordination with independently trained models. As for the parameter sharing choice, the performance of MADaFr-D-Shar and MA DIFF- D- D-NoShare is similar overall. Since MADDFFF-D-Shal has fewer parameters, we prefer MADIFF- D-Share, and use it as the default variant to be reported Table 1. Another advantage of sharing U-Net parameters is that the trajectories of various agents can be batched together and fed through the network. This not only decreases sampling time but also renders insensitive to an increasing number of agents. We provide a specific example in Appendix Section G.4.

---

5.6 Limitations
Scalability to many agents. MADIFF-D requires each agent to infer all teammates' future trajectories, which difficult and unnecessary in environments with large number of agents. Although have done experiments on a maximum number of 8 agents (SMAC 8m), MADIFF-D in general not suitable for scenarios with tens hundreds of agents. A potential solution is infer latent representation of teammates" trajectories.
Applicability in highly stochastic environments. Several theoretical and empirical studies [Paster et al., 2022, Brandfonbrener et al., 2022, Chen et al., 2021] have demonstrated that in offline RL, sequence modeling algorithms tend to underperform Q-learning-based algorithms in environments with high stochasticity. This is primarily because sequence modeling algorithms are more susceptible high-reward offline trajectories that are achieved by chance. Since MADIFF a sequence modeling algorithm, it shares this weakness. To assess how much MIDIFF is affected by environmental stochasticity, we conducted experiments on the terran_5 5 vs__ map in SMACv2 [Ellis et al., 2022]. The design principle of SMACv2 is to add stochasticity to the original SMAC environment, including randomized initial positions and unit types. We conducted experiments under four settings: the original version, without position randomness, without unit type randomness, and without both kinds of randomness. MADIFF performs worse than the Q-learning baaningg-basddd method only when both kinds of stochasticity present. In all settings, MADIFF outperforms the sequence modeling baseline. Detailed experimental settings and results can be found in Appendix Section H.1.
Conclusion
In this paper, propose MADIFF, novel generative multi-agent learning framework, which is realized with an attention-based diffusion model designed to model the complex coordination among multiple agents. To our knowledge, MADIFF is the first diffusion-based offline multi-agent learning algorithm, which behaves as both a decentralized policy and centralized controller including teammate modeling, and can be used for multi-agent trajectory prediction. Our experiments indicate strong performance compared with a set of recent offline MARL baselines on a variety of tasks.
Acknowledgements
The SJTU team partially supported by National Key R&D Program of China (2022ZD01 Shanghai Municipal Science and Technology Major Project (2021SHZDZX01022) and National Natural Science Foundation of China (62322603, 62076161).
References
Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint 17X1v:1910.01465, 2019.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola. and Pulkit Agrawal. Is conditional generative modeling all you need for decision-man decision-making? making? International Conference on Learning Representations, 2023.
Michael A Alcom and Anh Nguyen. baller2vec++: A look-ahead multi-entity transformer for modeling coordinated agents. arXiv preprint arXiv:2104.1198) 2021.
Itamar Arel, Cong Liu, Tom Urbanik, and Airton G Kohls. Reinforcement learning-based multi-agent system for network traffic signal control. IET Intelligent Transport Systems, 4(2):128-135, 2010.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does retum-condiitioned supervised learning work for offline reinforcement learning? arXiv preprint arXiv:2206.010799 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
10

---

Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Fo- erster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:22 12.04499, 2022.
Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivar, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. arXiv preprini arXiv: (709:043900000 2017.
Claude Formanck, Asad Jeewa, Jonathan Shock, and Arnu Pretorius. Off-the-grit marl: a framework for dataset generation with baselines for cooperative offline multi-agent reinforcement learning. arXiv preprint arXiv:2302.0052.1 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
Philippe HatrunnEsttuch, lya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 6851, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 1212-122666 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902-9915. PMLR, 2022.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2 10ti::2208.0000 088323 2021.
Bingyi Kang, Xiao Ma, Chao Du Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning, Advances in Neural Information Processing Systems, 36, 2024.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 179-1191,,,, 2020.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning, Advances in neural information processing systems, 33:19783-19799 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-competiivee environments. Neural Information Processing Systems (NIPS), 2017.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024.
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li. Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. arXiv e-prints, pages arXiv-2112, 2021.
Frans A Oliehoek and Christopher Amato, A concise introduction to decentralized POMDPs. Springer, 2016.

---

Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi- agent reinforcement learning with actor rectification. In International Conference on Machine Learning, pages 17221-17237. PMLR, 2022.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:19210-19222, 2021.
Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. Advances in neural information processing systems, 35: 2022.
Bei Peng. Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34: 1220--12222,, 2021.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pages 4218-4227
PMLR, 2018.
Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning, pages 4257 9257-4266. PMLR, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-7288) 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1ew,I.0020044, 2019.
Jascha Sohh-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang. Order matters: Agent-by-agent policy optimization. In The Eleventh International Conference Learning Representations, 2023.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusson policies as an expressive policy class for offline reinforcement learning. arXiv preprint 1XXi::2208.06993 2022.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi- agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35: 15:16-99-16521, 2022.
Timon Willi, Alistair Hp Letcher, Johannes Treutlein, and Jakob Foerster. Cola: consistent learning with opponent-lcarning awareness. In International Conference on Machine Learning, pages 23804-23831. PMLR, 2022.
Chenjun Xiao, Yifan Wu, Chen Ma, Dale Schuurmans, and Martin Müller. Learning to combat compounding-error in model-based reinforcement learning. arXiv preprint 19Xiv:1912.1 1206, 2019.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022.
12

---

Yigin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang. and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312,, 2021.
Stephen Zhao, Chris Lu, Roger B Grosse, and Jakob Foerster. Proximal learning with opponent- learning awareness. Advances in Neural Information Processing Systems,
2022.
13

---

A Outline
In this appendix, we provide a table to explain the main notations we used in Section B. Section C, we give the pseudocode of multi-agent planning and multi-agent trajectory prediction with MADIFF model. In Section D, we demonstrate how multiple agents' trajectories are modeled by MADIFF during centralized control and decentralized execution in an example three-agent environment. In Section E, we give additional information on offline datasets, including how they are collected, violin plots of return distributions, and minor issue of MPE dataset. In Section F, we briefly describe the implementation of baseline algorithms and links to related resources. In Section G, we provide details of the experiments, including the normalization used to compute the average score, the detailed network illustration unrolling each agent's U-Net, crucial hyperparameters, and examples of wall-clock time and resources required for training and sampling from MADIFF. In Section H, we demonstrate and analyze additional experimental results. Specifically, we provide experiment results on SMACv2 demonstrate how much MADIFF is affected by environmental stochasticity. We also provide ablation results to support the effectiveness teammate modeling in MADIFF-D, show the quality of teammate modeling by MADIFF-D D on SMAC tasks, and visualize predicted multi multi-player trajectories by MADIFF and the baseline algorithm on the NBA dataset.
B Notations
Table 3: List of main notations used in the paper.
Notation
Description
S,A,
state, action, and local observation spaces the discounted factor
Y
N
number of controlled agents state at step
St
of
action and local observation of agent at environment step t joint action and observation of all agents environment step t shared reward function joint trajectory of all agents
at O
r(s, a
y(T)
additional conditioning information parameters of the inverse dynamics model
0
parameters of the diffusion model
noised observation of agent at diffusion step and environment step
The
noised joint observation at diffusion step & and environment step predicted observation of agent at environment step noised joint trajectory all agents at diffusion step k: historical trajectory of agent up environment step /
of
fk
bg
h
historical joint trajectory of all agents up to environment step t.
C Algorithm
D Illustration of Multi-agent Trajectory Modeling
To provide a better understanding of how multiple agents' observations are modeled by MADIFF in centralized control and decentralized execution scenarios, we show illustrative examples typical three-agent environment in Figure 4. If the environment allows for centralized control, we can condition MADIFF on all agents' historical and current observations, and let the model sample all agents' future trajectories as a single sample, as shown in Figure 4a. Then the current and next observations are sent to the inverse dynamics model for action prediction. If only decentralized execution is permitted, as shown in Figure 4b, agent can only condition the model on its own information. The historical and current observations of other agents are masked when performing conditioning. MADIFF now not only generates agent I's own future trajectories but also predicts the
14

---

Algorithm Multi-Agent Planning with MADIFF
1: Input: Noise model inverse dynamies La guidance scale w, history length C, condition y
2: Initialize h t Queue(length = C); t t
// Maintain a history of length C
3: while not done do
4: Observe joint observation o; h.insert(o): Initialize
aI)
5: for h K...
6: Tul: length(h) +
// Constrain plan to consistent with history
7: if Centralized control then
8:
é <T( k) (ee y, k) c0(Ta,k))
I/ Classifier-free guidance
9:
(1 Ex-1) Denoise(TR, else if Decentralized execution then for agent E {1, NN do
11:
12:
+ co(Tk + (T y¹ c)(TT, (nk-1: L. t Demoise(T))) (*)
I/ Classifier-free guidance
13:
14:
end for
15:
end if
Tk-1 (1
17:
end for
18:
Extract (or, O.+1) from
19:
for agent E 21, N do
20:
of < fa (o,, o(+1)
21:
end for
22:
Execute a, in the environment; f f ++1
23:
end while
Algorithm 2 Multi-Agent Trajectory Prediction with MADIFF
1: Input: Noise model Eg, guidance scale w, condition historical joint observations with length C. predict horizon H
2: Initialize TK ~ (0,a.))
3: for k K...1 do
4: Tk|: 6
Constrain prediction to he consistent with history
5: é ē To. k) wTe y, k) <o(Tk,)))
// Classifier-free guidance
6:
(px-1, Et-1) e- (T,,
7: Tk-1 N((p-,,,
8: end for
9: Extract prediction (oc; 0C+:0 OC+H-1) from To
current and future observations of the other two agents. Due to the joint modeling of all agents during training, such predictions also reasonable and can be considered from agent I's perspective. A Although teammate modeling not directly used in generating agent I's ego actions, can help agent refine its planned trajectories to be consistent with the predictions of others.
form of teammate modeling
E
Additional Information on offline datasets
E.1
MPE Datasets
For MPE experiments, we use datasets and a fork of environment provided by OMAR [Pan et al., 2022]. They seem to be using earlier version of MPE where agents receive different rewards. For example, in the Spread task, team reward is defined using the distance of each landmark to its closest agent, which is the same for all agents. But when agent collides with others. it will receive the team reward minus penalty term. The collision reward has been brought into the team reward in the official repository since this commit'. However, the fork provided by OMAR still uses
htteppr//ggttuuu..cooooooea/a//p wwwpiaaagtnnntttitiiaaee c envs/commit// beerreeeeetoo000444400020hh.eeee
15

---

History Homoon
Homion
History Horizon
Horizon
Agent
Agent:
42
Observations
Agent
%
Agant.
63
42
CondRtoored Observations
Agent
Agent
65
Time step
Current
Time steps
Current Time stop
(a) MADIFF in centralized control.
(b) MADIFF in decentralized execution.
Figure 4: Illustration of how agents' observations are modelled by MADIFF in three-agent environ- ment. Note that figure (b) shows the situation when Agent is taking action during decentralized execution.
legacy version. For fair and proper comparisons, OMAR's dataset and environment where all baseline models are trained and evaluated.
We have to note that different rewards for agents only happen at very few steps, which might not contradict the fully cooperative setting much. For example, OMAR's expert split of the Spread dataset consists of 1M steps, and different rewards are recorded only at less than 1.5% (14929) steps.
E.2 MA Mujoco Datasets
For MA Mujoco experiments, we adopt the off-the-grid dataset Formanek al. [2023] and use Good, Medium and Poor datasets for each task. Each dataset is collected by three independently trained MA-TD3 policies, and a small amount of exploration noise is added to the policies for enhanced behavioral diversity.
For visualizations of the distribution of episode returns in each dataset, we provide violin plots of all datasets we used in Figure 5.
E.3 SMAC Datasets
For SMAC experiments, adopt the off-the-grid dataset [Formanek et al., 2023] and use Good, Medium and Poor datasets for each map. Each dataset is collected by three independently trained QMIX policies, and small amount of exploration noise added to the policies for enhanced behavioral diversity.
For visualizations of the distribution of episode returns in each dataset, provide violin plots of all datasets we used in Figure 6.
F Baseline Implementations
Here we briefly describe how the baseline algorithms are implemented. For MATP experiments, we use the implementation from the official repository of Baller2Vec++*. Baseline results on MPE datasets are borrowed from Pan et al. [2022]. According to their paper, they build all algorithms upon modified version of MADDPG which uses decentralized critics for all methods. Baselines on SMAC datasets are implemented by Formanek et al. [2023], and the performances are adopted from their reported benchmark results. The open-sourced implementation and hyperparameter settings can be found in the official repository
"tteg///gggtttooco////a//////llp 'nstpa///github.co
hhtppp///ggthii....aaaarrrgragaa cow///aaarrgggggga11////aaagggg phyro pytorch
'nttps://githur wtt.tt/b..ooo/maaaccpe///ggoooaa
16

---

3000
Epizode Raturn
6000
2000
2000
Poor
Medium
Good
Medium
Good
(a) MA Mujoco 2halfcheetah.
(b) MA Mujoco 2ant.
Edlodd Retuns
-500
Poor
Medium
Good
(c) MA Mujoco 4ant.
Figure 5: Violin plots of returns in MA Mujoco datasets.
G
Implementation Details
G.1 Score Normalization
The average scores of MPE tasks Table are normalized by the expert and random scores on each task. Denote the original episodic return S, then the normalized score Smorm is computed as
Snmm 100 x (S Spundom // (S Srandom)
which follows Pan et al. [2022] and Fu et al. [2020]. The expert and random scores on Spread, Tag. and World (516.8, 159.8), (185.6, -4.1), and (79.5, -6.8,, respectively.
G.2 Detailed Network Architecture
In Figure 7, we unroll the U-Net structure different agents.
We describe the computation steps of attention among agents in formal. Each agent's 's local embedding is passed through the key, value, and query network to form q', k and v², respectively. Then the dot product with scaling is performed between all agents' g¹ and ki, which is followed by Softmax operation to obtain the attention weight u Each v can viewed as the importance of j-th agent to the i-th agent at the current time step. The second dot product is carried out between the weight matrix and the value embedding n² to get é after multi-agent feature interactions. Then é skip-connected to the corresponding decoder block. The step-by-step computation of multi-agent
17

---

20
Edizodo Ratuma
Poor
Medium
Good
Poor
Medium
Good
(a) SMAC 3m.
(b) SMAC 5m, vs 6m.
15
Eploade Retuns
Epicada
L0
Poor
Medium
Poor
Medium
Good
(c) SMAC 2s3z.
(d) SMAC 8m.
Figure 6: Violin plots of returns in SMAC datasets.
attention in MADIFF can be written as
fquery (c'), k2 fray(c'), 0² =valuo(cc))
exp(e /vTT)
2 = Em explp<hp/ /v)))
8 Σ aii
where de is the dimension of k
G.3 Hyperparameters
We list the key hyperparameters MADIFF we used in Table 4, Table and Table 6. In all of our experiments, we use scaling factor of 0.5 and 8 of 0.25. Return scale is the normalization factor used to divide the conditioned return before input the diffusion model. The rough range of the return scale can be determined by the return distributions of the training dataset. We only tune the guidance weight w, return scale, planning horizon H, and history horizon. We tried the guidance weight of [1.0, 1.2, 1.4, 1.6), and found that different choices do not significantly affect final performances, we chose 1.2 for all experiments. For MPE tasks, we find it unnecessary to condition on history observation sequence; thus, set all history horizons to zero.
G.4 Computing Resources and Wall Time
The training of MADIFF does not involve an iterative process, and thus, the training time is not related to the total number of diffusion steps. Thanks the property that the sum of two independent
18

---

Attention
Attention
Dot Prddudt
Saikel Dei Rodalt Soltean
Guery Natwark
Valua Natwak
Attention
Figure 7: The detailed architecture of MADIFF. Each agent's U-Net is unrolled and lined up in the horizontal direction.
Table 4: Hyperparameters of MADIFF on MPE datasets.
TestBed
World
Replay
Expen
Learning rite
Unidimee wale w
Planing hanzon
Iillnsion stys
Reward disewant ?
(130
Optimiizer
Adam Oplimiser
Table 5: Hyperparameters of MADIFF on MA Mujoco datasets.
TesiBed
2halfcheetah
4ant
2ant
Dataset
Good
Medium
Poor
Good
Medium
Poor
Good
Medium
Poor
Return scale
1000
300
100
380
320
150
380
320
130
Learning rate
2e.4
Guidance scale
1.2
Planning horizon II
10
History horizon
18
Batch size
32
Diffusion steps K
200
Reward discount
0.99
Optimizer
Adam Optimizer
Gaussian random variables remains a Gaussian, the multistep forward process can be written in closed form [Ho al., 2020]:
(TO)
(9)
19

---

Table 6: Hyperparameters of MADIFF on SMAC datasets.
TestBed
3m
232
Smbn
8m
Darasct
Good
Medium
Poor
Good
Medium
Poor
Good
Medium
Poor
Good
Modium
Poor
Return scale
Learning
2e
Guidance scale w
1.2
Planning borizon
History horizon
20
Batch size
32
Ditfusion steps
Reward discount
1.0
Optimizer
Adam Optimizer
Therefore, the k-th step noisy trajectory in Equation (6) can be easily sampled from the Gaussian distribution above without an iterative process. We provide a concrete example to illustrate the time and resources required for training MADIFF. On a server equipped with an AMD Ryzen 9 5900X (12 cores) CPU and an RTX 3090 GPU, we trained the MADIFF-C model on the Expert dataset from the MPE Spread task, achieving convergence in approximately hour, The curve depicting Wall-clock time spent on training and the corresponding model performance shown in Figure 8.
Average Episode Return
600
500
400
300
200
25
50
75
100
125
150
Wall-clock Time (min)
Figure 8: Wall-clock time and corresponding average episode return (average over 10 episodes) during training MADIFF-C for MPE Spread task.
Table 7: The wall-clock time spent when generating multi-agent trajectories. We fix the dimension of observation space to 88 and use DDIM of 15 steps during sampling. The history horizon is set to 20, and the planning horizon is 8. The results are obtained on a server with an AMD Ryzen 9 5900X (12 cores) CPU and an RTX 3090 GPU, and are averaged over 1000 trials. The computation time does increase much with the number of agents thanks GPU-accelerated computing.
Num. Agents (Incl. Ego)
16
32
64
clock Time
124.25 ms
126.90 ms
127.65 ms
127.35 ms
In Table 7, we showcase the time required for sampling multi-agent trajectories with MADIFF as the that the sampling time does not differ much when generating
number of agents increases. We different number of trajectories. Since we use shared U-Net models for all agents in our experiments, different agents' trajectories be batched together and passed through the network. Therefore, using GPU-accelerated computing, the second part does not cost much more time than predicting each agent's trajectory during inference.
H Additional Experimental Results
H.1 SMACv2 Experiments
To understand how much MADIFF affected by environmental stochasticity, we conducted experi- ments the terran 5 map in SMACv2 [Ellis et al., 2022]. SMACv2 is built upon SMAC with
20

---

focus on higher stochasticity. Specifically, in SMACv2, the unit types and agent start positions are randomized at the beginning of each episode. As each agent can only observe a nearby area, such randomness results in increased stochasticity in environment transitions. 'here are two different types of starting positions, reflect and surround. In reflect settings, the map splitted into two sides. Allied units and enemy units are randomly and uniformly spawned on different sides. In surround settings, allied units are spawned at the center of the map, and enemy units are randomly stationed along the four diagonals. In terran_5 vs 5, there are three different unit types: marine, marauder, and medivac. The default sampling probabilities of these three types are 0.45, 0.45 and 0.1.
We design four settings with different degree of stochasicity: the original version, without position randomness (w/o PR), without unit type randomness (w/o TR), and without both kinds of randomness (w/o PR&TR). To reduce position randomness, we only use surrounding settings. Note that this does not mean the staring positions of all units are fixed, since enemy units are still randomized along the four diagonals. To remove unit type randomness, we set all units to be marines. The dataset for the original version is the terran_5 __rran___ss___ __vs_5 Replay dataset from Formanek al. [2023]. Datasets for other three stochasicity settings were collected by ourselves. We partially trained three MAPPO" models in each setting. Each model was then used to collect 500 episodes, resulting in a dataset comprising 1500 episodes for each setting.
Three algorithms are benchmarked under these four settings: MAICQ, which represents the state- of- in Q-learning- based algorithms; MADT, representative multi- sequence modeling baseline; and MADIFF-D. Results are presented in Table 8. We can see that MADIFF-D performs worse than MAICQ only when both kinds of stochasticity are present. As the environmental randomness diminishes, MADIFF-D's performance gradually catches up with and surpasses MAICQ. In all settings, MADIFF-D outperforms MADT.
Table 8: The average score on different settings of SMACv2 terran vs_5. Shaded columns represent our method. The mean and standard error are computed over 3 different seeds.
Setting
MAICQ
MADT
MADIFF-D
Original
13.7 + 1.7
8.2 0.2
10.1 + 0.8
w/o PR
16.0 1.6
14.3 + 0.8
16.1 0.3
w/o TR
18.4
1.3
18.6 + 0.2
w/o PR&TR
17.3 4.3300 0.3
16.8 0.3
0.2
H.2 Effectiveness of Teammate Modeling
To investigate whether teammate modeling can lead to performance improvements during decentral- ized execution, conduct ablation experiments on MPE Spread datasets. We compare MADIFF-I) with its variant that adopts the same network architecture but masks the diffusion loss on other agents' trajectories during training. We denote the variant as MADIFFD w/o TM. The results are presented in Table 9, which show that teammate modeling results in notable performance improvements on all four levels of datasets.
Table 9: Ablation results of teammate modeling on MPE Spread datasets across 3 seeds.
Dataset
MADIFF-D w/o TM
MADIFF-D
Expert
93.4 + 3.6
98.4 = 12.7
Medium
35.4 + 6.6
53.2 :: 2.3
Md-Replay
17.7 1 4.3
42.9 11.6
Random
5.7 = 3.1
19.4 2.9
policcy
21

---

H.3 Teammate Modeling on SMAC Tasks
We show and analyze the quality of teammate modeling by MADIFF-D on SMAC. Specifically, we choose two time steps from episode on 3m map analyze predictions on allies' attack targets and health points (HP), respectively.
On top of Figure 9a attacked enemy agent ID (0, 1, 2 stands for EO, E1, E2) of ally agents AO, AI, and A2. The first row the ground-truth ID, and the second and the third rows the predictions made by MADIFF consistent with the ground-truth ID, As can be seen from the agents tend to focus their firepower on the same enemies at the same time. And the accurate prediction of allies' attack enemy IDs intuitively can help execute such strategy.
from the other two allies' views. We can see that the predictions are in general
values of the attack enemy ID,
In Figure 9b, we visualize the HP change curve ally agents starting from another time step. From the environment state visualization below, agent A2 is the closest to enemies, its HP drops the fastest, Such a pattern is successfully predicted by the other two agents.
Attack Enemy ID
Ally HP
05
1.0
A2
(a) bround-truthh and predicted enemy's ID to attack by each ally agent.
(b) Ground-trut and predicted health points (HP) of each ally agent.
Figure 9: The ground-truth and predicted information of different MADIFF agents at two-time step. On the top of each figure, each column describes different agent. The first row shows the change curve of the real value, and the last two rows below are the information predicted by other agents.
H.4 Predicted Trajectory Visualization NBA Dataset
We visualize the players' moving trajectories predicted by MADIFF-C and Baller2Vec++ on the NBA dataset in Figure 10. In each image, the solid lines real trajectories and the dashed lines trajectories predicted by the model. The trajectories predicted by MADiff-C are closer to the real trajectories and are overall smoother compared to the Baller2Vect+ predictions.
22

---

MADIN-CC
2aller2Vec+++
Figure 10: Real and Predicted multi-player trajectories by MADIFF-C and Baller2Vec+++
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract and introduction accurately reflect contri- butions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We point out the limitations of our method in Section 5.6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We illustrate model architecture in Figure I and list important hyper- parameters Appendix Section G.3. We also provide the source code in supplementary
materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code, anonymous data download link, and necessary instructions in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
wrriiissssooooilloo onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc ggwpi/gg///////eeeeee /Ce///eeeubbiiic Sestsubmissiooos ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental details in Section 5.1, Section 5.2, and Appendix Section E, Section F, Section G.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide eтror bars in Table 1, Table 2, and Figure 3. Reported error bar is standard deviation calculated over trials with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide concrete instances of both training and sampling wall time and resources of our algorithm in Appendix Section G.4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://nent //te:r//nne.../eeeee wwwpuriis.cppppppeee cttppblic///pcppttccccccccccccct
http/puulcc/Ethicceeeeeee /Ettcic//eeeeeee csGuidelines
Answer: [Yes]
Justification:
Our paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The proposed algorithm is general solution for wide range of offline multi- agent learning problems. In our opinion, there no specific societal impact that should be stated explicitly.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited and mentioned open- open- sourced implementation we used in Appendix Section F.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not publicly release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "M2kTtRvrUl", "reviewer": "Reviewer_vCUA", "summary": "The paper introduces MADiff, a generative multi-agent learning framework that leverages diffusion models (DMs) to address coordination challenges in multi-agent scenarios. It extends the previous work of using DMs for single-agent decision-making tasks, where the direct application to multi-agent problems is limited due to the complexity of inter-agent interactions. MADiff bridges this gap by using an attention-based diffusion model to coordinate behaviors among multiple agents. This model operates both as a decentralized policy and a centralized controller, enabling it to generate desired trajectories during online execution while considering multi-agent interactions and teammate modeling effectively.", "questions": "* In decentralized execution with teammate modeling, agent $i$ cannot access other agents' current observations. Under this case, the imagined agent $j$ 's current state $\\tilde x^j_{K, t}$ could be very different from the ground-truth current observation $o^j_t$? How will this gap influence the performance of the ego agent? Some visualizations of the heatmap of the attention layer for communication and interaction among agents might be helpful.\n* Following the previous question, is the centralized training procedure exactly the same for both centralized control and decentralized execution with teammate modeling?\n* In section 5.4, how do you define two paths are consistent?\n* In Line 307 and 308, it is said that \"the red agent and the purple agent generate 307 inconsistent plans,\" but I think the inconsistent plans are generated by purple and green agents (i.e., planning agents) right?\n* How will planning horizon $H$ affect the performance?", "limitations": "* This method is hard to be scaled to problems with a large number of agents.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The idea of the paper is straightforward and easy to understand.\n* The proposed framework is extensively tested on multiple environments and datasets, demonstrating its robustness and superior performance compared to baseline methods.\n* The results provide detailed and interpretable analysis of the important components of the paper, teammate modeling and attention layer for inter-agent communication.", "weaknesses": "* The proposed method is mostly evaluated across domains with a small number of agents (up to 8) and MADiff with decentralized execution requires each agent to predict all agents' trajectories, which is difficult to be applied to larger scales.\n* It would be beneficial to provide some discussions about how attention help with the teammate modeling for decentralized execution.\n* The clarity of section 2.3 could be improved. For example, what is $w$ in Equation (4)?\n* The subscripts used in Equation (7) and (8) are not consistent with the ones in Figure 1 ($\\tilde x^i_{K, t+1}$, $o^i_{t+H}$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: \"The proposed method is mostly evaluated across domains with a small number of agents (up to 8) and MADiff with decentralized execution requires each agent to predict all agents' trajectories, which is difficult to be applied to larger scales.\"**\n\n**A1:** Thanks for your comment! \n\n- Most existing works in offline MARL do not consider tasks with a large number of agents. For example, MADT-KD [1] and OMAR [2] are tested on environments with up to 8 agents. Meanwhile, our MATP experiments are conducted on NBA datasets with 10 agents. \n- As predicting all agents' trajectories introduces more computational cost, as shown in Appendix Table 6, the computation time does not increase much with the number of agents thanks to GPU-accelerated computing.\n- When the number of agents is extremely large, teammate agents can modeled by a latent diffusion model [3] with the agent dimension reduced, thus avoiding the curse of dimensionality. We leave this direction to future work.\n\n**Q2: \"It would be beneficial to provide some discussions about how attention help with the teammate modeling for decentralized execution.\"**\n\n**A2:** In a decentralized execution setting, although different agents make decisions independently, they can use the same attention network to jointly infer other agents' future trajectories and plan their own trajectories. Since the attention network is trained in a centralized manner, all agents' trajectories should be reasonable, and tend to be consistent given enough information. Intuitively, it requires the ego agent to first think at a higher level, i.e., make a coordinated plan for all agents, and then place itself in that plan. We will include more discussion of the role of attention modules in CTDE settings in our paper.\n\n**Q3: \"The clarity of section 2.3 could be improved. For example, what is $w$ in Equation (4)?\"**\n\n**A3:** Thanks for the careful checking! Equation 4 should be $\\hat{\\epsilon} := \\epsilon_{\\theta}(\\hat{\\tau}_k,\\emptyset, k) + \\omega(\\epsilon\\_\\theta(\\hat{\\tau}_k, y(\\tau), k)-\\epsilon\\_{\\theta}(\\hat{\\tau}_k,\\emptyset, k))$ .\n\n**Q4: \"The subscripts used in Equation (7) and (8) are not consistent with the ones in Figure 1.\"**\n\n**A4:** We have revised the subscripts in Figure 1 to be the same as those in Equation 7 and 8.\n\n**Q5: \"In decentralized execution with teammate modeling, agent cannot access other agents' current observations ... Some visualizations of the heatmap of the attention layer for communication and interaction among agents might be helpful.\"**\n\n**A5:** Great question, thanks! In decentralized execution, teammate modeling is most important for reasonableness given local information, not correctness (which is hard to achieve). Figure 2 shows that the purple agent trajectory predicted by the red agent differs from it's own plan. However, the three agents still cover different landmarks in the red agent's imagination, and is valid in completing the teamwork. The model also has the ability to adjust the prediction given more information from other agents is observed later.\n\nRegarding the potential problem you mention, the most extreme case is when some teammates are completely outside the visible range, at which point their imagined current observations could be highly stochastic and the model could not use those estimated trajectories to make decisions.  We verified on SMACv2 that the attention mechanism mitigates this problem. Figure 1 of the supplement PDF shows the attention scores in two environment states, and we found that the teammates outside the visible range do have lower attention scores.\n\n**Q6: \"Is the centralized training procedure exactly the same for both centralized control and decentralized execution with teammate modeling?\"**\n\n**A6:** Yes, the centralized training procedure is the same for CTCE and CTDE.\n\n**Q7: \"In section 5.4, how do you define two paths are consistent?\"**\n\n**A7:** In Section 5.4, consistency is defined on plans made by different agents. Two agents' plans are consistent *if the ego trajectories in both plans cover different landmarks*. Therefore, if inconsistent plans are executed without runtime correction, two agents will cover the same landmark, and the 'Spread' task will fail. We have revised the corresponding part of the paper for better clarity.\n\n**Q8: \"In Line 307 and 308, it is said that 'the red agent and the purple agent generate inconsistent plans,' but I think the inconsistent plans are generated by purple and green agents (i.e., planning agents) right?\"**\n\n**A8:** In Figure 2, at t = 0, the plans (dashed lines) in the first and second plots are generated by the red and purple agents, respectively (the planning agents are represented by triangular marks). The ego trajectories (starting from triangular marks) are both heading towards the middle landmark. Thus, the red and purple agents generate inconsistent plans.\n\n**Q9: \"How will planning horizon $H$ affect the performance?\"**\n\n**A9:** We did not delicately tune the planning horizon, and it is decided for each environment and kept the same for datasets from the same environment. The choice of planning horizon is a tradeoff between planning ability and computational efficiency. A short planning horizon can not take full advantage of diffusion planning, and an overlong planning horizon requires more computing resources and potentially more training data. We report the performance of MADiff-C with different planning horizons in SMAC 3m-medium dataset in Table 2 of the supplement PDF.\n\n**References**\n\n[1] Pan, Ling, et al. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification.\n\n[2] Tseng, Wei-Cheng, et al. Offline Multi-Agent Reinforcement Learning with Knowledge Distillation.\n\n[3] Rombach, Robin, et al. High-resolution image synthesis with latent diffusion models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n0vxIh7yz4", "reviewer": "Reviewer_nRbJ", "summary": "This paper proposes a novel diffusion-based offline multi-agent learning framework called MADiff. It extends previous diffusion-based offline RL work to offline cooperative MARL, particularly in CTDE and centralized control settings. The main contribution is the inclusion of an attention module in the diffusion model for integrating information and facilitating coordination among agents during centralized training. MADiff can be applied in both centralized and decentralized execution settings and performs opponent modeling in decentralized execution. It is evaluated in various settings such as SMAC, MPE, and the NBA trajectory prediction dataset, and the experiments show its effectiveness against competitive baselines.", "questions": "1. How are “Decentralized policy and centralized controller” related to \"centralize training decentralize execution\"?\n\n2. What is condition y during training in Algorithm 1 and 2? Is it trajectory return or what? Doesn't seem to be mentioned in the text.\n\n3. Please check whether the Classifier-free guided generation part in Equations 4 and 6 is represented correctly or explained intuitively. Although it is known that these equations are similar to (Ajay, 2023), the paper should ensure its self-containedness.\n\n3.1) is the \\omega a scalar or a netowrk using \\tau, k, etc as its input in Eq. 4?\n\n3.2) why is there a \\beta before \\emptyset? It seems that no difference will be made by using this term.\n\nIn all, there readability of this paper should be improved and intuitive description should be included properly.", "limitations": "Limitations on scalability to many agents and applicability in highly stochastic environments have been discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It extends the application of diffusion models to offline multi-agent RL and modifies the U-Net architecture for better coordination, addressing a significant problem with potential applications in multiple fields.\n2. MADIFF is a principled framework without complex components yet achieving effective performance, and its code is available for reproducibility.\n3. The proposed attention mechanism for information interchange among agents is suitable and helpful and is important in the offline MARL, as confirmed by the ablation study.", "weaknesses": "The weaknesses of this paper include:\n1. The novelty of the work is somewhat limited, being seen as a simple extension of Decision Diffuser to the MARL setting with an additional attention layer. \n3. Writing quality needs improvement, especially to a table to explain the mathematical symbols would help.\n3. Is it unclear how sensitive is the MADiff's performance to some critical hyperparameters, e.g., \n\n3.1) do you need to balance the weights of the two losses in Eq. (6)? \n\n3.2) how many encode-decode layers are required to perform satisfactorily in different tasks?\n\n4. The self-containedness needs improved and many symbols in the paper have not been explained properly (e.g., \\emptyset, \\omega and \\beta*\\emptyset in eq.4 and 6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: \"The novelty of the work is somewhat limited, being seen as a simple extension of Decision Diffuser to the MARL setting with an additional attention layer.\"**\n\n**A1:** It is indeed that many successful design choices in single-agent diffusion learning have inspired us and guided us to derive our method. Similarly, much of Decision Diffuser (DD) [1] also referred to the design of Diffuser [2], but that does not take away from the fact that DD is excellent work.\n\nAlso, many of the well-acknowledged multi-agent RL methods inherit a lot from single-agent RL in both online (MADDPG, MAPPO) and offline (OMAR, MADT) settings. MADiff is the first to apply diffusion models in multi-agent settings, naturally designed for multi-agent problems and taking multi-agent coordination, opponent modeling, and trajectory planning in a unified framework.\n\n**Q2: \"Writing quality needs improvement, especially to a table to explain the mathematical symbols would help.\"**\n\n**A2:** Thanks for pointing it out. We have made a table that lists and explains all the mathematical symbols used. Due to the space limit, we are unable to include it in the supplement PDF. We will add the table in the Appendix of our paper.\n\n**Q3: \"Is it unclear how sensitive is the MADiff's performance to some critical hyperparameters, ...\"**\n\n**A3:** Thanks for the question. \n\n- The diffusion loss and inverse dynamics loss are independent and used to train different model parameters, so one does not need to balance between them.\n- We did not tune the number of encoder and decoder layers for each experiment setting. We use three encoder layers and three decoder layers for all our experiments. Intuitively, for tasks with smaller observation dimensions (e.g., MPE), a smaller number of network layers may suffice.\n\n**Q4: \"How are 'Decentralized policy and centralized controller' related to 'centralize training decentralize execution'?\"**\n\n**A4:** Our proposed algorithm has two variants depending on the use case. Both MADiff-C and MADiff-D are trained in a centralized way.\n\n- MADiff-C: If the task setting allows for a **centralized controller** to set actions of all agents, the diffusion model can be used to generate trajectories for all agents conditioned on all agents' current observations. This is the **\"centralized training with centralized execution\"** case.\n- MADiff-D: If the agents are required to make decisions independently using a **decentralized policy** based on their local observations, the diffusion model is used to generate ego trajectory along with predictions of other teammate agents. This is the **\"centralized training with decentralized execution\"** case.\n\n**Q5: \"What is condition y during training in Algorithm 1 and 2?\"**\n\n**A5:** Condition $y$ refers to other conditions besides the historical (and current) observations used in trajectory generation . \n\n- In offline MARL experiments, the condition $y$ is the cumulative discounted reward in the offline dataset. During online evaluation, it is set to a relatively large value to generate high-rewarded behaviors.\n- In MATP experiments on the NBA dataset, the condition $y$ includes the ball's historical trajectories, player ids, and a binary variable indicating the side of each player's frontcourt, as described in Line 256-257.\n\n**Q6: \"Please check whether the Classifier-free guided generation part in Equations 4 and 6 is represented correctly or explained intuitively. Although it is known that these equations are similar to (Ajay, 2023), the paper should ensure its self-containedness.\"**\n\n**A6:** Thanks for your suggestion! Regarding your questions:\n\n- As mentioned in **A3** to reviewer vCUA, there is a typo in Equation 4 and the correct form should be  $\\hat{\\epsilon} := \\epsilon_{\\theta}(\\hat{\\tau}_k,\\emptyset, k) + \\omega(\\epsilon\\_\\theta(\\hat{\\tau}_k, y(\\tau), k)-\\epsilon\\_{\\theta}(\\hat{\\tau}_k,\\emptyset, k))$ , and $\\omega$ is a scalar to balance the conditioned and unconditioned model output. If $\\omega$ is set to 1, it reduces to pure conditioned generation. Prior diffusion model studies reveal that an $\\omega$ larger than 1 can yield better condition quality [3], as it explicitly forces the sample to stay away from unconditioned regions.\n- The empty set means the model is not conditioned on any extra information $y$. In practice, it is implemented as zero out the embedding of $y$. \n- $\\beta$ is not a constant, but a random variable which has the Bernoulli distribution (we choose $p(\\beta=1)=0.25$ for all our experiments). It means during training, the embedding of $y$ has a probability of 0.25 to be masked with all zeros. Therefore, the trained diffusion model can be used to provide both conditioned and unconditioned outputs in Equation 4 with the same set of parameters.\n\nWe have revised the explanation of the above symbols in the paper to ensure sufficient self-containedness.\n\n**References**\n\n[1] Ajay, Anurag, et al.  Is Conditional Generative Modeling all you need for Decision Making?.\n\n[2] Janner, Michael, et al. Planning with Diffusion for Flexible Behavior Synthesis.\n\n[3] Ho, J., & Salimans, T. Classifier-Free Diffusion Guidance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lLohKyYDKP", "reviewer": "Reviewer_hh3p", "summary": "This paper presents a novel generative multi-agent learning framework named MADIFF. The framework employs an attention-based diffusion model (DM) to address the complex coordination problems in multi-agent settings. MADIFF combines centralized training with decentralized execution, enabling effective teammate modeling and trajectory prediction. The experimental results demonstrate that MADIFF significantly outperforms existing baseline algorithms across various multi-agent learning tasks.", "questions": "- The motivation for using DMs to solve the extrapolation errors in offline MARL is unclear to me. Why using the DMs can solve it? What are the strengths of DMs compared to previous works?\n\n- The attention mechanism and DMs for single-agent reinforcement learning (RL) have been extensively studied in previous works. This paper appears to simply integrate these two methods into MARL. Could you please provide more details on how this approach differs from directly applying these methods in MARL settings?\n\n- How does MADIFF perform with an increasing number of agents in MARL systems? Additionally, I am very interested in how MADIFF addresses the challenges of adding a new agent to the existing MARL systems.\n\nIf you can address my concerns, I would be happy to improve my rating.", "limitations": "- This paper presents an unclear motivation and methodology. It would be great to have a more holistic picture of the problem and solution.\n\n- The evaluation primarily focuses on performance metrics; additional analysis on the robustness and adaptability of the model in diverse scenarios could provide more insights.\n\n- Related works should discuss using diffusion for single-agent RL more.\n\n- It would be greater to add some discussions of real-world applications.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper introduces an innovative approach by integrating attention-based diffusion models for multi-agent learning, which is a novel contribution to the field.\n\nS2: The framework effectively addresses coordination among multiple agents, which is a significant challenge in MARL.\n\nS3: Comprehensive experiments are conducted, demonstrating the superior performance of MADIFF across a range of tasks and benchmarks.\n\nS4: The paper provides detailed ablation studies, showcasing the effectiveness of the attention mechanisms in the diffusion model.", "weaknesses": "W1: The paper lacks a detailed discussion on the computational complexity and scalability of the proposed approach, which is crucial for practical applications.\n\nW2: The motivation of this paper is unclear.\n\nW3: The evaluation primarily focuses on performance metrics; additional analysis on the robustness and adaptability of the model in diverse scenarios could provide more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: \"The paper lacks a detailed discussion on the computational complexity and scalability of the proposed approach, which is crucial for practical applications.\"**\n\n**A1:** The **computational complexity** of MADiff during sampling is $O(KN^3)$. The following three points warrant clarification:\n\n- Within decentralized execution settings, each agent must make decisions independently. Consequently, one factor of $N$ in $KN^3$ is inevitable for all methods.\n- As shown in Figure 1, we adopt one-layer attention only before each decoder layer. Most of the computations are still independently computed within each agent's U-Net. This majority of computation scales linearly with agent numbers. If the agent number is not very large, the attention operation will not cost much effort.\n- For the diffusion step K, many off-the-shelf fast sampling techniques can be adopted. For example, with DDIM samplers, we can reduce the sampling steps from 200 to 15 with negligible performance loss.\n\nTo ensure we address your concern accurately, could you please provide more specific details on what aspects of **scalability** you would like us to focus on? For instance, are you interested in the scalability in terms of computational resources, dataset size, or large number of agents? Any additional guidance would be greatly appreciated.\n\n**Q2: \"The motivation of this paper is unclear.\"**\n\n**A2:** In fact, you have summarized the motivation of our paper quite well in the summary and strengths of the review: \"introduces an innovative approach by integrating attention-based diffusion models for multi-agent learning\" to \"effectively address coordination among multiple agents, which is a significant challenge in MARL\". We are continually revising the paper, especially the abstract and introduction sections, to emphasize our motivation more prominently. We are a little confused regarding the unclear motivation issue you mentioned. We would appreciate it if you could clarify it more precisely.\n\n**Q3: \"The evaluation primarily focuses on performance metrics; additional analysis on the robustness and adaptability of the model in diverse scenarios could provide more insights.\"**\n\n**A3:** Thanks for the suggestion. The online evaluation return is the only optimization objective in the problem formation of offline (MA)RL. Thus, we choose the return as the measure to compare with baseline algorithms. Regarding robustness, MADiff is not sensitive to hyperparameters. As listed in Appendix F.3, most of the hyperparameters are kept the same for all datasets. We only tune the reward discount, return scale, and diffusion horizons. Regarding adaptability, could you please specify what aspects of analysis or testing you would like to see?\n\n**Q4: \"The motivation for using DMs to solve the extrapolation errors in offline MARL is unclear to me. Why using the DMs can solve it? What are the strengths of DMs compared to previous works?\"**\n\n**A4:** As the similar question was proposed by Reviewer joCT, please refer to the general response **A1** for our answer to this issue.\n\n**Q5: \"The attention mechanism and DMs for single-agent reinforcement learning (RL) have been extensively studied in previous works. This paper appears to simply integrate these two methods into MARL. Could you please provide more details on how this approach differs from directly applying these methods in MARL settings?\"**\n\n**A5:** Our contributions include but beyond the model design, and the most important one is that it allows the same model structure to handle various tasks (CTCE, CTDE, MATP) by flexible conditioning during evaluation and achieves superior results over baseline algorithms. We conducted ablation experiments on model design and case studies about teammate modelling with diffusion models in multiple environments, all of which are part of our contribution and are novel. We believe that our work, being the first to apply the diffusion model in multi-agent learning, is valuable. To better address your concern, could you specify what you mean by \"directly applying these methods in MARL settings\"? Thank you.\n\n**Q6: \"How does MADIFF perform with an increasing number of agents in MARL systems? Additionally, I am very interested in how MADIFF addresses the challenges of adding a new agent to the existing MARL systems.\"**\n\n**A6:** As shown in Table 1 and Table 2, our experiments are done on various sets of MARL systems ranging from 2 to 10 agents, and achieve superior performances on most of them. Adding a new agent to the existing MARL systems is a different problem known as ad hoc teamwork [1], and is beyond the scope of our paper. To the best of our knowledge, there is no offline algorithm that tackles the ad hoc teamwork problem.\n\n**Q7: \"Related works should discuss using diffusion for single-agent RL more.\"**\n\n**A7:** Thank you for your kind reminder. We have included an additional related work subsection to elaborate more on prior studies that using DMs in RL.\n\n**Q8: \"It would be greater to add some discussions of real-world applications.\"**\n\n**A8:** MADiff has the potential to be applied in scenarios such as multi-robot collaboration, autonomous driving, and multi-player match data analysis (similar to our NBA experiments). We'll include a discussion of real-world applications in the paper.\n\n**References**\n\n[1] Chen, S., et al. Achieving the ad hoc teamwork by employing the attention mechanism.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dSwt6a69Lx", "reviewer": "Reviewer_joCT", "summary": "The paper introduces MADiff, a generative multi-agent learning framework designed to tackle the challenges of coordinating multiple agents in offline MARL. Leveraging the attention-based diffusion model, MADiff effectively captures complex agent interactions, enabling effective teammate modeling and trajectory prediction. Experimental results demonstrate MADiff's superiority over baseline algorithms across various tasks, highlighting its effectiveness in managing multi-agent interactions.", "questions": "1. Beyond being index-free, what other advantages does the attention mechanism provide? Can the agent observations be concatenated as a single input?\n2. The inverse dynamics model takes the local observations $o^i$ of the agents as input. If other agents are considered part of the environment, it should be non-stationary. Can the inverse dynamics model learn the environment dynamics stably and effectively under these conditions?\n3. Can the authors elaborate on how DMs avoid extrapolation errors in offline RL?\n4. Should $\\mathbb{E}{(s^i,a^i,s^{'i})}$ in Eq. (6) be modified to $\\mathbb{E}{(o^i,a^i,o^{'i})}$?\n5. Please see Weaknesses for additional questions.\n\nIf the authors address the above concerns, I am happy to raise the score.", "limitations": "The limitations are explicitly discussed in Section 5.6 of the paper.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes an offline MARL algorithm based on DecisionDiffuser, incorporating an attention mechanism to integrate information from other agents.\n- The paper is clearly written and easy to understand.\n- The algorithm is compared with multiple baselines, showcasing superior performance.", "weaknesses": "- The novelty is limited, as this work extends the diffusion-based offline reinforcement learning method Decision Diffuser from single-agent to multi-agent settings. The new addition is the attention mechanism, which is not a novel concept in the multi-agent domain.\n- Why the win rate in SMAC is not reported, but instead the average return? From my perspective, a high return does not necessarily equate to a high win rate (e.g., high-return conservative policies may not achieve overall winning).\n- Experiments use only three random seeds, whereas current MARL works typically use five or more to validate algorithm effectiveness.\n- Why was a diffusion model based on the U-Net architecture with an added attention mechanism chosen? As far as I know, combining transformers with diffusion models, such as DiT [1], yields stronger performance, with transformers inherently possessing attention mechanisms.\n- Although the paper claims to be the first to introduce diffusion models to offline MARL, **is MADiff's performance mainly due to the attention mechanism**? Comparing the performance of the variant version without attention (Ind) in the ablation study with baseline algorithms, I observed that it often underperforms compared to OMAR across most datasets. Therefore, can I infer that the performance advantage of MADiff mainly derives from the attention mechanism, which is common in MARL algorithms?  Applying the attention mechanism to other baseline algorithms would help assess the source of MADiff's performance gains.\n- As the number of agents increases, MARL faces the dimensionality curse of the joint action space, which becomes more pronounced with computationally heavy diffusion models.\n\n[1] Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: \"The novelty is limited ... which is not a novel concept in the multi-agent domain.\"**\n\n**A1:** Many successful design choices of the decision diffuser (DD) inspired and guided our method. DD also referred to the design of the diffuser [1], but it remains excellent work.\n\nCompared with DM, which focuses on SAL, MADiff is naturally designed for multi-agent problems. By using different conditioning during evaluation, the same framework handles multi-agent coordination, teammate modeling, and joint trajectory prediction, supported by solid experimental results. We acknowledge the long-standing use of the attention mechanism in MAL and verified its effectiveness in conditional generative modeling of multi-agent trajectories.\n\n**Q2:** **\"Why** **the win rate in SMAC** **is** **not reported, but instead the average return? \"**\n\n**A2:** We adopted offline datasets and the performance of most baseline algorithms from OGMARL, which reported the average return of their baseline algorithms instead of the win rate. Therefore, we chose the average return as the evaluation metric in Table 1. We have included the win rate of MADiff and our self-implemented baseline, MADT, in three SMAC maps in Table 1 of the supplement PDF. We will re-run the other baseline algorithms and the 8m map and update the win rates to the main paper.\n\n**Q3: \"Experiments use only three random seeds, whereas current MARL works typically use five or more to validate algorithm effectiveness.\"**\n\n**A3:** We conducted experiments with five seeds on three SMAC maps, as shown in Table 1 of the supplement PDF. Results generally align with our previous reported performances, showing MADiff's robustness to random seeds. We will update all experimental results to five random seeds once finished.\n\n**Q4: \"Why was a diffusion model based on the U-Net architecture with an added attention mechanism chosen?\"**\n\n**A4:** We use U-Net to model each agent's trajectory and the attention mechanism to coordinate across agents for three reasons:\n\n- U-Net-based diffusion models are still predominantly used in offline RL due to the convolutional module's ability to fuse locally consistent subsequences into globally coherent trajectories [1].\n- Attention modules applied to U-Net's skip-connected features, which are shown to constitute high-frequency information [3], allow agent interactions on the most critical parts of the trajectory, effectively steering the generation process.\n- A prior study found that while transformers performed better on some imitation learning tasks, they were generally more sensitive to hyperparameters [4].\n\n**Q5: \"Although the paper claims ... would help assess the source of MADiff's performance gains.\"**\n\n**A5:** Both diffusion modeling and attention networks are essential in our algorithm. Baseline Q learning algorithms use single-agent independent actors, and only their centralized critics can incorporate attention modules. To our knowledge, **no prior work uses attention-based critics in offline MARL**. In online MARL, attention-based critics [2] do not outperform QMIX's mixing value network. MADT frames MAL problems as sequence modeling with a fully attention-based network, and MADiff outperformed MADT in most datasets.\n\n**Q6: \"As the number of agents increases, MARL faces the dimensionality curse of the joint action space, which becomes more pronounced with computationally heavy diffusion models.\"**\n\n**A6:** The potentially high computational cost of MADiff comes from two parts: the iterative denoising process and the modeling of teammate agents.\n\n- Many off-the-shelf fast sampling techniques can be adopted to reduce the steps. For example, with DDIM samplers, we can reduce the sampling steps from 200 to 15 with negligible performance loss.\n- As mentioned in Section 5.5, we choose to use a shared U-Net model for all agents, where different agents' trajectories can be batched together and passed through the network. It does not cost much more inference time as the number of agents increases with GPU-accelerated computing, as reported in Appendix Table 6. For self-attention among agents' embeddings, it can also scale with linear complexity [5].\n\nWhen considering performance improvements over baseline algorithms, the higher computational cost of MADiff is still tolerable in many scenarios. \n\n**Q7: \"Beyond being index-free, what other advantages does the attention mechanism provide? Can the agent observations be concatenated as a single input?\"**\n\n**A7:** Attention is conducted on U-Net skip-connected features, allowing agent interactions on crucial trajectory parts, effectively steering the generation. We compare MADiff-C with concatenating all agents' observations (ConcatDiff) on three MPE spread datasets. Table 3 of the supplement PDF shows MADiff-C consistently performs better.\n\n**Q8: \"The inverse dynamics model takes the local observations oi of the agents as input ... stably and effectively under these conditions?\"**\n\n**A8:** The non-stationarity in the transition function $p(o'|o,a)$ does not necessarily imply non-stationary inverse dynamics $p(a|o, o')$. For example, in MPE tasks, the next observation depends on all agents' actions. However, knowing the current and next local observations allows easy determination of ego movement. Most of our benchmarked tasks don't show high non-stationarity in inverse dynamics.\n\nIf the inverse dynamics of certain tasks depend significantly on other agents, one can use DM-generated current and next observations of other agents as additional input to the inverse dynamics model.\n\n**Q9: \"Can the authors elaborate on how DMs avoid extrapolation errors in offline RL?\"**\n\n**A9**: Due to the character limitation, we response to this common question from you and Reviewer hh3p in **A1** of the general response.\n\n**Q10: \"Should E(si,ai,s′i) in Eq. (6) be modified to E(oi,ai,o′i)?\"**\n\n**A10:** Thanks for your checking! We have fixed the typo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to express our sincere gratitude to all the reviewers for their valuable feedback and insightful comments.  We appreciate the recognition of our paper's strengths, highlighted by the reviewers: **novelty** (nRbJ, hh3p), **superior / effective performance** (all reviewers), and **extensive experiments and ablations** (vCUA, hh3p). 

We have uploaded a one-page pdf file for additional results. Specifically, Table 1 reports the win rate of MADiff and MADT on SMAC datasets. Table 2 includes ablation study on different planning horizons on the SMAC 3m Medium dataset. In Table 3, we compare MADiff to a model variant that uses a single U-Net to process concatenated multi-agent observations without using attention, which is denoted as ConcatDiff. Figure 1 illustrates the attention scores of other teammates from one agent's perspective when using its own U-Net embedding as the query in two different steps in SMACv2.

As Reviewer hh3p and joCT have a common question regarding how DMs avoid extrapolation errors in offline RL, we choose to response to this question in the general response.

**Q1: "The motivation for using DMs to solve the extrapolation errors in offline MARL is unclear to me. Why using the DMs can solve it? What are the strengths of DMs compared to previous works?""Can the authors elaborate on how DMs avoid extrapolation errors in offline RL?"**

**A1:** The term "extrapolation error" in the context of offline MARL refers specifically to the estimation error induced by the target policy selecting an out-of-distribution $a'$ to calculate the $Q(s', a')$ in offline Q learning [1, 2]. As using DMs as trajectory planners does not involve Q learning, the extrapolation error does not exist by nature.

There are other algorithms that similarly solve offline RL without Q learning, such as decision transformers. DMs achieve superior performances mainly for two reasons:

- As a powerful generative model, DMs are suitable for modeling potentially diverse and mixed-quality offline RL datasets.
- DMs are skilled at handling high-dimensional data, which enables them to generate multiple future steps simultaneously. This results in better temporal consistency and fewer compounding errors [3].

**In the following responses, we address each reviewer's comments and provide detailed explanations. We are willing to engage in further discussion regarding specific issues.**



**References**

[1] Fujimoto, S., et al. Off-policy deep reinforcement learning without exploration.

[2] Fujimoto, S., & Gu, S. S. A minimalist approach to offline reinforcement learning.

[3] Janner, Michael, et al. Planning with Diffusion for Flexible Behavior Synthesis.

Meta-review: This paper presents MADIFF, an offline multi-agent RL using diffusion models to address the challenge of complex coordination among agents. MADIFF utilizes an attention-based diffusion model to simultaneously function as a decentralized policy and a centralized controller, enabling effective multi-agent interaction modeling. Its CTDE approach enhances multi-agent trajectory prediction and teammate modeling without additional costs. Experimental results demonstrate that MADIFF outperforms existing baseline algorithms across various multi-agent learning tasks, proving its effectiveness in handling offline multi-agent coordination. The paper highlights the framework's robustness and applicability in environments requiring both centralized control and decentralized execution.

Most of the weaknesses and questions are resolved by revision during author-reviewer discussion period. Authors are highly recommended to incorporate the revision into the final version of the paper.
Decision: Accept (poster)