DoF: A Diffusion Factorization Framework for Offline Multi-Agent Reinforcement Learning
Authors: Chao Li
Ziwei Deng
Chenxing Lin
Wenqi Chen
Yongquan Fu
Weiquan Liu
Chenglu Wen
Cheng Wang
Siqi Shen
Venue: ICLR 2025 Poster (2025)
Keywords: multi-agent reinforcement learning; Diffusion Models;   Offline reinforcement learning
Primary area: reinforcement learning
TL;DR: A multi-agent diffusion model that can be factored into multiple small diffusion models

Abstract: Diffusion models have been widely adopted in image and language generation and are now being applied to reinforcement learning. However, the application of diffusion models in offline cooperative Multi-Agent Reinforcement Learning (MARL) remains limited. Although existing studies explore this direction, they suffer from scalability or poor cooperation issues due to the lack of design principles for diffusion-based MARL. The Individual-Global-Max (IGM) principle is a popular design principle for cooperative MARL. By satisfying this principle, MARL algorithms achieve remarkable performance with good scalability. In this work, we extend the IGM principle to the Individual-Global-identically-Distributed (IGD) principle. This principle stipulates that the generated outcome of a multi-agent diffusion model should be identically distributed as the collective outcomes from multiple individual-agent diffusion models. We propose DoF, a diffusion factorization framework for Offline MARL. It uses noise factorization function to factorize a centralized diffusion model into multiple diffusion models. We theoretically show that the noise factorization functions satisfy the IGD principle. Furthermore, DoF uses data factorization function to model the complex relationship among data generated by multiple diffusion models. Through extensive experiments, we demonstrate the effectiveness of DoF.  The source code is available at [https://github.com/xmu-rl-3dv/DoF](https://github.com/xmu-rl-3dv/DoF).

Published as a conference paper at ICLR 2025
DoF: A DIFFUSION FACTORIZATION FRAMEWORK FOR OFFLINE MULTI-AGENT REINFOR LEARNING
Chao Li Ziwei Dength, Chenxing Lin Wenqi Chen", Yongquan Fu", Weiquan Liu Chenglu Wen Cheng Wang Siqi Shen
FFujann Key Laboratory of Sensing and Computing for Smart Cities,
School of Informatics, Xiamen University (XMU), China "Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China "School of Computer, National University of Defense Technology, China "College of Computer Engineering, Jimei University, China "School of Information and Software Engineering,
The University of Electronic Science and Technology of China (chaoli, denggniwwee lincxlll 11 23) @ssu. xmu, edu. cn,
[siqishen, owang, clwen, wqliu) @xmu edu. cn, (yongquanf] ) @nudt .dd.. cn, [chenwenqi) dstd. wwstc. edu. cn.
ABSTRACT
Diffusion models have been widely adopted in image and language generation and are now being applied to reinforcement learning. However, the application of diffusion models in offline cooperative Multi-Agent Reinforcement Learning (MARL) remains limited. Although existing studies explore this direction, they suffer from scalability or poor cooperation issues due the lack of design prin- ciples for diffusion- is a popular design principle for cooperative MARL.. By satisfying this principle, MARI algorithms achieve remarkable performance with good scalability. In this work, we extend the IGM principle to the Sedividull-Globaliiin addentically-Distribbiee (IGD) principle. This principle stipulates that the generated outcome of multi- agent diffusion model should be identically distributed as the collective outcomes from multiple individual- diffusion models. We propose DoF, diffusion factorization framework for Offline MARL. It uses noise factorization function factorize a centralized diffusion model into multiple diffusion models. We theoretically show that the noise factorization functions satisfy the IGD principle. Furthermore, DoF uses data factorization function to model the complex rela- tionship among data generated by multiple diffusion models. Through extensive experiments, we demonstrate the effectiveness of DoF. The source code is available at http://gtuuuu.cooommuuu-13//////
MARL. The addvidual-GGlobe Max (IGM) principle
INTRODUCTION
Generative diffusion models (Ho et al., 2020; Song et al., 2021b) have achieved great success in multiple domains such as image generation (Rombach et al., 2022). Due to the powerful modeling ability of generative modeling, researchers have applied diffusion model (Ajay et al.. 2023) to generate decisions in the reinforcement learning domain. In this domain, offline reinforcement learning approaches (Fujimoto et al., 2019; Kumar et al., 2020; Shao et al., 2023; Yang et al., 2021) learn policies from offline data logged by the operational system. With access to such data, for reinforcement learning, diffusion models can be used to learn probabilistic model of trajectories or actions (Ajay et al., 2023; Janner et al., 2022; He al., 2023; Wang et al., 2023).
The success of diffusion model in offline reinforcement learning domain (Ajay et al., 2023) motivates us to apply it in cooperative multi-agent reinforcement learning (MARL). There are a few diffusionn based MARL approaches that exist. In MADIFF (Zhu et al., 2024), centralized diffusion process (CDG) is trained to generate joint trajectories. During execution, the same CDG used to generate
Equua contribution Corresponning author

---

Published as a conference paper at ICLR 2025
Ground Truth
Genration
Landnare Agert
(a) Diffusion with gral
Independent Diffusion
MADIFF
(c) DoF
Figure I: Landmark Covering Game: three agents must reach different landmarks, avoiding collisions. (a)
Each agent (Ajay et al., 2023) goes to distinet goal provided by human. (b) Independent Diffusion Agents independently trained (d) DoF: agents cooperatively plan trajectories closely match trajectories (a). circle is the position sampled point of an agent's 's trajectory. Each color represents a distinct agent.
tested. (c) MADIFF: multi multi-agent plan trajectories using diffusion (Zhu al., 2024).
trajectories for each agent. It suffers from the scalability issue that the state and the action space increase exponentially with the number of agents. Another avenue of applying diffusion (Li et al., 2023) to MARL is learn independent diffusion model for each agent. Although this approach scalable, suffers from poor-cooperative issues that each independent diffusion model does not fully consider cooperation. It is challenging to address the scalability and poor-cooperative issues of diffusion models in cooperative MARL (Chen et al., 2024).
In MARL, the centralized training with decentralized execution (CTDE) paradigm is widely adopted. Under this paradigm, the individual-ghee -looal--oo----none -ma (IGM) principle (Rashid et al., 2018) is proposed to address the scalability and poor-cooperative challenges. The IGM principle requires that the collective greedy selection of action of each individual agent equivalence to the optimal action of a whole multi-agent system. Many excellent algorithms (Rashid al., 2020a; Son al., 2019; Qiu al., 2021) satisfied the IGM principle have been developed. However, the IGM principle is developed for value-based MARL (Hernandez-Leal IGM principle is only applicable for methods that learn factorized policies (Rashid et al., 2018; Shen al., 2022; Wang et al., 2021; Rashid al., 2020a). is unsuitable for planning- based methods (Ajay et al., 2023) that generate (or predict) future outcomes and plan based on predictions. A more general principle than the IGM principle lacking for diffusion- bused MARI.
al., 2018), not for diffusion-bar based MARL. Moreover, the
To address the above limitations, we propose the Individual-Glober Seaee--Gooaal-idiiiiii iddenicalll--Dstrriiii principle, which a generalization of the IGM principle if the diffusion process can generate deterministic actions exactly. It requires that the collectively generated outcome of each individual agent follows the same distribution as the generated outcome of a whole multi-agent system. Given a diffusion method that satisfies the IGD principle, centralized diffusion model (CDM) can be used to generate high-return data (e.g., trajectories or actions). Once trained, the CDM, parameterized by Orot, is factored into multiple small decentralized diffusion models (DDM), each parameterized by 0,- During execution, each agent uses a decentralized diffusion model to generate data. The collection of each agent's ' generated data follows the same distribution the high-return data generated by the CDM. The IGD principle is flexible, applying to both factorized policies and planners.
(IGD)
In this work, propose DoF, diffusion factorization framework for offline MARL. The same as other diffusion models, the forward process of DoF gradually adds noise into data, whereas its backward process does the opposite. DoF utilizes noise factorization function to ensure that the noise of multi-agent is equivalent to the combination of the noise of each agent. We show theoretically that the noise factorization function satisfies the IGD principle. As shown Figure 1, DoF generates data that matches ground truth better than other methods, which demonstrates the effectiveness of the noise factorization function. Further, DoF utilizes a data factorization function to model the relationship among data generated by agents.
For evaluation, we conduct extensive experiments on the StarCraft II MARL tasks (Samvelyan et al., 2019; Ellis et al., 2023), the Multi-Particle Environment (MPE) (Lowe al., 201 Multi-Agent Mujoco (de Witt et al., 2020), and several illustrative examples. The experimental results demonstrate the effectiveness of DoF.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
2.1 DEC-POMDPS
We consider cooperative multi-agent reinforcement learning tasks, which can modeled as De- centralized Partially Observable Markov Decision Processes (Dec-POMDPs) (Olichoek & Amato, 2016). In this work, agents do not communication, The Dec-POMDPs represented as tuple G (S, (4. P.r. ( )o,) N, for N agents. Please refer to Appendix A.1 for details.
2.2 THE INDIVIDUAL-GLOBALL
PRINCIPLE
For Pee-POMDPs, value function factorization methods learn factorized Q value functions, which are used for the execution of each agent. The
(IGM) principle proposed in (Son et al., 2019) is essential for the realization value function factorization. It is defined as follows.
Definition (IGM). For a joint state-uction value function Qj= T x U + R, where Ttat T a joint action- cestonnobservation history and u U is the joint action, if there exist individual state- action functions [Q: T, x Us + B i=1 such that the following conditions satisfied
arg max 2j (Trot; u) = (arg max Q1 u1),
arg max QN(TN,
(1)
then, [Q satisfy IGM Qj: under T. We can state that Qj (Tiot, u) can be factorized in terms of [Q 1
2.3 DIFFUSION MODELS
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a type of generative model that learns the data distribution p(æ=") from dataset DD It consists of the forward noising process and the reverse denoising process. In the forward noising process, the data-generating procedure is modeled by *( (x*) NN (1-ag)) I)), where x is a data sample, a& E R determines the level trainable process which can be modeled as po(* x**) = (Nu( , k)), Σg), where ua( k) is function of æ and noise éø(=* ( k). N(,, Σ) a Gaussian distribution with its mean μ and variance Σ. DDPM (Ho al., 2020) uses the following loss function to train the reverse denoising process.
of noise add data a The reverse denoising process
((0)
co(x" (1)11
(2)
The noise model c (x k) estimates the noise NNO. I) added to x for æ². Once the noise model learned, can be used to generate data. Please refer to Appendix A.2 for details.
RELATED WORK
Offline reinforcement learning algorithms (Fujimoto et al., 2019; Yang et al., 2021; Kumar et al., 2020; Chen et al.. 2021: Janner et al., 2021; Meng al., 2023; Tseng et al., 2022) learn policies from static operational logs, which circumstances the need for costly online exploration. DoF learns policy from static operational logs too.
In cooperative multi-agent reinforcement learning (MARL), the widely used IGM principle requires agents work together for common goal. This work introduces the IGD principle a generalization of IGM. To satisfy IGM, researchers have proposed various multi-agent reinforcement learning (MARL) value factorization methods (Sunehag et al., 2018; Rashid al., 2018: Son et al., 2019; Rashid et al., 2020a; Sun et al., 2021; : Qiu et al., 2021; Haoyuan et al., 2024), which decompose joint Q-value function into individual Q-values for each agent. These factorization methods, along with value/policy regularization and heuristics, have also been adapted for offline MARL (Yang et al., 2021; Jiang & Lu, 2021; Fujimoto & Gu, 2021: Shao et al., 2023; Pan et al., 2022), but they often suffer from value function approximation and off-policy learning issues.
Diffusion- based approaches make decisions by generating trajectories or actions. Diffuser (Janner al., 2022) generates trajectories through classifier- diffusion (Dhariwal & Nichol, 2021) and
3

---

Published as a conference paper at ICLR 2025
acts according to generated trajectories. Decision Diffuser (Ajay et al., 2023) enhances Diffuser by using classifier-free guidance (Ho & Salimans, 2021). DiffusionQL (Wang et al., 2023) uses diffusion models to generate actions. Although these approaches are flexible and high-performing, they are not scalable for cooperative multi-agent scenarios. For multi-agent settings, DOM2 (Li et al., 2023) uses independent Di diffusion process to make decision for each agent without fully considering cooperation.
The closest work to us is MADIFF (Zhu et al., 2024). It learns a centralized diffusion model (CDM) to generate trajectories. During execution, each agent uses the same CDM to generate trajectories. DoF learns CDM, which can be factorized into multiple smaller diffusion models that are used by each agent. During decentralized execution, the input complexity of the MADIFF diffusion model is o(n), where n the number of agents, whereas the input complexity of the DoF diffusion model o(1). DoF achieves better scalability than MADIFF, thanks to the noise factorization function. Please refer to Appendix A.3 for more discussion.
4 DoF: A DIFFUSION FACTORIZATION FRAMEWORK FOR OFFLINE MARL
4.1
MOTIVATING EXAMPLE
In Figure 1, three agents need to cooperatively explore all three landmarks in a short time while avoiding collisions. The most ideal case is that each cooperative agent goes to its closest distinct landmark. Figure ((a) shows implementation of the ideal case. Each agent is trained using decision diffuser (Janner et al., 2022) with human-given goal. In this Figure, the positions of each agent from 10 episodes are depicted as colored dots. Figure ((b) shows the results for independent diffusion (ID), where cooperation is not considered. Each agent learns independently, which leads to many collisions. Figure (cc) shows the result for MADIFF. It performs better than ID but causes collisions, too. Figure (dd depicts the results of DoF. For MARL, it is important for agents to collaboratively generate data that mimic cooperative behaviors in ground truth data.
4.2 THE
DISTRIBUTED D PRINCIPLE
Under the centralized training with decentralized execution (CTDE) paradigm, the IGM principle is widely followed to address the scalability issues and to promote cooperation. Through satisfying the IGM principle, the collection of greedy local actions decentralized agents is equal to the optimal jointed actions of centralized multi-agents. Similarly, for diffusion- -basdd MARL, it is important to learn the decentralized diffusion process aligned with the centralized diffusion process. However, the methods.
IGM principle is designed for basued methods, so it is not suitable for diffusion- design principle that generalizes the IGM principle is needed. We extend the IGM principle to the hiloooaal--dnntilll-----a-
(IGD) principle, which is defined as follows.
Definition 2 (IGD). For a joint total distribution Po.... (xfa)) == P' (atol ddx 1:K which is called
the reverse process, defined as a Markov chain po... (a..:K == p(= I Po (att ( with learned Gaussian distribution starting p( b(=kk) (0. E R where Ftot is the generated data, N is the number of agent, d is data dimension, K is the diffusion steps. After pe (x is learned to model ground truth distribution, if there exists a joint individual distribution function (x?) = ʃ (x? ]xx I where x E Rd is the data generated by agent i, such that the following conditions are satisfied.
K ~ (O,
N
II po. (x)) Per. (x 0, C Ozot
It indicates that the collection of generated samples æQ. identically distributed as wiat. We can state that (po, (x satisfy IGD for pe... (x) and the diffusion model pe... (2%.at is generatively factorized by diffusion models [po, (a:))]
The IGD principle is a generalization of the popular Individual-Global-e (IGM) principle if the diffusion process generates deterministic actions exactly. Let's take an optimal discrete action genera- tion (OAG) case as an example. If we view the generated data Feot as the optimal joint action Uoot = arg maDu.ar Qior ( (Trot Mtot), and each I; as the optimal local action ã; arg mar., Qr uz). The IGD principle requires that N p(x;) p( For the OAG case, the IGD principle requires that
4

---

Published as a conference paper at ICLR 2025
Trajectory
Policy
Data Factorization ACA
12.12.4
Agents
Noise Factorization [66 (n)) Rab 2
(b)
(c)
Figure 2: DoF Overview: (a) Diffusion Factorization: in each diffusion (forward and backward) step 1 2 l: S K, the noise factorization function is used factorize the noises e and intermediate data x; 2 backward step = 0, we apply the data factorization function to model the complex relationship among data generated by each agent. (b) DoF Trajectory Agent: generating trajectory for planning. (c) DoF Policy Agent: generating actions for execution.
the last
I1 p(arg mar., Qi(T,, u4) p(arg maT Qtor (Tot,, Mtot)). )) which is extension of the IGM principle. For more details, please refer to Theorems 4 and 5 the Appendix B.
The IGD principle requires more than diffusion process factorization. It also requires that the diffusion process should match ground truth distribution by maximizing the likelihood of data via diffusion learning objective. After the diffusion processes satisfying IGD are learned, they be used to generate data with desired properties through guidance.
The IGD principle is flexible. If we use diffusion generate optimal policy, the IGD principle becomes the Individual-Global-Opuee -loballOptimall principle (Zhang et al., 2021). For generating optimal risk-sensitive action, the IGD principle becomes the Risk-sensitive IGM principle (Shen et al., 2023).
4.3 DIFFUSION FACTORIZATION
The overview of DoF method depicted in Figure In DoF, for the forward diffusion process, each agent samples noise eţ E Rdd and they are combined to form the joint noise < E RN Nxd to noisify the data Fzot. In the backward diffusion process, the multi-agent system uses cauu (x k)) R.xd to generate alo where wf.t is the data generated in the k-th step. The forward joint noise Et backward joint noise 01. and joint data Ftot is factorized through using a noise factorization function f and a data factorization h, where are described as follows.
Eio s(ei, ety)
0 k S K
(4)
Eot
(2
tot k) f(E") (xf k). 6x (2 k)) OSKSK
(5)
xtot f fa xN 1 k
(6)
2ho = h(x*)..., xx)
(7)
The noise factorization function f mixes the individual noise oi and data a& of each agent to form joint noise Ei and joint data afot- Thanks to f. after the noise model eleat (at k), parameterized by Otot is trained, it can be factored into multiple small noise models c, (a/ k), each parameterized by 0,. 0, C 2xat Vi, 0: a = 0 * j. During execution, agent uses the noise model ex to generate data, its input complexity (O(1)) is only 1/N of the input complexity (O(N) of the joint noise model 9tot, where N the number of agents. In the last diffusion step, a data factorization function h is applied to No: to form 2.
8.
4.3.1 NOISE FACTORIZATION FUNCTION f
Noise factorization function ff factorizes the joint noise Coot into multiple individual noises c,, In this work, consider two noise factorization functions: Concat and WConcat.
Concat uses the concatenation function 2 as f. Concat assumes that the noise can be decomposed by dividing them according data dimension. For example, given noise etot NH,, 0) E RdxN
5

---

Published as a conference paper at ICLR 2025
the function f can factorize it into [e|]/ where [e,] E R and e; Cloe [(i consists of the elements from the (i (2-1) 1) x d-th dimension to the x d 1th dimension of Etot-
1) d d|. E,
In diffusion probability models, the noise Etot must be Gaussian Noise with diagonal covariance. According to statistics, a concatenation of diagonal covariance Gaussian noises is still a Gaussian noise. So we can use f to concatenate diagonal covariance Gaussian [c,] into Coot. After training, each agent can use e, (a* (a..Ai k) generate data z2. which can be used for making decisions. Albeit
the Concat function is simple, we show in Theorem I that the Concat noise factorization function satisfied the IGD principle.
Theorem 1. multi-agent diffusion model po (artot)
Prro (ziot) = / Poror (2 (ot D: 1:K
dar
(8)
tot tt al OSKSK
k = = CEN(H,O) O
(9)
(10)
Eot (ar k))
@ ,
(11)
is generatively factorized by [pe, 1-= The noise (etot and eạ) and the transition probability
(po (nttn k-1 | and po, (ar h- ( ) diagonal Gaussian distributions. @ is the Concat func- tion. pe, (x]) == pe, (x.:K) dx! E₁ is the noise added during the forward process. co, (a k) is used for the denoising process to predict the source noise ei ~ N0, I) that determines x from aQ.
WConcat is weighted version of Concat. It assigns an agent-specific weight k; E We show in Theorem 2 that WConcat satisfies the IGD principle. Please refer to Appendix B for proofs.
Algorithm and Algorithm 2 present the pseudocode for the centralized training and decentralized execution phases of the DoF algorithm, respectively.
Algorithm 1 Centralized Training
Algorithm 2 Decentralized Execution
1: repeat
1: K ~ ,,(00,1) (Initialize for each agent i)
x q(Xix) (Sample global data)
2: for k- = K...... 1 do
3: ~ Uniform( (1...., K}) (Diffusion step)
3: cx k) (Noise prediction by each agent i) Update state for each agent
4: I)O E RaxN (Sample global noise)
5: x Va
x = xii Xix
+1-- a
xd:ixx (1,..,,
1
X
1- Oik
cx(xx, 1i)
tonz
7: Can (ch, (xt, k)...., G k))
VOLK
VI
The
Take gradient descent step on:
where Z (N(, 1, clse z end for return x² (Final trajectory or action for each agent i)
Volle Emal 13
9:
until convergence
4.3.2 DATA FACTORIZATION FUNCTION h
The noise factorization function f is used to learn factored diffusion processes. However, the modeling power of f adopted in this work is limited. Thus, the generated = may match closely as the real data. To improve the generation quality of Ftot, the data factorization function h is used to mix [ make xie match real data closely.
The data factorization h could model more powerful data relationships than the noise factorization function f. For example, if we consider each diffusion process generates individual Q value x! Qi, and Ftot as the joint Q value function Qior. then h can be viewed as value factorization function (Rashid al., 2020b) that modeling the relationship Qtot = hu(1,, Q.)). Besides Concat and WConcat, we explore the use of value factorization functions and their variants as h.
4.4 DOF AGENTS
DoF can be used for generating a trajectory for planning, and can also be used for generating actions for execution. To demonstrate the flexibility of DoF, implement agents based on agents of

---

Published as a conference paper at ICLR 2025
Decision Diffuser and DiffusionQL, respectively. When DoF is used for generating trajectories actions, we call these methods DoF-Trajectory or DoF-Policcy respectively.
DoF-Trajectory use observation history as the data for diffusion. The clean data afot used by the centralized diffusion process is defined as.
f,0 2 tot
I+1 = Lo Oot ooo =110
(12)
where is the time step of MARL trajectory, oea is the aggregated observations at t.
For each decentralized diffusion process i, the data 2 o o ..... o o is used during diffusion, where of is the observation agent at t. Following (Ajay et al., 2023), we derive a policy from Of by using an inverse dynamics model to estimate actions, which is defined as ui := (ar, 2i 1.0), where D determines actions based on x. and n; 1+0,0 Please refer the details of the agents in Appendix C.3.
DoF-Policy use continuous action as the data for diffusion. The data Fiot is the action whot for the multi-agent system. After the centralized diffusion process is trained, each agent uses a factored diffusion process, parameterized by 0, to generate its action ut. Please refer the details the agents in Appendix C.4.
For diffusion process i, we use condition y/ to guide generated data toward desired properties. In cooperative MARL, high-return value R suggests cooperative behaviors. Thus, R is included in y: to guide the diffusion process to generate high-return data. Further, the local observation history T, of agent is included in the condition y/ to make the generated data align with т;. For MADIFE, condition includes R and Too where Too (Z), Tay Zn ), 21, are random noises.
EVALUATION
In this section, we evaluate (1) the ability to generate data that match ground truth by comparing DoF against two diffusion-basee MARL methods, (2) the importance of satisfying the IGD principle, (3) the ability to learn effective MARL policies from offline data, (4) the scalability of diffusion-based MARL. We justify the use of noise factorization and data factorization functions. In addition, we study the impact of different diffusion methods (Appendix D.5), demonstrate the ability of DoF to generate novel behaviors that satisfy multiple constraints (Appendix D.6), justify the use of condition information (Appendix D.7).
Unless otherwise specified, WConcat is employed as both the noise factorization function and the data factorization function. In default, the results for DoF- Trajectoyy are reported. Each experiment repeated with five different seeds. Please refer to the Appendix for details.
5.1 ILLUSTRATIVE EXAMPLES
The ability to generate data that match the ground truth distribution is the core ability of diffusion- based MARL methods. The studied diffusion- based MARL approaches are DoF, MADIFF, and Independent Diffusion (ID). The agent of DoF is the same that of ID.
We evaluate the algorithms on three multi-agent cooperation tasks: (a) A matrix game generating two dimensional data, (b) A Landmark covering game, and (c) Q value generating Game. We demonstrate the superiority of the generation ability DoF thanks to the IGD principle and factorization functions.
5.1.1 A MATRIX GAME GENERATING TWO DIMENSIONAL DATA
The ground truth data consists of four two-dimension Gaussian distributed data. Their mean-value located in the tep-left, top-right, bottom-left, and bottom-right of a data plane, and their variance are the same. The probability for generating the four Gaussian 0.5, 0.2, 0.2, and 0.1, respectively.
Each agent is responsible for generating one dimension of the data. The closer the method mimics the ground truth, the better the algorithm. The scatter plot of the generated data from each method is depicted in Figure 3. The distribution of the data across quadrants is depicted in red in the center of the graph.
7

---

Published as a conference paper at ICLR 2025
0.5
0.08
8.14
0.31
0.23
(a) Ground Truth
(b) DoF
(c)
WADIFF
(d) Independent Diffusion
Figure 3: Generating two dimension data: (a) Ground Truth, (b) DoF, (c) MADIFE, (d) Independent Diffusion
(alAverage Return
[b]aerage Comparison
(c]Average Comparison
30
Dut
25
Independent deticaton
20
40
10
20
Training Steps
Tea Saiining Steps
Toa kii Ste
Figure 4: (a) Return and (b) Collisions for Trajectory Generation, (c) Return for Policy Generation.
Table 1: Payoff Matrix Games and Reconstructed Value Functions
A
Q1
A
M2
A
Q1
1.0
0.
18.0
1.0
17.9
1.2
B
14.0
2.0
B
13.9
2.1
Game Payoff Matrix
(b) DoF
(c) Game Payoff Matrix
(d) DoF
As can observe from Figure 3, DoF performs better than MADIFF and ID both visually and quantitatively. The data generated by DoF aligns most closely with the ground truth in both the scatter and probability distributions, with MADIFF performing the second. ID uses the same agent as DoF, but without using noise factorization and data factorization. ID does not satisfy the IGD principle. ID learns 2 separated diffusion processes, which cannot jointly model the data distribution well. MADIFF does not satisfy the IGD principle. As discussed in Section 4.4 and Appendix A.3, due to lack of factorization, each MADIFF agent uses a noisy condition y, to guide diffusion. It does not generate data match the ground truth data well.
5.1.2 LANDMARK COVERING GAME
In the landmark covering game, the motivating example, the trajectories of each method are plotted in Figure 1. It shows that DoF can generate trajectories similar to ground truth, whereas others do not. Figure 4(a) shows that more power modeling ability of DoF lead to higher rewards than others. And Figure 4(b) shows that the modeling ability of DoF lead to less collisions than others.
Further, we study the the action generation cases for the MPE dataset. As it is demonstrate in Figure 4 (c), DoF outperforms ID, which does not satisfy the IGD principle, by a large margins. This game and the above games demonstrate the ability of DoF to generate data that matches ground truth, and the importance of satisfying the IGD principle.
5.1.3 GENERATING VALUES
The goal of the two-agent game is to reconstruct a one-step payoff matrix Qtor (joint Q value function) through two agents. Agent i use the diffusion process to generate individual utility value Q; and they are mixed into Qtot = h(Q1, Q2), where à is the data factorization function h(Q1, Q2) k Q1 + K2Q2. 0 K 1 are modelled following QAtten (Yang et al., 2020). As it is depicted in Table 1, DoF can reconstruct the payoff matrix Quat well (including the optimal policies). This demonstrates the flexibility of the DoF framework in generating different types of
8

---

Published as a conference paper at ICLR 2025
Table 2: The Average Return of the StarCraft Multi-Agent Benchmark (SMAC) Scenarios
Maps
Data
MABCQ
MACQL
MAICQ
MADT
MADIFF
DoF
Good
3.721.1
19.1
18.720.7
19.0=0.3
19.320.5
19.8=0,2
3m
Medium
4.0+1.0
13.720.3
13.940.8
15.8+0.5
16,422.6
18.6#1.2
Ponr
3.4+1.0
4.240.1
8.442.6
4.2=0.1
10.3+6.1
10.9=1.1
Good
4.8+0.6
5,4+0.9
19.6+0.2
18.5+0.4
18.9+1.1
19.6#0.3
8m
Medium
5.640.6
4.5+1.5
17.9+0.5
18.2+0.1
16.8=1.6
18.6:0.8
Poor
3.6+0.8
3.5+1.0
11.2+1.3
4.8
9.8=0,9
12.0#1.2
Good
2.4+0.4
7.4+0.6
11,0#0.6
16,8+0.1
16.5+2.8
17.7=1.1
5m vs 6m
Medium
3.820.5
8.120.2
10.620.6
16.1
15,2+2.6
16.2=0,9
Poor
3.3+0.5
6.8+0.1
6.6+0.2
7.6=0.3
8.9+1.3
10.8=0.3
Good
7.7=0.9
17.4+0.3
18.3+0.2
18.1+0.1
15,9+1.2
18.5+0.8
2s3z
Medium
7.6+0.7
15.6#0.4
17.0=0.1
15.1
15,6#0.3
18. 18.9
Poor
6.6+0.2
8.4+0.8
9.9+0.6
8.9
8.5
10.0=1.1
Good 5.9+0.3
7.8+0.5
13.5+0.6
12.
7.1+1.5
12.8+0.8
3s5z vs 3s6z
Medium
6.5+0.5
8,540.6
11.5+0.2
11.6+0.3
5.720.6
11.900.7
Ponr
6.1+0.6
5.9+0.4
7.9+0.2
5.6+0.3
4.7=0.6
7.5+0.2
Good
10.1+0.2
12.9+0.2
14.2=0.3
13.8+0.3
14.7+2.2
16.1=0.8
2e vs 642g
Medium
9.9+0.2
11.6+0.1
12.0+0.1
11.8#0.2
12.8
13.920,9
Poor
9.0=0.2
10.2+0.1
9.8+0.3
10.1
10.8
11.5+1.1
data. Moreover, this demonstrates that DoF can promote agent coordination through monotonicity among individual generated content z; and joint generated content Ptot-
5.2 COMPARISON STUDY
We evaluate the ability of DoF to learn effective MARL policies on the SMAC (Samvelyan et al., 2019), SMACv2 (Ellis et al., 2023), MPE (Lowe al., 2017b), and MA-Mujoco (de Witt et al., 2020) environments against seven multi-agent algorithms.
The seven algorithms used for comparison are from three categories: (I) Offline MARL: MABCQ (Jiang & Lu, 2021), MACQL (Kumar et al., 2020), MAICQ (Yang et al., 2021), OMAR (Pan al., 2022), and MA-TD3-BG (Fujimoto & Gu, 2021). (II) Transformer-based offline MARL: MADT (Meng et al., 2023). (III) Diffusion- based offline MARL: MADIFF (Zhu et al., 2024).
5.2.1 SMAC AND SMACv2
As shown in Table 2. DoF achieves the best performance across most of the datasets. MABCQ perform poorly. MACQL and MAICQ achieve good results on some good datasets but failed on moderate and poor datasets. Compared to these value-based algorithm, through using diffusion to generate trajectories, DoF does not learn value functions thus does not suffer from the challenges of value function approximation and off- aff-policy learning. MADIFF, diffusion- approach, is able to model trajectory distributions and consider cooperation some scenarios. However, it performs poorly in heterogeneous environments (e.g., 3s5z_ vs_3s6z). DoF performs better than it thanks the use of noise and data factorization functions. See Appendix D.3.1 for details and the win rate metric.
SMACv2 improves SMAC with more stochasticity. Its contents procedurally generated with heterogeneous agents. In the Appendix D.3.2, Table 10 shows that DoF achieves the best results.
5.2.2 MULTI-AGENT PARTICLE ENVIRONMENTS (MPE) AND MULTI-AGENT MUJoCo
For MPE, the results are shown in Table 11 in the Appendix. MADIFF and OMAR perform the second and the third in most cases. MADIFF performs well in good-quality datasets but under-perform in low-quality datasets. DoF demonstrates the best performance across various settings thanks to the power modeling ability of diffusion and the effective collaborative strategies learned through noise factorization and data factorization functions.
In the Appendix D.3.4, Table 13 presents the HalfCheetah results from MA-MuJoCo. DoF performs best on the Medium dataset and ranks second on the Good and Poor datasets.

---

Published as a conference paper at ICLR 2025
Table 3: Scalability Experiment: Comparison of DoF and MADIFF for Different Numbers of Agents
Metric
Method
Agents
8 Agents
16 Agents
32 Agents
64 Agents
DoF MADIFF
1691
2123
2831
4322
5924
GPU Memory (MB)
3121
5387
8412
14981
21862
DoF MADIFF
8.2
11.3
14.9
18.1
24.3
Inference Time Cost
12.9
16.5
23.9
31.5
0OM
DoF MADIFF
60.12
75.91
120.31
154.62 148.34
210.42
Reward
63.78
70.42
113.49
0OM
Table 4: DoF with different Noise Factorization Function f
Decentralized
Centralized
Maps
Dataset
MADIFF
DOF+MADIFF
Concat
WConeat
Dec-Atten
QMix
Atten
Good
19.7+0.6
19.8s0.5
4.3+2.3
3.8
19.8+0.4
19.340.5
19.7+0.4
3m
Medium
17.8+2.1
18.0=1.0
4.5+1.8
4.2+1.5
18.0=1.4
16.4+2.6
18.2=1.1
Poor
10.6#1.6
11.4=0.7
3.2+1.5
3.5+1.4
11.3+1.3
10.3+1.5
10.8+1.2
Good
16.7+1.4
17.0+0.8
3.611.5
4.1+1.2
17.1
16.5+2.8
16.7+1.2
5m_vs_ _mm_v_66
Medium
15.6
15.9&1.2
2.5+1.6
2.9+1.4
15.9a0.6
15.242.6
15.7+0.9
Poor
9.8+1.1
10.7+0.8
2,911.4
2.3+1.1
10.2+0.7
8.9&1.3
10.0
5.3 SCALABILITY AND ABLATION STUDY
Scalability Evaluation We evaluate DoF and MADIFF in a customized environment developed based on MAgent (Zheng al., 2018) with increasing number of agents. The experimental results depicted in Table 3. It shows that through diffusion factorization, DoF achieves better scalability than MADIFF. When the number of agents reaches 64, MADIFF encounters out-of-memory (OOM) error, whereas DoF does not. We also conducted scalability evaluation on network parameters, inference time, and rewards. Please refer to Appendix D.3.5 for details.
Noise Factorization function f. We study the impact of different f. which can be categorized into decentralized and centralized execution functions. The experimental results are depicted in Table Additional comparisons of Noise Factorization Function f on more complex maps are provided in Appendix D.4. Please refer Appendix D.4 for more details.
For decentralized functions, after the noise model Got learned, it can be factored into multiple noise models e For centralized functions, the noise model cannot be factored, and it should be executed centrally. For decentralized functions, The WConcat function performs better than Concat. For Dec-Atten, is trained using an attention mechanism. During decentralized execution, the weights of considering other agents omitted, which causes its poor performance. For centralized functions, the Atten function based on the attention mechanism performs the best. The QMix function performs poorly. This is due the fact that through using QMix, the resulting noise may longer be Gaussian noise, which is required for diffusion. Please refer to Appendix C. for details of f-
To demonstrate the flexibility of DoF, we introduce f into MADIFF by replacing the DoF-trajectory agent with the MADIFF agent. The new method, DOF+MADIFF, performs better than MADIFF.
Data factorization function ht We study the impact of three functions: Concat, WConcat, and Atten. The results are depicted in Appendix Table 18. The experimental results show that Weight-Concat performs better than Concat but slightly weaker than Atten.
6 CONCLUSION
For diffusion- based multi-agent reinforcement learning (MARL), we extend the Individual-Glohal Max (IGM) principle to the Sevvidual-Global-deeeee -deeniiiaally---isrrbueeed (IGD) principle, requiring that outcomes from a centralized diffusion process match those from combined individual processes. By satisfying this principle, after diffusion model is learned, it can be factorized into multiple small diffusion models. We propose DoF, which employs the noise factorization function to decompose a joint noise into individual noises, and use the data factorization function to model data relationships. We prove that these noise factorization functions satisfy the IGD principle. Experiments on multiple benchmarks demonstrate DoF's effectiveness.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was partially supported by the Fundamental Research Funds for the Central Universi- ties (No. 20720230033); by the National Natural Science Foundation of China (No. 62401225); by PDL (2022-PDL-12); by Xiaomi Young Talents Program, by the Fujian Provincial Natural Science Foundation of China (No. 2024J01115); by Natural Science Foundation of Xiamen, China (No. 350222024720181 the Jimei University Scientific Research Spart-up Funding Project (No.7Q2024034). We would like to thank the anonymous reviewers for their valuable suggestions.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In ICLR, 2023.
Jiayu Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei, Tian Lan, and Vaneet Aggarwal. Deep generative models for offline policy learning: Tutorial, survey, and perspectives on future directions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview. Survey Certification.
neWforumm
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, volume 34, pp. 15084-15097, 2021.
Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. volume 19, 2020.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthe- sis. In NeurIPS, pp. 8780-8794, 2021. URL https///ooccddnnn..uuuussccppppp 1/hash/ 99ad23alecctsssbt8/77a00022 ids Aaatssfc--bbtttt..ttll
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. In NeurlPS, 2023.
Claude Formanek, Asad Jeewa, Jonathan P. Shock, and Arnu Pretorius. Off-the-grid MARL: datasets and baselines for offline multi-agent reinforcement learning. In AAMAS, pp. 2442-2444, 2023.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In NeurIPS, volume 34, pp. 20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1597-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In /CML, pp. 2052-2062. PMLR, 2019.
Qin Haoyuan, Ma Chennan, Deng Mian, Liu Zhengzhu, Mei Songzhu, Liu Xinwang, Wang Cheng, and Shen Siqi. The dormant neuron phenomenon in multi-agent reinforcement learning value factorization. In Advances in Neural Information Processing Systems, 2024.
Hado Hasselt. Double q-learning. volume 23, 2010.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. NeurIPS, 2023.
Pablo Hernandez-Leal. Bilal Kartal, and Matthew E. Taylor. Is multiagent deep reinforcement learning the answer or the question? A brief survey. CoRR, abs/1810.05587, 2018. URL http://arriv.ogggbbs///1.......
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS, 2021.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. In NewrIPS, 2020. URL
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence
modeling problem. In NeurlPS, pp. 1273-1286, 2021. URL http:///poceeddngggneuuiu..c/cpp
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In ICML, volume 162. pp. 9902-9915, 2022.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning, arXiv preprint arXiv:2108.01832 2021. URL http::/aariv.oo///a22210...11.11
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS, 2020. URL http://wwccceeiingg...errsssccpp 0d2b2061882a5ddf33) 1aaeaaaff2221111115aa11ae011 1655885660222 Abt.0522t--ttaa.ttmm.
Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusion policies offline
multi-agent reinforcement learning. arXiv preprint arXtv:2307.01472, 2023. URL https://arxiv. org/abs/2307.01472.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-competiive environments. In NeurlPS, pp. 6379-6390, 2017a.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatie-competttive environments. volume 30, 2017b.
Calvin Luo. Understanding diffusion models: unified perspective CoRR, abs/2208. 11970, 2022. doi: 10.48550AARXI.2208.111 11900. URL http://l....../00/00/oo//.......
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, and Bo Xu. Offline pre-trained multi-agent decision transformer. Mach. Intell. Res., 20(2):233-248, 2023. doi: 10.1007/5111333-22--1333 URL
https://oi.orr//10 (007//11633-022--1 1383-7.
Frans A. Oliehoek and Christopher Amato. A Concise Introduction Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In ICML, pp. 17221-17237. PMLR, 2022.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. In NeurIPS, volume 34, pp. 12208-12221, 2021.
Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang. Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. RMIX: learning risk-sensitive policies for cooperative reinforeement learning agents. In NeurlPS. pp. 23049-23066 2021. URL http:///meeeeiigg.ennrr.i.copppp 1bbae4c76f4b- hattee44e77.----batrttttt..ttt Abstract.html.
/hash/
c2626d850c80ea0777777 1
Tabish Rashid, Mikayel Samvelyan, Christian Schröder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In JCML, 2018
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. In NeurIPS. 2020a.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. CoRR, 088/2903.08889, 2020b. URL https//arkixxory//s///00.....33
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CVPR, pp. 10674-10685, IEEE, 2022. doi: 10.1 (w0.9//CCPS222822022222..
URL https///oboorrg//.00000ov/Ssssss
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In AAMAS, pp. 2186-2188, 2019.
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Coun- terfactual conservative Q learning for offline multi-agent reinforcement learn-
ing. In NeurlPS, 2023. URL http://papers.niss.co/mia hwtpsscccppppe_ehie wttpppsccpppppeflee//aapppeeeeee flerr_lllle/aaee/222/sshh 3312f19779badddeb$Sccaaeee 1099
10b99c-Abstrase Olbbee-AbbtraccCCCfeeeeee...m
Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. Resq:
A residual function-basee approach for multi-agent reinforcement learning value factorization. In NeurlPS, 2022.
Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu, Yongquan Fu, Songzhu Mei, Xinwang Liu, and Cheng Wang. Riskq: Risk-sensitive multi-agent reinforcement learning value factorization. In NeurlPS, 2023. URL http://papers.niss.c/// httpiipsscppppppp..ppp hlaeeell///apee2222aaahs 2tt.777//000005550000000///aaaa0
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256-2265. JMLR.org, 2015. URL http://proceedings..lr mir. press/v37/ prapspe// Sohl-dickstein15.heg
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML, 2019.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In ICLR. OpenReview.net, 2021a. URL http:://peeereeieie.ee/m/mmmmddd giarCCLPP
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR. OpenReview.nnt,, 2021b. URL http:://ppeenvvvvv.ee//oouutttrp
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 3221 32211-32252. PMLR, 2023. URL http:///wweeddnnn...rmerrs//////
htmml
Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Y Yi Lee. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. ICML, 2021.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanetot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decoopoositio networks for cooperative multi-agent learning based on team reward. In AAMAS, 2018.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. In NeurIPS, 2022.
Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John Dolan, Jeff Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. In ICLR. OpenRe- view.net, 2024. URL http://openeerie..efffommmmmm oddiiioonnnndd--Giiiiiiii..
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In ICLR, 2021.
Zhendong Wang. Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In ICLR. OpenReview.net, 2023. URL https: diopeneeiievne////PoodAAAAPPPPp
13

---

Published as a conference paper at ICLR 2025
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. CoRR, abs/2002.03939, 2020. URL https///aniivarr/abb220222233993
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In NeurIPS. pp. 10299-10312. 2021. URL https:/proceecdnnsseurrisspsp
Tianhao Zhang, Yueheng Li, Chen Wang. Guangming Xie, and Zongqing Lu. FOP: factorizing optimal joint policy maximum-entropy multi-agent reinforcement learning. In ICML, volume 139, pp. 12491-12500. PMLR, 2021. URL http://prooceedins..ll pl..ssss//v 13tp/hang22/n11 mm.mmll.
Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In AAAI, pp. 8222-8223, 2018. doi: 10. V0.1609/AAAL.V321..111 URL https///oii.or// 1609/aaii.
v3311.11371.
Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang. Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models. CoRR, abs/2305.
2024.
14

---

Published as a conference paper at ICLR 2025
Appendix
A BACKGROUND AND COMPARING RELATED WORK
A.1 DEC-POMDPS
In this study, we investigate cooperative multi-agent reinforcement learning scenarios, which can be effectively modeled using Decentralized Partially Observable Markov Decision Processes (Dec- POMDPs) (Olichoek & Amato, 2016), a framework that handles environments where multiple agents must make coordinated decisions based partial observations and incomplete information. A Dec-POMDP is formally represented by the tuple G = (S, (4 P.r. [ (o N, for N agents.
Here, S represents a finite set of states, encapsulating all possible environmental configurations. Each agent i interacts with the environment using of discrete actions U4 At any discrete time the joint action of all agents is u' E 4 = U2 x x Uy, leading to state transition to g E S based on the transition function s P(- s u²). Each agent receives reward r from this transition, critical for learning optimal policies.
Due to partial observability. each agent i receives local observation of (s"),, reflecting limited state information and complicating decision-making. The enviromment's partial observability is captured each agent's local action-observation history Tę = (O; x x4)", extending from the start to the time horizon T. The joint policy π = (T1,..., .T) maps each agent's history to action probabilities.
A.2 BACKGROUND DIFFUSION MODELS
Diffusion models have recently emerged as a powerful class of generative models, particularly successful in image and audio synthesis (Sohl-Dickstein et al., 2015; Ho al., 2020; Song et al., 2021b). A diffusion model typically involves two main components: a forward (noising) process that gradually perturbs data, and a backward (denoising) process that learns to invert this perturbation to recover generate new samples.
Forward Noising Process. Given an initial data point x g(x²) from the empirical data distribu- tion, the forward diffusion process adds noise step by step to produce sequence of latent variables fx+1 x2 ,...,,:ff .. Each transition follows Gaussian: (1-a*)1) ,
g(x - Naa Va==*~1, (1
k= I, K,
(A.1)
where a E (0, 1) is a schedule parameter controlling the extent of noise injection at step k. After K steps, x becomes nearly an isotropic Gaussian variable if a is chosen such that (1 a²) sums to sufficiently large variance.
Backward Denoising Process. // I) and sequentially denoises via the reverse Markov chain:
To generate new samples, one starts from Gaussian noise æ
po (=
|
N (2*-1, p k), E,(x*, 1:)),
(A.2)
where encapsulates the learned parameters. The mean nx( k) and variance (o k) are trained (e.g., via variational bounds (Ho et al., 2020) or score matching (Song et al., 2021b)) to
approximate the true reverse distribution axa+~1 | x*) as closely as possible. After K denoising steps, one obtains z" ~ 5 as a generated sample.
The choice of Gaussian transitions in both forward and backward processes not arbitrary but rather arises from practical and theoretical considerations. First, isotropic Gaussian noising is mathematically convenient, allowing closed-form derivations and efficient sampling. Second, the reverse-time formulation exploits properties of Gaussian distributions make the training objective tractable. For more detailed derivations and discussions, see DDPM.
Hence, our use of Gaussian transitions in both the forward and backward stages of the Markov process aligns with the standard setup in DDPM and related diffusion-based generative models, rather than introducing novel contribution in this work.
15

---

Published as a conference paper at ICLR 2025
A.3 DIFFERENCES AMONG DoF, MADIEF, AND INDEPENDENT DIFFUSION
Here discuss the differences among DoF, MADIFF and Independent Diffusion.
Table 5: Comparison of DoF, MADIFF, and Independent Diffusion Algorithms
Methods
Decentrally executed
Input complexity
IGD
Num. of Processes
DoF (f=Concat)
Yes
O(1)
Yes
DoF (f=WConcat)
Yes
O(1)
Yes
MADIFF-D
Yes
O(n)
No
1
MADIFF-C
No
O(n)
NA
1
Independent Diffusion
Yes
O(1)
No
In Table 5, Decentrally Executed indicates whether the trained agent can be executed decentrally without communication. Input Complexity the input dimension for each trained backward diffusion step with respect to the number of agents n. All the methods listed in this table use U-net as its main neural network. Different from DoF and Independent Diffusion, the U-net used by MADIFF consists of attention mechanism. IGD indicates whether following the IGD principle. NA is short for not applicable. For methods that cannot be decentrally executed, the IGD principle is not applicable for them. Num, of Processes indicates whether the number of factored diffusion processes.
is the noise factorization introduced in Section 4.3.1, and their details are described in Section C.I. In this table, we compare different noise factorization functions f for DoF.
In DoF (f=Concat), the per-step backward-diffussee noises eg. It is expressed as c = (ei, ..... ).). In other words, eior [(i 1) d x d = < For execution, each agent decentrally generates its noise ef using its noise model to generate 24, which could be an action or a trajectory.
noise eoor is the concatenation of individual
f=WConcat extends the basic f=Concat by introducing learnable weight variable k; making the overall noise term etor (ior[(i- ( - x d :d:i: x di = kec, f=WConcat follows the paradigm of centralized training and decentralized execution. During execution, each agent generates z; based the noise c; and the corresponding weight variable k;, i.e., k;c, FonCot satisfies the IGD principle.
==Concat and f=Wconcat learn diffusion processes. Each process is used by separate agent to generate x; conditioned on observation тı and a return value R. For each agent, its input complexity O(1) with respect to the number of agents. The agents of f=Concat and f=WConcat can be executed decentrally.
MADIFF trains diffusion process for all the agents. After the diffusion process is trained. Each agent uses the same diffusion process to generate data. There are two variants of MADIFF; they differ in the way the process is used for execution.
The centralized execution variant, MADIFF-C, conditions on aggregated observation- history Teot = (TI,..,, Tn.) and a return value R to generate data Tiot. MADIFF-C can generate data that matches the original data distribution. If R is high, MADIFF-C can sample high-return data. Its input complexity O(n).
The decentralized execution variant (the default variant). MADIFF-D conditions on Tiat and return value generate data T₁. Tiat (21, ..., Tt ..., 2n), where 7, the local observation abservation-history history of and Zn are random noises. As agent can only observe except for the i-th element, all the other elements in Tio are filled with random noise. Its signal-to-noise ratio is low, only 1/n. Thanks to the modeling ability of the diffusion model, MADIFF can still generate data. However, it is unclear how the generated data matches the original data distribution. Moreover, we show theoretically that MADIFF does not satisfy the IGD principle. The input complexity of MADIFF-D is O(n). The input complexity is n times higher than that for DoF(f=Concal and DoF (f=WConcat).
If consider the combination of the diffusion model and inverse dynamics as the MADIFF-D agent network, then the condition Tie can be regarded as the input for this agent network. What we mean by random noise is that there is large amount of noise E EN(O,) 7) in the input =i.e for the
16

---

Published as a conference paper at ICLR 2025
MADIFF-D agent. We have borrowed the Formula of MADIFF as follows, where the left part is denoted as X and the right part as X
0g.
0 H-1
K,D
K-++H-1
X
Og
T --
Rerative K diffusion steps X
O H+H-1
IN KK
TR
ON
NN 1-
(A.3)
The input Tios for the MADIFF-D agent network corresponds to the first column of X. In Fiot except for the i-th entry Oj the other N entries consist of random noise ã E (O, T). Therefore, MADIFF-D contains more noise compared to DoF, which limits its ability to fully utilize the learned coordination. Once the input Tiot is provided to the agent network, the left column of X (the condition) is improved through multiple diffusion steps (the teammate modeling mechanism of MADIFF), shown in the first column of X However, because MADIFF-D'S 's condition consists of sigmificant noise, its performance is unsatisfactory despite the use the attention mechanism inside the diffusion process during training.
The random noise we refer to is the noise present in the input to the MADIFF-D agent network, not the noise generated during the diffusion process within the network itself. This excess noise in the input reduces the effectiveness of the coordination learned through diffusion. In contrast, MADIFF-C performs better than MADIFF-D as suffers from less input noise. However, MADIFF-C is not scalable and faces limitations such as the curse of dimensionality in Multi-Agent Reinforcement Learning (MARL).
In Independent Diffusion (ID), each diffusion process is trained to mimic the behavior of all the agents rather than a specific agent due the lack of noise and data factorization. For example, for two-agent scenario, process 1 learns the behaviors of agents 1 and 2, and so does process 2. With the use of noise and data factorization, the diffusion process i will learn the behaviors of agent rather than that of all the agents. As ID does not tried to model the collective behaviors, it does not satisfy the IGD principle.
DOM2 (Li et al., 2023) adopts actor-critic MARL approach. Each actor is developed base on DiffuionQL. (Wang et al., 2023). The corresponding critic (the Q function) of each actor are learned independently without considering others. As the critic does not fully taken into account the other agents, the learned agents do not fully consider cooperation.
B THE AUNNIDDA-GGLOAADDDDTTIAALLLDDLD
PRINCIPLE
Definition (IGD). For joint total distribution po (olo)) == poie (aotot ydo which is called the reverse process, defined as a Markov chain Po... (x = p(x II po.. (2 k- (x)a)) with learned Gaussian distribution starting as p( p NN0, () E RNxxd where Feot is the generated data, N is the number of agent, d is data dimension, K is the diffusion steps. After po (wx 0 is learned, if there exists a joint individual distribution functions [pe, (x?) = pe, (x. Idx where D; E Rd is the generated data, ax ~ NO, I), such that the following conditions are satisfied.
N II
po: (x.() po (ar 0: Ozot
(B.4)
indicates that the collection of generated samples x!, identically distributed as xf We can state that (po, (xx)) satisfy IGD for po (x) and the diffusion model pe... (x%o0 is generatively factorized by diffusion models (po, (ar))]
17

---

Published as a conference paper at ICLR 2025
Theorem 1. multi-ager agent diffusion model prloi (flot)
Pr (2iot) = / Po (ooo
0:K 1:K
dat
(B.5)
h = =b aic CEN(H,O) 0 K
(B.6)
tot
k + a OSkSK
(B.7)
Octt Etol (orot) k:)
2 E (aff, 1)
(B.8)
generatively factorized by [pe, (a:)) The noise (Exot and Ex) and the transition probability (Po... (atot b and pe, (at (x))) follow diagonal Gaussian distributions. @ is the Concat func- tion. pe, (2 = pe, pe, (e::K) dæ! E, is the noise added during the forward process. Eo. (af; , k) is used for the denoising process to predict the source noise eQ ~ NO, I) that determines x from aQ.
Proof. In the forward diffusion process, the global data too is modified as follows.
k Vataton
+ v1 V1-a8 ahe
(B.9)
a² = IIIat a
(B.10)
i=1
not = 1 OSKSK
(B.11)
We view the data xa as it is polluted by adding noise to x k-1 according to (B.13), where a is pre-specified hyper-parameter k is the diffusion step, # (0,
xx =roel(( [( 1) d: i x d i E [1,..N
(B.12)
x = va + v1 ake k-1
(B.13)
After the model trained, the noise model tot (Iiat, k) is factorized into multiple noise models 0s (zz, k). 0₃ C Otot 0 7 02 = 0. Pco = + [.], The noise model e (Ix, k) can be used for backward diffusion of Tg. For the backward diffusion step, the probability Pe, (x, (x/-- (1=]) of generating an based on - is defined as Gaussian distribution described in (B.14). The mean ue, (x, k) and the variance (kk) of the Gaussian distribution is defined in (B.15) and in (B.16). Formulas (B.15) and (B.16) are adapted from Luo (2022).
pe, ( - 1 = (a
1-1 Me, (x, k), Ekk))
(B.14)
a
Me, (x) kt) V x
E (a
(B.15)
VI kk Vak
E(k) = (1 - ac)(1 1)
(B.16)
The backward probability pa, (a (/+)) depends on 0 only. For the factorization function Concat, 0, n 02 = 0 i & j. Thus, Po, (x/ is independent from po, (af-"([a)) We can derive in equation (B.17) the joint conditional probability is equal to the product of two independent conditional probabilities.
palue, (,,-),,5
..) k- " a²)) (2) '1o/) x pa, (a) 1a3)) i i
(B.17)
The probability backward denoising step of atot written as follows.
Po (o/ l = pe, (xr k- ..... xx k 10 /loof,.., ..., x%))
(B.18)
Pe, (z) k- | x (2 8-1 (22))))))) x ... (a k- (1xx))
(B.19)
I pe. (x) - 1
(B.20)
18

---

Published as a conference paper at ICLR 2025
The generation process of ælot, parameterized by Ocot. can be written as Markov chain follows.
/ poroe (afot )da )dx
Prior (olot)
(B.21)
= planto)) / Poroe (artot Frot dar 1:
(B.22)
N
= [(att) / poror (otot aoot daat 1:k (ar ~(0, x ~ ~N(00,) (B.23)
1=1
N
N
I(axt
II xm
pa, po,(ar Mar )dx
(B.24)
2m1
II p(a)))
II pa, (ar 1 d [ar
ai.kk 1:K
1:K
(B.25)
6=1
N
/
N
II pla, 1=1
II pe,
(x) k- doo day dockk do -
(B.26)
p(xff() / po; (xx-- datt ...p(xx))) / " (B.27)
po, (zs) (c:)).. (2.))
(B.28)
II po, (x
(B.29)
We have shown that the probability paay (#lot) of centrally generating alot is equal to the product of probability po, (az)) for generating decentrally.
Theorem 2. multi-agent diffusion model pe (2tot)
Pr (F(at) = / 10 (ot darkok 1:K
(B.30)
di WE EE NEN(H, OSKSK
(B.31)
atot Waat osksk
(B.32)
Pcot (rh k) (af k)
(B.33)
Pcot= @00,1
(B.34)
is generutively factorized by [Po, (x:)) The noise (Exat and 6s) and the transition probability (po... (xt (xf and po, (xp-1) )) follow diagonal Gaussian distributions. W is the WConcat function, and @ is the Concat function. Po, (x:") = f po, (xt::K da E is the noise added during the forward process. Eo, (2 k) is used for the denoising process predict the source noise <f N(00 I) that determines x. from a.
Proof.
This theorem can be proved in the same way as the proof for Theorem 1
Theorem 3. The multi-agent diffusion model po... (a learned by MADIFF does not satisfy the IGD principle.
Proof. The diffusion model learned by MADIFF is defined as follows.
Po (x = f
po
O:K / da 1:K
(B.35)
After the model is trained, the diffusion model used by agent i is pe, (æ?), it is the same Pe... (xc ) parameterized by 0 Thus, 0; = o does not meet the factorization requirement the IGD principle that 0, Acot
19

---

We can use the IGD principle generate multi-agen actions. Let us treat the generated data æ as us where is the action taken by agent Then, the probability pe, (u:) becomes T; (u; Ti), where æi the policy of agent and r; is the local observation. Further, let us treat the generated total data Published as a conference paper at ICLR 2025
Theorem 4. IGD Generalization of the IGO Principle
The definition of the IGO Principle (Zhang et al., 2021): For an optimal joint policy toat (Utot
Ttot.) T + [0, 1], where Thot T is a joint trajectory, if there exist individual optimal policies
[mf(u; (u; mi) T x U + [0, Ill such that the following holds:
Tiot (Uto Ttot)
II
क T Tz),
(B.36)
then we say that [ satisfy IGO for Teot under Ttot. The IGD principle requires that
N II Po. (2() = Poene (2)at)) 1-1
(B.37)
where x! the generated data of agent i, and ufor is the generated data of whole multi-agent system. Here, po, (a:)) is the probability of z.., and po.. (#2oat is the probability of =hot-
xfot utot- The probability Po (ueot) becomes Too (ueot Teot), where toa is the policy of the multi-agent system, and Too is the aggregated observations. Then the IGD principle becomes the following formulas:
II pe, (u:) Proot (utot),
(B.38)
II < Ti) Troo (Utot Ttot).
(B.39)
The above formula requires that for any policy æ. the total policy Teot is equal to the product of its per-agent policies. By substituting the optimal policy xf for the policy and the optimal total policy Ti for the policy Toot, we obtain the following formula:
II
T (1
Ti) Tiot (Ueot Ttot),
(B.40)
N II 1=1
Too (Uoot
Teot.)
प (u: Tz)-
(B.41)
The formula is exactly the requirement of the IGO principle. Thus, we have shown that the IGD principle is generalization of the IGO principle.
Theorem 5. IGD Generalization of the IGM Principle.
We have shown that the IGD principle is generalization of the IGO principle. And the IGO paper Zhang et al. (2021) shows that the IGO principle a generalizatioin of the IGM principle, thus the IGD principle is generalization of the IGM principle.
Theorem 6. IGD Generalization of the RIGM Principle.
The definition of the RIGM Principle (Shen et al., 2023): Given risk metric v.a, set of individual return distribution utilities [zs (7r, u;)];" and a joint state-action return distribution Zror (Trot, Utat)) the following conditions are satisfied:
arg max dia (Zroe ZZooe(Tor. (Tror, upot)) = arg max vaa (Z1(11, u1)) ...., arg max d/a (Zx(TN uN)) uN
(B.42)
20

---

Published as a conference paper at ICLR 2025
where Va x R. - R risk metric such the VaR distorted risk measure, is its risk level. Then, [7g (T; ui)l=z satisfy the RIGM principle with risk metric va for Zye under under We can state that Zior (Teot: Utot) can be distributionally factorized by [Z, (T,, uz ut)] with risk metric vaa
Let's define probability functions Tool Urot Ttot) and (T, TT). Teod Urot ( Ttot) 1. when
Usot arg maxu Vex (Zeot(Tee (Toot: Utot)) )) and is O otherwise. For Ty (u: | Ti), <i(4; T.) = 1, when M4 arg maxu Va (T, u:),, otherwise O. The RIGM principle becomes the following formula:
N II 1=1
Trot (tto Ttot)
T TC)-
(B.43)
The same as the IGO case, we can use a diffusion model to generate risk-sensitive multi-agent action. Let's s treat the generated data xQ as risk-sensitive action of agent i. Then, the probability Po, (x.) = Pe: (u,) becomes Fy (u4 7,), where т,, the risk-sensitive policy of agent and т; is the local observation. Further, let's treat the generated total data xfot Utot. The probability Pr (Xtot") ) Ph (Urot) becomes Tror (Utot Ttot). where Toot is the risk-sensitive policy of the multi-agent system, and Toot is the aggregated observation.
N
II pa, (x 1=1
PPcce (ztot)
(B.44)
po. (u:) Pocor (utot);
(B.45)
N
II 1=1
Tu Ti) = Toot (Utot Ttot),
(B.46)
N
Teol Utot Ttot)
=
II 2m
T Tz).
(B.47)
We have shown that the IGD principle is generalization the general RIGM principle. Thus, the IGD principle is generalization of the RIGM principle.
C DOF ALGORITHM
In this section, we will provide a detailed explanation of the noise factorization functions proposed in the main text. We will discuss the basic principles of each method, explaining the mathematical models and algorithms that form the core of their functionality.
C.I NOISE FACTORIZATION FUNCTIONS
C.1.1 f=CONCAT
In this section, we provide detailed overview of the f=Concat noise factorization function. We start with noise term étot M(,, 0) RdxN where Eto is modeled Gaussian noise with mean vector and covariance matrix 0, covering d x N dimensions. The mixer function f decomposes Eoo into N smaller noise vectors [ each 6 Fd This decomposition is specifically done as ezo (i - 1) x d x d). meaning each e contains the elements of eat from the (i 1) x d-th dimension to the i x 1-th dimension. By this partitioning. we ensure that each e, retains the properties of the original Gaussian noise within its respective subspace.
In diffusion probability models, the noise Étor must be Gaussian with diagonal covariance. This requirement ensures that the noise components are uncorrelated and independently distributed across different dimensions. A notable property of Gaussian noise with diagonal covariance is that the concatenation of multiple such Gaussian noises remains Gaussian. Mathematically, if each e, follows a Gaussian distribution with diagonal covariance, then the combined noise vector eoo formed by the mixer function f also maintains a Gaussian distribution with diagonal covariance.
21

---

Published as a conference paper at ICLR 2025
We further analyze the statistical properties of Gaussian distributions. Assume ( 8,) for i = 1,2,.... ... .N. where each 8 represents a diagonal covariance matrix. When these noise vectors are concatenated, the resulting noise vector Eto @ c has a mean vector Moot M2: M and a block-diagonal covariance matrix 2zt.. where each block on the diagonal corresponds to 0z. This confirms that éto remains Gaussian with diagonal covariance.
C.1.2 f=WCONCAT
In this section, we provide detailed overview of the f=Wconcat noise factorization function, which utilizes learnable weight variables to combine Gaussian noise. The mixing function f for f=Wconeat is defined as follows:
Goo (ar k y, (i 1 x d: : x d] = k,,,((;; k-)
(C.48)
Here, the coefficients k22 are trained through learnable weight variables. For the concon case, since each E; follows Gaussian distribution with diagonal covariance, the scaled noise term k also retains a Gaussian distribution with diagonal covariance. Therefore, when the scaled vectors KiG still follows a Gaussian distribution with K,G are concatenated, the resulting vector Got = &ic diagonal covariance.
C.1.3 f=ATTEN
In this section, we provide a detailed overview of the f=Atten noise factorization function, which utilizes an attention mechanism combine Gaussian noise. Specifically, the mixing function f for f=Atten is defined as follows:
Coor y, k) [ x d: = Σ
w (.
(C.49)
Here, wf are weights computed using multi-head attention mechanism, reflecting the relative importance of each source of noise in the current context.
The multi-head attention mechanism widely used in natural language processing and computer vision due to its ability to dynamically assign different weights to different imputs, thereby highlighting significant information. In the f=Atten, the multi-head attention mechanism captures information across different dimensions through multiple attention heads. Each attention head computes a set of weights, and the results are then aggregated to generate the final combined noise. This design enhances the model's 's expressiveness and flexibility, making it more effective in handling complex tasks.
Crucially, despite the complexity of the attention mechanism, the f=Atten function maintains the statistical properties of Gaussian noise. Since a linear combination of diagonal covariance Gaussian noises remains Gaussian, the total noise Coot generated using the mixing function f retains its Gaussian distribution. This characteristic is vital for ensuring the stability and consistency of the generation process.
However, the f=Atten method has certain limitations when it comes to decentralized execution. During the backward denoising steps, each agent needs access the noise information from other agents to generate its state xzf This requirement limits the applicability of the f=Atten method in decentralized settings, as each agent cannot independently complete the denoising process.
C.1.4 f=QMIX
Similar to f=Atten, through using f=QMIX, each agent cannot independently complete the denoising process. The f=QMIX noise factorization function adapts the QMIX architecture, to combine Gaussian noise. This method employs mixing network that takes individual agent noises input and produces combined total noise. The mixing network's 's weights are generated by a hypernetwork, allowing for state-dependent mixing of noises.
C.1.5 Loss FUNCTION FOR NOISE FACTORIZATION
In this section, we discuss the loss functions for different noise factorization functions: f=Concat, WConnatt
22

---

Published as a conference paper at ICLR 2025
For f=Concat, the loss function is defined as follows:
I(0, 6) : D., (r.B((p)))))
|||e
6x not (1- 80, -,0
E (
||ue fo(o,
(C.50)
where
Etot (x y, [(i - x xd: (x
(C.51)
eg represents the parameterized noise model applied across all agents.
For each trajectory т, noise is sampled from a normal distribution (0, /), and timestep k: is selected from uniform distribution uf Kj. A noisy array of states 2x((7) is constructed, and the model predicts the noise as Eg == EØ ((() y(r), k). With probability the conditioning information y()) may be ignored, and the inverse dynamics model fø o') trained to predict the action leading from observation to o'. The term ef is predicted noise function, and B,, sampled from Bernoulli distribution, determines whether the condition y(r) is used or replaced by empty set. The dataset D contains the trajectories utilized for training. This loss function effectively balances the model's capability to predict noise and capture the underlying dynamics of the system.
For WConcat,, the loss function follows:
T) (,,rED, ||| - c0a0
90, A]]]
+ (s ||u fo(o,
(C.52)
where
Etot (x* ky, [(i 1) d d kec y:
(C.53)
k is trained through learnable weight variables, assigning specific weight each noise model.
C.2 DATA FACTORIZATION FUNCTIONS h
In this work, we use the data factorization function h to model relationship among generated data. All the noise factorization function f (Concat, WConcat, Atten, and QMIX) can be used the data factorization h.
For Coneat, the data factorization function is follows:
Srot [( 1) d: d| 2;
(C.54)
For h = WConcat, the data factorization function is as follows:
Ftor ( - 1) x d d] = kyr,
(C.55)
k is trained through learnable weight variables.
For h Atten, the data factorization function is follows:
N
Ftot [[i - 1) x d: d:
Σ
w
(C.56)
h is trained through learnable weight variables.
For Qmix, the data factorization function as follows:
Ptot h(11 #2. ,.,)
(C.57)
Where, h is monotonic non-linear function.
23

---

Published as a conference paper at ICLR 2025
C.3 TOF-TRAJECTORY
In the trajectory-based version of our reinforcement learning model, we focus on modeling states using diffusion processes. Since action sequences in reinforcement learning are typically discrete and noisy, while states are continuous, we apply the diffusion model to states rather than actions. The state sequence within trajectory segment represented as:
(T) == [0t; 1t
H1+
(C.58)
where & denotes the time step and specifies the time of accessing states in trajectory т. The sequence x²(r) is treated as noisy state sequence over the H-step prediction horizon. In the forward training process, leverage the diffusion model for planning, the diffusion process is conditioned on the trajectory return v(T), employing unsupervised classification and low-temperature sampling, thereby extracting shgh-probbbility optimal trajectories from dataset containing suboptimal paths.
For multiple agents, we introduce Noise factorization function f(-) to integrate the noise generated by each agent's diffusion model:
f cg..
(C.59)
Noise factorization functions can stacking, attention mechanisms, or adaptive dynamic program- ming, among others. To derive policy from the sampled states, we use an inverse dynamics model to estimate actions:
:= Do(o,, ot+1)
(C.60)
The designed loss function aims to minimize noise prediction and action prediction error:
[|| c/ (ax (T), 3)v(T) 30, })]]"]
L)0,0) : D,,rED,
(C.61)
+ L.o,,DD'
||u o')
The loss function L(0, Φ) optimizes the model's 's decision- capability complex environments.
Next, we will provide a detailed explanation of the implementation of DoF in Trajectory. The DoF algorithm is described in Algorithm
C.4 DOF-POLICY
DoP-Policy is a combination of the DiffusionQL algorithm and the noise and data factorization functions. The DoF-Policy agent built using the DiffusionQL (Wang et al., 2023) algorithm. It is designed for continuous action domains only. DiffusionQL initially ensures that the Actor's behavior policy closely aligns with the offline dataset and subsequently enhances policy performance through policy gradient optimization based on the Critic's estimations. DOM2 Li et al. (2023), extend the diffusion-QU to Offline MARL by learning a separated critic for each diffusion process. It does not address inter-agent cooperation and credit assignment in multi-agent reinforcement learning. Moreover, they overlook the environmental instability caused by interactions among agents in multi-agent scenarios.
In DiffusionQL, the diffusion policy represents each agent's action u, based on its observation, as Gaussian distribution æ;. This is achieved by utilizing reverse process of conditional diffusion model to represent тæ, as
K II
K. 0. To. (u: o;) = (u 0:K ( = Nuu
(u
uti o;).
(C.62)
k=
where Po (u k- whose mean constructed as
u oi) can be reparameterized as N(u R- no. (u, os, k), Eo, (u o,, k))
1 (ut
Be --
noo (u. 0z, Vas
(u (afuuo; 0g k))
(C.63)
To gain ground action u of each agent, we need to start sampling from w ~ N' I) to u via
k-1
Eo. (ut os k) + VBLE, N), for k= K, 1.
Okk
Var(1 σk)
(C.64)
24

---

Published as a conference paper at ICLR 2025
Algorithm 3 DoF
# Training Process
Initialize: Offline dataset D, Agent Nums N, Inverse Dynamic fa. Batch Size M.
1: for = to n_epoch do
2: Sample trajectory sequence [T:]E= of H and condition [y/ (r") from D with batch size M
3: Sample noise ())
4: for each Agent i N do do
5: Sample timestep ~ uu1 .T}
6:
Construct noisy array states 24 (7,)
Predict the noise <) = (7 (1)),
7:
8:
Omit the condition y(Ti) with probability B, ~ Bern(p.)
9:
end for
10:
C sot + f(c01, c0.1...,, cox) 02
Otot -1,0,,0,....... ON,..,,NN (or the local observation of agent i, it the first element of r.) -
Get the L(0, 6) = D,
[||e ||e-E; ex (xx(7), (1 3)v(T) (0, k)l? +Bs
D [[lu fo, o') o[)
13:
Update [e'],=] model
14: end for
# Trajectory sampling Process
Input: Noise model co, Inverse Dynamic [æ. guidance scale w. History length C, condition y
2: Initialize h Queue(length = C); t
3: while not done do do
4: Observe joint observation o; h.insert(o); Initialize Tx NN0, af)
5: for = to do do
6: Tyl: length (h)] <
7: for agent E [1.2, 11.,,....77 N) do
8:
E t (T(,t) + w (eb (Ti. (T),))
9:
end for
10:
stot t f(eo, 20gs..., 80N)
11:
stot (ul Mee ) + Denoise (Te, 2gr tot NN(
12:
13:
end for
14: Ocot t hioo O2,...,00 (o; is the local observation of agent i, it is the first element of =?)
for agent E 21,2,.... ,N,,,N N) do
16: u, +- for (o), 01+1)
17: end for
18: Execute in the environment;
19:
end while
Through the noise factorization function f. DoF-Policy factorizes the global noise e into noises E9, generated by each agent's diffusion model via
EØ2 : C8, (v,aa + vI - T&E oz, ki, e (1.n)
(C.65)
E/ tot
- Es....., Eg.)
(C.66)
where the f is the noise factorization function, enabling the agent to account for the non-stationarity of the environment caused by other agents' behaviors during training. Consequently. in both policy learning and action value function training, we attained global training goal policy x² with parameters by considering global information to minimize the loss function L(f):
[(0) Lan + Cpg
(C.67)
(Ee.. [||e 6/ too [13]
Eg Dmu (Q wn)]
(C.68)
where the second term Lpe can utilize various methods while in this case we follow the DiffusionQL (Wang et al., 2023) to learn (s u²) for policy improvement, and the E(e [J] is a hyperparameter to balance the two loss terms.
25

---

Published as a conference paper at ICLR 2025
During execution, each agent samples its action through the diffusion model based the IGD principle and Equation(C.633) This allows for decentralized execution of the agents.
The DoF-P algorithm is described in Algorithm 4.
Algorithm DoF-POLICY
Training Process
Initialize: Offline dataset D, Agent Nums N, Policy network no, Critic network Qo (double Q- learning Hasselt (2010) could be added), and target network Ter, Qu,
1: for to n_epoch do {(ON ,u, rr, (2,01
2: Sample mini-batch
# Critic learning
3: for each Agent i NNo do
4: Sample action uh 11413 To, (U++1, 0++1,))
5: end for
6:
Weat < h(u1, Ug...., uN!
7:
Update Qs by Q-learning method
# Policy learning
8:
for each Agent i E N do do
9:
Sample timestep k 2111 K} Sample random noisy distribution of action x
10:
Predict the noise C0, == co, (Vaku, vI 6g 0g 0z,h) end for
12:
13:
Ep
f(E), ( EB2 Update policy network ma by Equation C.67
14:
15:
Update the two target networks wer and Q+ through soft update.
16:
end for
D EXPERIMENT DETAILS
D.1 EXPERIMENTAL SETUP
We select four categories of MARL algorithms for comparison: (I) uses online algorithms to train offline datasets, such as QMIX (Rashid et al., 2018). (II) offline multi-agent reinforcement learning algorithms based on the Centralized Training with Decentralized Execution (CTDE) paradigm, in- cluding MABCQ (Jiang & Lu, 2021), MACQL (Kumar al., 2020), MAICQ (Yang al., 2021), and OMAR (Pan et al., 2022), and MA-TD3-BC (Fujimoto & Gu, 2021). These methods optimize multi-agent collaborative strategies by combining advantages of centralized training and decen- tralized execution. (III) Offline multi-agent algorithms based on the Decision Transformer, such as MADT (Meng et al., 2023). (IV) Offline multi-agent reinforcement learning algorithms based on diffusion models: MADIFF (Zhu et al., 2024), DOM2(Li et al., 2023), and Independent Diffusion. These methods use diffusion models to generate more effective multi-agent strategies. To demonstrate robustness, was tested with experiments using five different seeds.
The work used for comparison is listed as shown in table
1.1.1 COMPUTING RESOURCES
The experiments are conducted on nigh-peeformance computing cluster equipped with multiple NVIDIA GeForce RTX 3090 GPUs, which provide the necessary computational power. The CPUs
http:://ggiuub.oo///nttadeeeeee marl marl
htttps//ggihub..oo//ntttada
httpss//gituub.oom//iiyyunuu Yang/ICC
httpss//ggihub..co/iiiggaaam
DecisiDee---------- Transformer
26

---

Published as a conference paper at ICLR 2025
Table 6: Baseline algorithms
Algorithms
Brief Description
QMIX! (Rashid al., 2018)
Facilitates monotonic combination of individual agent utilities.
Offline decentralized auuli--agent reinforcement learning using value deviation and transition normalization for coordinated policies.
MABCQ² (Jiang & Lu, 2021)
Prevents overestimation by adjusting Q-values for policy samples and dataset state-action pairs.
MACQL (Kumar al., 2020)
Mitigates extrapolation eroor by trusting only datasct-provided state-action pairs
MAICQ' (Yang al., 2021)
Combining first-order and zero-order methods improves conservative value function optimization.
OMAR (Pan et al., 2022)
behavior cloning term added to TD3 (Fujimoto al., 2018)
MA-TD3+BCC (Fujimoto & Gu, 2021)
regularize the policy.
uses transformer- based offiine RL integrate global information into agents' policies via centralized critic gradients.
MADT" (Meng et al., 2023)
incorporates atlention mechanisms into Unet diffusion models to model trajectories. The version adopt is MADIFF-D.
MADIFI (Zhu al., 2024)
used in the cluster are Intel(R) Xeon(R) Silver 4216 processors, each running at 2. 2zz To ensure the robustness our results, we run the DoF algorithm five times with different random seeds in each experimental setup. In the MPE environment, a single run of the DoF algorithm takes approximately 5 hours to complete, with convergence typically achieved within the first to 2 hours.
D.1.2 HYPERPARAMETERS OF DoF
We implement DoF based on the source code of MADIFF, Decision Diffuser, and DiffusionQI.
The hyperparameters of the DoF for engjectory-genrratie are shown in Table 7. The hyperparameters of DoF algorithm for selley-generation are shown in Table 8.
DoF- -ra-arajyyy As shown in Table 7, the learning rate is set to 0.0002, which dictates the step size for each parameter update. The condition guidance weight is selected from {1.2, 1.4, 1.6, 1.8). depending on the task requirements. The number of diffusion steps is chosen from (100, 200, 300]. based on the specific task, and adjusts the model's reliance conditional information during generation. The planning horizon is set to 20, representing the number of future time steps considered by the model for planning and prediction. The history horizon is set to 8, indicating the number of past time steps used for decision making. The condition dropout is set to 0.25, randomly dropping parts of the conditional information during training prevent overfitting and enhance the model's generalization ability. The agent share noise is set to False, meaning different agents do not share noise during training, which helps improve the diversity and robustness of the multi-agent system. The discount factor is set to 0.99 for calculating the discounted future rewards, allowing the model to consider both immediate and future rewards in long-term planning. The loss type mean squared error as the loss function, which penalizes larger deviations more heavily and helps reduce significant prediction errors. The batch size set to 32, indicating the number of samples used in each training iteration.
to L2, using
DoF-Policy As shown in Table 8, we use values of {1e-3, 1e_ 1e=5 for the policy learning rate of the Adam optimizer, while fixed learning rate of 3e-1 is used for the Q-networks Adam optimizer. The parameter is set to 0.005, which is the update rate for the target networks. For in the loss function L(O), we use a fixed value of 0.5 to balance the two loss terms. The diffusion step is set to K 10 for action inference. We normalize the action space to [-1, 1] using the hyperparameter max action. The model is evaluated 200 times in total, with evaluations occurring every 1,000 training steps.
Next, we will provide a detailed introduction of the experimental environments for MPE, SMAC, and MA mujoco in Sections 1.1.4, D.1.3, and 1,1.5, and explain the sources of the offline datasets.
27

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameter Settings for DoF in Trajectory
Hyper-parameter
Value
learning rate horizon
0.0002
20
history horizon
8
condition dropout
0.25
condition guidance weight number of diffusion steps discount loss type batch size agent share noise optimizer
1.2
200
0.99
L2
32
False
Adam optimizer
Table Hyperparameter Settings for DoF in Policy
Hyper-parameter
Value
diffusion Step discount max action beta schedule
10
0.99
1.0
linear
0.005
"/ learning rete eval iterations train iterations step optimizer
0.5
0,0003
200
1000
Adam optimizer
D.1.3 STARCRAFT MULTI-AGENT CHALLENGE (SMAC) AND SMACv2
The SMAC environment (Samvelyan et al., 2019) is widely used benchmark for evaluating co- operative multi-agent reinforcement learning (MARL) algorithms. In SMAC, two teams engage in real-time battles, with one controlled by built-in AI and the other by learned multi-agent policies. Agents must independently make decisions while cooperating to achieve victory. The environment includes diverse scenarios with varying difficulty based unit composition, terrain constraints, and team balance. It features both homogeneous settings, where all agents share the same unit type, and heterogeneous settings, where different unit types require coordinated strategies. We evaluate approach on six SMAC maps: 3m, 8m, 5m_vs. pn__v_66m, 2c_vs_642g 2s3z, and 3s5z_vs_it $1si__s_3362, covering both symmetric and asymmetric matchups well as homogeneous and heterogeneous unit composi- tions. This variety allows for a comprehensive assessment of an algorithm's 's capability to coordinate agents in diverse combat settings.
SMACv2 (Ellis et al., 2023) is an updated version of the SMAC benchmark, designed specifically for research in cooperative multi-agent reinforcement learning (MARL). The update introduces three major changes: randomizing start positions, randomizing unit types, and adjusting unit sight and attack ranges. The first two changes address key limitation of the original SMAC benchmark, which is its lack of sufficient randomness in many maps, making it less challenging for modern MARL algorithms. The adjustment to unit sight and attack ranges increases agent diversity and aligns these attributes more closely with their actual values in StarCraft, enhancing the benchmark's 's realism and complexity, We conducted experiments on three SMACv2 scenarios: terran __vs__ zerg _0_vs____ vs 10, and zerg 5 5
We utilized the datasets from the off-the-gre grid offline dataset (Formanek et al., 2023), where each map is divided into three datasets: good, medium, and poor, based on the quality of the joint policies. This dataset enhances diversity by leveraging several different joint policies and adding a small amount of exploration noise.
Dataset:
28

---

Published as a conference paper at ICLR 2025
1.1.4 AULNI-AGENT PARTICLE ENVIRONMENT (MPE)
MPE (Lowe et al., 2017a) is straightforward multi-agent particle environment where particles can perform continuous observations and discrete actions. The experiments described in this study utilized three distinct environments. The Spread environment comprises three agents and three landmarks; the agents must learn to avoid collisions while covering all landmarks. The Tag enviromment includes one pre-trained prey, three predators, and two obstacles. The predators must cooperate to apprehend the faster prey. The World environment also includes one pre-trained prey and three predators; the prey agent needs to locate food on the map and can hide in forest avoid detection.
The dataset used in this study, collected by OMAR (Pan et al., 2022), consists of multiple datasets of varying quality, developed by introducing noise into the behavioral policy in MATD3 to enhance diversity. Random-quality datasets were generated using a randomly initialized policy for one million steps. The medium replay dataset was obtained by recording all samples in the buffer when training reached medium performance level. The medium and expert datasets were derived from either a partially pretrained policy with a medium performance level or fully trained policy.
Dataset:
We normalized the average scores of MPE tasks better compare the performance of different algorithms, as shown in Table 1. We used expert scores and random scores as the benchmarks for normalization. Let the original episodic return be S. The normalized score Snorn is calculated using (8 Sualeee
the formula Snorm = 100 X [Sigann Saaston where Spandom is the score obtained by a random policy and Sexpet is the score obtained by an expert policy. This normalization formula follows the work of Pan et al. [2022] and Fu et al. [2020], ensuring the method's reliability and validity. For specific MPE tasks, we used the following expert and random scores: for the Spread task, the expert and random scores are 516.8 and 159.8, respectively; for the Tag task, the expert and random scores are 185.6 and -4.1, respectively; and for the World task, the expert and random scores are 79.5 and -6.8, respectively.
D.1.5 MULTI-AGENT MUJOCO (MA MUJOCO)
MA Mujoco (Peng al., 2021) is based on the Mujoco physics engine and provides high-precision multi-agent simulation platform. The robots are composed of multiple intelligent agents that must learn to cooperate to move faster while maintaining balance. We conducted experiments using the 2-agent halfcheetah (2halfcheetah) configuration, where two different agents control the front half and the back half of the cheetah, respectively.
Dataset: We utilized the datasets from the off-the-grid offline dataset (Formanek et al., 2023), where each map is categorized into three datasets: good, medium, and poor, based on the quality of the joint policies.
D.2 ILLUSTRATIVE EXAMPLES
We demonstrate the superiority of DoF's generation capability under the IGD principle through three tasks: (a) matrix game generating two-dimensional data, (b) the Landmark covering game, and (c) the Q-value generation game. We will now provide a detailed introduction to the setup of each environment.
D.2.1 A MATRIX GAME GENERATING TWO DIMENSIONAL. DATA
The Matrix-like Game is a simple experimental environment designed to study the generation capabilities of three algorithms: DoF, MADIFF, and Independent Diffusion. In this game, we developed a multi-agent system where each agent is responsible for generating a different dimension of the data, learning to reproduce the ground-truth data. The ground-truth data consists of four sets cruusian--dsstrbuun
data, with their means located at the top-left (-0.75, 0.75), of two-dimensional bottom-left (-0.75, -0.75), and bottom-right (-0.75, 0.75) of the data plane, all with variance of 0.05. These four sets of data are generated with different probabilities: 0.5 for the top-left, 0.2 for the tip-right, 0.2 for the bettom-left, and 0.1 for the bottom-right.
top-right (0.75, 0.75),
In generative multi-agent reinforcement learning (MARL), generating data that matches the ground truth is a key metric for evaluating the performance of algorithms. We used the DoF, MADIFF, and Independent Diffusion algorithms to generate data, and assessed their performance by analyzing the
29

---

Published as a conference paper at ICLR 2025
Table Payoff matrix of one-step matrix games and reconstructed value functions approximate the optimal policy.
u2
A
B
A
Q2
A
B
A
1.0
0.0
A
0.6
0.0
09
0.0
(a) Game Payoff matrix 1
(h) h=Concat
(c) A=Atten
u1
B
Q1
Q1
A
0.0
0.2
14,0
2.0
15.2
1.4
B
13.9
211
Game Payoff matrix
2
h=Concat
h=Atten
(a) 3m
(b) 3m_vs_6m
20
Nudeen
Hodion
Average Return
Average Return
200K
400K
600K
B0OK
1M
200K
400K
600K
800K
1M
Training Progress
Training Progress
Figure 5: SMAC Return Curves: (a) 3m environment, (b) 5m, _t__v__60 vs_6m environment.
data distribution generated by each method and how well it matched the ground truth. The result are depicted in Figure 3.
D.2.2 LANDMARK COVERING GAME
In this game, three agents must cooperative cover three landmarks without collision in short-time. This game developed based the mpe-spread environment. order ensure the uniformity of the assessment and reduce random interference, we fixed the initial position of the agent and landmarks in the game. For each algorithm, we ran 10 iterations of planning, each iteration with 10 trajectory samples per agent, and visualized these trajectory points using different colors. The goal was for agents to learn to select the nearest landmark while avoiding overlap, exhibiting cooperative behavior.
D.2.3 GENERATING Q VALUE
The goal of the game is to reconstruct the one-step payoff matrix Qtot through two agents. Agent i use the diffusion process to generate individual utility value Q, and they are mixed into Qtoe h(Q1, Q2). where h is the data factorization function. We consider two data factorization functions: Concat and Atten. The results are depicted in Table 9, DoF can reconstruct the payoff matrix Qtot well. The Atten function performs better than the Concat function.
D.3 COMPARISON RESULTS
Table 10: The Average Return of the SMACv2 Scenarios
Map
Data
MABCQ
MACQL
MAICQ
MADIFF
DoF
terran. 5.
replay
7.3+1.
13.814.4
11.840.9
13.7+1.7
13.3+18
15.4+1.3
Zerg 5 vs_5
replay 6.8+0.6
10.3+1.2
10.343.4
10.6#0.7
10.2+1.1
12.0=1.1
terran 10 vs
replay
7.440.5
12.742.0
11.842.0
14.420.7
13.8+1.3
14.6+1.1
30

---

Published as a conference paper at ICLR 2025
Table 11: The Average Return of the Multi-agent Particle Environments (MPE)
Dataset
Task
MATCQ
MA-TD3+BC
MACQL
OMAR
MADIFF
DoF
DoF-P
Spread
101.443.4
110.343.3
85.3+4.6
113.942.6
120.1=6.3
136.4=3,9
126.3+3.1
Expert
Tag
95.2410.1
113.1211.6
84.3=10.2
115.8
120.8+11.3
125.628.6
120.126.3
World
98.5+21.8
95.3+18.3
65.4+20.2
113.4#23.1
124.7=20.1
135.2+19.1
138.4+20.1
Spread
29.3+5.5
32.3+3.8
35.3+10.3
45.0+18.8
67.5+8.5
75.6+8.7
60.5+8.5
Medium
Tag
58.3+18.0
63.3+25.6
62.3+27.8
15.3+16.7
78.6+12.3
10.3
83,929.6
World
69.9+20.1
72.4+9.3
56.446.4
69.2+21.5
80.1
85.2&11.2
16.4
Spread
13.745.6
14,4
19.2+6,4
35.3+14.0
48,43.44
57.4+6.8
48.1+3.6
Md-Replay
Tag
29.5+21.8
25.7+20.1
23.9+16.2
12.**1833
57.4=13.4
65.4
11.7=10.1
World
12.049.1
1
21.3+10.3
42.6428.2
51.6+12.1
58.6+10.4
18.1
Spread
5.343.4
8.844.4
20.5+5.8
30,4+8.2
20.6=7.6
35,926.8
34,5+5,4
Random
Tag
2.2+2.6
3.743.5
2.744.4
10.9#3.8
13.343.4
16.526.3
14.8
World
1.042.2
2.8+3.5
2.4+3.2
9.243.6
6.122.2
13.142.1
15.1+3.0
Table 12: The win rate for the SMAC Scenarios
Maps
Data
QMIX
MABCQ
MACQL
MAICQ
MADT
MADIFF
DoF
Good
0.0
3m
Medium
0.0
0.0
0.25
0.28
0.60
0.63
0.82
Poor
0.0
0.0
0.0
0.0
0.0
0.06
Good
0.0
0.0
0.0
0.93
0.86
0.90
0.94
Mediam
0.0
Poor
0.0
0.0
0.0
0.05
0.0
0.0
0.08
Good
0.0
0.0
0.0
0.04
0.68
0.66
0.72
$m_vs_ 6m
Medium
0.0
0.0
0.0
0.0
0.62
0.58
0.62
Poor
0.0
0.0
0.0
0.03
Good
0.0
0.76
2s3z
Medium
0.0
0.0
0.56
0.68
0.50
0.56
0.70
Poor
0.0
0.0
0.0
0.0
0.0
0.0
Good
0.0
0.0
0.0
0.21
0.18
0.0
0.18
365x_ Jefi
Mediam
0.0
0.0
0.0
Poor
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Good
0.0
0.0
0.18
0.38
0.24
0.42
0.61
64_zg
Medium
0.0
0.0
0.14
0.17
0.06
0.18
0.23
Poor
0.0
D.3.1 SMAC
In offline multi-agent reinforcement learning, the SMAC return metric is commonly used to evaluate performance. To provide more comprehensive assessment of our algorithm, we also evaluated the win rate. While win rate is meaningful indicator of performance, especially environments with well-structured data, offline multi-agent reinforcement learning presents additional challenges, often resulting in lower win rates compared to online methods. In cases where the dataset quality is poor and returns are consistently low (c.g., below 11), the win rate may drop to O, reducing the comparability between algorithms these settings.
The SMAC win rate results we present are shown in the table 12. In the Good and Medium datasets, DoF achieved the best win rate most environments. In simpler environments, such as the 3m and 8m environments in the Good dataset, DoF's win rate was about 96%, followed by the MADIFF and MAICQ algorithms.
the 3m environment, MADIFF's win rate was 94%, while in the 8m environment, MAICQ's win rate was 93%. In the 5m_ vs_ vo_6smmm and 2c_ s0__s__6222 644g environments, DoF achieved the best win rates. In heterogeneous environments, such as the 2s3z and 385z vs 3562 environments, both the DoF and MAICQ algorithms achieved the best win rates.
To further analyze, we plotted the return curves for all datasets on the 3m and 5m_ vs 6m maps. During training, saved the models periodically throughout the diffusion process, training for total of million steps and saving a model every 30,000 steps. In the sampling phase, we evaluated each saved Diffusion model and recorded the corresponding return data, shown in Figure 5.
As shown in Figure 5, the Good dataset, DoF converges around 200,000 to 300,000 steps, with some fluctuations around the mean afterward. In the Medium dataset, the DoF algorithm only begins
31

---

Published as a conference paper at ICLR 2025
spread
Emp Jeapert
smate (ospert
140
1.00
122
Avanng Ruture
300
9
Averge
3
B0OK
3,34
3.4M
200K
ROOK
3.3M
660K
BOOK
3.3H
METTOI sCrmed (medum!
simple (medium)
simp woword (medum)
Arurage farturn
Rvsrage Raturn
Ausrage
Tranning Progress
Training Progrems
Training Progrwa
Dag reolay)
Raturrn
Avenuee Ruturn
Average Raturn
20-
-10
609K
BO
1.3M
1.0M
4.0KK
BOOK
3.20
3.0M
2M
600K
DOOK Truining Pregimas
Truining Progress
simpk_.ppraa (randam))
simple nnlom))
00000
Avsrage Beturn
Averrgee Return
Average
-2.6
Thainteg Progress
Thainting Progrems
Trabiing Progress
Figure 6: Training Curves of DoF-P, MAICQ. MA-TD3+BC, and OMAR in MPE
slowly converge around 600,000 steps. In comparison, for the Poor dataset, DoF exhibits larger fluctuations, but the return curve still shows an upward trend.
D.3.2 SMACV2
We further evaluated our algorithm on the SMACv2 replay dataset in three scenarios: terran _s_ org_10_vs_IMh vo000,,_____00 and 7erg_5_vs_$ 5__s___ As shown in Table 10, DoF algorithm outperforms other ap- proaches, achieving state-of-the-art (SOTA) performance across these benchmarks.
D.3.3 MPE
In Figure 6, we present the training curves of DoF-Policy (DoF-P) in the Multi-Agent Particle Environment (MPE). These graphs cover the performance across three different environments and four distinct datasets. We compare the training results of four algorithms: DoF-P, MAICQ, MA- TD3+BC, and OMAR. The solid lines in the graphs represent the mean values, while the shaded areas indicate the variance, providing insight into the central tendency and variability of the model performance. It is worth noting that the DoF-P algorithm displayed here employs the DoF-concat noise decomposition method.
32

---

Published as a conference paper at ICLR 2025
Table 13: The Average Return of the Multi-Agent MuJoCo Benchmark
Maps
Data
MACQL
MAICQ
OMAR
MA-TD3+BC
DoF
Good
2886.2#651.7
3044.2+3114
3124.2+4114
3412.7+281.
3400.1=310.5
HalfCheetah-v2
Medium
1243.2+455.1
262142814
2864.3+322.4
1011.2+178.6
3123.1+161.7
Poor
1045.3+376.7
744.3+141.7
1968.1+141.77
1651.9#156.1
1869.9=129.8
As shown in Figure 6, it is evident that DoF-P achieves superior performance in the presented experimental results, consistently outperforming the MAICQ, MA-TD3+BC, MACQL, and OMAR algorithms. Notably, DoF-P demonstrates a faster convergence rate, particularly in the simple world environment, where it converges in just 400,000 steps. In contrast, OMAR achieves the second-best results in most environments.
D.3.4 AUENT-AGENT MUJOCO (MA MUJOCO)
Table 13 depicts the experimental results for the HalfCheetah task of MA-MuJoCo (de Witt et al., 2020). DoF performs the best in the Medium dataset, and ranks second in the Good and Poor datasets.
D.3.5 SCALABILITY EVALUATION
We have developed a customized combat game based on MAgent (Zheng et al., 2018) environment, a grid-world specifically designed for large-scale multi-agent reinforcement learning. The combat game is drone combat game, where multiple drones fight against other drones controlled by in-game AI The goal of the game is to train RL controlled drones to defeat all the opponent drones. In this game, each drone agent has 120 observation field and a smaller 120 attack range. It is cooperative MARL game, all the agents share the same reward: point for hitting an enemy drone, 10 for neutralizing one, and 50 for eliminating all enemy drones. Negative rewards to following scenarios: -1 for being hit, -10 for being neutralized, and -3 if no enemy drones are taken down in timestep. This large-scale environment presents significant challenges in agent coordination, requiring drones to cooperate effectively to maximize enemy drone elimination while minimizing their own casualties. We consider multiple game scenarios with different number of drones fighting against the same number of agents. For example, the 64x64 scenario indicates 64 drones fighting against 64 opponents. For each scenario, we train agents using MADDPG (Lowe et al., 201 7a) for 2,000 episodes (with 550 steps on average), and the replay logs are collected for offline training-
assigned
We conducted the Scalability Experiment: Comparison of DoF and MADIFF for Different Numbers of Agents in the main text. As shown in Table 3, this experiment compares GPU Memory, Inference Time Cost, and Reward for both DoF and MADIFF across different numbers of agents. Specifically, Inference Time Cost represents the total time taken for a single inference, which includes the time for the entire process of generating samples using the model, with batch size of 32 and 200 diffusion steps. GPU Memory measures the memory usage for running the model, and Reward reflects the performance of the agents during the game. The results demonstrate that through diffusion factorization, DoF achieves better scalability than MADIFF. DoF consistently uses less GPU Memory and achieves faster Inference Time Cost compared to MADIFF, particularly as the number of agents increases. In terms of Reward, DoF maintains high performance, outperforming MADIFF in larger-scale scale scenarios. These results validate the efficiency and scalability of DoF in comparison to MADIFF.
To further evaluate the scalability, we compare the network parameter efficiency of MADIFF and DoF. Table 14 presents comparison of network parameter counts (in MB) across different numbers of agents, aimed evaluating the model's 's parameter efficiency and scalability. We measure the network parameter counts per agent directly compare how both methods scale as the number of agents increases.
We also investigate the inference time costs of various baseline methods. Table 15 provides comparison of inference time (in seconds) for DoF with enhanced sampling techniques alongside other baseline approaches. Specifically, and Consistency Model, and include comparisons with non-diffusion methods (e.g.. MACQL and MABCQ) offer more comprehensive view of the computational trade-offs involved.
explore sampling acceleration techniques such DDIM
33

---

Published as a conference paper at ICLR 2025
Table 14: Network Parameter Count (in MB) Comparison Between DoF and MADIFF for Different Numbers of Agents
Metric
Method 4 agents
8 agents
16 agents
32 agents
64 agents
MADIFF
109 MB
135 MB
174 MB
228 MB
310 MB
Network Parameter Count
DoF
71 MB
72 MB
76 MB
81 MB
91 MB
Table 15: Inference Time Cost (s) for Different Methods Across Agent Configurations
Method
agents
agents
16 agents
agents
agents
DoF(DDPM)
8.2s
11.3s
14.9s
18.1s
24.3s
DoF(DDIM)
5.1s
7.8s
9.6s
12.2s
14.8s
DoF(consistency model)
1.4s
1.9s
2.4s
Inference Time Cost (s)
MACQI
1.1s
1.2s
1.2s
1.38
1.5g
MABCQ MADIFF
1.1s
1.2s
1.3s
1.4s
1.6s
12.9%
16.5s
23.9s
31.58
Out Of Memory
20
MADIF
Conot
A-Attan
f- A-IMIX
MARIFY
P-Comeet
F-OMEX
38:22
18:41
16.82
32422
34.33
11.94
11.00
Time
5.16
Map
(b) Inference Time
Map
Figure 7: Diffusion Inference Time in SMAC Environment
Table shows the reward comparisons using the same sets of methods. By comparing rewards across different sampling techniques (e.g., DDPM, DDIM, and Consistency Model) and non-diffusion methods, we can observe variations both performance and efficiency. These experiments high- light the effectiveness of advanced sampling strategies and underscore the importance of selecting appropriate techniques under different computational constraints.
Table 16: Reward Comparison for Different Methods Across Agent Configurations
Metric
Method
4 agents
8 agents
16 agents
32 agents
64 agents
DoF(DDPM)
60.1
75.9
120.3
154.6
210.4
DoF(DDIM)
61.3
73.8
118.5
151.7
208.3
DoF(consisenncy model)
55.3
70.2
116.3
148.9
202.1
Reward
MACQL
50.7
65.6
100.3
135.1
190.6
MABCQ
42.1
49.4
90.4
119.2
162.3
MADIFF
63.8
70.4
113.5
148.3
Out Of Memory
D.3.6 INFERENCE TIME
We evaluate the inference time, which refers the duration taken by the diffusion model to complete one reverse process during sampling. This process involves N diffusion steps, and for this experiment, set N = 200. Our primary focus is to compare the inference times of DoF using different noise factorization function f (f=Concat, fornncttt=Aaa fAAtten, and f f=QIIX) and MADIFF.
Experiments were conducted the 3m and 8m maps of the SMAC environment. The detailed results are shown in Figure As illustrated, the inference time for f=Concat is faster than both f=Atten and /=QMIX. On the 3m map, executing single reverse process takes approximately 7.32 seconds,
34

---

Published as a conference paper at ICLR 2025
Table 17: Performance Comparison Across Different Maps and Methods
Maps
Dataset
f=
Concal
f WConcat
f = Alten
Good
12.0+0.8
12.8+0.8
14.7=0.7
v3366
Medium
10.4+0.7
11.9=0.7
12.6+0.6
Poor
7.0+0.2
7.5+0.2
8,440.3
Good
15.7+0.9
16.1+0.8
18.0#0.6
2c 64zg
Medium
13.3+0.8
13.920.9
14.7=0.7
Poor
11.2+0.9
11.5+1.1
12.0+0.6
while on the 8m map, it takes around 13.82 seconds. The inference times for f=Attn and S=QMIX are similar, with around 11 seconds on the 3m map and approximately 17 seconds on the 8m map. MADIFF has the slowest inference time, taking around 12.65 seconds on the 3m map and 18.55 seconds on the 8m map. This experiment confirms that DoF achieves faster inference times than MADIFF, and that the noise factorization method f=Concat outperforms f=Atten and f=QMIX in terms of inference speed. f=Concat performs the fastest thanks to its ability to factor diffusion process.
D.4 ABLATION STUDY
How does the Noise Factorization Function f Affect Agent Performance? In the main text (see Table 4), we briefly compared different noise factorization functions f and observed that decentralized functions like WConcat generally outperform simpler approaches such Concat, while centralized attention-based functions Atten excel but require centralized execution.
To further investigate the impact of noise factorization functions S. Table 17 presents the results of using the noise factorization functions
WConcat and f = Atten in the homogeneous 2c ss_vv___.. 2s_64zg and heterogeneous 3s5z_vs, vs_3s6z environment. Notably, the centralized attention-basee method (f = Atten) demonstrates a clear performance advantage over the WConcat approach (f WConeat), particularly in this challenging heterogeneous setting.
How does the Data Factorization Function h Affect Agent Performance? In this experiment, we compare three data factorization methods: Concat, WConcat, and Atten. Concat uses a simple concatenation operation combine individual agent data; WConcat builds upon Concat by adding learnable weights; while Atten employs an attention mechanism for more sophisticated data integra- tion. We conducted tests on the SMAC 3m, 5m_ 6m_,mmm. 3s57_ vs __66,, and 2s3z. scenarios, using datasets of varying quality (Good, Medium, and Poor). As shown Table 18, WConcat and Atten consistently outperform Concat across all four scenarios and dataset qualities. The performance gap is particularly noticeable in more complex maps, indicating that the weighted concatenation method WCCnnat and the attention-based approach Atten more effective at capturing inter-agent relationships and extracting relevant information from the joint state.
How Does Parameter Sharing Affect Agent Performance? Similar other MARL approaches, we use parameter sharing for the agent network. The parameters of the noise prediction network of shared among agents too. In this way, 0; 0,, i + j. To discern whether the performance improvement is due to parameter sharing or noise factorization, we study the performance of DoF without parameter sharing for the noise prediction network on the SMAC 3m and 5m_ vs_ ____v__6mmm_____ scenarios. The results are depicted in Table 19. DoF without parameter sarameter-sharing is depicted as DoF (No share). As shown in the Table 19, there is no significant difference between the aurameter-shring and the saapparrmmter-shrinng approaches. This suggests that parameter sharing plays minor role in multi-agent cooperation tasks.
How sensitive is DoF to different hyper-parameters? We investigate the impact of different hyper- parameters for the DoF trajectory. Figure shows the results for different diffusion steps, history horizons, and conditional guidance scales. As shown in Figure 8 (a), Longer diffusion steps does not always lead to higher returns. Using a longer history horizon does not always lead to better performance, as it is depicted in Figure 8 (b). Horizon H 24 performs better than H 36. As it depicted in Figure 8 (c), the conditional guidance scale does not lead to significant performance differences.
35

---

Published as a conference paper at ICLR 2025
Table 18: Ablation Study Data Factorization Functions / Across Different Maps
Maps
Dataset
h= = Concat
h WConcat
h Atten
Good
19.7=0.6
19.8=0.2
19.9#0.1
3m
Medium
17.8+2.1
18.6#1.2
18.7+1.0
Poor
10.6
10.9#1.1
10.8+0.9
Good
15.8#1.4
17.7
18.2+0.9
5m_ [o__s__mm 66
Medium
14.9+1.1
16.2=0.9
16.8+0.8
Poor
9.8+1.1
10.8+0.3
11.0#0.5
Good
11.320.9
12.8+0.8
15.2+).7
3s5z 3s6z
Medium
9,420.7
11.9=0.7
12.8+0.5
Poor
6.8+0.3
7.5+0.2
8,2+0.3
Good
15.5+1.0
18,5+0.8
19.5+0.3
2s3z
Medium
14.8+0.8
18.1
18.5+0.3
Poor 9.6+1.1
10.0#1.1
10.240.7
Table 19: DoF w/wo for Parameter Sharing
Maps
Dataset
Share
No_share
Good
19,8+0.2
19.7+0.5
3m
Medium
18.6+1.2
180.7
Poor
10.9#1.1
11.240.4
Good
17.7=1.1
17.5+0.8
5m vs
Medium
16.2:0.9
16.340.7
Poor
11.420.7
10.8+0.9
263
Sape
Grllawe Oen
Training
(a) Diffusion Steps
(b) History Horizon
(c) Conditional Guidance Scale
Figure 8: Sensitive analyze: Diffusion Steps, (b) History Horizon, and Conditional Guidance Scale.
20.0
Average Return
7.5
Cood hef heur
DuRisrtc Peur
0.6
Training
0.4
Linear Non-linear
Figure 9: Comparison of linear and non-lineat noise combination methods in the SMAC 3m environment. The non-lincar method underperforms due to non-Gaune nn-Gaussian noise profiles. Each subplot shows the average episode reward over training steps. The shaded regions indicate the standard deviation of the rewards.
Can we use Monotomic-inerresinng Noise Factorization Methods? The normal distribution is linearly additive,
the linear combination of noises in DoF to form larger noise. However, it is unclear whether monotonic increasing mixers such as QMIX (Rashid et al., 2018) be used for noise factorization. To investigate the impact of the monotonic increasing noise
36

---

Published as a conference paper at ICLR 2025
factorization combination method, we conducted experiments in the SMAC 3m environment using good, medium, and poor datasets. In this experiment, we use QMI as the noise factorization method. It is depicted as DoF (QMIX) in Figure 9. As it is shown in the figure, using the monotonic increasing function as the mixer hurt the performance of DoF significantly. This due to the fact that through using such monotonic increasing mixer, the resulting noise may longer be Gaussian noise, which is required during the diffusion process.
D.5 DIFFERENT DIFFUSION GENERATION PROCESSES
The diffusion model is computationally intensive despite its flexible modeling ability; in DoF, we leverage the modeling ability of the diffusion model model the cooperative behaviors among agents. As generation -based MARL approach, our work relies on generation models to generate data. In this work, we use DDPM (Ho et al., 2020). The major testing time of DDPM is spent on the long diffusion steps to sample data. Our work can be built recent advancements of the diffusion model to accelerate the sampling time. For example, DDIM (Song et al., 2021a) can reduce the sampling steps significantly with performance drop. The consistency model (Song et al., 2023) requires only one sampling step.
The experimental results for DoF and MADIFF using DDPM and DDIM on SMAC 3m, using different steps, are shown in Table 20 and Table 21. As shown in Table 20, the testing time is lower with fewer diffusion steps. The diffusion step 50 achieves the lowest testing time at the of the lowest return. The testing time of MADIFF is always longer than that of DoF. And MADIFF always performs weaker than DoF. For Table 21, shows that using DDIM leads faster testing time. We observe similar trends demonstrated in Table 20.
Table 20: Comparison for DoF MADIFF across Difftusion Steps with DDPM in SMAC 3m.
Diffusion Step
Time (DoF)
Time (MADIFF)
Reward (DoF)
Reward (MADIFF)
50
515.4
689.4
13.5
12.7
100
625.8
925.8
16.4
15.1
200
1018.8
1413.6
19.3
19.1
300
1492.8
1679.4
19.2
19.0
Table 21: Comparison for DoF and MADIFF Across Diffusion Steps with DDIM in SMAC 3m.
Diffusion Step
Cost Time (DoF)
Cost Time (MADIFF)
Reward (DoF)
Reward (MADIFF)
50
367.2
475.2
11.8
11.3
100
533.4
738.6
15.8
14.7
200
813.6
1185.6
19.1
19.0
1124.4
1304.4
18.8
The training time and testing time of DoF can be further reduced by following the approach of StableDiffusion (Rombach et al., 2022) and LatentDiffusin (Venkatraman et al., 2024). They use Variation Auto Encoder (VAE) to encode input data into latent space with lower dimensions than before. The diffusion is happens in the latent space. In the end of diffusion, the data is recovered using a decoder from the latent space. We implement this idea and named it as DoF+VAE, and conduct experiment on the SMAC 3m good dataset. The VAE compress the data from 33 dimensions to 17 dimensions. The experimental results are shown in table 22.
Table 22: Comparison of Training Time and Return for DoF and DoF+VAE on SMAC 3m Good Dataset
Method
Training Time
Return
DoF
48h 39h
18.96
DoF+VAE
15.80
Table 22 shows that with the use of VAE, the training time is reduced from 48h 39h, decrease of approximately 27%, with a 16.6% decline in performance. This suggests that within an acceptable range of performance loss, using VAE to compress data effectively reduce training time and
37

---

Published as a conference paper at ICLR 2025
Training Datasets
Generation
Now
(a) a a
360"
Figure 10: able to generate novel behaviors through using multiple constraints. (a) Dataset data
located in upper half-circle (09 S a < 180"), (b) Dataset 2: data is located in lower-half-circle 180" s a < 360", and (c) Generated data: full circle 0 360
enhance training efficiency, particularly in scenarios where resources limited or there is high demand for rapid training.
D.6 SATISFY MULTIPLE CONSTRAINTS
Consider a multi-agent system comprising two agents: one responsible for learning the angle a, and the other for learning the radius These two agents work collaboratively to generate point distributions in a wio-dimensional plane. We are given two datasets, each representing different constraint conditions. As shown in Figure 10: (a) O' constrained to learn within the upper half-circle. (b) 180 < a < 360%: In the second dataset, the angle agent is constrained learn within the lower half-circle. (c) 0 phase, we demonstrate how the two agents collaborate to generate a complete circular distribution.
< 180%: In the first dataset, the angle agent
360": In the generation
In this work, we train diffusion models for the datasets: model A for the upper half-cirele dataset, with constraint (1,0), O), model for the lower half-circle dataset, with constraint (0, 1). Each model can be factored into two diffusion models for generating data. After factorization, one model is used to generate the radius and another model is used to generate the angle During generation, we utilize the model A and B to generate novel data, the full circle, with the constraint (1, 1). We follow the approach of Decision Diffusion (Please refer to its Appendix D) to guide the factored diffusion processes to generate data following multiple constraints.
Figure 10 illustrates that the multi-constraints diffusion processes generate point distribution covering the entire circle. This example demonstrates the flexibility and adaptability our approach in handling multi-agent systems, where different agents are responsible for different parameters but can collaboratively satisfy complex constraint conditions.
D.7 CONDITIONING ON LOCAL OBSERVATIONS
After the diffusion processes satisfying the IGD principle are learned, they can be used to generate data with desired properties with guidance. Researchers (Ho & Salimans, 2021) have shown that using classifier-free guidance can lead to better performance. In this work, we adopt classifier-free guidance to guide the agent to learn cooperative behaviors.
For diffusion process i, we use condition y; to guide the generation process toward desired properties. In cooperative MARL high-return value R suggests cooperative behaviors, thus R is included into y/4 to guide the diffusion process to generate high-return data. Further, we include the local observation history of each agent into the condition y to make the generate data aligns with its local observation.
Table 23 presents ablation study the condition of our proposed method. We compare the performance of DoF with two different types of conditions: return value R only, return and local observation 7;- The results demonstrate that using both R and 7; condition yr consistently
38

---

Published as a conference paper at ICLR 2025
Table 23: Ablation Study Condition Components
Datasets
DoF I
DoF R and
3m medium
7.34 0.89
18.58 = 1.22
5m6m medium
4.79 + 0.64
16.22 = 0.83
8m medium
6.66 0.86
18.64 = 0.86
outperforms the retur-oonyy condition across all tested scenarios. This justify the selection of using both and 7; as guidance to guide generation process toward desired cooperative behaviors.
E DISCUSSION
E. CIFFUSION-BASEI METHODS AND OUT-OF-DISTRIBUTION (OOD) SCENARIOS
Diffusion-based methods exhibit certain advantages in handling out-of-distribution (OOD) situations. When using DoF-Trajectory, from the perspective trajectory modeling, diffusion methods bypass the traditional Q-value estimation step commonly employed in reinforcement learning. This bypass naturally alleviates the issue of Q-value overestimation, which is particularly prominent in offline scenarios with limited biased data. By mitigating this overestimation, diffusion methods effectively address OOD challenges. On the other hand, when using DoF- DoF-Policy, from the perspective of policy modeling, diffusion models model the distribution of the underlying policy well. In this way, effectively reduces extrapolation errors caused by sampling out-of-distribution actions.
E.2 POTENTIAL. APPL.CATIONS IN COMPETITIVE AND MOXED-MOTIVATION SCENARIOS
While our current work focuses on cooperative multi-agen reinforcement learning tasks, extending diffusion-based methods to competitive mixed- mixed-motivanet motivation scenarios is valuable direction for future research. In competitive settings, the diffusion process could condition on adversarial strategies payoff structures during sampling to model the interplay between competing agents. For mixed- motivation scenarios, diffusion methods could incorporate both individual motivations and shared objectives into the conditional space.
This enhancement would allow the model to capture the complexity and dynamics of such envi- ronments more effectively. For example, in competitive game, the diffusion model could learn to anticipate and counter adversarial moves, while in mixed- -ixeed-moivvation tasks, it could optimize individual agent goals while ensuring collective success. Incorporating motivations into the diffusion conditions provides flexible mechanism to extend the applicability of our framework to these challenging scenarios, paving the way for broader applications in multi-agent systems.
F LIMITATION
Our method may face challenges when applied to tasks with -igh-dimensional observations. While our current experiments focus on standard MARL benchmarks, extending the approach to handle more complex, high-dimensional observation spaces remains an open direction for future work.
39


Reviews:
[{"review_id": "pPKEprF2CS", "reviewer": "Reviewer_StRZ", "summary": "Please see the strengths and weaknesses sections.", "questions": "- Can you intuitively explain why the agents trained with MADiff still suffer from collisions and perform worse than the ones trained with DoF in Figure 1? I suppose their main difference lie in scalability which shouldn't be an issue for the simple illustrative example.\n\n- Functionally, \\(f\\) and \\(h\\) appear to be similar. Why is it necessary to train an additional \\(h\\)?\n\n- (This is only a suggestion for future works.) Policy distillation methods can be an important category of baselines to compare with. Specifically, you can apply CDG to learn a centralized policy and distill a set of decentralized policies for each agent from the centralized agent, which is also a form of factorization.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper tries to solve an important problem in Diffusion-model-based offline MARL, that is, the use of the centralized training with decentralized execution (CTDE) paradigm. As noted in [1], this is a research gap in this field.\n\n- The algorithm design is simple yet effective, which is demonstrated in a series of benchmarks through comparisons with various baselines.\n\n### Reference:\n\n[1] Chen, et al. \"Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions.\" TMLR, 2024.", "weaknesses": "- The definition of IGD seems to assume the independence among agents' behaviors/rollout trajectories. This assumption is also shown in the selection of the noise factorization function, which is a simple concatenation operator.\n\n- The first paragraph of Section 4.3 is a little bit of confusing. A refined version of the overview should be provided.\n\n- The provided codebase does not involve a detailed instruction to reproduce **ALL** numerical results in the paper. The code files are also not complete. If the authors fix this, I will improve my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pGZrO0d1mN", "reviewer": "Reviewer_5VM5", "summary": "The paper proposes to use Individual-Global-Identically-Distributed (IGD) principle for the training of diffusion models in multi-agent decision making (MADM). This IGD, requiring the outcome of multi-agent diffusion model should be identically distributed as the collective outcomes from multiple individual agent diffusion models, is a generalization of the individual global max (IGM) principle. The authors proposes DoF a way to do factorization in this setting and validated their proposed approaches via experiments.", "questions": "-  How long did the proposed methods take to train compared to other methods? The paper reported inference time, but not the training time. \n- I was a bit confused by the overall setup, are the agents doing global training and then individual sampling of the noise after doing the factorization? Is there a way for the agents to train simultaneously? \n\nI just got the invitation to review this paper last night, and I understand that the authors probably won't have time to address all of my questions/comments. I hope these comments will be helpful for improving the manuscript for publication (at ICLR or elsewhere).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "+ IGD is a very interesting and promising idea. I can see its potential in multi-agent diffusion.\n+ IGD captures recovers IGM or risk-sensitive IGM principle.", "weaknesses": "- While the ideas are interesting, presentation of the paper makes it difficult to fully appreciate the contribution. For example, the algorithms are missing from the main manuscript. \n- Related to the previous point, it is unclear how to formulate a diffusion algorithm based on Theorem 1.\n- Some of the notations are not well defined, such as $\\epsilon$ in the theorem, and what role $\\theta$ play in Eq (2). \n- The POMDP setting should be described a bit more in the main text, for instance, there is one environment that all agents are interacting with and the transition probability depends on the product space of all agent's actions. \n- Typos: I list a few here, but please go through the paper one more time to double check.  For example, in definition 1, should be \"there exist ... functions\", the sentence before section 3, \"to for $x^k$\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oGjulJgVMc", "reviewer": "Reviewer_zeea", "summary": "This paper proposes an individual-global-identically distributed (IGD) principle (property) for model design in cooperative multi-agent decision-making.  The paper claims that the property generalizes the popular individual-global-max principle (IGM).  They further propose a diffusion factorization framework that 1) satisfies the IGD principle and 2) performs well on several datasets.", "questions": "Can the authors provide formal statements and proofs showing that IGD generalizes IGM, individual global optimal principle or risk-sensitive IGM principles?\n\nAre there any nontrivial examples that satisfy IGD beyond the Gaussian case (or a component-wise independent Markov chain with a stationary initial distribution)?\n\n---\nAfter discussing with the author, I remain deeply concerned about the paper’s mathematical writing, which requires substantial improvement. First, the claim that IGD generalizes IGM is neither formally stated nor proved, even after multiple responses. This is not merely an issue of correctness; the statement is fundamentally unfalsifiable and lacks rigor. Second, the definition of IGD is unnecessarily convoluted and appears redundant. A more straightforward would be that a diffusion model exhibits IGD if each component of the initial state is mutually independent. While the empirical results may be interesting, the mathematical statement is seriously deficient and demands thorough and precise revision to meet an acceptable standard.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The empirical results show their method perform slightly better than other methods.", "weaknesses": "I am unfamiliar with this domain of literature, and my comment would mostly focus on mathematical writing.  Overall, I have strong concerns about the claim that IGD generalizes IGM, and I feel the theory about diffusion factorization framework under Gaussian assumption satisfying IGD uninteresting.  Thus, I suggest rejection.\n\nThe technical writing quality needs significant improvement. Here are some comments.\n1. Does IGD property apply to the initial distribution $p_{\\theta_{tot}}(x_{tot}^0)$  or the whole series $p_{\\theta_{tot}}(x_{tot}^{0:K})$? Does $x_{tot}^K$ need to be sampled from Gaussian?  Does the reverse process need to be a Markov chain?\n2.  The claim that IGD generalizes IGM is not well-defined.  IGM needs a value function but does not necessarily have a distribution over action or even under a diffusion model.  Please give an explicit statement on the generalization: For instance, given a value function $Q_{jt}$ satisfies IGM what is the corresponding $p_{\\theta_{tot}}$ such that Eqn.(1) holds for any history $\\tau_{tot}$.\n3.  At the end of section 4.2, there should be formal statements of IGD vs individual global optimal principle or risk-sensitive IGM principles.\n\nIf I understand correctly, Theorem 1 basically states that each component of isotropic Gaussian is independent.  I feel this result is very limited and hardly holds in general.  For instance, it’s unclear whether this would still apply even when the Gaussian has a non-diagonal covariance.\n\n\n\nMinor comments\n- In eq (3), instead of $\\theta_i\\subset \\theta_{tot}$, $(\\theta_1,\\dots, \\theta_K) = \\theta_{tot}$ would be better, as the order of coordinate matters.  \n- $\\epsilon^{\\theta_1}_1$ in eq (5) is not clearly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qmYCNsu2PM", "reviewer": "Reviewer_y4iS", "summary": "The paper presents Diffusion Factorization Framework (DoF), a new method for offline cooperative multi-agent decision-making (MADM) using diffusion models. DoF generalizes the traditional Individual-Global-Max (IGM) principle with a new Individual-Global-identically-Distributed (IGD) principle. This IGD principle aligns decentralized outcomes in multi-agent diffusion models with centralized outcomes, fostering effective cooperation. DoF includes two key functions: a noise factorization function that decentralizes a centralized diffusion model per the IGD principle, and a data factorization function that models inter-agent data relationships to ensure alignment with ground truth. Extensive experiments demonstrate DoF’s advantages in scalability, cooperation, and alignment with ground truth, outperforming existing MADM methods across various benchmarks.", "questions": "1.\tGiven that QMIX is an online algorithm not designed for offline reinforcement learning, could you clarify the rationale for including it in the comparisons? What insights does this comparison provide?\n2.\tCan diffusion-based methods effectively handle out-of-distribution situations? Additionally, can these methods be applied to competitive or mixed-motivation scenarios?\n\nPlease respond to the aforementioned weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper generalizes the well-known IGM principle to the IGD principle, which broadens the applicability of diffusion models in multi-agent decision-making and improves coordination among agents. It addresses the limitations of existing cooperative MADM methods, enhancing scalability and cooperative performance.\n2.\tDoF introduces novel noise and data factorization methods that align agent-generated data with a centralized diffusion model while enabling decentralized execution. This factorization process addresses the complexity of multi-agent coordination and enhances resource efficiency across a range of multi-agent tasks.\n3.\tThe extensive experiments conducted on the SMAC and MPE show that DoF outperforms competing methods in multi-agent coordination tasks. Experimental results indicate that DoF performs well with an increasing number of agents. This scalability suggests DoF’s potential for large-scale applications in complex multi-agent environments.", "weaknesses": "1.\tThe paper lacks clarity in some important parts. It introduces DoF-Policy and DoF-Trajectory but does not specify the suitable scenarios for each. The results section also fails to indicate which algorithm was used, and Figure 6 only includes DoF-Policy results, making it difficult to assess the relative strengths of the two methods. Additionally, implementation details for the inverse dynamics model in DoF-Trajectory are missing.\n2.\tIn Figure 2(a), there is a mislabeling of the reverse diffusion process: the output should be  x^{k-1} instead of x^k , which may mislead the understanding of the process. Additionally, the caption of Figure 2 lacks necessary descriptions, which reduces its readability.\n3.\tThe paper's benchmarks are based on state vectors, making it unclear whether the proposed method can be applied to environments with high-dimensional observations. The Scalability Experiment section only compares GPU memory and time with MADIFF, without providing comparisons to other baseline algorithms. Since diffusion-based methods require multiple iterations to generate the final output, does this lead to significantly higher computational costs compared to traditional methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0b85b2qXw", "reviewer": "Reviewer_YHzY", "summary": "The paper studies the use of diffusion models in offline cooperative multi-agent decision-making (MADM). The authors propose the Individual-Global-identically-Distribute (IGD) principle, a generalization of the IGM principle. IGD requires that collectively generated outcome of individual agents follows the same distribution as the generated outcome of a whole multi-agent system. The authors propose DoF, a diffusion factorization framework for offline MADM which adopts a noise factorization function to factorize a centralized diffusion model into multiple diffusion models. The authors show theoretically that the noise factorization functions in DoF satisfy the IGD principle. Experiments are used to shown the effectiveness of DoF.", "questions": "- Table 6 compares decentralized methods (WConcat) with 'centralized' methods (attention), showing similar performance. To properly interpret these results:\n  - How is the centralized attention method implemented during execution?\n  - What observations/information does centralized attention have access to versus decentralized methods? Is centralized attention meant to serve as an upper-bound baseline (with access to information not available in real deployments) rather than a competing method?\n- The authors claim in section 4.3.2 that 'the data factorization h could model more powerful data relationships than the noise factorization function f'. However, Table 14 shows minimal performance improvement when using attention (a more complex function) versus WConcat for data factorization h. Are there theoretical or empirical evidence that supports the claim that a more complex h can compensate for using simple noise factorization functions f (as is mainly used in the paper)?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper provides a novel definition of the IGD principle to generalize the IGM for the use of diffusion models.\n- The authors theoretically show that the proposed DoF added with particular factorization functions satisfy the IGD principle.\n- Strong empirical section showing the effectiveness of the proposed method", "weaknesses": "Potential limitations of noise and data factorization functions - the primary options used in the paper are Concat and Wconcat, which might lack modeling power.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)