DoF: A Diffusion Factorization Framework for Offline Multi-Agent Reinforcement Learning

Authors: Chao Li, Ziwei Deng, Chenxing Lin, Wenqi Chen, Yongquan Fu, Weiquan Liu, Chenglu Wen, Cheng Wang, Siqi Shen
Primary area: reinforcement learning
Keywords: multi-agent reinforcement learning; Diffusion Models;   Offline reinforcement learning

Abstract:
Diffusion models have been widely adopted in image and language generation and are now being applied to reinforcement learning. However, the application of diffusion models in offline cooperative Multi-Agent Reinforcement Learning (MARL) remains limited. Although existing studies explore this direction, they suffer from scalability or poor cooperation issues due to the lack of design principles for diffusion-based MARL. The Individual-Global-Max (IGM) principle is a popular design principle for cooperative MARL. By satisfying this principle, MARL algorithms achieve remarkable performance with good scalability. In this work, we extend the IGM principle to the Individual-Global-identically-Distributed (IGD) principle. This principle stipulates that the generated outcome of a multi-agent diffusion model should be identically distributed as the collective outcomes from multiple individual-agent diffusion models. We propose DoF, a diffusion factorization framework for Offline MARL. It uses noise factorization function to factorize a centralized diffusion model into multiple diffusion models. We theoretically show that the noise factorization functions satisfy the IGD principle. Furthermore, DoF uses data factorization function to model the complex relationship among data generated by multiple diffusion models. Through extensive experiments, we demonstrate the effectiveness of DoF.  The source code is available at [https://github.com/xmu-rl-3dv/DoF](https://github.com/xmu-rl-3dv/DoF).

Decision: accept (poster)
Reviewer scores: 6, 8, 3, 5, 8

Reviews and discussion:
<thread>
creation date:
1729705565075

thread_type:
official_review

<thread_object>
creator: reviewer_yhzy

<content>
summary:
The paper studies the use of diffusion models in offline cooperative multi-agent decision-making (MADM). The authors propose the Individual-Global-identically-Distribute (IGD) principle, a generalization of the IGM principle. IGD requires that collectively generated outcome of individual agents follows the same distribution as the generated outcome of a whole multi-agent system. The authors propose DoF, a diffusion factorization framework for offline MADM which adopts a noise factorization function to factorize a centralized diffusion model into multiple diffusion models. The authors show theoretically that the noise factorization functions in DoF satisfy the IGD principle. Experiments are used to shown the effectiveness of DoF.

strengths:
- The paper provides a novel definition of the IGD principle to generalize the IGM for the use of diffusion models.
- The authors theoretically show that the proposed DoF added with particular factorization functions satisfy the IGD principle.
- Strong empirical section showing the effectiveness of the proposed method

weaknesses:
Potential limitations of noise and data factorization functions - the primary options used in the paper are Concat and Wconcat, which might lack modeling power.

questions:
- Table 6 compares decentralized methods (WConcat) with 'centralized' methods (attention), showing similar performance. To properly interpret these results:
  - How is the centralized attention method implemented during execution?
  - What observations/information does centralized attention have access to versus decentralized methods? Is centralized attention meant to serve as an upper-bound baseline (with access to information not available in real deployments) rather than a competing method?
- The authors claim in section 4.3.2 that 'the data factorization h could model more powerful data relationships than the noise factorization function f'. However, Table 14 shows minimal performance improvement when using attention (a more complex function) versus WConcat for data factorization h. Are there theoretical or empirical evidence that supports the claim that a more complex h can compensate for using simple noise factorization functions f (as is mainly used in the paper)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713637825

thread_type:
official_review

<thread_object>
creator: reviewer_y4is

<content>
summary:
The paper presents Diffusion Factorization Framework (DoF), a new method for offline cooperative multi-agent decision-making (MADM) using diffusion models. DoF generalizes the traditional Individual-Global-Max (IGM) principle with a new Individual-Global-identically-Distributed (IGD) principle. This IGD principle aligns decentralized outcomes in multi-agent diffusion models with centralized outcomes, fostering effective cooperation. DoF includes two key functions: a noise factorization function that decentralizes a centralized diffusion model per the IGD principle, and a data factorization function that models inter-agent data relationships to ensure alignment with ground truth. Extensive experiments demonstrate DoF’s advantages in scalability, cooperation, and alignment with ground truth, outperforming existing MADM methods across various benchmarks.

strengths:
1.	The paper generalizes the well-known IGM principle to the IGD principle, which broadens the applicability of diffusion models in multi-agent decision-making and improves coordination among agents. It addresses the limitations of existing cooperative MADM methods, enhancing scalability and cooperative performance.
2.	DoF introduces novel noise and data factorization methods that align agent-generated data with a centralized diffusion model while enabling decentralized execution. This factorization process addresses the complexity of multi-agent coordination and enhances resource efficiency across a range of multi-agent tasks.
3.	The extensive experiments conducted on the SMAC and MPE show that DoF outperforms competing methods in multi-agent coordination tasks. Experimental results indicate that DoF performs well with an increasing number of agents. This scalability suggests DoF’s potential for large-scale applications in complex multi-agent environments.

weaknesses:
1.	The paper lacks clarity in some important parts. It introduces DoF-Policy and DoF-Trajectory but does not specify the suitable scenarios for each. The results section also fails to indicate which algorithm was used, and Figure 6 only includes DoF-Policy results, making it difficult to assess the relative strengths of the two methods. Additionally, implementation details for the inverse dynamics model in DoF-Trajectory are missing.
2.	In Figure 2(a), there is a mislabeling of the reverse diffusion process: the output should be  x^{k-1} instead of x^k , which may mislead the understanding of the process. Additionally, the caption of Figure 2 lacks necessary descriptions, which reduces its readability.
3.	The paper's benchmarks are based on state vectors, making it unclear whether the proposed method can be applied to environments with high-dimensional observations. The Scalability Experiment section only compares GPU memory and time with MADIFF, without providing comparisons to other baseline algorithms. Since diffusion-based methods require multiple iterations to generate the final output, does this lead to significantly higher computational costs compared to traditional methods?

questions:
1.	Given that QMIX is an online algorithm not designed for offline reinforcement learning, could you clarify the rationale for including it in the comparisons? What insights does this comparison provide?
2.	Can diffusion-based methods effectively handle out-of-distribution situations? Additionally, can these methods be applied to competitive or mixed-motivation scenarios?

Please respond to the aforementioned weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730784871992

thread_type:
official_review

<thread_object>
creator: reviewer_zeea

<content>
summary:
This paper proposes an individual-global-identically distributed (IGD) principle (property) for model design in cooperative multi-agent decision-making.  The paper claims that the property generalizes the popular individual-global-max principle (IGM).  They further propose a diffusion factorization framework that 1) satisfies the IGD principle and 2) performs well on several datasets.

strengths:
The empirical results show their method perform slightly better than other methods.

weaknesses:
I am unfamiliar with this domain of literature, and my comment would mostly focus on mathematical writing.  Overall, I have strong concerns about the claim that IGD generalizes IGM, and I feel the theory about diffusion factorization framework under Gaussian assumption satisfying IGD uninteresting.  Thus, I suggest rejection.

The technical writing quality needs significant improvement. Here are some comments.
1. Does IGD property apply to the initial distribution $p_{\theta_{tot}}(x_{tot}^0)$  or the whole series $p_{\theta_{tot}}(x_{tot}^{0:K})$? Does $x_{tot}^K$ need to be sampled from Gaussian?  Does the reverse process need to be a Markov chain?
2.  The claim that IGD generalizes IGM is not well-defined.  IGM needs a value function but does not necessarily have a distribution over action or even under a diffusion model.  Please give an explicit statement on the generalization: For instance, given a value function $Q_{jt}$ satisfies IGM what is the corresponding $p_{\theta_{tot}}$ such that Eqn.(1) holds for any history $\tau_{tot}$.
3.  At the end of section 4.2, there should be formal statements of IGD vs individual global optimal principle or risk-sensitive IGM principles.

If I understand correctly, Theorem 1 basically states that each component of isotropic Gaussian is independent.  I feel this result is very limited and hardly holds in general.  For instance, it’s unclear whether this would still apply even when the Gaussian has a non-diagonal covariance.



Minor comments
- In eq (3), instead of $\theta_i\subset \theta_{tot}$, $(\theta_1,\dots, \theta_K) = \theta_{tot}$ would be better, as the order of coordinate matters.  
- $\epsilon^{\theta_1}_1$ in eq (5) is not clearly defined.

questions:
Can the authors provide formal statements and proofs showing that IGD generalizes IGM, individual global optimal principle or risk-sensitive IGM principles?

Are there any nontrivial examples that satisfy IGD beyond the Gaussian case (or a component-wise independent Markov chain with a stationary initial distribution)?

---
After discussing with the author, I remain deeply concerned about the paper’s mathematical writing, which requires substantial improvement. First, the claim that IGD generalizes IGM is neither formally stated nor proved, even after multiple responses. This is not merely an issue of correctness; the statement is fundamentally unfalsifiable and lacks rigor. Second, the definition of IGD is unnecessarily convoluted and appears redundant. A more straightforward would be that a diffusion model exhibits IGD if each component of the initial state is mutually independent. While the empirical results may be interesting, the mathematical statement is seriously deficient and demands thorough and precise revision to meet an acceptable standard.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732335535255

thread_type:
official_review

<thread_object>
creator: reviewer_5vm5

<content>
summary:
The paper proposes to use Individual-Global-Identically-Distributed (IGD) principle for the training of diffusion models in multi-agent decision making (MADM). This IGD, requiring the outcome of multi-agent diffusion model should be identically distributed as the collective outcomes from multiple individual agent diffusion models, is a generalization of the individual global max (IGM) principle. The authors proposes DoF a way to do factorization in this setting and validated their proposed approaches via experiments.

strengths:
+ IGD is a very interesting and promising idea. I can see its potential in multi-agent diffusion.
+ IGD captures recovers IGM or risk-sensitive IGM principle.

weaknesses:
- While the ideas are interesting, presentation of the paper makes it difficult to fully appreciate the contribution. For example, the algorithms are missing from the main manuscript. 
- Related to the previous point, it is unclear how to formulate a diffusion algorithm based on Theorem 1.
- Some of the notations are not well defined, such as $\epsilon$ in the theorem, and what role $\theta$ play in Eq (2). 
- The POMDP setting should be described a bit more in the main text, for instance, there is one environment that all agents are interacting with and the transition probability depends on the product space of all agent's actions. 
- Typos: I list a few here, but please go through the paper one more time to double check.  For example, in definition 1, should be "there exist ... functions", the sentence before section 3, "to for $x^k$".

questions:
-  How long did the proposed methods take to train compared to other methods? The paper reported inference time, but not the training time. 
- I was a bit confused by the overall setup, are the agents doing global training and then individual sampling of the noise after doing the factorization? Is there a way for the agents to train simultaneously? 

I just got the invitation to review this paper last night, and I understand that the authors probably won't have time to address all of my questions/comments. I hope these comments will be helpful for improving the manuscript for publication (at ICLR or elsewhere).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732501778255

thread_type:
official_review

<thread_object>
creator: reviewer_strz

<content>
summary:
Please see the strengths and weaknesses sections.

strengths:
- This paper tries to solve an important problem in Diffusion-model-based offline MARL, that is, the use of the centralized training with decentralized execution (CTDE) paradigm. As noted in [1], this is a research gap in this field.

- The algorithm design is simple yet effective, which is demonstrated in a series of benchmarks through comparisons with various baselines.

### Reference:

[1] Chen, et al. "Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions." TMLR, 2024.

weaknesses:
- The definition of IGD seems to assume the independence among agents' behaviors/rollout trajectories. This assumption is also shown in the selection of the noise factorization function, which is a simple concatenation operator.

- The first paragraph of Section 4.3 is a little bit of confusing. A refined version of the overview should be provided.

- The provided codebase does not involve a detailed instruction to reproduce **ALL** numerical results in the paper. The code files are also not complete. If the authors fix this, I will improve my rating.

questions:
- Can you intuitively explain why the agents trained with MADiff still suffer from collisions and perform worse than the ones trained with DoF in Figure 1? I suppose their main difference lie in scalability which shouldn't be an issue for the simple illustrative example.

- Functionally, \(f\) and \(h\) appear to be similar. Why is it necessary to train an additional \(h\)?

- (This is only a suggestion for future works.) Policy distillation methods can be an important category of baselines to compare with. Specifically, you can apply CDG to learn a centralized policy and distill a set of decentralized policies for each agent from the centralized agent, which is also a form of factorization.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tries to solve an important problem in Diffusion-model-based offline MARL, that is, the use of the centralized training with decentralized execution (CTDE) paradigm. The algorithm design is simple yet effective, which is demonstrated in a series of benchmarks through comparisons with various baselines. The authors and reviewers had multiple interactions, and the responses seem to address the key questions by the reviewers.