INS: Interaction-aware Synthesis to Enhance Offline Multi-agent Reinforcement Learning
Authors: Yuqian Fu
Yuanheng Zhu
Jian Zhao
Jiajun Chai
Dongbin Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-agent reinforcement learning
Offline reinforcement learning
Diffusion models
Primary area: reinforcement learning
TL;DR: We propose an interaction-aware approach to synthesize high-quality datasets for offline MARL.

Abstract: Data scarcity in offline multi-agent reinforcement learning (MARL) is a key challenge for real-world applications. Recent advances in offline single-agent reinforcement learning (RL) demonstrate the potential of data synthesis to mitigate this issue.
However, in multi-agent systems, interactions between agents introduce additional challenges. These interactions complicate the synthesis of multi-agent datasets, leading to data distortion when inter-agent interactions are neglected. Furthermore, the quality of the synthetic dataset is often constrained by the original dataset. To address these challenges, we propose **INteraction-aware Synthesis (INS)**, which synthesizes high-quality multi-agent datasets using diffusion models. Recognizing the sparsity of inter-agent interactions, INS employs a sparse attention mechanism to capture these interactions, ensuring that the synthetic dataset reflects the underlying agent dynamics. To overcome the limitation of diffusion models requiring continuous variables, INS implements a bit action module, enabling compatibility with both discrete and continuous action spaces. Additionally, we incorporate a select mechanism to prioritize transitions with higher estimated values, further enhancing the dataset quality. Experimental results across multiple datasets in MPE and SMAC environments demonstrate that INS consistently outperforms existing methods, resulting in improved downstream policy performance and superior dataset metrics. Notably, INS can synthesize high-quality data using only 10% of the original dataset, highlighting its efficiency in data-limited scenarios.

Published as a conference paper at ICLR 2025
INS: INTERACTION-AWART SYNTHESIS TO ENHANCE OFFLINE MULTI-AGENT REINFORCEMENT LEARNING
Yugian Fu Fu?.? Yuanheng Zhu Zhu Jian Zhao Jiajun Chail Dongbin Zhao 1.2,
1 Institution of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence, University of Chinese Academy of Sciences
3 Polixir
fuuugiia22222, chai yuanhen yuanheng. zhu, dongbin. zhao) @ial ac en
jian. zhao@pol peppoliiiiaat
ABSTRACT
Data scarcity in offline multi-agent reinforcement learning (MARL) is a key chal- lenge for real-worid applications. Recent advances in offline single-agent reinforce- ment learning (RL) demonstrate the potential of data synthesis to mitigate this issue. However, in multi-agent systems, interactions between agents introduce additional challenges. These interactions complicate the synthesis of multi-agent datasets, leading to data distortion when inter-agent interactions are neglected. Furthermore, the quality of the synthetic dataset is often constrained by the original dataset. To propose INteraction-awarr Synthesis (INS), which
address these challenges, synthesizes high-quality multi-agent datasets using diffusion models. Recognizing the sparsity of inter-agent interactions, INS employs sparse attention mechanism to capture these interactions, ensuring that the synthetic dataset reflects the under- lying agent dynamics. To overcome the limitation of diffusion models requiring continuous variables, INS implements bit action module, enabling compatibility with both discrete and continuous action spaces. Additionally, we incorporate a select mechanism to prioritize transitions with higher estimated values, further enhancing the dataset quality, Experimental results across multiple datasets in MPE and SMAC environments demonstrate that INS consistently outperforms ex- isting methods, resulting in improved downstream policy performance and superior dataset metrics. Notably, INS can synthesize high-quality data using only 10% the original dataset, highlighting its efficiency data-limited scenarios. The source code is available at here.
INTRODUCTION
Recent advances in offline multi-agent reinforcement learning make possible to train policies for real-world applications, such as autonomous driving (Zhang et al., 2024), robotics (Li et al., 2024a), and network control (Ma et al., 2024), entirely from previously collected multi-agent datasets. This approach climinates the need for online interactions with environments, mitigating the high costs associated with trial-and-error in online MARL. However, offline MARL faces challenges due to the fixed dataset with incomplete coverage of the state-action space (Fujimoto al., 2019), leading to the distribution shift problem and suboptimal policy performance. Furthermore, data collection in real-world scenarios expensive and potentially dangerous (Gao et al., 2024; Canese et al., 2021), restricting the dataset scalability and resulting in policy overfitting (Agarwal et al., 2020). Consequently, learning effective MARL policies from limited datasets remains critical challenge.
The data scarcity challenge also exists in offline single-agent reinforcement learning, where using generative models for data synthesis is prevalent approach overcome this issue. Despite the promising results in single-agent settings, extending this approach to multi-agent scenarios presents unique challenges. Firstly, multi-agent systems involve complex inter-agent interactions (Georgeff, 1988), such as collisions and information exchange, which influence agent dynamics. Therefore, merely applying existing data synthesis methods by treating each agent independently often results
"Corresponding Authors

---

Published as a conference paper at ICLR 2025
in lower-quality synthetic datasets, these methods fail to capture the true agent dynamics. In Appendix A, we present a motivating example to illustrate this phenomenon. Moreover, an agent typically interacts with only subset of other agents (DeWeese & Qu, 2024; Liu et al., 2024), and considering interactions with irrelevant agents can introduce distractions into the synthesis process. Secondly, research shows that both the diversity and quality of datasets are crucial for effective policy training (Corrado & Hanna, 2024). While generative models aim to preserve diversity by approximating the original dataset distribution, their effectiveness constrained by the dataset quality, Low- Low-quality original datasets often lack sufficient high-value transitions, leading to imbalanced synthetic datasets and ultimately hindering effective policy learning. Finally, since most diffusion models are limited to generating continuous data, existing synthesis methods primarily focus on continuous action spaces, such as D4RL (Fu et al., 2020) and MPE (Lowe et al., 2017). This restricts their applicability to multi-agent environments with discrete action spaces, such as SMAC (Samvelyan et al., 2019) and FightingICE (Tang et al., 2023). In summary, there is critical need for a method that can synthesize high-quality multi-agent datasets applicable to both continuous and discrete action spaces in the offline MARL domain.
To meet this need, propose INteraction- aware Synthesis (INS), which leverages powerful generative model, diffusion model (Ho et al., 2020), to synthesize multi- agent datasets. Diffusion models excel ensuring broad coverage of the data distri- bution while maintaining high quality, mak- ing them ideal for modeling complex multi- agent transitions. To model the inter-agent interaction during the synthesis process, INS integrates sparse attention mecha- nism (Martins & Astudillo, 2016). This mechanism assigns different weights to agents based on their interactions and fil- ters out irrelevant ones. To further improve dataset quality, we design a value-based select mechanism that estimates transition values and prioritizes higher-value transitions for the synthetic dataset. To ensure compatibility with discrete action environments, INS implements a bit action module, bridging the gap between continuous diffusion models and transitions with discrete actions. We conduct preliminary evaluation of INS with existing synthesis methods on various multi-agen datasets. As illustrated in Figure our method outperforms baseline methods in both policy performance and dataset metrics. In contrast, baseline methods that ignore inter-agent interactions yield performance comparable to, or even worse than, that of the original datasets. More details of the preliminary evaluation provided in Appendix B.
Addictive Noise
VAE Augmented
MA-SynthER
Our Metheds
640
Marginal
Our Methodk
Menn
600
153
Corregatnn
stack Reward
(outiora))
(dxure))
Dynamic MSE
Novelty
Figure I: Preliminary evaluation. Our INS con- sistently outperforms existing data synthesis methods across policy performance and dataset metrics.
Our contributions are three-fold: (1) We propose an interaction-aware synthesis method, INS, which leverages diffusion models to synthesize high-quality multi-agent datasets, applicable to both continuous and discrete action spaces. INS is compatible with any offline MARL methods, as synthesizes data without modifying the policy learning algorithmss To the best of our knowledge, INS is the first diffusion-based data synthesis approach for offline MARL. (2) We introduce a sparse attention mechanism to capture inter-agent interactions in the synthesis process, effectively reducing the irrelevant interaction modeling. Additionally, we design a select mechanism to improve dataset quality by increasing the proportion of high-value transitions. (3) We conduct extensive evaluations on MPE and SMAC datasets, showing that our method enhances both dataset metrics and policy performance across various offline MARL methods. Moreover, INS can synthesize high-quality data with only 10% of the original dataset, underscoring its effectiveness in data-limited scenarios.
RELATED WORKS
Data Synthesis in Offfine MARL. In offline reinforcement learning, agents frequently encounter challenges stemming from dataset scarcity (Wang et al., 2022). To mitigate this challenge, recent advances leverage diffusion models for data synthesis, e.g., MTDiff-S (He al., 2024) and Syn- thER (Lu et al., 2024). MTDiff-S generates trajectory-level data for multi-task settings, producing trajectories for unseen tasks. SynthER, closely related to our method, synthesizes data at the transition
2

---

Published as a conference paper at ICLR 2025
level for subsequent policy training. In offine MARL, data synthesis has not yet achieved the same level of performance as that observed in single-agent RL. This disparity is pronounced due to the complex dynamics in multi-agent systems (Tian et al., 2023; Chai et al., 2024), where multiple agents interact within a shared environment. Our method diverges from existing approaches by explicitly considering inter-agent interactions and improving dataset quality.
Diffusion Models in Offline MARL. Diffusion models (Ho et al., 2020; Karras et al., 2022), as powerful class of generative models, have recently been adopted in offline reinforcement learning as planners and policies (Zhu et al., 2023b). The application of diffusion models in the field of MARL remains underexplored. Building on the foundational work of Janner et al. (2022) and Wang et al. (2023), recent studies extend diffusion models the MARL domain, such MADiff (7hu et al., 2023a) and DOM2 (Li et al., 2023). Specifically, MADiff formulates planner as a return-conditional diffusion model designed to maximize cumulative rewards in multi-agent decision-makiin on the other hand, focuses on enhancing policy expressiveness and diversity by leveraging diffusion models. In contrast to these methods, which focus on the policy level and aim to maximize rewards or optimize action selection, INS leverages diffusion models to synthesize multi- datasets, thereby improving offline MARL performance from data-centric perspective. Further details regarding the differences between our method and related works can be found in Appendix D and Table A1.
DOM2,
PRELIMINARIES
3.1 OFFLINE AGENI-AGENT REINFORCEMENT LEARNING
We focus on the fully cooperative MARL, which can be modeled as Decentralized- Partialll Observable Markov Decision Process (Dec-POMDP) (Olichoek et al., 2016), defined by the tuple (S, A. P, O, 22 R, n. 7). Here, s E S denotes the global state of the environment and A represents the action set for each of the n. agents. At each time step, each agent i E E == 21,22 ................................ n.} receives a local observation O E O, generated by the observation function [2x i) S x X - O. Each agent i selects and executes its own action a, E A, resulting in a joint action E A that induces the next state according to the state transition function P(s, a) S A + S. The environment provides global reward r shared by agents, determined by the reward function R(s, a)) S A" - IR. The objective is to find the optimal policies π = (T,,... Tp) that maximize the discounted return G -i= Y'r,, where T denotes the horizon and 7 E 10, represents the discount factor.
In the offline MARL setting, access to each agent's transition restricted to a fixed multi-agent dataset Dori. := (w+ x² ...., x |Po.. I]. Each joint transition is defined æ = (o, a, o' .r) and the transition of agent i is given by = (o,, 0; o,', .) E Rdd The dataset is typically collected by arbitrary policies like human experts or random policies. However, these policies fail guarantee sufficient coverage of the whole state-action space, which grows exponentially with the number of agents. As result, learning effective decision-making policies that surpass the behavior policy from scarce and potentially suboptimal multi-agent datasets poses a significant challenge. Existing offline MARL methods (Yang et al., 2021; Pan et al., 2022) attempt to address these issues by incorporating policy constraints and conservative estimation techniques. Moving beyond these methods, another intuitive yet underexplored solution to dataset scarcity is data synthesis (Levine et al., 2020).
3.2 DIFFUSION MODELS
Diffusion models (DM) (Ho et al., 2020) represent a class of generative models that generate data Do by incrementally removing noise from a Gaussian distribution. These models comprise two primary processes: a forward noising process and reverse denoising process. Given the data distribution p(ær and the noise standard deviation a, the data distribution with added noise denoted by p(x; or.. The forward process is characterized by a Markov chain, starting from the data æ ~ p(x) and gradually increasing the noise level to o max. The reverse process involves K times sequential denoising, starting with noise æ sampled from Gaussian distribution A(0 o T), and following decreasing fixed sequence of noise levels d maax = ou > 01 > > ок = 0. As a result, the endpoint Zy of this process aligns with the original data distribution (.))
In this work, we adopt the Elucidated Diffusion Model (EDM) architecture (Karras et al., 2022), which implements both forward and reverse processes through the continuous schedule of noise
3

---

Published as a conference paper at ICLR 2025
Phase A: Train Diffusion Model
Phase B: Synthesize Datasets
Phase C: Train Offline MARL
xB
Multi-agent Joint Policy
π (T), (T1....;To) To)
Embedding Layer
Downstream Offline
Sparse Irannformer Encoder
K denoising steps
MARL A Algorithms
MA-BCQ
OMAR
Bit Action (for discrete actions)
MA-ICQ
MA-CQL
Transition Candidates
xB
Output Transitions
value
training
Value Estimator
Select Mechanism
Transition Samples
trninable
Original Dataset
add to
Synthetic Dataset
sample
frozen
D
D
Figure 2: Overview of INS framework. The framework consists of three phases: In the first phase, we train a diffusion model. Following this, we synthesize and select transition data. Lastly, we employ the synthetic transitions to train offline MARI poliey
levels in probability flow ordinary differential equation (ODE);
dæ - log:p(:(p;)))) (kk)ddk
(I)
where the dot notation represents the time derivative, and okk denotes the noise level time Ve log p(:; (k)) is the score function, which can be computed via the connection between score matching and denoising: V. log p(x; = (Do(2;; z)/o². Do(x;o) denotes the denoiser with parameters 0. The Lg loss objective for denoising, applied to the denoiser Do σ) is defined as:
aa (0) = Ee--ptt). DD + c, - ==1
(2)
Subsequently, we can sample transitions via solving Eq. (I) with the trained denoising network Da(:; 0) and score function v log (x: (l:)).
4 METHOD
In this section, we present interaction-aware synthesis (INS), which synthesizes high-quality multi- agent transitions and is compatible with any offline MARL methods. As illustrated Figure 2, our method consists of three phases: (i) train the diffusion model, (ii) synthesize and select transition data, and (iii) train offline MARL policy.
4.1 PHASE A: TRAIN DIFFUSION MODEL
Diffusion Model Architecture. In order to synthesize multi-agent transitions, we employ diffusion model to learn the distribution of the original dataset Dori.. While existing methods typically use a U- Net (Ronneberger et al., 2015) denoising network, INS adopts Transformer encoder (Vaswani et al., 2017) as its basic architecture, incorporating a sparse attention mechanism along the agent dimension. This design offers two key advantages: First, multi-agent systems, interaction relationships are often sparse (Liu et al., 2024; Fu et al., 2025; Li et al., 2019), with agents typically interacting only with specific nearby agents. The conventional dense attention mechanism in the Transformer encoder assigns non-zero weights to all agent pairs, implicitly assuming universal interaction. This mechanism inefficient and complicates learning the true interaction distribution. Moreover, modeling irrelevant- agent interaction can distract diffusion models, potentially hindering the denoising process. In contrast, the sparse attention focuses on critical inter-agent interactions, thereby simplifying the learning task for models. Second, prior research (Dong et al., 2024) indicates that Transformer- based diffusion models exhibit improved training stability and achieve superior performance in reinforcement learning contexts. The diffusion model architecture of INS detailed in Appendix K.1.
4

---

Published as a conference paper at ICLR 2025
Sparse Attention for Inter-agent Interaction. In the denoising process, the transition sample 24 = [21.k..... _n, E [R at denoising step k is initially embedded into 2k,e E R=xd, via an embedding layer. For notational simplicity, denote the embedding matrix X instead of Kk.e. The embedding features of each agent are then projected into query Q XWQ, key K XWK, and value V = X XWV representations, where Wº, WK and W are learnable parameters. The product of query and keys is denoted Z QKT R comprising rows {=1,-. ..... )n) To focus on the critical interactions, we replace the dense Softmax in conventional dense attention with Sparsemax (Martins & Astudillo, 2016), which tends to produce sparse attention weights:
Sparsemax(2) = arg min | 2112
(3)
PÊA"
where: z E IR" is the input, p R" the output, and A ~ = {p e R
p 2 0, [|pl] = 1] is the
(n 1) diimensional simplex. The solution to Eq. (3) is given as follows:
p
[z)]]++
where (z) is the Sparsemax threshold, 1 E Rn is the all-one vector, and [++ denotes the ReLU function. In order get the solution p. we calculate the threshold o (z) as follows:
(z) (Eismte & -
(5)
m(²)
where é is sorted in descending order, and m(²) is the number of non-zero elements in p:
m(²) = arg max ().......))
m2m
2 1
(6)
iS m
Thus, the resulting sparse attention weight is given by P Sparsemax Z2) E RR*n, The output of sparse attention is (P/v /d)) V. which preserves key properties of Softmax while assigning zero probability to irrelevant agents. This approach allows the diffusion model concentrate on critical inter-agent interactions, effectively mitigating distractions during the denoising process. A detailed discussion of Sparsemax and its comparison with related work can be found in Appendix E.
Bit Action. While the current diffusion model inherently generates continuous data, many multi-agent environments, such as SMAC (Samvelyan et al., 2019), feature discrete action spaces. To bridge this gap, we introduce a bit action module inspired by Bit Diffusion in Chen et al. (2023). The bit action module maps a discrete action space containing M actions to a real-valued bit vector of length L == (M)], f0.0, 1.0) During the denoising process, this bit vector is concatenated with other continuous variables the transition. After that, the output vector corresponding to the action is thresholded into binary bit vector via quantization mechanism. Then this vector is mapped back to the original discrete action space. The bit action design offers two key advantages. Firstly, it enables INS be universally applicable to multi-agent environments with either discrete or continuous action spaces without requiring additional modifications. Secondly, compared to alternative discretization techniques such as one-hot encoding, the bit action module effectively reduces the action space dimensionality from M to [log,(M)) thereby decreasing computational complexity.
Diffusion Model Training. Given the original multi-agent dataset Dori. containing transitions æ and the denoising network Do(æ; a), we train diffusion model Pam((a) to approximate the joint distribution p( of multi-agent transitions. The parameters of the denoising network are optimized by minimizing the L2 loss for denoising score matching, as described in Eq. (2). With the trained model, we can generate numerous transitions from the original dataset, as shown in the next phase.
4.2 PHASE B: SYNTHESIZE DATASETS
The synthesis process begins by sampling an initial noisy transition æK. which is considered the result of multiple noise additions to the original transition. Following the diffusion steps described in Eq. (1), the final joint transition is obtained as:
6, à. o'.f)
steps
(o, a, o
(7)
x
5

---

Published as a conference paper at ICLR 2025
In offline MARL, both dataset diversity and high value transitions crucial, implying that the relative importance of each transition varies (Schweighofer et al., 2022). Previous approaches primarily focused on diversity, overlooking the role of high-value transitions, which led to an imbalance in transition quality within the synthetic dataset Dsy.. To mitigate this imbalance, we propose transition select mechanism that prioritizes transitions based on their estimated values. Specifically, state value estimator is trained simultaneously with the diffusion model during Phase A estimate transition values:
5% (o) ((o,,a))
VVz(o)) - Vo(o))²
(8)
where á and é are parameters of the value estimator, and y is the discount factor. During the synthesis process, B transitions are generated as candidates. The value estimator evaluates each transition, and candidates within the proportion 1) are selected for inclusion in the final output synthetic dataset. In this paper, we propose two select mechanisms:
Top-K:
All candidate transitions are sorted by their estimated values, and the top 7/ proportion of them are selected. While this approach intuitively selects the most valuable transitions for training, it may reduce dataset diversity and degrade policy performance.
Softmax: A Softmax probability calculated based on the estimated values of transitions:
exp(Va(o())
p(ar))
Σj=1 exp(VV (o())
Then the proportion of candidates are sampled according to these probabilities. Despite the additional computational cost, empirical results show that enhances downstream policy performance. Therefore, the Softmax select mechanism is employed in the primary experiments.
The selection proportion represents a trade-off between dataset diversity and quality. Increasing this proportion tends concentrate more high-value transitions but at the cost of reduced diversity. This parameter can be fine-tuned according to the specific requirements of the synthetic data.
4.3 PHASE C: TRAIN OFFLINE MARL
In the final phase, the high-quality dataset synthesized in the previous stages is used for offline multi- agent policy training. As previously mentioned, our approach is orthogonal to existing offline MARL methods. Therefore, rather than designing new policy training algorithms, we directly integrate INS with these methods to assess its effectiveness. The overall algorithm is given in Appendix C.
5 EXPERIMENTS
In this section, we evaluate INS across different multi-agent settings, including both discrete and continuous action spaces. We initially evaluate our method on multiple datasets, replacing the original datasets with synthetic datasets to isolate and highlight the impact of synthesis methods. We also investigate how the quality and size of synthetic data affect downstream policy performance in offline MARL. Additionally, conduct an extensive ablation study to empirically validate the effectiveness of key components and hyperparameters within our method. We provide visualizations coverage, attention weights, reward distributions, and value distributions to illustrate the effects of our method. Finally, we demonstrate the advantage of INS when trained on small original datasets.
data
5.1 EXPERIMENTAL SETUP
We conduct experiments on two widely used offline MARL environments: Multi-Agent Particle Environment (MPE) (Lowe et al., 2017) for continuous action space tasks and StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) for discrete action space tasks. We use a diverse range of original datasets, including 12 from MPE and 9 from SMAC. All datasets are categorized based on the quality of the policies used for collection. In MPE, datasets are categorized as expert, medium, medium replay (md-replay), or random, while in SMAC, they are classified good, medium, or random. In our primary experiments, the size of the synthetic dataset is fixed at 5.M. More information about datasets provided in Appendix H.

---

Published as a conference paper at ICLR 2025
Table I:
The normalized average scores of INS and baselines in MPE and SMAC. Original means training policies on the original datasets. We show the mean and standard deviation of scores averaged over 8 seeds. The best performance is highlighted in bold.
MA-CQQ
MA-CQL
OMARMA-BC
Origiral
MA-Syaafi
INS
Original
MA-Syshili
INS (vur)
Original
MA-Syrhhiit
INS
Export
3.3
104.7 4:2
107.0 2:00::00
97.9 53
96.
98.5
114.1::33
111.3 199
11566 3.1
Specul
Medium
268 48
27.911
50.
34.1 7.1
35.6 6.2
36.6
47.4 13.3
45..11 11.2
48.0 11.3
Mikerplay
129
18.2
20.
18.1
21.1
57.8 10.5
17.1
88 13.0
230
Exen
131.7:151
114.1
1142
93.8 12.2
92.5
93.
117.2 18.8
1144 20.3
119.1 196
MIP
Teg
Meduu
623 19.6
61.6 18.6
64. 24.3
60
602 0 24.3
61.
66.8 20.9
64.1 23.4
07.5
(cottinuuu))
Milerplay
33.9
362 =:
36.4 30.3
24.8 17.5
25.2 15.1
26.1 15.1
46.1 13.5
440 16.3
48.6 14.9
13
1.4
4.7
9.
11.0
10.5
Papen
110.7
108.
25.8
Wirl
Mediun
712 19.2
73.6 19.4
72. 17.9
17.9
$5.0 10.6
58.8 10.1
24.32:
12.1.
75.0 16.3
Milerplay
122.11 9.8
12.6 6.64
11.9.177
29.3 12.8
27.1 152
90.4 14.
43.0 20.5
42.6 21.2
46.2 14.5
Rendom
08
03:1.29
088
0.2.4
69
Total Seore
361.9
510.3
Good
18.1 06
186.1 1.5
19. 1 1.4
18.5
18.2 0:09
20.1
3.4:2 0.8
0.8
4.1 0.5
3m
169
179.1
185
163.1.08 0.8
162.1
17.9
4.1.1
1.3
Good
1.
1.
17:2 1.7
12
13.-4:43
14.3
2.14004
30:
SMAC
fm, ki_ 5m
Mediun
0.
1.0
139 1.1
148 1.7
1.91
16.1 211
09
0.
4,3: 0.7
(diccrete)
Pooer
05
005.400
1..04.1.0
12.3
02-406
19.3
20:2::2.0
30.7 ::
11.12:511
1:.1:::599
12.5
4.5
8m
19.3
1.4.133
06
Poor
0.
10.2 1.4
10.1
4.
3,8
5.4 19
34
3.7 1.2
29
Tatal Score
135.6
137.8
1453
109.6
105.9
138
33.5
54.3
35.
MA- MA-SynthER
Our Methods w/o Select
Our Methods w/o Attention
Our Methods
Sinnlanty
Similarty
Constaio
REAL Reaad
Dymene
Nardly
Curriaaion
Nuvelty
(a) Spread expert
(b) Spread medium
(c) 3m good
(d) 3m medium
Figure 3: Dataset metries of the synthetic dataset. We compare datasets synthesized by INS and MA-SynthER across different datasets. Details of metrics provided in Appendix F.
We compare our method against SynthER (Lu et al., 2024), a recently proposed synthetic method for offline single-agent RI. To ensure fair comparison, implement multi-agent version of SynthER (MA-SynthER) and evaluate it in the same settings. The detailed implementation of MA-SynthER, is provided in Appendix J. The data synthesis methods used in the preliminary evaluation (Additive Noise and VAE Augmented) are excluded from the main experiments due to their poor performance. In order to demonstrate the broad applicability of our method, we select four widely used offline MARL algorithms for downstream policy training: MA-ICQ (Yang et al., 2021), MA-CQL (Kumar et al., 2020), OMAR (Pan et al., 2022), and MA-BCQ (Wu et al., 2019). More information on these algorithms is provided in Appendix I.
5.2 COMPARATIVE EVALUATION
We begin our evaluation by comparing the performance of INS combined with different offline MARL algorithms across multiple environments. Due to constraints in the official code implementations, we evaluate OMAR only in MPE and MA-BCQ only in SMAC. The main results are presented in Table 1, which demonstrate that offline MARL algorithms using INS synthetic datasets outper- form all other baseline datasets across all algorithms on the total score. Among different types of original datasets (expert/good, medium, md-replay, and random), synthetic datasets generated from expert/good and md-replay datasets lead to more substantial improvements in downstream policy
7

---

Published as a conference paper at ICLR 2025
Our Mathhoss
Better
wie Altcaation
14.0
Nocmonted 16
Avematted
12.
110
11.5
Selection Propartion (a) Spread (MPE)
Selectien Propartee
Spreed
SSL.V, (SMAC)
(b)
(SMAC)
Figure 4: Module ablation. The nor- malized average score of the down- stream policy with ablation variants.
Figure Parameter ablation. Ablation studies on the selection proportion and the select mechanism type across different datasets.
performance compared with those generated from medium and random datasets. This discrepancy can be attributed to two key factors. First, the narrow coverage of medium datasets constrains the potential improvements in data coverage achievable by the diffusion model of INS, thereby limiting the synthesis fidelity. Second, the inherently low quality of random datasets hinders the capacity of synthetic datasets to enhance downstream policy performance. These findings underscore the critical role of initial dataset quality in synthesis methods. Additionally, the limited performance gains of MA-SynthER, and in some cases, performance degradation (such as when combined with OMAR in the MPE environment), can be attributed to its failure consider the inter-agent interaction. To summarize, these results underline the effectiveness of INS across various offline MARL algorithms.
5.3 DATASET QUALITY ANALYSIS
In order to analyze the dataset quality of synthetic data, we employ multiple data metrics to evaluate the synthetic data synthesized by INS and MA-SynthER. Inspired by prior works (Lu al., 2024; Lee et al., 2024), we conduct comprehensive analysis using five metrics: Similarity measures the distance between synthetic and original data distributions, while Correlation evaluates their statistical relationships, whether casual or not; Oracle Reward represents the true reward of the synthetic data, which indicates optimality; Dynamic MSE evaluates how well the synthetic data aligns with the environment dynamics; Novelty quantifies the uniqueness of the synthetic transition, reflecting the extent data coverage. Detailed definitions and calculation methods for these metrics are provided in Appendix F. We exclude Oracle Reward and Dynamic MSE for the SMAC environment due the unavailability of true environment rewards and environment dynamics. Figure 3 illustrates the metrics of synthetic datasets generated from original datasets of different quality. The results demonstrate that our method consistently outperforms MA-SynthER across all metrics and datasets. This superior performance indicates that INS not only enhances the true reward and data coverage of the synthetic data but also preserves its similarity and correlation with the original dataset. Furthermore, it demonstrates INS's ability capture the dynamic information inherent in the original dataset.
5.4 ABLATION STUDY
To verify the effectiveness of each component in our method and to analyze the impact of hyperpa- rameters and dataset size, we conduct a series of ablation experiments. All experiments performed both MPE and SMAC environments, with MA-ICQ serving as the downstream policy algorithm.
Module Ablation. We evaluate the contribution of key modules through several ablations. These include replacing sparse attention with dense attention (w/o sparse), removing the select mechanism retaining all synthetic data (w/o select), and replacing the attention module with an MIP, allowing each agent to independently synthesize transitions (w/o attention). Figure illustrates the impact of each module on dataset quality and policy performance. The results demonstrate that the attention mechanism significantly enhances the ability of INS to learn inter-agent interactions, improving synthetic data accuracy and policy performance. The select mechanism effectively prioritizes high- value transitions, which proves crucial in complex enviromments such SMAC. Additionally, sparse attention reduces redundant information in agent interactions, mitigating distraction impact in the denoising process. We also analyze the metrics of synthetic data generated by different ablation
8

---

Published as a conference paper at ICLR 2025
variants. As shown in Figure 3, the select mechanism enhances the true reward of synthetic data, while sparse attention improves the accuracy and state scte-action coverage of the method. Details of the ablation variants and more results are provided in Appendix G.3.
Parameter Ablation. We investigate the effect of selection proportion 1) and select mechanism type within INS. As mentioned in previous sections, the selection proportion represents a trade-off between dataset diversity and quality, with higher proportions favoring high-value transitions at the cost of diversity. As shown in Figure 5, the relationship between policy performance and the selection proportion varies across different environments. In the Spread environment, the offline MARL algorithm benefits from viguevalue transitions in the dataset, while in 5m_ vs 6m, agents require more diverse offline dataset to achieve optimal performance. We find that the selection proportion of 0.8 provides sufficient performance on different datasets, so we fix it in our primary experiments. Additionally, our experiments on select mechanisms show that the Softmax approach used in primary experiments balances high-value distribution and dataset diversity. Across various selection proportions in different environments, the Softmax select mechanism achieves superior performance compared to the Top-K K select mechanism.
Synthetic Dataset Size. In order to inves- tigate the relationship between synthetic dataset size and downstream policy per- formance, we conduct experiments with varying dataset sizes. We select five differ- dataset sizes spanning two orders of magnitude: M0. 0.5M, 1.0M, 5.0.M, 10.0M) transitions. As shown in Fig- ure 6, as the dataset size increases, policy performance improves. However, the im- provement does not scale linearly with the dataset size. Similar to findings reported by He et al. (2023) and Lu et al. (2024), the performance tends to saturate as the dataset size increases beyond certain threshold. Based on these observations, we conclude that dataset size of 5M offers reasonable balance between performance and computational cost.
Mi-aCQ
MA-CO
Seraaled Star
055 3,0
31.
8.1
0L5
30.
Nuaker of Suaephs
Nander Seaaphe
(a) Spread (MPE)
(b) 5m vs. 6m (SMAC)
Figure 6: Impact synthetic dataset size. Each curve displays the mean and standard deviation of the normal- ized score across 5 random seeds.
5.5 VISUALIZATION
To provide intuitive insights into the effectiveness of INS in synthesizing multi-agent interaction datasets, we present four types of visualizations: data coverage, attention weights, value distributions, and reward distributions. These visualizations aim to demonstrate the improvements in dataset quality, the impact of the sparse attention mechanism, and the effect on value and reward distributions.
Data Coverage. To assess the improvement in dataset coverage, we employ t-SNE (van der Maaten & Hinton, 2008) to project both the generated data and the original (o. a) pairs onto 2D space for both MPE and SMAC, as illustrated in Figure 7. Our analysis reveals that INS not only fills in parts of the data distribution that are not well explored by the real interaction but also augments the empty spaces around the existing transition distribution. In other words, it demonstrates an interpolation effect, potentially filling gaps in the original dataset distribution.
A Attention Weights. To elucidate the impact of our attention mechanism on agent interactions, we visualize the attention weights during the synthesis process across multiple multi-agent scenarios. Specifically. we use the real transitions as input for the final diffusion step in INS, then synthesize transition using sparse and dense attention mechanisms, visualizing their respective attention weights. Additionally, we compute the inverse distance 1/d(i.j) between each agent pair (i.j) to reflect the true interaction relationships between agents. As shown Figure 8, sparse attention effectively focuses on agents that are more likely to interact based on their proximity. In contrast, dense attention distributes focus more broadly, disregarding agents' relative distances, which introduces irrelevant interactions and reduces the fidelity of the synthetic dataset. Overall, sparse attention offers two key advantages: lower computational cost during synthesis and less distraction from irrelevant agents. These advantages contribute to more efficient and accurate multi-agent interaction synthesis.

---

Published as a conference paper at ICLR 2025
Spread
5m, vs 6m
8m
Spread (MPE)
(SMAC)
Distance
Sparse
Dense
Figure 7: Data coverage visualization. Visualization of a t-SNE projection on the observation-action space of the original dataset and INS synthetic dataset.
Figure 8: Attention visualization. Vi- sualization of attention weights in the synthesis process.
Oracle Reward and Value Distribution. To further evaluate the impact of our method on the synthetic dataset, we visualize the oracle reward distribution (in Appendix G.1) and the actual value distribution (in Appendix G.2) of the original dataset and synthetic datasets. The results indicate that the select mechanism introduced in INS's Phase guides the synthetic dataset toward regions with higher rewards, effectively improving the actual value distribution of the offline MARL policy.
5.6 SYNTHESIZE FROM SMALL DATASETS
We evaluate the ability of INS to synthe- size transitions from limited-size datasets by subsampling each dataset in proportion to its original size. Specifically, we select (10%, 50%, 100%) ) of the original dataset as training data for INS, and then assess the pol- icy performance thetic data. The denoising network uses the same hyperparameters as in the main evalu- ation. The experimental results in Table show that our method can synthesize effective transitions even when using only 10% of the original data, with the performance comparable to that of the original dataset. These results validate the capability INS to synthesize high-quality datasets from small-size multi-agent datasets, taking a step towards addressing the data scarcity in real-world offline MARL tasks.
Table 2: Synthesize from small datasets. We only
present two expert/good dataset results for brevity, with additional results provided in Appendix G.4.
Dataset
Original
10%
50%
100%
the corresponding syn-
Spread
102.8
102.3
106.3
107.0
5m vs 6m
16.3
16.3
16.8
17.2
6 CONCLUSION
In this paper, we propose INteraction-awant Synthesis (INS), a method that leverages diffusion models synthesize high-quality datasets for offline MARL. INS employs sparse attention mechanism for inter-agent interactions and selects transitions based value estimator. The method is orthogonal to existing offline MARL algorithms and adaptable datasets with both discrete and continuous action spaces. Our experiments demonstrate that datasets synthesized by INS lead to superior policy performance and improved dataset metrics compared to the original datasets and those synthesized by baseline methods. Additionally, INS can synthesize datasets using little 10% of the original data while maintaining comparable policy performance. INS enables MARL agents to benefit from synthetic datasets, taking step towards effective training scarce real-world datasets.
Limitations. Real-world multi-agent systems often involve variable number of agents, making dataset synthesis for varying agent numbers promising research direction. Additionally, given the high computational demands of diffusion models, acceleration techniques will be employed to scale our method to large-scale multi-agent systems. Finally, to balance diversity and fidelity, we did not apply guidance in INS. However, applying guided diffusion models to specific domains (c.g.. safety MARL) presents an interesting future direction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant XDA27030400, in part by the National Natural Science Foundation of China under Grant 62293541 and Grant 621 36008, in part by Beijing Natural Science Foundation under Grant 4232056, and in part by Beijing Nova Program under Grant 20240484514.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020. 1
Paul Barde, Jakob Foerster, Derek Nowrouzezahrai, and Amy Zhang. A model-based solution to the offline multi-agent reinforcement learning coordination problem. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 141-150, 2024. 18
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: A review of challenges and applications. Applied Sciences, 11(11)):948, 2021.
Jiajun Chai, Yuqian Fu, Dongbin Zhao, and Yuanheng Zhu. Aligning credit for multi-agent co- operation via model-based counterfactual imagination. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 281-289, 2024. 3
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog Bits: Generating discrete data using diffusion models with self-conditioning. In International Conference on Learning Representations, 2023. 5
Daesol Cho, Dongseok Shim, and Jin Kim. S2P: State-conditioned image synthesis for data
augmentation in offline reinforcement learning. Advances in Neural Information Processing Systems,
2022. 17
Nicholas Corrado and Josiah P. Hanna. Understanding when dynamics-invarian data augmentations benefit model-free reinforcement learning updates. In International Conference on Learning Representations, 2024. 2
Alex DeWeese and Guannan Qu. Locally interdependent multi-agent MDP: Theoretical framework for decentralized agents with dynamic dependencies. In International Conference Machine Learning, 2024. 2
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yi Ma, Pengyi Li, and Yan Zheng. CleanDiffuser: An easy-to-use modularized library for diffusion models in decision making. arXiv preprint arXiv:2406.09509, 2024. 4, 17
Edgar C Fieller, Herman O Hartley, and Egon S Pearson. Tests for rank correlation coefficients. i.
Biometrika, 44(3/4):470-881, 1957. 19
Claude Formanck, Asad Jeewa, Jonathan Shock, and Amu Pretorius. Off-the-Gii Grid--rrd MARL: Datasets with baselines for offline multi-agent reinforcement learning. arXiv preprint arXiv:2302.00521, 2023. 23
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020. 2. 24
Yuqian Fu, Yuanheng Zhu, Jiajun Chai, and Dongbin Zhao. LDR: Learning discrete representa- tion to improve noise robustness in multiagent tasks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 55(1):513-55 2025.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062, PMLR, 2019.
Yinfeng Gao, Qichao Zhang. Da-Wes Ding, and Dongbin Zhao. Dream to drive with predictive individual world model. IEEE Transactions Intelligent Vehicles, pp. 1-16, 2024.
11

---

Published as a conference paper at ICLR 2025
Michael Georgeff. Communication and interaction in multi-agent planning. In Readings in Distributed Artificial Intelligence, pp. 200-204. Elsevier, 1988.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10122, 2018. 18
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue-
long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in Neural Information Processing Systems, 36:64896-644 64917, 2024. 2, 17
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? International Conference on Learning Representations, 2023. 9
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840-6851, 2020. 2, 3, 17
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022. 3
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577 2022. 3, 17, 25
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-11991, 2020. 7, 23
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884-19895, 2020. 16
Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. GTA: Generative trajectory augmentation with guidance for offline reinforcement learning. arXiv preprint ar arXiv:2405 16907, 2024. 8
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020. 3
Haoran Li, Guangzheng Hu, Shasha Liu, Mingjun Ma, Yaran Chen, and Dongbin Zhao. NeuronsGym: A hybrid framework and benchmark for robot navigation with sim2real policy learning. IEEE Transactions on Emerging Topics in Computational Intelligence, pp. 1-15, 2024a.
Haoran Li, Yaocheng Zhang, Haowei Wen, Yuanheng Zhu, and Dongbin Zhao. Stabilizing diffussio model for robotic control with dynamic programming and transition feasibility. IEEE Transactions on Artificial Intelligence, 5(9):4585 4594, 2024b. 17
Wenhao Li, Bo Jin, and Xiangfeng Wang. SparseMAAC: Sparse attention for multi-agent reinforce- ment learning. In Database Systems for Advanced Applications: DASFAA 2019 International Workshops: BDMS, BDQM, and GDMA, Chiang Mai, Thailand, April 22-25, 2019, Proceedings 24, pp. 96-110. Springer, 2019. 4, 18
Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning, arXiv preprint arXiv:2307.01472, 2023. 3
Shunyu Liu, Jie Song, Yihe Zhou, Na Yu. Kaixuan Chen, Zunlei Feng, and Mingli Song. Interac- tion pattern disentangling for multi-agent reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-15, 2024. 4,
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-onn cooppetive-oomptttiiie environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6382-6393, 2017. 2, 6, 22
Cong Lu, Philip Ball, Yee whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36:46323 46344,, 2024, 2, 7, 8, 9, 16, 17, 23, 25
12

---

Published as a conference paper at ICLR 2025
Chengdong Ma, Aming Li, Yali Du, Hao Dong, and Yaodong Yang. Efficient and scalable rein- forcement learning for scage-scale network control. Nature Machine Intelligence, 9 2024. ISSN 2522-5839.
Andre Martins and Ramon Astudillo. From Softmax to Spaasemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pp. 1614-1623. PMLR, 2016. 5,5
Frank J Massey Jr. The kolmogorov-smirnov test for goodness of fit. Journal of the American Statistical Association, 46(253):68-78, 1951. 19
Frans A Oliehock, Christopher Amato, et al. A concise introduction to decentralized POMDPs,
volume 1. Springer, 1st edition, 2016.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi- agent reinforcement learning with actor rectification, In International Conference on Machine Learning, pp. 17221-17233 PMLR, 2022. 3, 7, 22, 23,
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Proceedings of the 20th International Conference on Neural Information Processing Systems, pp. 1177-1184, 2007. 25
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-asiisted Intervention, pp. 241. Springer, 2015. 4
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The Star- Craft multi-agent challenge. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188, 2019. 2, 5, 6, 23
Kajetan Schweighofer, Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-zadeh, and Sepp Hochreiter. A dataset perspective on offline reinforcement learning. In Conference on Lifelong Learning Agents, pp. 470-517. PMLR, 2022. 6
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pp. 216-224. Elsevier, 1990. 18
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991. 18
Zhentao Tang, Yuanheng Zhu, Dongbin Zhao, and Simon M. Lucas. Enhanced rolling horizon evolution algorithm with opponent model learning: Results for the fighting game Al competition. IEEE Transactions Games, 15(1):5-15, 2023. 2
Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang. Learning from good trajectories in offline multi- agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11672-1 11680, 2023. 3
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lueas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mix-Mixer: An alI-MLP architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272, 2021. 16, 23
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 986))2579-2665. 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 6000-6010, 2017. 4
13

---

Published as a conference paper at ICLR 2025
Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Bootstrapped Transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 15:34748-347611 2022.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023. 3
Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 11Xiv:1911. 1361, 2019. 7, 23
Yigin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang. Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021. 3, 7, 23
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14411 14142, 2020. 18
Ruiqi Zhang, Jing Hou, Florian Walter, Shangding Gu, Jiayi Guan, Florian Röhrbein, Yali Du, Panpan Cai, Guang Chen, and Alois Knoll. Multi-agent reinforcement learning for autonomous driving: A survey. arXiv preprint arXiv:2408.099 09675, 2024.
Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. MADiff: Offline multi-agent learning with diffusion models. arXiv preprint arXiv:2305.17330 2023a. 3, 17
Zhenghang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint 2xXi:2311.00222 2023b. 3, 17
14

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Motivating Example: Fireboy and Watergirl
B Details of the Preliminary Study
16
C Pseudocode of Interaction-aware Synthesis
17
D Main Differences between INS and Related Works
17
E Comparison of the Sparsemax in Related Works
18
F
Dataset Quality Metries
F.
Similarity
19
F2
Correlation
F3
Oracle Reward
19
R4
Dynamic MSE
F.5
Novelty
19
G Additional Experimental Results
20
G.1 Oracle Reward Distribution Visualization
20
G.2 Actual Value Distribution Visualization
................................
20
G.3 Module Ablation
21
G.4 Synthesize from Small Datasets
21
G.5 Computational Efficiency of Dataset Synthesis
22
H Information on Offline MARL Datasets
22
H.1 MPE Datasets
22
H.2 SMAC Datasets
23
Offline MARL Algorithm Implementation
23
J MA-SynthER Implementation
23
Implementation Details INS
24
K.1 Detailed Network Architecture
24
K.2 Score Normalization
24
K.3 Hyperparameters...
25
Computational Resources
25
15

---

Published as a conference paper at ICLR 2025
A
MOTIVATING EXAMPLE: FIREBOY AND WATERGIRL
Single-agent Synthesis
Multi-agent Synthesis
2
3
(1 - 2 P
(1-2-3-4-5]
1
8
7 6
(5 + 4 3
6-6-77--11
Figure A1: Motivating example, Fireboy and watergirl are situated in a grid world where they must collaborate retrieve the diamonds located behind each other. In this environment, fireboy and watergirl cannot occupy the same grid cell, requiring them to follow different paths to reach their respective diamonds.
In this section, we present a simple motivating example to illustrate the importance of inter-agent interactions during data synthesis. As shown in Figure A1, two agents must select different paths to reach opposite ends of the grid, as they cannot occupy the same grid cell. The original dataset contains transitions where each agent follows either the upper or lower path. However, applying single-agent data synthesis methods independently, without considering inter-agent interactions, leads the scenario depicted in Figure A1 (mid), where both agents (fireboy and watergirl) are synthesized to follow the same upper path. This transition is infeasible in the real environment, resulting in distorted synthetic dataset. By considering inter-agent interactions, multi-agent data synthesis can effectively avoid this issue and synthesize high-quality datasets, as shown in Figure A1 (right).
B DETAILS OF THE PRELIMINARY STUDY
In this section, we provide more details of the experimental setting of our preliminary study, as illustrated Figure! 1. We compare our method against three baseline approaches: Additive Noise, VAE Augmented, and MA-SynthER, The Additive Noisc method (Laskin al., 2020) augments the original dataset by introducing Gaussian noise. The VAE Augmented approach employs variational autoencoder to learn from the original dataset and subsequently generate new data through sampling. MA-SynthER, our multi-agent implementation SynthER (Lu et al., 2024) utilizes a diffusion model based on the MLP-Mixer architecture (Tolstikhin et al., 2021). These models synthesize transitions for each agent independently, without considering interactions between multiple agents. To adapt the Additive Noise, VAE Augmented method, and MA-SynthER for discrete action environments, incorporated our proposed bit action module into these methods.
Specifically, Figure (left) compares the performance of MA-ICQ using various purely synthetic datasets in MPE and SMAC environments. The dashed line indicates the policy performance trained on the original dataset. Our method demonstrates superior performance compared both the original dataset and the baseline synthetic dataset, indicating the efficacy of INS synthetic dataset in enhancing downstream policy training. Additive Noise and VAE Augmented methods yield results comparable to or even worse than the original dataset. MA- MA-SynthERR shows modest improvement over the original dataset but underperforms relative to our proposed method, potentially due to its limited considcration of inter-agent interactions and dataset quality enhancement. Figure 1 (right) presents comparison of various dataset metrics across different synthetic datasets. INS demonstrates consistently high performance across all dataset metrics, suggesting the effectiveness of our method in synthesizing high-quality synthetic datasets.
16

---

Published as a conference paper at ICLR 2025
C PSEUDOCODE OF INTERACTION-AWARE SYNTHESIS
We describe the training and synthesis processes of INS, as shown in Algorithm 1.
Algorithm 1 Interaction-awarc Synthesis
Input: Offline MARL dataset Dori. number of candidates B and selection proportion y)
Output: Synthetic dataset Payn. and trained joint policy π
Diffusion model training process
1: Transform Doni. into (|Pori.l, dz)) shaped tensor
2: Update diffusion model Pam (x) and state value estimator (o with samples from Dori. using
Eq. (2) and Eq. (8), respectively
I/ Dataset synthesis process
3: Initialize synthetic dataset Dayn. = 0 and transition candidate set Dean. Ø
4: for i ,,...,,...... B do
5: Sample the i-th new joint transition x² = (o' a', o" rr) using diffusion model pam (x)
6: Add transitions into the candidate set Deaa..
7: Calculate the value of each candidate transition using state value estimator (o()')
8: end for
9: Select proportion 17 of the candidate set by their values
10: Add all transitions of selected candidates to Dayn.
// Policy training process
11: Initialize and train the offline MARL policy using Dayu.
return Dayn. and π
D MAIN DIFFERENCES BETWEEN INS AND RELATED WORKS
Diffussion Models Offline MARL. Diffusion models (Ho et al., 2020; Karras et al., 2022), as powerful class of generative models, have recently been adopted in reinforcement learning as planners and policies (Zhu et al., 2023b; Li et al., 2024b). In this section, we discuss related works, MTDiff-S (He et al., 2024), SynthER (Lu al., 2024) and MADiff (Zhu et al., 2023a), clarify the differences between our approach and existing methods. Motivationally, both MTDiff-S and SynthER are designed for single-agent settings. MTDiff-S generates data at the trajectory level for unknown tasks, focusing on multi-task scenarios, which is why it is not included as baseline in our work. SynthER, on the other hand, generates data at the transition level and is applicable both online and offline reinforcement learning. Unlike these methods, INS is specifically designed for multi-agent environments, incorporating inter-agent interactions during synthesis to produce enhanced datasets for improving offline MARL performance. Technically, MTDiff-S adopts Transformer architecture, treating trajectory generation as a sequence modeling problem, while SynthER employs an MLP- Mixer architecture. In contrast, INS uses Sparse Transformer Encoder architecture, treating each agent as an individual token input. Additionally, neither MTDiff-S nor SynthER can handle discrete action spaces, whereas INS achieves this compatibility action spaces through bit action. MADiff an offline MARL method, which models a planner as a return-conditional diffusion model to maximize the cumulative reward. At each time step, MADiff generates trajectory of length H 1 based the agent's 's current or historical observations and then uses only the observations (or, 0z+1) along with an inverse dynamics model to determine the action at.
Theoretically, data synthesis in reinforcement learning often refers to the ability to generate data that can used to train policies, rather than directly engaging in decision-making (as a planner or policy) (Dong et al., 2024; Cho et al., 2022). In two overviews diffusion models for RL (Dong et al., 2024; Zhu et al., 2023b), MADiff is consistently categorized as a planner (notably, the first author of Zhu et al. (2023b) is also the first author of MADiff). In contrast, INS and SynthER generate synthetic data without maximizing rewards or selecting actions, and should therefore be categorized under Data Synthesizers. Thus, we believe that MADiff and method
17

---

Published as a conference paper at ICLR 2025
Table Al: A comparison of INS and the related works.
Key
Input
Outgut
Requires
Requires Policy
Space
SynthER
Tatn Symthoxirer (singlo-agen))
None
fo ty olrr)
No
No
Continoou
MIDDUI 5
Muhii tok Data Synthesizer (single agent)
Ae
[oces) Predess faed fred -o
Yes
No
Continoous
MADIT
Planser
(orsc)
Yes
No
None
MOMA-PPO
Model-troe MARL (Dyus)
Yes
Yes
None
INS (vurs)
Daca Synthesizer
None
[0.0,0l/r]
No
No
Contimon/Disereee
fundamentally different. Furthermore, MADiff generates observation sequences based on the agent's 's current state and only uses the next observation (ot, 0t+1) to produce an action, meaning it cannot generate dataset from scratch that can be used for policy training. In contrast, method requires no input and can generate a complete dataset of transitions {(ot, Gt, rz)} suitable for subsequent offline multi-agent reinforcement learning.
Technically, while both MADiff and our approach leverage the generative capabilities of diffusion models, we believe this similarity may cause confusion. In terms of the learned targets, MADiff trains observation sequences or, while INS learns from transitions (o,, at, 0,+1; When it comes to the generated output, MADiff produces observation sequences for inverse decision- making, while INS generates multiple transitions (0t, dty ; Te) that form dataset. In terms of training, MADiff uses classifier-fres guidance, requiring the agent's current observation and team rewards as imputs, whereas our method needs no input and can generate qulti--gent datasets from scratch.
Model-based MARL, Model-based multi-agent reinforcement learning involves interacting with a predictive model of the environment, often referred to world model (Ha & Schmidhuber, 2018), to facilitate policy learning (Sutton, 1991). I). One class of these methods relevant to INS optimizes policy by unrolling trajectories "in the imagination" of the world model (Yu et al., 2020; Barde et al., 2024). They employ the world model as a simulator, which takes the current observation and action as input to predict the next observation and reward. We note that our method shares some similarities with model-based MARL, particularly with Dyna-like algorithms (Sutton, 1990), which involve unrolling trajectories "in the imagination" " of the world model optimize a policy. In contrast, INS is a data synthesis method that generates new experiences without starting from a real state or the current policy, and the generated experiences follow the distribution of the dataset. Moreover, INS is an orthogonal method that can be combined with forward dynamics models by generating initial states via diffusion, potentially enhancing diversity. summary of the differences between INS and the related works mentioned above is provided in Table Al.
E COMPARISON OF THE SPARSEMAX IN RELATED WORKS
In OPT (Liu et al., 2024) and SparseMAAC (Li et al., 2019), sparsemax is used to extract relationships between the real observations of multiple agents, enabling agents to focus on specific targets during inference, which helps in effective Decision-making. Therefore, in these works, the input to sparsemax is the encoding of agent observations or observation-action pairs, and the output is used for subsequent action generation or Q-value computation. In contrast, our work a generative task, where sparsemax is applied during the denoising process of transitions (which are initially sampled entirely from noise). During training. sparsemax learns the interaction patterns present in the dataset, and during synthesis, it progressively incorporates agent interactions into the noise, rather than extracting pre-existing relationships. Therefore, we believe the role of sparsemax INS fundamentally different from its role in the aforementioned works.
F DATASET QUALITY METRICS
In this section, we define the metrics employed to assess the quality of the synthetic dataset. Our evaluation framework consists of five distinct metrics: Similarity, Correlation, Oracle Reward, Dynamic Mean Squared Error (MSE), and Novelty. The Similarity and Correlation metrics quantify the statistical fidelity and relational consistency between the synthetic and original datasets, while the Oracle Reward metric evaluates the true reward magnitudes associated with the synthetic data.
18

---

Published as a conference paper at ICLR 2025
The Dynamic MSE metric assesses the fidelity of the synthetic data to the environment dynamics, quantifying how accurately represents the transitions. The Novelty metric quantifies the uniqueness and diversity of the synthetic dataset, providing insights into its coverage of the observation- space. All metrics are computed from 200K samples from each synthetic dataset. The formal mathematical definitions and computational procedures of these metries provided below.
F.1 SIMILARITY
This metric means Kolmogorov-Sminne (KS) distance (Massey Jr, 1951), which measures the maximum distance between the cumulative distribution functions of the original and synthetic datasets. The Similarity is defined as:
Similarity Daya.) = ssup Frti. (x) - Fayu. (z))
(A1)
where Fori. (x)) and Fsya.()) (x) are the cumulative distribution functions of the original and synthetic datasets, respectively.
F.2 CORRELATION
This metric is pairwise rank Pearson correlation (Fieller et al., 1957), which measures the statistical relationship between the original and synthetic datasets. The Correlation is defined as:
Σ (rankori. (xn - Fori.) (ranksyn. (æ.) Fayn.)
Correlation(Por.. Dayn.)
(A2)
Σ (rankori. (ax) Fori.) Σ (ranksyn. (xt:) Peyn.)
where rank(x;;) is the rank of joint transition 2;, and F is the mean rank of the dataset.
F.3 ORACLE REWARD
This metric the average true reward of the synthetic dataset, which measures the optimality of synthetic data. The Oracle Reward is defined as:
(racleReward(Degn...
=
[
Ttrue:
(A3)
syn.
DEDD.n.
where the true reward Turue obtained by querying the environment with the generated joint observa- tions and actions (o, a).
F.4 DYNAMIC MSE
This metric the mean squared error (MSE) between the predicted next observation and the true next observation, which evaluates how well synthetic data fits the environment dynamics. Prior to error computation, we normalize each observation. The Dynamic MSE is defined as:
DynamicMSE(Dsga...
1
Σ Dann.
(P(o.a)-o'}?
(A4)
[Daym
where P(o, a) is the true next state by the environment dynamics.
F.5 NOVELTY
This metric the average distance between synthetic data and the closest original dataset, which quantifies the uniqueness of datasets. The Novelty is defined as:
Novelty(Dsgn. Dori.) [Dayal
Σ
min DeDon.
((o,a)
(A5)
RED
where z is the nearest neighbor of in the original dataset.
19

---

Published as a conference paper at ICLR 2025
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 ORACLE REWARD DISTRIBUTION VISUALIZATION
We visualize the reward of the original dataset and real reward computed from the environment using observation and action generated from MA-SynthER and INS to visualize that INS can improve the data distribution to high reward region while preserving environmental dynamics. Figure A2 presents these distributions. The results demonstrate that our method effectively enhances the proportion of high-reward transitions in the dataset while simultancously broadening the overall range of rewards compared the original dataset. This dual improvement is particularly beneficial for subsequent policy learning, as it provides a richer and more diverse reward landscape for agents to learn from.
Original
MA-SyuthER
Our Methods
Prohability Denstry
Probability Deasity
Orack
Spread-Expeee
(b) Spread- Meerad
Figure A2: Oracle reward distribution visualization. Comparison of the oracle reward distributions among original, MA-S MA-SynthER, and INS datasets.
G.2 ACTUAL VALUE DISTRIBUTION VISUALIZATION
To further evaluate the impact of our method on dataset quality, we visualize the actual value distributions of the original, MA-SynthER, and INS datasets, shown in Figure A3. We first train policy on the dataset and then obtain the actual value for the trained policy by running it in the real environment. The results demonstrate that method effectively enhances the value distribution of transitions in the dataset under the policy, while the value distribution of MA-SynthER synthetic data exhibits a decrease compared the original dataset.
Original
MA-SyalkER
Methods
Probubiliry Denity
Probability Denity
Actual Value
Actual Value
(a) Spread-Expert
(b) Spread-
Figure A3: Actual value distribution visualization. Comparison of the actual value distributions among original, MA-SynthER, and INS datasets.
20

---

Published as a conference paper at ICLR 2025
G.3 MODULE ABLATION
We conduct more ablation studies on the components of INS across 14 datasets from 4 different scenarios. These include replacing the sparse attention mechanism in the Transformer Encoder with a dense attention mechanism (w/o sparse), removing the select mechanism retaining all synthetic data (w/o select), and replacing the attention module with an MLP, allowing each agent to independently synthesize transitions (w/o attention). In the w/o attention variant, concatenate the transitions of individual agents and replace the attention module in the Transformer Encoder with multiple MLPs, removing the dot-product and Sparsemax components. This modification results in each agent synthesizing transitions independently, disregarding interactions between agents. The results shown in the Figure A4. The results demonstrate that the atlention mechanism significantly enhances agents' ability to learn inter-agent interactions, improving synthetic data accuracy and policy performance. The select mechanism effectively screens high-value transitions, which is particularly crucial in complex environments like SMAC. Additionally. the sparse attention mechanism reduces redundant information in agent interactions, mitigating distraction impact.
Methods
w/o Sparse
w/o Seleet
Atteation
Normalized
Spread
Tag
3m
5m vs,
Figure A4: Module ablation on different datasets. The normalized average score the downstream
policy with INS ablation variants.
G.4 SYNTHESIZE FROM SMALL DATASETS
We evaluate the scalability our method across datasets from 4 different scenarios. The results show that, when using only 10% of the original dataset, our method often achieves performance comparable to that of the original dataset. With 50% of the dataset, the policy consistently outperforms the original dataset. The quality of the original dataset also impacts scalability. On expert-level datasets, where higher proportion of high-quality samples remain even after downscaling, our method demonstrates better scalability, particularly when using just 10% of the data.
Table A2: Synthesize transitions from a small dataset. We use four different scenarios for exploration, each scenario includes two different quality datasets.
Dataset
Original
10%
50%
100%
Expert
102.8
101.9
106.8
107.0
Spread
Medium
26.8
24.5
28.4
30.2
Expert
111.7
110.9
113.1
114.2
Tag
Medium
62.3
61.2
63.5
64.7
Good
18.1
18.0
19.3
19.9
3m
Medium
16.9
15.6
17.8
18.5
Good
16.3
16.3
16.8
17.2
5m vs _s_6m
Medium
12.8
11.9
12.9
13.9
21

---

Published as a conference paper at ICLR 2025
G.5 COMPUTATIONAL EFFICIENCY OF DATASET SYNTHESIS
To evaluate the computational efficiency of INS, we measured the time taken by INS to generate 1M transitions and compared with the time required collect 1M transitions through environment interactions using trained policies. The results in Table A3 show that INS is significantly more time-efficient than collecting data via direct environment interactions, while still maintaining high dataset quality. We believe this demonstrates that dataset synthesis is promising avenue for MARL, where generative methods like INS can be employed to synthesize multi-agent datasets, reducing the time and challenges associated with data collection in real-world environments.
Table A3: Computational efficiency of INS.
Scenario
INS
Collecting by agents
spread
1540s
2728s (MAPPO)
3m
2396s
9243s (QMIX)
H INFORMATION ON OFFLINE MARL DATASETS
:
(a) MPE
(b) SMAC
Figure A5: MPE and SMAC environment examples.
H.1 MPE DATASETS
Multi-Agent Particle Environment (MPE) (Lowe al., 2017) is multi-agent particle environment where particles can execute continuous observations and actions, as illustrated in Figure A5(a). Our study employed three distinct MPE scenarios. The Spread environment consists of three agents and three landmarks, where the agents' objective is to learn collision avoidance while ensuring complete landmark coverage. The Tag scenario involves one pre-trained prey, three predators, and two obstacles, requiring the predators to collaborate in capturing the prey. The World environment, similar to Tag, includes one pre-trained prey and three predators; however, the prey agent must locate food within the map and can utilize forest areas for hiddening.
For our MPE experiments, we used dataset! provided by OMAR (Pan et al., 2022). This compre- hensive dataset comprises multiple subsets of varying quality. The random datasets are generated using a randomly initialized policy for one million steps. The medium replay (md-replay) dataset is obtained by recording all samples in the buffer when training achieves medium performance level. The medium dataset was derived from a partially trained policy with medium performance, while the expert dataset was generated using fully trained policy. It is worth noting that OMAR utilizes an earlier version of MPF, which allows agents to potentially receive different rewards. However, we find that the different rewards for agents only occur infrequently and do not significantly impact the predominantly cooperative nature of the environments.
'htpps: thub. com/1 hitp//innnnannnnnenn pan/CMAR
22

---

Published as a conference paper at ICLR 2025
H.2 SMAC DATASETS
The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) is a widely recognized benchmark in the MARL field, shown in Figure AS(b). The SMAC environment simulates battle scenarios between two competing teams. One team is controlled by built-in artificial intelligence, while the other is managed by policies learned through MARL algorithms. Our experiments utilized three distinct maps: the 3m map, where both sides command three marines; the 8m map, featuring eight marines on each side; and the 5m_vs 6m map, where our team controls five marines against an enemy force of six.
For our SMAC experiments, we employ the off-the-gric dataset' (Formanek et al., 2023), which provides good, medium, and poor quality datasets for each map. These datasets were generated using three independently trained QMIX policies. To enhance behavioral diversity, a modest amount of exploration noise was introduced to the policies during data collection.
1
OFFLINE MARL ALGORITHM IMPLEMENTATION
This section details the implementation of various offline MARL algorithms.
MA-ICQ
(Yang et al., 2021) implicit constraint Q-learning to multi-agent settings, introducing a decomposed multi-agent joint-policy under implicit constraints. By relying solely on state-action pairs provided in the dataset, this approach effectively mitigates extrapolation errors.
MA-CQL (Kumar et al., 2020) adapts the conservative Q-learning framework to multi-agent environments. This algorithm addresses the overestimation problem by adjusting Q-values for both policy samples and dataset state-action pairs, resulting in more conservative and reliable policy updates.
(Wu et al., 2019), the multi-agent version of BCQ, implements decentralized multi- agent reinforcement learning using value deviation and transition normalization to achieve coordinated policies.
MA-BCQ'
OMAR (Pan et al., 2022) employs zeroth-order optimization in conjunction with multi-agent CQL, enhancing coordination among agents' policies. In implementing these offline MARL algorithms, adhere to the hyperparameters specified in their respective original papers to ensure consistency and reproducibility of results.
J MA-SYNTHER IMPLEMENTATION
We implement MA-SynthER, a multi-agent implementation of SynthER (Lu et al., 2024), which extends the synthetic data generation approach to multi-agent settings. MA-SynthER employs diffusion model based on the MLP-Mixer (Tolstikhin et al., 2021) architecture to generate transition- level data (observation, action, reward, and next observation) for multiple agents. This model synthesizes transitions for each agent independently. without considering interactions between multiple agents. We built MAA-SynthRR upon the code" provided by authors, adapting the multi- agent context. Our implementation maintains consistency with the original SynthER by adopting the same hyperparameters and training settings.
htttees//aattubb.com////ijj////a
huttpe://gttnbb.oom////nggg////
'https: htcrpee/gggthhooccooo//oo/ggg//9
23

---

Published as a conference paper at ICLR 2025
K IMPLEMENTATION DETAILS OF INS
K.1 DETAILED NETWORK ARCHITECTURE
In this subsection, detail the diffusion model architecture of INS, shown in Figure A6. The architecture begins with shared embedding layer that processes each agent's 's transition, denoted x; (oi, d,, r; o)). Subsequently, the embedded transition is projected into query, key, and value vectors, which serve imputs to a sparse attention mechanism. The output of the attention mechanism combined with the imput through a residual connection and normalized, after which is transformed by feedforward neural network with ReLU activation. The block's final representation obtained by applying layer normalization the feedforward network's output. The transformer encoder typically stacks this block N times enhance model depth. Our implementation utilizes a stack of 4 such blocks. To enhance the model's expressive capacity, we implement multi-head attention with 4 heads, extending the sparse attention mechanism. The diffusion model incorporates temporal information by integrating a step embedding vector k. This embedding is concatenated with the agent's 'ss transition before being fed into the network.
Extending to Synthesize Global States. For certain MARL algorithms that require global state s, extend the input of the diffusion model to generate the global state Specifically, instead of modeling joint local transitions with diffusion tuples of the form (o, a, o', r). we could model expanded tuples of the form (o, a, o' s, r), and on. This allows the method be applied value decomposition models that require global state.
for. dh
for oo.
Nx
Add Norm
Sparse Transformer Encoder
Feed Forward
k iterations
Add & Norm
Sparsemax
Dot Product
Scaled Dot Product
Embedding Laver
.C
Figure A6: The detailed diffusion model architecture of INS.
K.2 SCORE NORMALIZATION
Following Pan et al. (2022) and Fu et al. (2020), the average scores of MPE datasets in Table 1 are normalized by the expert and random scores on each task. Denote the original episodic return as S, then the normalized score Snorm is computed as
S Srandom
Snorm = 100 x Sexpert Srandom
(A6)
24

---

Published as a conference paper at ICLR 2025
where Srandom and Sexpert the average scores of the random and expert datasets, respectively. Besides, the score corresponds to random policy performance and 100 corresponds to an expert policy performance.
K.3 HYPERPARAMETERS
In this subsection, we first list the key hyperparameters of INS in Table A4. The noise level of the diffusion model encoded by Random Fourier Feature (RFF) (Rahimi & Recht, 2007) embedding.
Table A4: INS Hyperparameters.
Hyperparameter
Value
Selection proportion
0.8
Embedding dimension
Number of attention heads
4
Number of blocks
Dropout
0.1
Batch size
1024
Optimizer
Adam
Learning rate
2 x 10
Weight decay
10
Learning rate schedule
Cosine annealing warmup
RFF dimension
16
Training steps
1e6
For the diffusion sampling process. we use the SDE sampler of Karras et al. (2022) with the default hyperparameters used for SynthER (Lu et al., 2024), given in Table A5. In the implementation of downstream offline MARL algorithms, we use the hyperparameters specified in their respective original papers to ensure consistency and reproducibility of results.
Table A5: Diffusion Sampling Hyperparameters following EDM (Karras et al., 2022).
Hyperparameter
Value
Diffusion steps
128
min
0.002
mmax
Scham
80
Smin
0.05
Smax
50
S Spoise
1.003
K.4 COMPUTATIONAL RESOURCES
Most experiments are conducted on a server equipped with an Intel(R) Xcon(R) Gold 6442Y and two NVIDIA RTX A6000 GPUs. The training time of INS about hours for each task, the synthesizing time is about 1.5 ~ 3 hours, and the training time of offline MARL policies varies depending on the specific algorithm.
25


Reviews:
[{"review_id": "oF8udquSuJ", "reviewer": "Reviewer_HKfA", "summary": "The paper addresses data scarcity in offline multi-agent reinforcement learning (MARL), emphasizing the unique challenges in synthesizing high-quality multi-agent datasets due to complex inter-agent interactions. The authors propose Interaction-aware Synthesis (INS), a method that uses diffusion models with sparse attention to accurately model these interactions and a bit action module to support both discrete and continuous actions. INS also includes a selection mechanism to prioritize high-value transitions. Experiments in MPE and SMAC environments show that INS outperforms current methods, enhancing downstream policy performance and dataset quality. Remarkably, INS can synthesize effective data using only 10% of the original dataset, demonstrating its efficiency in data-scarce scenarios.", "questions": "- Is the proposed approach scalable in terms of a large number of agents?\n- It seems that the objective of the trajectory generation process only guarantees whether the generated trajectories are realistic or not. There is no mechanism to implicitly guide the diffusion model to enhance the quality of the trajectories (accumulated rewards of trajectories).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper was well-written. It's easy to understand the motivation, contributions, and methodology of the proposed work\n- Collecting datasets for multi-agent reinforcement is hard. Also, generating datasets in the multi-agent scenario is non-trivial due to the difficulty of modelling the interaction between the agents.\n- The experiments are complete and promising.", "weaknesses": "- In terms of novelty, the proposed solution looks like simply applying a diffusion model for generating trajectories with additional attention mechanisms across the agents, which is a lack of novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGtFZo3v7i", "reviewer": "Reviewer_jHSc", "summary": "This paper introduces INS, a method for synthesizing high-quality multi-agent datasets to enhance offline multi-agent reinforcement learning. INS leverages diffusion models and incorporates a sparse attention mechanism to capture inter-agent interactions, ensuring the synthetic dataset reflects the underlying agent dynamics. The method also includes a select mechanism to prioritize high-value transitions, improving dataset quality. The experimental results demonstrate the effectiveness of INS.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-organized.\n- Considering both dataset diversity and high-value transitions is valuable.", "weaknesses": "- On Line 087, the authors claim that \"INS is the first data synthesis approach for offline MARL.\" However, on Line 118, in the related works section, the authors mention that \"recent studies extend diffusion models to the MARL domain, applying them to trajectory generation [1].\" This statement suggests that the authors are not the first to engage in data synthesis work within the offline MARL domain. Furthermore, upon reviewing the relevant literature, the reviewer found additional papers that have also conducted data generation tasks in offline MARL [2]. Therefore, the reviewer believes that the authors have overclaimed their contribution. It is important to acknowledge the pioneering work in the field, and the authors should provide a more accurate representation of their contribution in relation to prior art.\n\n> [1] MADiff: Offline multi-agent learning with diffusion models.\n\n> [2] A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem.\n\n- The authors claim to introduce a sparse attention mechanism to capture inter-agent interactions, utilizing the method sparsemax, which was originally proposed in the NLP field. However, upon reviewing the literature, the reviewer found that sparsemax has already been applied in the MARL domain [3,4], reducing the novelty of this aspect of the work. Furthermore, the formulation of the sparsemax (equations 3-6) shows overlap with [4]. The authors should acknowledge the existing use of sparsemax in MARL and clarify how their implementation differs from or improves upon previous applications. A detailed comparison with related works would strengthen the paper's contribution and novelty.\n\n> [3] SparseMAAC: Sparse Attention for Multi-agent Reinforcement Learning.\n\n> [4] Interaction pattern disentangling for multi-agent reinforcement learning.\n\n- The authors propose the use of Bit Action as an alternative to one-hot action representation for action generation. However, the reviewer has a concern regarding the orthogonality of each action in the Bit Action representation. Non-orthogonal actions may lead to ambiguity in the action space, potentially affecting the outcomes.\n\n- The selection proportion parameter in the authors' method lacks flexibility and requires tuning for different tasks.\n\n- The synthetic method comparison is limited to SynthER. A more comprehensive comparison should be conducted, including both single-agent methods mentioned in the related works and multi-agent methods [1,2].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCzEEsSviW", "reviewer": "Reviewer_8UPd", "summary": "The paper makes a significant high-level contribution by extending data synthesis from single-agent reinforcement learning (RL) to multi-agent reinforcement learning (MARL), which is an important advancement. However, I have some concerns and suggestions.", "questions": "1.Clarity of Motivation: The detailed motivation for addressing the unique challenges of data synthesis in MARL seems somewhat unclear. While agent interactions are indeed a key issue, other challenges such as partial observability and environmental non-stationarity also play significant roles. I encourage the authors to delve deeper into these challenges and explain how their proposed method tackles them.\n\n2.Modeling Agent Interactions: Relying solely on an attention mechanism to model agent interactions may be overly simplistic. In the MARL domain, modeling agent relationships is a complex and standalone research topic. Would employing more sophisticated methods for agent relationship modeling enhance the model's effectiveness? \n\n3.Generation of Global States: I noticed that the generated data includes joint actions and joint observations but seemingly not the global state. How is this generated data applied to value decomposition models like QMIX that require global state information? \n\n4.Consistency with Environment Dynamics: How does the method ensure that the generated data conforms to the environment's dynamic model? If the generated state transitions are invalid, could this negatively impact the pre-training process? I recommend including theoretical analysis or experimental results to demonstrate the validity and effectiveness of the generated data.\n\n5.Computational Efficiency: Utilizing diffusion models in MARL might introduce computational efficiency challenges. Could the authors provide experimental data or analysis to illustrate the computational efficiency of the proposed method?\n\nOverall, the research shows potential, but addressing the above issues is essential to enhance the credibility and impact of its contributions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper represents the first attempt to apply data synthesis techniques to multi-agent reinforcement learning (MARL). The experiments are carefully designed and provide strong support for the proposed method. Additionally, the writing is clear and accessible, making the paper easy to follow.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2yDWAex2v", "reviewer": "Reviewer_PNEG", "summary": "In this paper, the authors propose INS, a novel multi-agent reinforcement learning (MARL) data synthesizer aimed at enhancing the performance of offline MARL algorithms. INS leverages a diffusion model to generate synthetic data and employs a sparse attention architecture and a value-based selection mechanism to improve the data quality. Compared to the naive extension of a single-agent diffusion-based data synthesizer to multi-agent tasks (MA-SynthER), INS demonstrates significant improvements.", "questions": "- See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly presented and easy to follow. The idea is simple yet effective.\n- The authors made several incremental improvements (Bit Action, Sparse Attention, Value-Based Selection) to progressively transform SynthER into INS. The motivation and methods for each improvement are convincing and predictably effective.\n- In the ablation studies, the authors use many figures and tables to illustrate the contribution of each improvement and explain the reasons behind these gains. This level of detail provides valuable insights for future research.", "weaknesses": "- In my view, the novelty is somewhat limited. INS is essentially an improved version of SynthER in the MARL context, and the ideas of sparse attention and value-based selection are neither difficult to conceive nor novel, as these methods have already been widely applied.\n- The experimental results are somewhat disappointing. First, many results in Table 1 do not show significant improvements over MA-SynthER. For example, in the SMAC-8m-good task, while the mean score improves by 0.1, the standard deviation is around 1.0, making the increase in the mean unconvincing. Second, SynthER has demonstrated its ability to significantly improve the sample efficiency of Online RL in the experiments. Including similar experiments for INS would help enrich the paper’s content and better showcase its capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)