Multi-agent Coordination via Flow Matching
Authors: Dongsu Lee
Daehee Lee
Amy Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-agent reinforcement learning
Flow matching
Primary area: reinforcement learning
TL;DR: Fast and expressive multi-agent cooperation via flow matching

Abstract: This work presents MAC-Flow, a simple yet expressive framework for multi-agent coordination. We argue that requirements of effective coordination are twofold: *(i)* a rich representation of the diverse joint behaviors present in offline data and *(ii)* the ability to act efficiently in real time. However, prior approaches often sacrifice one for the other, *i.e.*, denoising diffusion-based solutions capture complex coordination but are computationally slow, while Gaussian policy-based solutions are fast but brittle in handling multi-agent interaction. MAC-Flow addresses this trade-off by first learning a flow-based representation of joint behaviors, and then distilling it into decentralized one-step policies that preserve coordination while enabling fast execution. Across four different benchmarks, including $12$ environments and $34$ datasets, MAC-Flow alleviates the trade-off between performance and computational cost, specifically achieving about $\boldsymbol{\times14.5}$ faster inference compared to diffusion-based MARL methods, while maintaining good performance. At the same time, its inference speed is similar to that of prior Gaussian policy-based offline MARL methods.

Published as a conference paper at ICLR 2026
MULTI-AGENT COORDINATION VIA FLOW MATCHING
Dongsu Lee"
Daehee Lee²
Amy Zhang'
'Tee University of Texas at Austin SSungkyunkwan University dongsu. lee@utexas.
ABSTRACT
This work presents MAC-Flor a simple yet expressive framework for multi-agent coordination. We argue that requirements of effective coordination are twofold: (i) a rich representation of the diverse joint behaviors present in offline data and (ii) the ability to act efficiently real time. However, prior approaches often sacrifice one for the other, ie., denoising diffusion-based solutions capture complex coordination but are computationally slow, while Gaussian policy- solutions are fast but brittle in handling multi-agent interaction. MAC-Fl Flow addresses this trade-off by first learning a flow-based representation of joint behaviors, and then distilling it into decentralized one-step policies that preserve coordination while enabling fast execution. Across four different benchmarks, including 12 environments and 34 datasets, MAC-Flow alleviates the trade-off between performance and computational cost, specifically achieving about X 14.5 faster inference compared diffusion- MARL methods, while maintaining good performance. At the same time, its inference speed is similar to that of the prior Gaussian policy.
Code: https: /ttt::///ttttttt com http://onnngeecc///eoeoooaa mtt///accaaaee flow
TL;DR: MAC-Flow alleviates performance-inference time trade-off, achieving a 14.5x speedup compared to SOTA
Performance. Inference time. (Benchmarks: SMACv1 and SMACv2)
15
MAC-Flow
DoF
Performance
MACQL
MADiff
Diffusion BC
BC
Flow BC
10
ABCO
5
30
100
Inference time (ms)
Figure 1: Summary results. This summarizes performance vs. inference speed selected algorithms on widely-used MARL benchmarks. SMACv1 and SMACv2. We plot aggregate mean performance and inference time across 18 datasets for 8 scenarios related to the SMAC maps. More precisely, we measure inference time based on the total computation performed each algorithm and report it using milliseconds (ms) unit and log As result, our proposed solution, MAC-F cw,
scale, where higher value indicates greater computational achieves x 14..5 faster inference speed on average with comparable performance compared to previous SOTA.
INTRODUCTION
Multi-agent reinforcement learning (MARL) has been actively studied to handle real-world problems in multi-agent systems, such modular robot control (Zhang et al., 2024; Peng et al., 2021), multi- player strategy games (Carroll et al., 2019; Samvelyan et al., 2019), and autonomous driving (Lee et al., 2024a; Vinitsky al., 2022). However, because seminal MARL approaches heavily rely extensive online interactions among agents, their applicability to real-world domains is severely limited. Moreover, training from scratch is both expensive and risky in data collection and computation (Levine

---

Published as a conference paper at ICLR 2026
et al., 2020). These challenges motivate methods that import the safety and efficiency benefits of offline reinforcement learning (RL) into MARL (Wang et al., 2021).
Offline RL learns decision-making policies from fixed datasets, thereby avoiding expensive and risky environment interactions by optimizing returns while staying close to the dataset's 's state-action distribution (Bhargava al., 2024; Kumar et al., 2020; Wu et al., 2019). However, the transition from a single-agent to a multi-agent system is not trivial, introducing a unique set of formidable challenges. For example, the joint action space grows exponentially with the number of agents, making it difficult to learn coordinated behaviors that adapt to diverse interaction patterns (Li et al., 2025c; Liu et al., 2025; Barde al., 2024). Despite several efforts to address these problems (Yang al., 2021; Pan et al., 2022; Wang et al., 2023), they still struggle with the complex multi- molti-modality of joint action distributions, as Gaussian policies are prone failure by generating out-of -fitt---iittiuttion coordination.
To tackle this challenge, seminal works have turned to powerful generative models based on denoising diffusion in MARL (Zhu al., 2024; Li et al., 2025a). While diffusion policies capture complex multi-modal distributions, directly optimizing them with respect to learned value function remains non-trivial. Backpropagating value gradients through a multi-step denoising chain is expensive and unstable (Park et al., 2025; Ma et al., 2025; Yang et al., 2025), so several previous RL solutions employ diffusion for offline imitation or distillation based solution (Ding & Jin, 2023; Chen et al., 2024b). Furthermore, their reliance an iterative denoising process, which requires numerous neural function evaluations per forward pass, makes them computationally expensive at inference time. This latency precludes its practicality in scenarios that require time-critical decision-making.
At its core, the question motivating this study can be phrased as follows:
How should offline MARL algorithms balance the trade-off between expressiveness for multi-agent distributions and computational efficiency?
Building on this question, this work introduces a novel offline MARL framework, dubbed MAC-Flow. that bridges the gap between expressive generative policies (simply coordination performance) and computational demands. Our main idea is to first learn a flow-based joint policy through behavioral cloning, which captures the complex multi-modal distribution of joint actions in offline multi-agent datasets. For decentralized exccution, we factorize the flow-based joint policy into a one-step sampling policy by optimizing two objectives: (i) distilling the flow-based joint action distribution and (ii) maximizing the global value function under the indiidual-globbl global-max -max (IGM) principle (Son et al., 2019). By decoupling expressiveness from value maximization, MAC-Flow avoids the instability of backpropagation through time (Park et al., 2025; Wagenmaker et al., 2025).
Our contributions are summarized follows.
We propose MAC-F FFoo,, a novel MARL algorithm that enables scalable multi-agent systems by alleviating the trade-off between coordination performance and inference speed.
We introduce flow-based joint policy factorization method that decouples expressiveness from optimization, decomposing a bow-based joint policy into efficient one-step sampling policies while jointly optimizing RL and imitation objectives with mathematical guarantees.
We demonstrate the efficiency of MAC-F Flow across four widely used MARL benchmarks, 12 environments, and 34 datasets, encompassing diverse characterizations, especially in 14.5x speedup compared to previous diffusion- solutions in SMAC benchmarks.
2 RELATED WORK
Offline MARL. The goal of offline RL is to extract policy using static operational logs, without active interaction with the environment. In multi-agent settings, inter-agent dependencies amplify distribution shift. Even slight deviations in single agent's policy can trigger cascading mismatches in joint behavioral patterns, complicating efforts to maximize returns while preserving alignment with the dataset's, joint state-action distribution. Naive approaches typically involve extending single-agent offline RL methods to multi-agent contexts (Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto & Gu, 2021). Yet. these extensions often fall short in incorporating global regularization to facilitate cooperation. More sophisticated methods, tailored for mulli--gent settings, tackle coordination issues via value decomposition, policy factorization, and model-based optimization (Yang et al., 2021; Pan
2

---

Published as a conference paper at ICLR 2026
et al., 2022; Wang al., 2023; Barde et al., 2024; Liu et al., 2025). Despite these advances, such techniques struggle to capture the multi-modal distributions inherent in multi- offline datasets, resulting in imprecise credit assignment. To better capture joint action distribution, generative modeling has gained traction, for example, diffusion-based trajectory modeling (Zeng et al., 2025) (Li et al., 2025a; Zhu et al., 2024), cittusiionhaseed policy (Qiao et al., 2025; Li et al., 2023), and transformer-based modeling (Wen et al., 2022; Tseng et al., 2022; Meng et al., 2023b). However, they frequently require substantial computational resources and may struggle to learn optimal patterns, owing to their reliance on behavioral cloning (BC) paradigms. This work introduces a novel MARL algorithm that increases efficiency and takes advantage of both RL and generative modeling.
Diffusion and Flow Matching in RL. Drawing on the robust capacity of iterative generative modeling frameworks, such as denoising diffusion processes and flow matching techniques, recent efforts have explored the diverse ways to employ them for enhancing RL policies. Examples of RI. with an iterative generative model include world modeling (Rigter al., 2023; Ding et al., 2024; Alonso et al., 2024), trajectory planning (Janner et al., 2022; Ajay et al., 2022; Zheng al., 2023; Liang et al., 2023; Ni et al., 2023; Chen et al., 2024a; Lu et al., 2025), policy modeling (Alles et al., 2025; Zhang et al., 2025b; Chi et al., 2023), data augmentation (Lu et al., 2023; Huang et al., 2024; Lee et al., 2024b; Yang & Wang, 2025), policy steering (Wagenmaker et al., 2025). and exploration (Ren et al., 2024; Liu et al., 2024; Li et al., 2025b). Such iterative generative models show powerful performance, but their inference is prohibitively slow for real-world deployment.
Our approach extracts an expressive flow policy to capture the multi-modal distribution of mixed behavioral policies from offline multi-agent datasets. Algorithmically, this is motivated by flow distillation (Frans et al., 2024; Chen et al., 2025) and flow Q-learning (FQL) (Park et al., 2025), which distills a one-step policy with an RL objective to model complex action distributions via flow matching in single-agent RL. Instead, we leverage a flow matching- baacchin--based joint policy that explicitly models the joint action distribution across agents and pioneering the integration of the IGM principle (Son et al., 2019) with flow matching in MARL,, ensuring individual policies align with the global optimal joint policy for enhanced coordination and scalability.
3 BACKGROUND
Problem Formulation. This work posits the MARL problems under a decentralized partially observable Markov decision process (OecP) (Bernstein et al., 2002) M defined by tuple SI, A;, TT, 74 ri, 7), where I = {1, 2, ... global state space; O, and A, correspond to the observation and action spaces specific to agent i, respectively. The state transition dynamics are captured by T(s'ls a) S x Ai X .... Ar + S, where a is a joint action [a1, ....... az] and we color the gray to denote placeholder variables. Next, (,(o.(s): S + O₂ specifies the observation function for agent i. Each agent receives individual rewards according to its reward function re(8, dg a-:): : S x A X ... x Ar + R. The goal of offline MARL under cooperative setups is learn of policies IT {m,, m2, . that jointly
I} denotes set of agents. Here, S represents the
[s'|s,
maximize the discounted cumulative reward (r- Σl "1 ,r,, af. a..)) from an offline multi-agent dataset D (T(1)) (T(()) [1e,,,..... without environment interactions, where ? e [0, 1) is a discounted factor, T denotes a joint trajectory {TI, 72, ,T TÜ = (of, a.. ........ off aff). and p" (T) represents probability distribution over joint trajectories induced by set of policies II.
Individual- -Global--ooo Principle. The IGM principle serves as a foundational approach in MARL, offering a method to ensure globally consistent action selection through factorized Q-value functions for each agent (Son et al., 2019; Rashid et al., 2020). By aligning individual agent policies with a shared objective, IGM simplifies the complexity of joint action spaces, making it scalable solution for multi-agent systems. This principle can be mathematically defined as follows.
arg max
Qio a) arg max Qi (1),,, arg. max (0)) d1
as
Here, Qo a) represents the global Q function, while Qi(o;, a.) denotes an individual Q function for agent The IGM ensures that optimizing its local Q, remains consistent with the global optimum.
rehaaiizra Offline RL. Behavioral regularization (Wu et al., 2019; Fujimoto & Gu, 2021; Kostrikov et al., 2021; Tarasov et al., 2023; Hom et al., 2024) is simple and powerful way to
3

---

Published as a conference paper at ICLR 2026
alleviate the out-of-diitributtion issue in the offline RL setting. Most seminal works leverage both actor and critic penalization, whereas critic penalization may deteriorate the Q function additional online training. Therefore, secure the versatility in both offline and online, minimize the most vanilla loss functions of the behavioral-regullrieed actor-critic framework as follows.
(o) Blo,a,o', r))D, dT [Qo(o, a) - (roooold,
(Ø) Blo.a))D, a² a'M
2 + (a((ra(()))
(3)
behavioral regularization
Herein, 0 and ó is a parameter of critic and actor networks respectively, 0 is a target parameter of the critic network (Mnih et al., 2013), a is a weight coefficient (Fujimoto & Gu, 2021), f(.,)) represents the function that captures the divergence between a trained policy {s(alo) and offline policy p(alo), which used collect the dataset D. The simplest implementation of f(.-)) the entropy regularization or behavioral cloning in the soft-actor critic algorithm log (a(a)o) (Haarnjj et al., 2018). Within such an RL framework, we introduce flow-based MARL solution.
Flow Matching and Flow Policies. Flow matching (Lipman et al., 2022; Gat et al., 2024) of- fers an alternative denoising diffusion models, which rely on stochastic differential equations (SDEs) (Ho et al., 2020; Song et al., 2020a; Nichol & Dhariwal, 2021). It simplifies training and speeds up inference while often maintaining quality, as based on ordinary differential equations (ODEs) (Papamakarios et al., 2021; Wildberger al., 2023; Lipman al., 2024).
The objective flow matching is simple: to transform simple noise distribution po, = (O, into given target distribution p1 p(z) over d-dimensional Euclidean space X' c Rd More preciscly, finds the parameter ø of a time-dependant velocity field no (f, x)) [0, 1] x IRd + RRd to build a dime-dependent flow (Lee, 2003) 1ho (t, x) [0, 1] x Rd + Rd via ODE as follows:
d dt ( (1 (l),)))) where x0,
Here, the terminal state vhe (1, with 20 Po. is expected follow the target distribution PL. To make the learning problem tractable, we follow an interpolating probability path (pt)osrs) between Po and Pl, where each intermediate sample is obtained by linear interpolation xf (1 (I-t)at t)x tt.
A timestep t is sampled from uniform distribution Unif([0, (0, 1)) corresponding to flow step.
The velocity field then trained to approximate the displacement direction (a) x)) at intermediate points along this path. Formally, the training objective is defined as:
[())= Eg slopp, (~Um)(( (t, (x)
[[]]
which encourages to recover the underlying transport field from source to target.
In this work, extract policy via the simplest variant of flow matching (Equation 4) (Park et al., 2025; Zhang et al., 2025a). Specifically, the flow policy is extracted to minimize the following loss.
LC
E.gorp (o,0)~D, t~Unnif((o.1] [llog(t, 0, x²) (a x²)]
Flow policies extend flow matching to policy learning by conditioning the velocity field (t o, a) on the observation o. The resulting flow eig o, 2), with a noise & ~ Pu. defines a deterministic mapping 20) 2) from observation and noise to actions by ODEs. Since is stochastic, this induces a stochastic policy T (alo), enabling flow matching to serve as generative policy model.
4 MULTI-AGENT COORDINATION VIA FLOW MATCHING (MAC-FLOW)
In this section, we introduce a novel MARL algorithm, dubbed MAC-Flow, which is a simple and expressive tool for extracting multi-agent policies via flow matching.
4.1 How DOES MAC-FLOW EXTRACT ONE- STE- POLICIES FOR COORDINATION?
Our desiderata are threefold: (i) to capture the distribution of coordinated behaviors, thereby preserving inter-agent dependencies under multi-agent dynamics; (ii) ensure high practicality by
4

---

Published as a conference paper at ICLR 2026
Stage I: Train vector field Us in BC flow-based joint policy mp
xN/2
2
2
flow step
flow step
Multi-agent dataset
Joint noise
Joins action
Joint distribution
Train critic Qo.
Stage II: Flow policy factorization for one-step policias {Tw,,". Tayy
Q
3
Flow policy
Pap.
Bow distillation
maximiiatioo Lioe:
Divvesity
Individual function((s)
AP
Paformmas
Quor a.)
Zi
Pull flo pollcy psi
Spoal
Zi
Ong-stop policy
Divemeite
One-step policy Mu, (0).
Individduoo-4 Settridoo--Globa--MMM Globai-MMor
noieefe)
Sceed
Figure 2: Overview diagram of proposed solution. Our solution, MAC-Flow. composes of stages. The first stage models joint action distribution via now-matching capture inter-agent dependencies, thereby facilitating the extraction of coordination behaviors more effectively than treating individual policies. For next stage, individual critics are trained under the individual-glon -glaaal--oo--aoa principle, thereby embedding behaviors for multi-agent coordination. At the second stage, practicality highlighted deriving individual policies for decentralized execution from flow-based joint policy via maximization and BC distillation.
enabling decentralized execution and fast inference at test time; and (iii to maintain algorithmic simplicity by avoiding unnecessary architectural overhead. Therefore, we adopt two-stage strategy, which trains joint policy via flow matching for (i), then distills it into set of individual policies for (ii). Thanks to the simplicity of flow-matching and BC distillation, MAC-E Flow directly fulfills (iii).
Overview. Figure 2 shows overview diagram for MAC-Flow. To achieve our goal, the first stage learns a joint observation- and time-dependent vector field va(4, o, to capture the multi- modal action distribution from the multi-agent dataset D. This vector field serves as a joint policy uo (o, z). Before proceeding to the next stage, we train individual critics (Q...... Qo...... Qo based the IGM principle. In the second stage, distill the flow-based joint policy into one- step sampling policies {Mo, (01, 21). ... Man, (0s, 2;) ... Hu, (01,21)) } for each agent i, where w; represents parameter of an individual policy network for i-th agent. This relies on three key properties: Definition 4.1, the joint action distribution can be factorized into independent individual policies; Proposition 4.2, the mismatch between the joint distribution and its factorized approximation is upper-bounded by the distillation loss; and Proposition 4.3 the resulting performance gap is controlled via a Lipschitz bound on the value function. Together, our solution preserves the multi- modal structure of the joint policy while extracting individual policies for fast inference.
4.2 PROPOSED SOLUTION
Joint Policy Extraction via Flow Matching, The objective of the first stage is to build a flow-based joint policy (,(0,)) via solely BC objective that accurately captures the joint action distribution in the offline multi-agent dataset D. Concretely, we train it by expanding the flow-BC loss function L (Equation 5) into the joint observation-action data sample follows:
Crlow-BC(es Ex (o,a)~D, (-Unif((o.1] [t. -a xx)
(6)
where x [ 2 and Pu I'_ (N Ia,) denote the random sampled joint noise and the noise distributions for all agents. The trained vector field ug defines a joint flow th e,, o, z) and hence stochastic joint policy (a through reparameterization with Po-

---

Published as a conference paper at ICLR 2026
Flow-based Joint Policy Distillation. Since execution under the CTDE framework must be fully decentralized, a joint policy conditioned on global observation is infeasible to deploy. We therefore factorize the flow-baeed joint policy into individual policies that approximate the individual action distribution while preserving coordination. This connection can be formalized by extending the IGM principle to the action distribution follows.
Definition 4.1 (Action distribution identical matching). For a joint observation o and action a with agent-wise dimensions d, let raaa o) be the joint action distribution. If each agent i admits an individual distribution Ty Ti(a; o:) such that x(a o) = I1 m<(a; 0z), then we say that action distribution identical matching holds.
This condition implies that decentralized execution via independent local sampling from F. (a | os) is distributionally equivalent to centralized execution of the joint policy x(a o). In practice, to approximate this factorization, we introduce a distillation loss that aligns the product of individual policies with the flow-based joint policy as follows:
Σ Mmn (os; 2:) - (uo 2)]el(
Distill-Flow (w) = EanDD z-Po
where [:]c is the i-th subvector of joint variables, and w is the set of individual policy parameters [w1,-.. wy]. Importantly, distillation is merely heuristic. Given both joint and factorized policies with the same noise, we obtain the following bound:
Proposition 4.2 (2-Wasserttiin upper bound of distillation). Fix a joint observation Let Po be a noise variable, and define the joint policy mapping Ho z) E A and the factorized policy mapping pw(o, z) [ (1),2))... ..)................... , Howy (or, 21]] E A. Denote by (o) and ( the porh-forward distributions of po through Hig and Pww respectively. Then, the 2- 2-Wasserstein distance between the joins policy and its factorization is upper-bounder by the square root of the distillation loss:
(Eaz-po [ll 2) no (o, 2) 1/2
W2(Tw(O),TToO) <
8
Full Objective for Policy Factorization. The goal of the second stage to factorize a flow-based joint policy (o(o, z) into a set of one-step sampling policies {Mw, (01, 21),... -/w (01, 21)] for T agents under the IGM and Definition 4.1. Formally, the full loss function can be defined as follows:
au + Σ [1 (or, 2;) (u, 2)
C (w) = EoWD, ar-Tw; zr-pa
As mentioned in Section 4.1, we design this function for one-step sampling policies to maximize the Q function and minimize the BC distillation losses. We train a critic network with a parameter 0, for agent i. In practice, minimizing the La (w) in turn upper-bounds the performance difference in terms of the global value function Quot- The following proposition formalizes this guarantee.
Proposition 4.3 (Lipschitz value gap bound). Fix joint observation o and assume Qur -) is Lq-Lipschitz in the action: QQmr(o, (T())) (o)))) - Qua(o, (o))) < LQ ko]la, - awl12. Yas, aw E A. Denote by (0) and (o) the push-forwarl distributions of the joint noise Po through pa .) and (o. [01,(01.2) (01.21). ,w,, (0z, 21)], respectively. Then, the performance gap satisfies
EaNTw(o) Qun( a)] (a) Qmn a)]
S La ()(Tm(()) (o())
S &Ly (Bz-po |I 2) - us 20(0)))113) (10)
For all mathematical derivations of the provided Propositions, please see Appendix D. Note that these properties collectively characterize bounded performance degradation under explicit assumptions rather than perfectly global optimality preservation.
4.3 DIDACTIC EXAMPLE: VALIDATING THE WASSERSTEIN- VALUE GAP RELATION
To further understand the theoretical idea, study toy example with grid world, landmark covering task. This subsection aims to show how policy factorization, distributional mismatch, Wasserstein distances, and value gaps manifest during learning.

---

Published as a conference paper at ICLR 2026
8.3
R 1or0
2.5
Natual intormation
0.2 500
0.3-
1.0
1.5
0.22
1.0
0.1
0.5
0.0
6.8
5.0
0.9
0.2
5.5 Steps
0.6
9.5
3.6
9.5
1.9
3.5
Steps
Steps
Wa(Blur
(4) Dietill loss and valoe
(h) Mutual Information
Uppechite upper bound
Value gop and Wassersten discrepancy
Figure 3: Theoretical validation a didactic example. (a) Distillation loss and corresponding value gap hetween the joint and factored policies over training. (b) Inter-agent mutual information during training, comparing dependency strength in joint policy vs. factorized policies. (c) Empirical value gap alongside the theoretical Lipschitz upper bound predicted by Proposition 4.3. (d) Point wise scatter of value gap Wasserstein discrepancy, showing all checkpoints lie below the theoretical bound.
Didactic task setup. In 2D plane environment, three agents aim cover three fixed landmarks. Each agent observes its own position and all landmark positions, but not other agents. The action of agent i is a movement vector a' IR2 The reward is defined through optimal assignment. The reward is a negative distance to its assigned landmark. We provide visualization in Appendix H.4.
Empirical evidence for propositions. Proposition 4.2 and 4.3 relate the performance deviation between the joint policy and its factorized approximation to the distillation discrepancy measured in Wasserstein distance under controlled conditions. First, Figure (a) plots both the distillation loss and the empirical value gap over training. As distillation proceeds, the value gap decreases in tandem with the loss, confirming the expected contraction. Second, improved distributional alignment results in reduced performance degradation. Next, Figure 3(b) shows that the joint policy exhibits strong inter-agent mutual information MT(a', a²) during the early phase of training, reflecting the ambiguity in how agents initially partition the landmarks. As training progresses and the landmark assignments become more stable, this MI gradually decreases and eventually converges. In contrast, the factored policy starts from O MI, then gradually becomes similar the joint policy. It maintains MI values below approximately 0.1 throughout training. which indicates that its independent parameterization cannot capture the interaction-indue induced dependencies without distillation. Finally, Figure (c) shows that the empirical value gap stays below the theoretical upper bound at all checkpoints, with the gap tightening as learning stabilizes. A point-wise analysis in Figure 3 (d) further shows clear monotonic trend: all samples lie beneath the linear envelope defined by LQ W2, indicating that the bound is both valid and informative in practice.
4.4 ALGORITHM SUMMARY
Algorithm I outlines MAC-Flow: we learn BC flow-based joint policy pa 2) via flow matching to model joint action distributions; train individual critics (Qe.) } under the IGM principle; and factor- ize ji into decentralized one-step policies {Mw; via Q guidance and BC distillation. During training, policy action for TD backups are sampled from (o 2) using the Euler method (Algorithm 2), whereas at deployment, actions are generated directly by [Mwc]. This design enhances inference speed while preserving coordinated expressivity, offering tractable value estimation under IGM, performance guarantees via Propositions 4.2 and 4.3, and single-step execution for fast inference.
EXPERIMENTS
The following subsection presents a suite of experiments designed to assess the effectiveness of MAC-Flow and experimental results via the following research questions (RQ) and answers.
RQL. How good is MAC-Flow
continuous and discrete action spaces of MARL benchmarks?
RQ2. How fast is the inference speed of MAC-Flow compared to liffusion-based solutions?
RQ3. Can MAC-Flow be extended beyond offline pretraining to online fine-tuning?
RQ4, How effective is the two-stage strategy in MAC-Flow framework?
RQS. How effective is the IGM--base critic training in MAC-Flow framework?
Please Appendix F, G, and H check additional RQs and detailed description for experiments.

---

Published as a conference paper at ICLR 2026
Algorithm MIAC-loWM
Algorithm Sampling
while not converged do
function pa(o, d 1/M
Sample hatch {(oi, ri of)
t+ 0
TRAIN FLOW-BASHD JOINT POLICY ne
for k E (0.... , do x (t x,j
Set variables x
p", + a, (ni (0, 4))
Calculate noise point X2 + (1 - f)xo 1x
1-i+d
Update using Equation (6)
return X
# TRAIN INDIVIDUAL CRITIC Qo,
for 1,.., do
Sample noise =i < zi ~ pf
Sample joint action a' + Ho (o x²) Update {9:) ElQo, (oz, 0i) -i -2 (d:
b. Algorithm 2
EXTRACT INDIVIDUAL POLICY Tun
for = 1,... ,
Sample noise =! < 24 pp
Sample action a; + (Og, =?])
Update {ux using Equation (9)
return of one-step policies (Twys Tayy)
5.1 ENVIRONMENTAL SETUPS
We evaluate the proposed solution. MAC-Flow. on four widely used MARL environments: StarCraft multi-agent challenge (SMAC) v1, SMACv2, multi-agent MuloCo (MA-MuloCo), and the multiple- particle environment (MPE). description of the testheds and datasets provided below.
SMACv1 (discrete action) provides real-time combat environment where two teams compete, with one controlled by built-in AI and the other by learned policies. incorporates both homogeneous and heterogeneous unit settings, thereby enabling diverse strategic coordination requirements. For offline datasets, we use the assets from off-the-grid benchmark (Formanek et al., 2023), including three quality datasets for each map, e.g., Good, Medium, and Poor.
SMACv2 (discrete action) extends SMACv1 by addressing the limited randomness of SMACv1 through three major modifications: randomized start positions, randomized unit types, and adjusted unit sight and attack ranges. These changes increase the diversity of the scenarios, making the tasks more challenging. For offline datasets, et al., 2023), including a dataset for each map, e.g., Replay.
the assets from off-the-gris benchmark (Formanek
MA-MuJoCo (continuous action) decomposes single robotic systems into multiple agents, each responsible for controlling a specific subset of joints. This design enables agents to coordinate in achieving shared objectives. For its datasets, we leverage the asset from Wang et al. (2023), including four datasets for each robotic control, e.g., Expert, Medium-Expent Medium, and Medium Medium-Replay.
MPE (continuous action) is a lightweight benchmark commonly used for studying cooperative coordination. Agents are represented particles moving in a two-dimensional continuous space, where they must coordinate to achieve goals. We leverage the offline datasets collected by Pan et al. (2022), including four quality datasets, e.g., Expert, Medium, Medium-
and Random.
Baselines. For offline MARL experiments, we use the three categories for baselines. For Gaussian policies, we consider the extension of SARI,, e.g., BC, BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and 3D3BC (Fujimoto Gu, 2021), and standard offline MARL solutions, e.g., ICQ (Yang et al., 2021), OMAR (Pan et al., 2022), and OMIGA (Wang et al., 2023). For diffusion policies, select recent offline MARL algorithms, e,g., diffusion BC, MADiff (Zhu et al., 2024), and DoF (Li et al., 2025a). Lastly, the flow policies include Flow BC and our proposed solution.
5.2 EXPERIMENTAL RESULTS AND RESEARCH Q&A
Contribution overview with RQ1 and RQ2. Diffusion baselines, e.g., DoF, demonstrate strong coordination performance, but they incur substantial cost due to iterative denoising process. In contrast, Mac-Flow trades small amount of expressiveness for dramatically faster optimization.
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance evaluation for discrete action control. We present a performance comparison across benchmarks, 8 tasks, and 18 datascts. These results are averaged over 6 seeds, and we report the two standard deviations after the sign. We highlight the hest performance bold and the second best in underlined
Sccmarios
Dataset
Cacsslan policies
Diffusion pelicies
Flow policies
BC
MABCO
N6.00
DI nidfusion on
MADI
DoF
Flow BC
NAC-Flov
Good
16.0 #1.0
8.7 +1.1
19.1 +0.1
19.5 +0.5
19.8 20.2
20.0 +0.0
19.8
3m
Mediam
8...
11-8
13.7 1033
13.3 1.077
16.4 13.6
18.6
14.7 11.
15.0 13.2
Poor
4.4.00.1
3.4 +1.0
4.2
4.2
10.3
10.9 +111
4.5
10.6 =2.0
Good
16.7 10.4
4.8 10.5
18.9 10.9
19.4 10.5
18.9
19.0 19.3
19.5 10.2
19.7 10.3
Medium
10.7
49.8
15.3 21.3
18:6
16.8
18.6
18.2
19.4
5.9.20.1
10.9
2:.0
20.
10.9
SMACS3
Good
+0.4
7-7 +0.9
17.4 +0.5
18.0 +1.0
15.9 +1.2
18.5 +0.8
19.5 #0.1
19.5 +0.3
2dh
Medium
12.3 10.7
7.6 10.7
15.6 10.4
13.4 11:4
15.6 10.3
18.1 10.3
15.1 12.0
17.6 10.5
Pote
6.7.20.3
6.6 #0.2
8.4
6.2
8.3
10.0 #111
6.9 =0.8
8.0
Good
15.8
10.4
16.8
16.6
18.0
5m
Mediam
20.5
15.1 =2.0
200.
Pooc
7.5.+0.2
+0.5
10.3 #3.1
8.0
8.9++13
10.8 +00.3
9.8 =2.1
Goad
17.5 10.4
10.1 13.2
12.9 2002
17.8 11.3
14.7 2.7.2
16.1 10.0
11:3
19.1 10.6
2c.vs. 64kcg
Medium
12.5 +0.3
0.9 +0.2
11.6 =0.1
10.5 =1.1
12.8 +12
13.9++00
11.8 #2.0
14.9 +4.1
Paxx
9.7 10022
9.0 19.2
10.2 $0.1
10.2 22.
10.8
11.6 13.1
10.0 19.3
11.4 20.4
Average rewards
12.2
13:1
13.0
13.8
SMACY2
terran. 5.35.5
Replay
7.3
13.8
11.8 20.9
9.3
13.3
45.4
8.3
16 2:333
5.v..5.vv.55
Raplay
10.3
10.2
12.0
4.6
terran. 10.v0.000. 10
Replay
2.0
43.0
11.8
#2.0
5.5
13.8 #1.3
14.6
3.8
24.
Average rewards
7.2
12.3
11.3
7.6
12.4
14.0
6.2
18.1
Table 2: Performance evaluation for continuous action control. We present a performance comparison across 2 benchmarks, 4 tasks, and 16 datasets. Results are reported following the conventions of Table 1. For readability, we use the acronyms M-E and M-R for Medium- Medium-Expert and Medium- Medium-Replay, respectively.
Sccoartos
Dataset
Extension
adcline SARL
Detline MAR1
MATOUSC
200
CHSR
CHEGA
MADSEE
MIC-7.0V
Expert
4401
4589.G =do.
2050.0
200.7 #:00..1
3383.G #a02 7
4711 2215.8
4050.0
2711
HallChowtah
M-R
Mediun
3828.9
2620.8
69-9
3189.4
3500.7
1235.
1336.9
1022.4
2540.3
1412.9
196.
-255.4.1350 1:354.
1:347.0
8008.1 1237
2504.7 283.5
2060101 365.4
2890.5-20
3030.2 1-43..8
M-ll
2818-3
Export
$300.0 65.5
$369.1 1533.8
754.7 1303.3
2.4-11.5
850.6 1700.5
2863.3+663.
3892.1
Happer
Medium
4 169.7
119.0
MEE
14.9
709.0
2046.9
2050.0
+297.5
2065.5
2060.0+10.55
2000:2 #2000
Ant
M-lk
Medium
9908. 3 192.8
431.6 _100.3
1412.4
1016.7 153.5
2014 1864.7
1905.1 198.9
1418.4
1296.51.00.2
1453
1.198. 4 1.90.3
M-B
1606.1
1800.2 123.5
1500.2 188.0
2092.88
1720.3 1330.6
1760.21156
2055.22 35 1.0.4
Average ruaall
2290.7 7
2367.7
15:1.5
-611.5
1856.1
2:50.21
2749.4
95.2 #5.2
96.0
AIPE
Spread
M-I
Mezium
29.3 1.4
15.4 11..6
$1.1 1.7.9
20.0 25.4
17.9 118.9
$7.9 112.3
28.3 -5.5
13.6 -5.7
30.1 1164
5.4 $.4.113.0
61.9 1777
36.3
NO.1 1904
50.4 1a3.
Raidon
9.8 24.9
24.0.298
34.4.20
6.3 28.b
-3.8 #12.3
6.9.5311
31.1 29.8
Arerage rewards
40.7
44.1
56.6
38.3
28.1
49.2
05.8
Additionally,
incurs a modest increase in training time over Gaussian models, but still trains far faster than diffusion baselines. Full results are provided in Appendix H.2
A1: For RQ1 (Performance) MMC-Fllw achieves best second-best average performance across four benchmarks regardless of continuous or discrete action space. Table 1 and 2 summarize the performance comparison across four benchmarks. These results demonstrate that MAC-F1 ow matches the performance of DoF by combining expressive flow-based modeling of joint action distribution with an efficient distillation step into decentralized one-step policies, thereby preserving coordination quality while ensuring scalability across discrete and continuous benchmarks.
A2: For RQ2 (Inference speed) MAC-Flor low achi- eves averaged x 14.5 faster inference than diffusion policies with comparable performance. Figure 4 shows its faster inference while maintaining competi- tive performance relative to MADi ff and DoF. More- over, MAC-FI 1 ow matches the inference speed of prior offline MARL algorithms but significantly outperform- ing them in performance. Theoretically, the per-agent inference complexity of FAC-Flow is low, especially O11. in contrast to (KK) for DoF and O(IK) for MADIFF, where K is the diffusion steps and is the number of agents. Full details are in Appendix C.
Detereece Soaed
1.80
MAT100
MOCEL
UPZZAA
BACOK
6,21
SMAD22
HEFE
Teferance Spand
12.5
MMC-PPu
MACOL
Figure 4: Inference time. These results are aver- aged over each benchmark's 's scenarios.

---

Published as a conference paper at ICLR 2026
PATAC
dist:llation
MPN
SMAAV
SABACV
Melliam
fem.
1.50
07000
Figure 5: Offline-to-online experiments. Online fine- tuning starts 0.5 normalized steps.
Figure 6: Ablation study for RQ4. We test the effect of the distillation phase and its RL. objective.
MAC-F can seamlessly use online rollouts to fine-tune itself, enabling it to achieve better results than previous methods. Figure shows the training curves of MAC-FloW and previous baselines, where online fine-tuning begins after 0.5 iteration steps. Some baselines fail to account for exploration and thus remain limited to their offline performance, while MA-COL exhibits the reported issue of a sharp performance drop at the initial stage of the online phase. In contrast, our approach effectively improves performance in the online stage by leveraging newly collected rollouts. In practice, MAC-Fl ow can fine-tune its networks under the same objective used in offline learning by simply augmenting the offline dataset with additional online rollouts.
A3: For RQ3 (Offtine-to-onlinn
A4: For RQ4 (Ablation on distillation with Q maximization). both the full objective and training scheme of MAC-F Flow are essential. Figure 6 presents the ablation study on Q maximization in Equation (9) and the distillation stage in the two-stage training scheme. Specifically, w/o Q corresponds to one-step sampling policy without an RL objective, while w/o distillat on refers to pure BC flow policy that requires an ODE solver. Across all scenarios, MAC-Flow consistently outperforms its ablated counterparts, demonstrating that removing either Q-maximization or the distillation phase substantially limits policy learning. These results highlight that such components are critical to achieving strong performance across diverse tasks.
MAC-Flow with 1GM
Flow without 1GM
MA-MujoCo (Scenario: Halfcheetah. Dataset: Medium.)
MPE (Scenario: Spreadi Dataset: Medium.) )
Perfortance
4.0
Avg- Q value
Perfortance
vallue
200
250
2.5
280-
Avg.
0.0
-200
0.8
150
1.0
Steps
Steps
Steps
Steps
Figure 7: Ablation study for RQS.
This figure shows learning curves for performance and Q value. Performance's 's y-axes of MA-MuloCo and MPE are scaled as 1000 and 20 units, respectively.
A5: For RQS (Ablation IGM) IGM-based critic training is crucial for ensuring stability in critic optimization, consistency in Q-value estimation, and superior performance in the MAC-Flow framework. Figure 7 presents learning curves for the ablation study comparing MAC-Flow with IGM and without IGM. The IGM-based variant achieves substantially higher performance, while the non-IGM counterpart stagnates at suboptimal levels. In particular, IGM leads significantly lower and more stable loss curves in a multi-agent setting; additionally, the non-IGM baseline exhibits a collapse of Q estimates as training progresses.
6 CONCLUSIONS
In this work, we propose KAC-Flow, novel MARL algorithm that learns flow-based joint policy to capture the multi-modality of multi-agent datasets, and then distills into decentralized one- step sampling policies using combination of RL and BC objectives. Our experiments show that MAC-Flow addresses the trade-off between inference efficiency and performance in offline MARL.
Future Directions and Impact Statement. While our approach demonstrates strong performance and efficiency gains, extending MAC-Flow to more diverse and dynamic environments remains an important direction. We should develop an algorithm that can integrate other pre-trained distributions to enhance the diversity of decentralized policies. This can enable more flexible role adaptation and generalizability improvement, ensuring that agents adapt new scenarios. Such directions increase the stability of multi-agent systems. This line of research provides a foundation for future advances in generalizable MARL and its deployment in real-world domains.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. NSF 2340651, NSF 2402650, DARPA HR001 12490431, and ARO I- 0193. This research used the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing computational resources.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. con- ditional generative modeling all you need for fecision-making? arXiv preprint arXiv:2211.15657, 2022.
Marvin Alles, Nutan Chen, Patrick van der Smagt, and Botond Cseke. FlowQ: Energy-guided flow policies for offline reinforcement learning. arXiv preprint arXiv:2505.14139 2025.
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in Atari. NeurlPS, 37: 58757-58791, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In ICML, pp. 1577-1594, 2023.
Paul Barde, Jakob Foerster, Derek Nowrouzezahrai, and Amy Zhang. A model-based solution the offline multi-agent reinforcement learning coordination problem. AAMAS, 2024.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of Markov decision processes. Mathematics of operations research, 27(4): 819- 840,840,222 2002.
Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard. Shagun Sodhani, and Amy Zhang. When should we prefer decision transformers for offline reinforcement learning? ICLR, 2024.
Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-AI coordination. NeurIPS, 32, 2019.
Bo Chen, Chengyue Gong, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Mingda Wan. High-order matching for one-step shortcut diffusion models. arXiv preprint arXiv:2502.00688, 2025.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401 0066444, 2024a.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. ICLR, 2024b.
Cheng Chi, Zhenjia Xu, Siyuan Feng. Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273668. 2023.
Tom Danino and Nahum Shimkin. Ensemble-MIX: Enhancing sample efficiency in multi-agent rl using ensemble methods. arXiv preprint arXiv:2506. 028411 2025.
Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint 10Xiv:20/1.0953: 2020.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984 2023.

---

Published as a conference paper at ICLR 2026
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model: Future model- ing beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03570, 2024.
Chanin Eom, Dongsu Lee, and Minhae Kwon. Selective imitation for efficient online reinforcement learning with pre-collected data. ICT Express, (0(6))1308-0 -314444,, 2024.
Claude Formanek, Asad Jeewa, Jonathan Shock, and Arnu Pretorius. Off-the-grid MARL: Datasets with baselines for offline multi-agent reinforcement learning. arXiv preprint urXtv:2332.0055/ 2023.
Juan Formanek, Callum R Tilbury, Louise Beyers, Jonathan Shock, and Armu Pretorius. Dispelling the mirage of progress in offline marl through standardised baselines and evaluation. NeurlPS, 37: 139650-139672, 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410.125577 2024.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. NeurIPS, 34:20132-2001 20144,,, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In /CML, pp. 2052-2062, 2019.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. NeurIPS, 177:133345-13338 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, pp. 1861-1870, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840-6851, 2020.
Xingshuai Huang, Di Wu Member, and Benoit Boulet. Goal-conditioned data augmentation for offline reinforcement learning, arXiv preprint arXiv:2412.20519, 2024.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint "xXi:222550099) 2022.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.66166 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. NeurlPS, 33:1 179-1191,, 2020.
Dongsu Lee and Minhae Kwon. Scenario-fire autonomous driving with multi-task offline-to-online reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 2025.
Dongsu Lee, Chanin Eom, and Minhae Kwon. AD4RL: Autonomous driving benchmarks for offline reinforcement learning with value-based dataset. In ICRA, pp. 8239-8245. IEEE, 2024a.
Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. GTA: Generative trajectory augmentation with guidance for offline reinforcement learning. NeurlPS, 37:56766-56888 56801, 2024b.
John M Lee. Smooth manifolds. In Introduction to smooth manifolds, pp. 1-29. Springer, 2003,
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
Chao Li, Ziwei Deng, Chenxing Lin, Wenqi Chen, Yongquan Fu, Weiquan Liu, Chenglu Wen, Cheng Wang, and Siqi Shen. Dof: A diffusion factorization framework for offline multi-agent reinforcement learning. In /CLRR 2025a.
12

---

Published as a conference paper at ICLR 2026
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint arXiv:2507.077 00066, 2025b.
Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusson policies in offline
multi agent reinforcement learning. arXiv preprint arXiv:2307.01422 2023.
Zhuoran Li, Xun Wang, Hai Zhong, and Longbo Huang. OM2P: Offline multi-agent mean-flow
policy, arXiv preprint 2518:25000066669 2025c.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXtv:2302.0187, 2023,
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprini arXiv:2412.06264, 2024.
Yijing Liu, Chao Du. Tianyu Pang, Chongxuan Li, Min Lin, and Wei Chen. Graph diffusion policy optimization. NeurIPS, 37:9585-961 2024.
Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu, Yile Liang, Donghui Li, and Xuetao Ding. Offline multi- agent reinforcement learning via in-sample sequential policy optimization. In AAAI, volume 39, pp. 19068- 2025.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. NeurIPS, 30, 2017.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. NeurIPS, 36:46323-46344, 2023.
Haofei Lu, Dongqi Han, Yifei Shen, and Dongsheng Li. What makes good diffusion planner for decision making? ICLR, 2025.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2 100::21011023000 02388, 2021.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. ICML., 2025.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR, 2023a.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, et al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Reseurch, 20(2):233-248, 2023b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 102.52331 2013.
Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-RL. In ICML, pp. 26087-26105, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pp. 1162-8171,00 2021.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In ICMI., pp. 17221-17237 2022.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
13

---

Published as a conference paper at ICLR 2026
Scohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-constrainee unsupervised skill discovery. In ICLR, 2022.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline r!? NeurlPS, 37:79029-79053 2024.
Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning, JCML, 2025.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Plerre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. FACMAC: Factored multi-agent centralised policy gradients. NeurIPS, 34:12208-12221, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression: Simple and scalable pollcc reinforcement learning. arXiv preprint
2019.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-88) -55, 1992.
Dan Qiao, Wenhao Li, Shanchao Yang. Hongyuan Zha, and Baoxiang Wang. Offline multi-agent reinforcement learning with sequential score decomposition. OpenReview; 2025.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, (1 2020.
Allen Z Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. CoRL Workshop, 2024.
Mare Rigter, Jun Yamada, and Ingmar Posner. World models via policy-guided trajectory diffusion. arXiv preprint arXiv:2312.08553 2023.
Stephane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning, arXiv preprint 123.v:1203.1007, 2012.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005522 2022.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: /902.04043, 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pp. 2256-2265, 2015.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICML, pp. 5887-5896, 2019.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13345 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. ICMI, 2023.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Valop-deconposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1706.05266, 2017.
14

---

Published as a conference paper at ICLR 2026
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. NeurIPS, 36:11592-11620, 2023.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. NeurIPS, 35:226-237, 2022.
Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang. Brandon Amos, and Jakob Foerster. Nocturne: scalable driving benchmark for bringing multi-agent learning one step closer to the real world. NeurlPS, 15:3962-3977 2022.
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Scohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506. 15799, 2025.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex ducling multi-agent Q-learning. ICLR, 2021.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. NeurlPS, 36:52413-52429 2023.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. NeurIPS, 35:16509-16521, 2022.
Jonas Wildberger, Maximilian Dax, Simon Buchholz, Stephen Green, Jakob H Macke, and Bernhard Schölkopf. Flow matching for scalable simulation-basee inference. NeurIPS, 36: 36:16837-16864 2023.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:191..11386 2019.
Ningyuan Yang, Jiaxuan Gao, Feng Gao, Yi Wu, and Chao Yu. Fine-tuning diffusion policies with backpropagation through diffusion timesteps. arXiv preprint arXiv:2505. 10482, 2025.
Qianlan Yang and Yu-Xiong Wang. RTDiff: Reverse trajectory synthesis via diffusion for offline reinforcement learning. In ICLR. 2025.
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. NeurIPS, 14:10999-13,,, 2021.
Xianghua Zeng, Hang Su, Zhengyi Wang, and Zhiyuan Lin. Graph diffusion for robust multi-agent coordination. In ICML, 2025.
Qinglun Zhang, Zhen Liu, Haoqiang Fan, Guanghui Liu, Bing Zeng, and Shuaicheng Liu. Flowpolicy: Enabling fast and robust 3d In AAAL, volume 39, pp. 14754 14762. 2025a.
policy via consistency flow matching for robot manipulation.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighte flow matching for offline reinforcement learning. arXiv preprint arXiv:2503.04977 2025b.
Yuyou Zhang, Yanı Niu, Xingyu Liu, and Ding Zhao. Composer: Scalable and robust modular policies for snake robots. ICRA, pp. 10800-10806. IEEE, 2024.
Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman. Aditya Grover, and Ricky TQ Chen. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311.1344. 2023.
Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. MADiff: Offline multi-agent learning with diffusion models. NeurIPS, 37: 4177-4206, 2024.
15

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
A Miscellaneous
17
A.1 Summary of Notations
17
B Extensive Related Works
18
C Complexity Analysis
19
Mathematical Derivation
20
D.1 Proof for Proposition 4.2
20
D.2 Proof for Proposition 4.3
20
E Training Details of MAC-Flow
21
Implementation Details
22
F1 Baseline Algorithms
22
F22 Git Repository for Baseline Implementation
23
Experimental Details
24
G.1
SMACv] and SMACv2
24
G.2 MA-MuJoCo
24
G.3 MPE
24
G.4 Hyperparameters
25
H Additional Results
26
H.I Ablation Study
26
H.2
Training Time
27
H.3
Learning Curves of MAC-Flow
28
H.4 Landmark Covering Game: Scaling Up The Number of Agents
29
H.5 Pure Coordination Game: Better Point than BC Generative Modeling
30
H.6 XOR Stress Test: Failure Mode Analysis
31
H.7 Payoff Game: Analysis According Interaction Strength
32
16

---

Published as a conference paper at ICLR 2026
A
MISCELLANEOUS
A.1 SUMMARY OF NOTATIONS
Dec-POMDP elements
Notation
Description
Notation
Description
I
set of agents number of agents
agent index discount factor state at time local observation of agent action of agent i observation function of agent return of agent time offline datasct (replay buffer)
Y E 1)
S
global state space observation space of agent action space of agent state transition function reward function of agent trajectory
8
O
0f
A
4
T
f2
R
7
D
Algorithm elementa
(Flow Matching Policies)
Notation
Description
Notation
Description
Po
prior noise distribution noise sample from Pa interpolated point (1 f) + twa
P1
target action distribution target action sample continuous flow step flow trajectory noise for agent i
I1
t
Us 0)
velocity field conditioned on
<t,,
2
joint noise variable
21
(g(0, z)
flow-based joint policy
(alo)
induced stochastic joint policy
Mw, (Os, zi)
one-step policy for agent
Tw: (aglos)
induced one-step sampling policy
Qeot (
global Q-function
Qo, (os, as)
individual Q-function
RL Training
Notation
Description
Notation
Description
critic parameters
target critic parameters parameters of one-step policies batch size
BC flow policy parameters
{w:}
77
learning rate
number of updates
T
episode horizon divergence measure
balancing cocfficient (regularization)
f(--)
17

---

Published as a conference paper at ICLR 2026
B EXTENSIVE RELATED WORKS
Behavioral-regularizze Actor Critic. A closely related line of our research is behavioral-regularized actor-critic (BRAC) (Wu et al., 2019). The BRAC is one the simplest and most powerful policy extraction strategies among offline RL solutions (Park et al., 2024). These approaches solve the distif-disttiuttiin sample issue by constraining the learned policy to remain close to the behavior policy. In general, this is implemented by adding divergence penalties regularization terms in the actor and critic updates, thereby stabilizing policy improvement in offline settings (Tarasov et al., 2023; Fujimoto & Gu, 2021; Kostrikov et al., 2021). Although these methods are primarily designed for single-agent RL (SARL) and rely on relatively simple Gaussian policies to model the action space, they are often adopted as baselines in MARL by being naively extended multi-agent settings (Pan et al., 2022; Formanek et al., 2023; Wang et al., 2023; Zhu et al., 2024; Li et al., 2025a; Qiao et al., 2025). As a result, they struggle capture the multi- multi-modality of joint action distributions that are inherent in cooperative MARL scenarios, thereby being behind on their performance compared to advanced architecture-based RL solutions.
In contrast, our proposed solution embraces the same principle of behavioral regularization, striking a balance between Q maximization and fidelity to the offline dataset. MAC-Flow, specifically, leverages flw-matching introduces a distillation loss analogous to the behavioral penalty BRAC at the individual policy extraction phase. This allows decentralized one-step policies to inherit both expressiveness from the flow-based policy and regularization from the dataset distribution, concurrently guaranteeing coordination in multi-agent settings.
extract a rich generative model of the joint action space, and then
Short-cut Diffusion or Flow. Another relevant line and motivation of this work is shorteut generative modeling, e-g., shortcut diffusion and shortcut flow matching (Frans et al., 2024; Park et al., 2025). These approaches alleviate the inefficiency of iterative generative models (denoising diffusion or ODE/SDE solvers)
al., 2015; Ho al., 2020; Song & Ermon, 2019; Song et al., 2020b; Nichol & Dhariwal, 2021) by either reducing the number of denoising steps (Song et al., 2020a; Salimans & Ho, 2022: Meng et al., 2023a; Song et al., 2023) or by learning direct mappings that approximate the multi-step generative process with fewer evaluations (Frans et al., 2024; Luhman Luhman, 2021; Lipman et al., 2022). In the context of RI. shortcut generative methods have been explored to accelerate policy sampling while retaining the expressive capacity of diffusion or flow models to address complicated problems.
Our work shares the motivation of achieving fast inference with expressive policies, but introduces a key adaptation for the multi-agent setting. Rather than merely shortening generative trajectories, MAC-Flow factorizes the flow-based joint policy into decentralized one-step sampling policies, supported by theoretical guarantees under the IGM principle. This perspective extends shortcut flow approaches, where the shortcut lies not only in time complexity but also in the structural factorization of multi-agent policies. By combining flow-based modeling with policy distillation guided by the IGM principle, MAC-Flow generalizes shortcut generative techniques address the scalability and coordination challenges unique to offline MARL.
18

---

Published as a conference paper at ICLR 2026
C COMPLEXITY ANALYSIS
To theoretically support the empirical inference efficiency of MAC-Flow, we provide big-O analysis comparing its per-agent and total inference-time complexity against two diffusion-based SOTA baselines, DoF and MADiff. We analyze the computational complexity of generating one joint action at a single environment step.
Setups. For this discussion, let I denote the number of agents and T the number of iterative steps required by diffusion or flow policies. The input dimensions for per-agent observation and action treated as do,, du, (11). Although
solution is based simple [512, 512, 512, 512] -sizdd MLP network, and MADi f f and DoF employ a U-net-based temporal architecture, we posit that all constant factors absorbed in asymptotic notation.
Table 3:
Asymptotic inference time complexity analysis. This table reports big-O analysis about input dimension, per-agen cost, and total cost of producing one joint action single environment step.
Method
Decentralizeation
Input dimension
Per-agent complexity
Total complexity
MAC-Flow
Yes
(1)
((1)
(
DoF
Yes
((1)
O(K)
O(1K)
MADifE-D
Yes
O(I)
O(IK)
O(IK)
MADiff-C
No
(())
O(IK)
O(IK)
MAC-Flow. Our solution extracts decentralized one-step sampling policies for each agent. Therefore, at the inference phase, each agent can produce its action with a single forward pass using simple MLP networks. The per-agent complexitiy is (1), and the total complexity (I).
DoF. This decomposes the centralized diffusion process into decentralized per-agent processes. Each agent must execute full -tep denoising chain, although the factorization ensures that the per-step input dimension is (1). independent of I. Thus, the per-agent complexity is O(K) and the total complexity is (TK)). We follow the default setups, -ra-errjec,,yy, W-Concat factorization, and 200-steps denoising iterations.
MADiff. We report two variations for MADiFf, centralized and decentralized versions. For MADiff-D. each agent conditions the diffusion model on its own local observation. Neverthe- less, due to the model's architecture, the diffusion process generates a full joint trajectory that includes both the agent itself and its teammates. As result, the input size at each denoising step scales with the number of agents, i.e., O(I). Moreover, since the denoising process requires K iterative steps, the per-agent complexity is O(IK). As all I agents must perform this computation independently, the total complexity reaches O(T²K), which highlights a quadratic dependence on the number of agents. Next, for MADi MADiff-C, ff- single diffusion model jointly generates the actions for all agents in one forward chain. At each denoising step, the input dimension remains O(I), and the process requires K iterations, yielding a per-agent complexity of O(IK). However, because the joint model executes once for all agents, the total complexity is limited to O(IK). This distinction emphasizes that MADiff-C avoids the quadratic blow-up observed MADiff-De albeit the cost of requiring centralized execution and communication during deployment.
Discussion Summary. Table summarizes the theoretical inference complexities. Our analysis shows that MAC-F Flow achieves constant-time inference per agent, independent of both I and K, thereby scaling linearly with the number of agents. contrast, diffusion-based methods inherit the iterative overhead of K denoising steps. While DoF alleviates input scaling via factorization, its complexity remains O(IK) MADi ffffff-C- also maintains O(I)) complexity but requires centralized execution. Finally, MADi F F-D is the least efficient, with quadratic dependence the number of agents due to per-agent joint inference, resulting in (T2 K). These asymptotic distinctions theoreti- cally underpin the empirical findings reported in the main text, where MAC-Flow demonstrates 13.7 ~ 21.4x speedup over diffusion-basee baselines.
19

---

Published as a conference paper at ICLR 2026
D MATHEMATICAL DERIVATION
This section provides a mathematical derivation for two propositions. Before looking deeper into them, we first show the Lemma related to the comparability joint and factorized policies.
Lemma D.1 (Comparability of Joint and Factorized Policies). Let o be a joint observation and let z ~ Po denote the joint noise variable. Consider the flow-based joint mapping (o, that induces the push-forward distribution (o) and the factorized mapping pw z)
|mm, (01, .1).....,A.. .,mu (or, z1) thut induces (o). By Desinition 4.1, if action distribution iden- tical matching holds, then the joint distribution can be factorized as a product of individual policies.
Even when exact matching does not hold, both T (0) and (0) are defined as push-forwarl dis- tributions of the same base noise po- Hence, T (O, and Tw (o) are comparable within the same probability space, and their discrepancy be measured via a divergence function.
D.1 PROOF FOR PROPOSITION 4.2
Let be a joint observation and Po a joint noise variable. Define the joint pol- icy mapping Pi (o, x) Ai ... Ar and the factorized policiy mapping (4o z) (pa,, (01, 21)."". ... -Bw, (os, 21]] Denote by ~ no(o) and a ~ (o) the push-forward distributions of Po through ne and Mvu respectively.
Following Lemma D.I, the squared 2-Wasserstein distance is defined as
W (o), To(o)) XEA(Fu.Ta)
inf B(ay)>> [|la yl1
where A(Tw; π&) denotes the set of coupling distributions between Tw and πt-
By choosing the specific coupling À induced by sampling Po and pairing (o, 2) with pa z).
we obtain
W (To (o), ne (o))
Eg-pa |l (o, z) m=(0,
Then, taking square roots on both sides yields the desired inequality:
W2 (T.) mo())) S (B [llmm 2) mo(o, 2)11))
The proof is completed.
D.2 PROOF FOR PROPOSITION 4.3
Fix joint observation o and assume the global Q-function Qwe(o, is Lo Lipscchtii in its action argument, i.e..
(o, (o, y) Lolla yll2, Va,y E A .... Az.
Let (o) and (o) denote the fosh-forward distributions of po under na(o,-) -) and (w, :), respectively. Then,
Ea) (0, a)] (aa=,,(o)luuu < LQ ()(
where the inequality follows from the dual formulation of Lipschitz functions and Wasserstein distances. Finally, applying Proposition 4.2 gives
(o) Ear (o, s Lo (0ampoll/ww00. 2) (00 (o(0,2))33) 1/2
The proof is completed.
20

---

Published as a conference paper at ICLR 2026
E TRAINING DETAILS OF MAC-FLOW
This section describes the implementation details of MAC-Flow.
Network architectures. MAC-Flow is implemented on multi-layer perceptrons (MLPs) with hidden sizes [512, 512, 512, 512] for all networks, including the joint flow policy, the crities, and the factorized one-step policies. Layer normalization applied consistently to further improve stability.
Flow matching. As described in Section 4.2, we adopt the simplest flow-matching objective (Equation 6) based on linear interpolation and uniform time sampling. For all environments, we use the Euler method with step count of 10 to approximate the underlying ODE dynamics (Algorithm 2). This ensures that the joint flow policy (1 2) captures the mudti-modal structure of coordinated behaviors while remaining computationally efficient at training and inference.
Value learning. We train individual critics [Qa,y=n under the IGM principle with TD error update. We basically follow the mean Q2) method, instead of min(Q1, Q2), to avoid pessimism in an offline RI setting. The one-step policies are optimized to maximize the global Q-function while simultaneously minimizing the distillation loss between the flow-based joint policy and the factorized policies. To calculate global Q, we use the average mixer for each agent's 's Q value (Sunchag et al. 2017; Danino Shimkin, 2025). To practically enforce the Lipschitz constraint required in Proposition 4.3, we apply layer normalization (Ba et al., 2016; Park et al., 2022) to all critic networks, which we found to be crucial for stabilizing value learning in multi-agent coordination.
One-step policy learning, After training the flow-based joint policy μe(o, 2). we factorize it into decentralized one-step policies {my! (os, 24 2 This stage jointly optimizes two objectives in Equation 9: (i) O-maximization and (ii) BC distillation. In general, we set the BC distillation coefficient a = 3.0 as the default. In practice, we alternate between updating the critics and the one-step policies, using the same batch of transitions (Algorithm 1).
Flow matching and policy for discrete action space. For SMACvl and SMACv2, which are discrete action control tasks, we model actions as one-hot vectors and learn a continuous vector field over the simplex. Specifically, for each transition, we form a linear path from Gaussian noise 2g ~ NN I) to the one-hot actions 2 onehot(a). sample t ~ Unif( (0, 1)). set x, (1 z))ng + fx and supervise the BC flow field Us (o, F4, t) with the target velocity #1 - To via an MSE loss. To obtain a distill the multi-step Euler integration of 0s into single-step flow head that outputs logits over actions. During actor updates, we add Q-guidance term that maximizes the mixed value of the actions proposed by the one-step policy, with an optional normalization of the guidance magnitude. At the training phase, target actions for TD backups are sampled from the flow policy; at deployment, we use arg max with temperature control value, and optionally apply legal-action masking before the softmax.
one-step sampling policy,
Online fine-tuning. For the fffline-to-onlinn experiments (RQ3), we do not consider symmetric sampling. which reuses the offline dataset during online training (Ross & Bagnell, 2012), unlike prior research (Ball et al., 2023; Eom et al., 2024; Lee & Kwon, 2025). Instead, the agent is trained purely on newly collected online rollouts for an additional 500K gradient steps, continuing from the offline pretraining checkpoint.
Training and evaluation. We train MAC-Flow with 1M gradient steps for SMACv1 and SMACv2, and 500K steps for MPE and MA-MuJoCo. For offline-to-online training, we first perform 500K steps of offline training, followed by 500K steps of online training. We evaluate the learned policy every 50K steps using 10 evaluation episodes. For the main results in Tables and 2, we report average performance and two standard deviations across 6 random seeds for the table and tolerance interval for the learning curves.
21

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION DETAILS
The main objective of this work is to alleviate the gap between performance and inference speed in multi-agent setting. Therefore, we deliberately adopt a simple network architecture, such as a multi- layer perceptron (MLP), rather than resorting more complex or specialized designs. The simplicity of MLPs provides several advantages: (i) they allow for faster inference and lower computational overhead, which is critical for scalability in multi-agent settings: (ii) they facilitate stable training and clear evaluation of the proposed algorithmic contribution without the confounding effects of intricate architectures; and (ili) they serve as a neutral baseline architecture, demonstrating that the observed improvements stem from our framework itself, not from architectural sophistication.
F.1 BASELINE ALGORITTM
BC. This is a simple behavioral cloning (also known as imitation learning) algorithm. For the continuous action domain, we design it as a Gaussian policy with unit standard deviation. For the discrete action domain, we parameterize the policy as a categorical distribution, where the policy network outputs unnormalized logits over all possible actions and the resulting action probabilities are obtained via a softmax function. Our network scheme is [512, 512, 512, 512] sized MLPs, which is also our default network architecture, for all environments.
Diffusion BC. This is diffusion- based extension of BC. Instead of directly regressing expert actions, Diffusion BC learns a denoising diffusion process: given an observation and a noisy version of the expert action, the policy predicts the noise to recover the clean action. For the continuous action domain, we model actions as Gaussian vectors, where the training objective is to predict Gaussian noise added during the forward diffusion process. At inference, the policy generates actions by reverse diffusion conditioned on current observations, starting from Gaussian noise. For the discrete action domain, we represent actions as one-hot vectors and apply diffusion in the relaxed continuous space. For both domains, we use [512, 512, 512] -ized MLPs, augmented with a sinusoidal timestep embedding. We consider a Gaussian diffusion scheduler with 200 timesteps to govern the forward and reverse processes for all environments.
Flow BC. This is implemented on top of the same codebase as MAC-Flow, sharing the same flow- matching implementation. However, Flow BC does not consider Q maximization and distillation; in other words, it directly uses a trained full flow policy in training and deployment. We train individual vector fields for individual BC flow policies to distribute them into decentralized setups. For discrete control, actions are represented as one-hot vectors and flow matching applied between Gaussian noise and the one-hot target along linear path, with the policy trained to predict the corresponding velocity field. We consider 10 flow steps and [512, 512, 512, 512]-sizee MLPs for all environments.
MATD3BC (Fujimoto & Gu, 2021). l). We reimplement it on our codebase by referring to the official open- open-source implementation of TD3BC. This is a multi-agent extension of TD3 with an additional BC regularization term. The BC parameter is set as 2.5 for all environments. We train two critic networks via clipped double Q-learning, and target critic networks via the Polyak averaging with 0.005 (Polyak & Juditsky, 1992). For multi-agent settings, we employ the mixer for Q-value networks. The policy and value networks are parameterized as [512, 512, 512, 512]-sized MLPs.
MABCQ (Fujimoto et al., 2019). We reimplement it on our codebase by referring to the official open-source implementation of BCQ. We employ the mixer for Q-value networks. For discrete control, we design the policy as softmax policy network with twin Q-networks. It masks out actions whose probability falls below BC threshold a = 0.4 and selects the maximum Q-value only among admissible actions. The critic is trained with TD targets using the masked actions set, and the actor is updated via cross-entropy loss against the action of the replay buffer. We use [512, 512, 512,512]
MI.ss for both actor and critic, with a target update period of 200 steps.
MACQL (Kumar et al., 2020). We reimplement it on our codebase by referring to the official open-source implementation of CQL. The critic consists of twin Q-networks with TD loss and a conservative penalty that lowers Q-values on OOD actions via a log-sum-exp term. Additionally, we employ the mixer for Q-value networks to consider multi-agent coordination. For discrete controls, the policy is set categorical distribution, and the illegal actions are masked during selection; for continuous controls, we use the Gaussian policy. We use [512, 512, 512, 512]-sized MLPs for both actor and critic, target update period of 200 and every iterations (T = 1.0 for discrete domains) and
22

---

Published as a conference paper at ICLR 2026
(T 0.005 for continuous domains), conservative weight 3.0, and 10 sampled actions per state for calculating the conservative loss for all environments.
OMAR (Pan et al., 2022). We reimplement it codebase by referring to the official open-source implementation of OMAR. This algorithm employs CQL-style conservative critic regularizer, a cross- entropy method (CEM)-based policy improvement head. The policy collects candidate actions via iterative CEM and is trained to imitate the best-Q candidate and maximize Q via a small L.2 penalty. We [512, 512, 512, 512]-sized MLPs and the hyperparameter follows: target update rate 0.005, CQL regularizer (10 OOD samples, and a = 3.0), and CEM process (3 iterations, 10 samples, 10 elites per step, and mixing coefficient 0.7).
OMIGA (Wang et al., 2023). We reimplement it on our codebase by referring to the official open- source implementation of OMIGA. The critics use twin Q-networks combined by learnable state- dependent mixing network, while separate V-network provides baseline for variance reduction. The policy is updated with dlvaattae--wiighte regression (Peng et al., 2019). Target networks for Q, V, and the mixer are updated via Polyak averaging with 0.005. We use [512, 512, 512, 512] MLPs for all networks, a mixer embedding dimension 128, advantage scaling coefficient a 10.0, gradient clipping at 1.0, and weight clipping for policy updates 100.0.
MADiff (Zhu et al., 2024). We use the official open-source implementation of MADi ff. This reposi- tory provides two variants: MADiff-C and MADiff-DD for centralized and decentralized versions, respectively. Given that our problem formulation is Dec-POMDP we select decentralized variants, that is, MADiff-D. We train conditional diffusion policy on joint demonstration trajectories with centralized data, then deploy with decentralized execution. Each agent at deployment conditions only on its own local observation and optional history and samples its action via reverse diffusion while jointly predicting teammates' trajectories. We keep the same architecture and hyperparameters in the original paper and set the diffusion scheduler to Gaussian DDPM with 200 denoising steps for all environments.
DoF (Li al., 2025a). We use the official open-source implementation of DoF. We use the DeF ReF-TrE TeE-Trajectov agent with the W-concat factorization. Training and deployment settings are identical the original paper. For sampling, we use Gaussian DDPM scheduler with 200 denoising steps across all environments.
F.2 GIT REPOSITORY FOR BASELINE IMPLEMENTATION
The implementation adheres closely to the aforementioned official code as follows.
BCQ: 3CC2ttpp.///sthhbb.oo///////////
CQL: h0g.:/ttpps//ggttu..cco///////as
OMAR: /0gR::tttsss/gghhhooooo////////2
MADifE: MDDEE//.tte///tttnnooo///////222
DoF: htt.tt/sse//ttttiooooo/rrra/raar
23

---

Published as a conference paper at ICLR 2026
G EXPERIMENTAL DETAILS
G.1 SMACVI AND SMACV2
SMACv1, introduced by Samvelyan et al. (2019), serves as prominent benchmark for assessing cooperative MARL algorithms. Built on the StarCraft II real-time strategy game, SMAC simulates decentralized management scenarios where two opposing teams engage in combat scenarios, with one team controlled by built-in Al and the other by learned multi-agent policies. Agents receive partial observations restricted to local sight range (e.g,, nearby allied and enemy units, distances, health, and cooldowns), while the complete state information is available only for centralized learning of some algorithms. The discrete action space includes moving four directions, attacking visible enemies within range, stopping, and a no-op, which makes coordination strategies such as focus fire, kiting, and terrain exploitation critical for success. The testbed defines suite of combat maps of varying difficulty (e.g., homogeneous battles like 3m and heterogeneous battles such as 2s3z), enabling systematic evaluation of algorithms across easy, hard, and super- hard scenarios. Reward signals are shaped by combat outcomes, including damage dealt, enemy kills, and victory, making SMACvl rigorous benchmark for addressing key MARL. challenges such as credit assignment, cooperation under partial observability, and scalability to larger teams. Our evaluation focuses five SMAC maps: 3m, 8m, 2s3z, 5m_vs $m.vs.6m 6m, and 2c.vs v8.64zg.
SMACv2 extends the original SMAC benchmark to provide a more robust and challenging testbed for MARL. Unlike SMACv1, where difficulty mainly arises from heterogeneous unit compositions and map layouts, SMACv2 introduces environment stochasticity and increased diversity in scenarios to better approximate real-world complexity. In particular, unit placements, initial health, and enemy strategies randomized across episodes, requiring policies that generalize beyond fixed configurations. The benchmark also rebalances reward signals to reduce overfitting to deterministic strategies and to encourage learning more adaptive coordination behaviors. By encompassing broader range of maps and stochastic battle conditions, SMACv2 provides more rigorous evaluation of MARL algorithms in terms of robustness, generalization, and sample efficiency. Our evaluation focuses on three SMAC maps: terran 5 vs_5, terran 10_ Ss_ 10, and zerg_5_vs.55
SMAC datasets. Experiments utilized datasets from the off-the- the-grid offline dataset (Formanek et al., 2023), which provides offline trajectories for SMACv1 and SMACv2. The benchmark contains variety of trajectories generated by different policies with varying quality, including Good, Medium, and Random behaviors, thereby covering wide performance spectrum (whereas there is only the Replay dataset for SMACv2). Each dataset consists of observations, actions, legal action information, and rewards, following the decentralized agent structure of SMAC.
G.2 MA-MuJoCo
MA-MuJoCo is continuous-contre benchmark that extends the classical MuJoCo locomotion suite to multi-agent settings (Peng et al., 2021). In MA-MuJoCo, single robot, e.g., Halfcheetah, Hopper, and Ant, is decomposed into multiple controllable parts, each assigned to an individual learning agent. Each agent receives local observations corresponding to its controlled joints and must produce continuous actions to coordinate with others for effective global locomotion. Rewards typically shared among all agents based on the overall task performance, creating a cooperative continuous- action control problem. MA-MuJoCo focuses fine- aineggrinedd coordination in high-dimensional continuous dynamics, making it complementary for evaluating scalability and cooperation in MARL..
Dataset. For offline MARL, we employ the dataset collected by Wang et al. (2023). The dataset provides trajectories collected under a variety of policies with different quality levels, including Expert, Medium, Medium-Expert, and Medium- thus covering wide spectrum of data distributions for 6-Halfcheetah, 3-Hopper, and 2-Ant. Heading number is the number of agents.
G.3 MPE
MPE, originally introduced by Lowe et al. (2017) as a suite of simple particle-basee -baiilee-baedd worlds with continuous observation-action space and basic simulated physics, serves foundational benchmark for cooperative and competitive MARL tasks. features two-dimensional scenarios where agents move, interact, communicate, and observe each other within partially observable setting. This
24

---

Published as a conference paper at ICLR 2026
emphasizes coordination, communication, and emergent behaviors in multi-agent settings. Common scenarios include Cooperative Navigation (also known as Spread), where agents must cover landmarks while avoiding collisions; Predator-Prey, where predator agents pursue and capture prey agents; and World tasks, which involve more complex interactions like gathering resources or navigating with environmental elements. These environments widely adopted for their scalability, ease of customization, and ability to test algorithms on communication-oricnni MPE is typically normalized using the following equation: 100 (S Srando)) (Seepet Srandom). where S' is the score of the evaluated policy, Standom is the performance from a random policy (159.8), and Sespen the performance of an expert-level policy (516.8).
problems, Performance in
Dataset and Scenario Selection. In our experiments, we focused solely on the Spread scenario dataset (including Expert, Medium, Medium-Replay, and Random), as implemented in a JAX-based framework to ensure efficient computation and compatibility with modern acceleration tools. This choice was necessitated by challenges in accessing the Predator- Prey (PP) and World (WD) datasets, despite their use in prior works by Pan et al. (2022) for offline MARL. Furthermore, adapting customized environments for PP and WD proved difficult due to the requirement loading pre- trained policies, compounded by limited documentation integration processes, which hindered compatibility with widely used pre-trained models (Formanek et al., 2024).
G.4 HYPERPARAMETERS
Hyperparameter
Value
Gradient steps
10
(SMACv1 and SMACv2), 2 x 10S (MA-MuJoCo and MPE)
Batch size
64
Flow step
10
BC coefficient
3.0
Network configuration
[512, 512, 512,
Polyak averaging coefficient
0.005
Discount factor
0.995
Optimizer epsilon
10-5
Weight decay
0
Policy learning rate
3 10
Value learning rate
3 x 10-1
Layer normalization
True
Optimizer
Adam
25

---

Published as a conference paper at ICLR 2026
H ADDITIONAL RESULTS
This section provides supplementary analyses that further complement the main results presented in the paper. Specifically, we first include several additional ablation studies that examine the robustness of our framework under varying design choices, such alternative policy parameterizations and critic configurations. Second, we check the training time of our approach and baselines. Third, we report the full learning curves of MAC-Flow corresponding to the performance tables in the main text.
H.1
ABLATION STUDY
Individual with Centralized Q
MA-MujoCo (Scenario: Halfcheetah.)
Expert
Medium-
Medium
Medium-Replay
Perforrance
4.0
Perforsance
4.0
Perfomance
3.0
2.8-
2.8
2.9
1.8
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Steps
1.0
Steps
Steps
Figure 8: Ablation on critic configuration. This ablation shows the differences between individual Q training under IGM and centralized Q training. The reported point and shaded area represent the average and tolerance interval from 6 random seeds.
Individual Q with IGM vs. Centralized Q. In Figure 8, across all MA-MuJoCo datasets, the individual Qs based on IGM consistently outperform the centralized counterpart Q. While the centralized variants often suffer from lower stability and suboptimal convergence, the individual Q based on IGM achieves higher performance and maintains table learning curves. This demonstrates that the IGM formulation enables reliable value estimation in multi-agent settings by preserving individual-glan -lobbll consistency, which is critical for cooperative policy learning. We conjecture that such empirical observations originate from a representation bottleneck of simple MI.P networks. Specifically, the centralized critic should extract coordination patterns from the joint observation- action space. In practice, shallow MLP architectures struggle to capture such high-order dependencies, leading to severe information compression and bottleneck in representing coordination. To sum the limitations observed in centralized critics likely stem not from the principle of centralization itself, but from the restricted capacity of MLP-based function approximators when faced with large joint inservation-actiinn spaces.
MAC-Flow - FMC-Flow with IndIvidual Flow
MA-MuJoCo (Scenario: Halfcheetah.
Expert
Medium- Medium-Expert
Medium
Medium-Replay
4.0
Performance
Performance
Peri Frforrnnce
4.8
Porformance
2.0
2.0
2.0
2.0
0.0
8.0
0.0
1.0
0.5 Steps
1.0
0.0
0.5
0.9
8.5 Steps
1.0
Steps
Steps
Figure 9: Ablation on policy type in Stage I. This ablation shows the differences between individual flow policies and joint flow policy for stage I. The reported point and shaded area represent the average and tolerance interval from 6 random seeds.
Individual BC Flow Joint BC Flow (Stage D). Figure 9 investigates the effect of policy configuration in Stage by comparing individual flow policies against our joint flow policy. Across all MA-MuJoCo datasets, our basic configuration and its variant show stable and powerful performance.
26

---

Published as a conference paper at ICLR 2026
Interestingly, unlike the centralized Q ablation, the joint flow policy does not entirely collapse. We attribute this difference to the expressive capacity of flow-macching methods, which are specially designed approximate complex distributions. Whereas centralized critics implemented with shallow MI.ss face a severe representation bottleneck when mapping from joint observation-actioo inputs, flow-based policies retain stronger inductive biases for capturing distributional structure. This expressiveness enables joint flows to remain viable in principle, though their optimization often remains more challenging than individual flows. Comprehensively, given the trade-offs, we opt for the joint policy to enhance sample efficiency, reduce memory usage (with respect to network load), and facilitate faster training.
MAC- FLow
MAFQL
MA-MuloCo
MPE
SMACV1
SMACx2 Replay.
Spread Medium.
5.0
2.0
Performance
4.0
Performance
Performance
Perfornance
2.5
1.0
1.8
2.0
0.0
0.0
0.0
0.0
8.5
Steps
Steps
Steps
Stops
Figure 10: Performance comparison with MA-FQL. We compare solution and the naive extension of the FQL across four benchmarks. The reported point and shaded represent the average and tolerance interval from random seeds.
MAC-Flow vs. MA-FQL. Figure 10 compares the performance between our proposed solution and the multi-agent extension of FQL (Park et al., 2025), which is a close connection with our work. Comparing MAC-Flow against MA-FQL highlights that the transition from single-agent to multi-agent system is not trivial. In MA-MuloCo, MPF, and SMAC benchmarks, MAC-F ow substantially outperforms MA-FQL in both convergence speed and final performance. In contrast, MA-FQL either stagnates at suboptimal levels or exhibits unstable progress, particularly in more complex environments. Note that we can simply put that FQL is fully decentralized version of MAC MAC-Flow, Flow, especially in decentralized Q without IGM and flow decentralized policy.
H.2
TRAINING TIME
Discrete control
Continuous control
Training Tire (harr
Trasning (hour)
MAC Flow
MAC MAC-FFoo FLow
DoF
NAD ff
MABCQ
MACOL
(OMIGA
MACOL
ipc
Figure 11: Wall-clock time for training, Reported numbers are measured on average of all tasks, which are included in discrete or continuous benchmarks.
We report the wall-clock training time for both discrete and continuous control benchmarks in Figure 11. MAC-Flow achieves substantially lower training time compared to diffusion-basee methods such as DoF and MADifÉ. In discrete control, MAC-Flow trains nearly an order of magnitude faster than MADiff, while in continuous control, it also outperforms strong baselines like MA-CQU or OMAR, which require many bootstrapping steps. These results confirm that the one-step flow formulation of MAC-F) F1 ow yields not only efficient inference but also significantly reduced training cost. Note that COL is implemented with independent Q learning (De Witt et al., 2020) in discrete control, unlike continuous control.
27

---

Published as a conference paper at ICLR 2026
H.3 LEARNING CURVES MAC-FLOW
Anc Expert
Art Medium Rapert
Art Mediun
Anc Medium Mesliun-Rpsky
Performance
2000
Performance
1.00
Performance
1420
1500
1508
1208
1250
1.000
2009
1000
0.3
0.5
0.5
Steps
Steps
Steps
Steps
Hupper Expert
Meppec
Hopper
Happer Madiaes
Perforsmce
3000
3008
Perforaince
Perffor Performine
2000
2000
500
1800
1000
1000
0.5
Hlalichutaa's Expert
Vediar
Hatcharah Medium
4000
Nefformnc
Purforsunce
Perforannce
4000
Perforracc
4000
2000
2000
2000
0.5
0.5
MPE
sore, Molton
Performance
Performance
Performance
Performance
200
50
50
0.0
0.5
2.5
1.0
6.5 Steps
1.0
0.0
0 Steps
Steps
Steps
Ten- Good
Fra Croof
3Gl--Good
vi. Good
temon 3t wiu Seep-yy
Par Formerncce
18
20
20
Per 2
0.0
0.5
1.8
0.0
6.5 Stops
6.5
1.0
6.5
6.5 Stops
1.8
Steps
Stops
Stops
Smn Maliam
fer. Medium
Lale Malium
Maduur
20
forrunce
Per Performance
10
sey, S-mppay
Nur For sance
18
15
0.5 Steps
8.5 Stops
1.9
Stopa
Stopa
Figure 12: Full learning curves for MAC-Flow.
28

---

Published as a conference paper at ICLR 2026
0.2-
2.0
0.5
8.0
0.0
1.6
1.6
1:9
-1.6
-1.
-1.6
1.5
1.0
1.6
1.0
0.0
0.0
-1.4
1.0
Figure 13: Landmark covering game visualization. (Top) Trajectories sampled from the dataset. (Middle) Trajectories sampled from the flow joint policy. (Bottom) Trajectories sampled from the individual policy.
H.4 LANDMARK COVERING GAME: SCALING UP THE NUMBER OF AGENTS
To further examine how MAC-Flow scales with the number of agents, extend the landmark- covering game (used Section 4.3) up to 40 agents from three agents. We generate an offline dataset consisting of 50 trajectories, and train both the joint flow model and the distilled one-step policies for 1000 iterations with 64 batch size.
Figure 13 visualizes the sampled trajectories from the dataset, flow joint policy, and individual policies, respectively. Up to 40 agents, the flow joint policy successfully captures the multi-agent's joint action distribution and the appropriate partitioning of agents across landmarks model occasionally produces slightly dispersed trajectories, given our training setup, this could be mitigated by scaling up the hyperparameter. Next, the individual policies learn clean trajectories, since each agent only needs to reproduce its own trajectories, which is similar to expert demonstrations.
mark). Although the flow
Summary. This scaling analysis confirms that MAC-Flow continues capture stable coordination up to 40agents, with this model capturing joint structure. The results indicate that MAC-Flow can retain its effectiveness (according flow policy) even as multi-agent dimensionality grows.
29

---

Published as a conference paper at ICLR 2026
0.01
0.50
0.02
0.47
0.14
0.23
0.24
0.28
Action
Action
Action
Action
0.46
0.04
0.43
0.08
0.21
0.42
0.21
0.27
Action
Action
Action
Action
joint policy
One-step policy
One-step policy
Figure 14: Policy comparison in pure coordination game, (a) Datasct distribution. (b) Distribution of flow
joint policy. (c) Distribution of one-step policy with IGM. (d) Distribution of one-step policy without IGM.
H.S PURE COORDINATION GAME: BETTER POINT THAN BC GENERATIVE MODELING
This section presents setting where IGM offers clear advantage over BC-style generative modeling. This experiment isolates a simple yet revealing coordination structure where value information plays decisive role in recovering the optimal joint behavior, whereas BC alone fundamentally fails due dataset imbalance.
minimal pure coordination game. We consider two-agent binary-action environment with unique high-value coordinated action, (1, 1). yielding reward +22 All partially coordinated actions, (1. 0) and (0, 1). give lower reward +1, while (0, 0) yields zero. Critically, the offline dataset is intentionally biased, that is, it consists almost of the two asymmetric suboptimal modes (0, 1) and (1, 0), with only small fraction of rare samples (1, 1) and (0, 0). This generates a combinatorial ambiguity that challenges BC-style modeling, which must extrapolate optimal coordination from extremely weak statistical evidence (Figure 14 (a)).
Flow joint policy vs. MAC-Flow vs. Factored policy without IGM. In this setup highlights common failure pattern BC generative modeling (Flow joint policy). Because BC methods optimize likelihood without considering value structure, the learned joint flow simply reproduces the dominant dataset frequencies, faithfully capturing (0, 1) and (1, 0) while ignoring the much rarer yet more valuable (1, 1) mode. Even though this expressive enough to represent the correct distribution, its objective provides no incentive to amplify the crucial but low-frequency coordination (Figure 14 (b)).
In contrast, introducing IGM during distillation (MAC-Flow) substantially changes the learning dynamics. When the joint flow is mapped into factorized per-agent policies through value-guided guided IGM constraints, each agent receives consistent signal that selecting the action is slightly more aligned with high-value coordination, even though this is rarefied in the dataset. The IGM mechanism exploits these weak cues by forcing individual policies agree on action choices that maximize the centralized value function, effectively amplifying faint optimality information that pure BC cannot utilize. As a result, the distilled factorized policy concentrates significantly more probability mass on the optimal (1, configuration (Figure 14 (c)).
Whereas the version without IGM collapses toward an almost uniform product distribution that fails to represent any coordinated behavior. In this case, each agent's 's marginal policy distributes probability mass nearly evenly over its two actions, causing the joint distribution to degenerate into four nearly equal-probability modes (Figure 14 (d)).
Summary. This example shows when IGM is strictly better than pure BC modeling. In settings where demonstrations contain only weak hints of coordinated optimal behavior, the IGM mechanism reliably extracts and amplifies value-relevant structure that generative models alone cannot recover. This controlled setup thus clarifies why MAC-Flow's value-guided distillation often outperforms BC-based generative modeling in more complex multi-agent domains.
30

---

Published as a conference paper at ICLR 2026
Disribution dataset
Distribution of joint policy
Distribution of factorized policies
1.0
1.0
1.0
8.5
8.5
0.5
0.0
9.0
0.0
-8.5
-0.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
a1
a1
Figure 15: XOR toy environment. (Left)The dataset contains two anti-aligned modes. (Center) The joint flow recovers this non-factorizable structure, (Right) Factorized policies collapse into an almost uniform product distribution.
H.6 XOR STRESS TEST: FAILURE MODE ANALYSIS
Although we provide theoretical justification for our bound, this subsection presents empirical counterexamples for transparency and trust, demonstrating how the method behaves in environments that intentionally violate the assumptions underlying the theory.
When does IGM break? We analyze settings where the IGM factorization is provably invalid. While MAC-Flow performs well in typical cooperative tasks, separability could fail under strong inter-agent coupling. To build trust in our theoretical claims, we present controlled stress test where factorization guaranteed to break.
XOR coordination examples. We introduce minimal two-agent continuous XOR task in which optimal joint actions are anti-aligned, e.g., (-1,+1) and (+1,-1) Because the reward depends entirely the relative actions of the agents, per-agent Q-function recover the optimal strategy independently, making IGM mathematically impossible in this domain. This setting provides a clean, analyzable failure case for coint-to-factorieen policy distillation.
Empirical counterexamples. We train the MAC- Flow joint policy on an offline dataset exhibiting two sharp, disconnected high-density XOR modes. The learned flow successfully reconstructs these modes and captures their fundamentally non-separable geometry, confirming that the Now-matching stage is expressive enough model complex, multi-modal joint behaviors. However, when this joint flow is distilled into per-agent policies, the factorized representation collapses into a near-independent product distribution, failing reproduce the anti- anti-aligned high-density regions and instead placing probability mass around the center of the action space. This degradation arises despite the joint model being correct, indicating that the failure is due to the intrinsic non-separability of the coordination structure rather than optimization or training artifacts.
Summary. The XOR stress test provides exactly the concrete failure mode. In this environment, the optimal joint action is intrinsically anti-alige aliina,, causing the assumptions behind IGM factorization to break. As a result, value bounds cease to be predictive because the global optimum cannot be decomposed into per-agent optima. Consequently, MAC-Fl underperforms not due to model capacity or training issucs, but because the coordination structure is fundamentally non-separable. This controlled setting, therefore, offers transparent counterexample.
31

---

Published as a conference paper at ICLR 2026
Interaction strength
(a)
Weak
Strong
:eee
9:52
6:31
0.25
9:34
9.15
0.90
Dalasel
0.00
1.14
0.24
0.35
9.20
0.32
(b)
DebatBill pallo wlly
Joint policy
0.03
0.30
0.39
0.24
0.32
0.19
0.02
Ratur
actiin
0.50
0.50
1.14
8.25
0.37
0.51
1.0:
9.9
0.5
1.0
strengt
(e)
Individual policy
0.27
3.20
5.20
82
5.17
0.01
0.20
00
0.02
3.68
1.50
17
15
8. 84
75
60
92
3.56
0.8
8.5
3.0
Action
Action
Act:con
Action
Action
Interaction Streugth
Figure 16: Payoff game according to interaction strength. (a) Sampled distributions. Return differences.
(c) Wasserstein distance hetween joint and individual policies. Correlation between the dataset and joint policy.
H.7 PAYOFF GAME: ANALYSIS ACCORDING TO INTERACTION STRENGTH
To validate the value gap and W2, we design pay-off game according to the interaction strength, as an extension the pure coordination game in Appendix H.S. The MDP setup is the same as the pure coordination game: each agent can select an action in {0, 1}, and the team reward is defined as (0,0) = 0, (1,0) or (0,1) (1, 2.
How do define interaction? In this example, we define interaction strength as the degree to which one agent's 's optimal action depends on the other agent's action, or equivalently, how far the joint action distribution deviates from factorized form p(a' a²) p(a' )p(("). Under this view, diagonal joint actions (0, 0) and (1, represent strong interaction, because the optimal behavior requires perfectly matching actions and thus induces full coupling. In contrast, off-diagonal actions (1, and (0, 1) correspond to weak interaction, where each agent's 's marginal behavior remains nearly independent. To probe this, we generate datasets by mixing diagonal and off-diagonal modes with controllable interaction parameter Ç [].
= (00,)) 0), (1, 1)) (1 () x ((0, 1), (1,
Herein, a = 0 yields purely weak interaction, and 1 yields purely strong interaction.
Results. Figure 16 compares the flow joint policy and individual policies trained under increasing interaction strengths ((0.00, 0.25, 0.50, 0.75, 1.00}) using the sampled joint-action matrices, the return, and the corresponding Wasserstein distance W2. In Figure 16 (a), across all interaction levels, the flow joint model reproduces the offline dataset distribution. It means that the BC flow learning is effectively capturing the observed joint-action modes. On the other hand, the individual policy shows different pattern driven by the IGM- based Q maximization rather than simply BC. As interaction increases, the IGM objective encourages the individual policies to concentrate probability mass on the optimal joint actions rather than merely reproducing the empirical frequencies.
To quantify its differences, Figure 16 (b) clarifies how interaction strength influences the learned policies. The flow joint policy achieves the near-optimal reward of approximately 1.0, matching the reward level the dataset since it directly models the full joint action distribution. In contrast, the return of the individual policy increases gradually as the interaction strength rises. This occurs because the number of samples in the optimal condition increases, thereby making the conditions for identifying coordinated behaviors more pronounced. Next, Figure 16 (c) confirms that W2 rises with the interaction strength, reflecting the growing structural mismatch between the expressive joint flow and the restricted factored representation.
Summary. As interaction strength increases, the Wasserstein distance W2 between the joint and individual policies grows, reflecting the structural mismatch that factorized policies cannot eliminate.
32


Reviews:
[{"review_id": "aMRoCQDaQH", "reviewer": "Reviewer_u2dH", "summary": "This paper introduces MAC-Flow, a framework for multi-agent coordination that learns a flow-based joint policy from offline data and distills it into decentralized one-step policies to balance performance and inference speed. It achieves ~14.5× faster inference than diffusion-based MARL methods across 4 benchmarks while maintaining good performance, with ablation studies confirming the value of its two-stage strategy.", "questions": "1. Can you provide some explanations on why MAC-Flow generally underperforms compared to DoF on SMACv2? Does it mean that individual policies are unreliable when facing complex and stochastic environments?\n2. How is the training cost of MAC-Flow compared to other baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. MAC-Flow effectively balances the trade-off between multi-agent coordination performance and inference speed.\n2. The resulting individual policy supports seamless offline-to-online fine-tuning.\n3. The ablation studies show the components of MAC-Flow are effective.", "weaknesses": "1. There's still a small performance gap compared to diffusion policies (DoF), especially on SMACv2, showing room for improvement in handling highly stochastic multi-agent environments.\n2. It requires offline datasets with diverse joint behaviors for effective training, and its performance may degrade when using low-quality or limited offline data.\n3. The baselines for continuous control are weaker due to the absence of diffusion and flow-based policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZe06Z54oq", "reviewer": "Reviewer_4G4R", "summary": "This paper tackles a long standing tension in offline multi agent RL between expressive centralized training and fast decentralized execution. The authors first learn a centralized joint policy over all agents’ actions via flow matching, then distill it into one step per agent policies suitable for test time. Empirically, MAC-Flow performs competitively on standard cooperative benchmarks while delivering large gains in inference speed relative to diffusion style generators, and it stays comparable to simpler Gaussian MARL baselines in wall clock.", "questions": "1. Could you include tasks where interaction strength is dialed up and report both $W_2\\big(\\pi^{\\text{joint}},\\prod_i \\pi_i\\big)$ and the corresponding return gaps?\n2. Can you provide training cost curves and memory footprints as functions of agent count and action dimension, along with sensitivity to the number of flow integration steps during training. A head to head with an autoregressive joint policy would be especially informative.\n3. How does MAC-Flow adapt when observation is local rather than global? If policies condition on $o_i$ at test time while the joint flow used $s$ and a central critic at training time, do your guarantees change, and what additional assumptions would be needed for the value gap bounds to carry over?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learning a rich joint policy with flow matching, then distilling to per agent policies, directly addresses the coordination vs speed trade off that many of us have run into in offline MARL. I think the training to deployment narrative is easy to follow and feels usable.\n2. The comparisons include both diffusion based policies and conventional MARL baselines, and the results highlight a strong reduction in inference latency while retaining returns. Given how often test time latency matters in multi agent control, this is an impactful result.\n3. The writing is clear, the algorithmic pipeline is spelled out, and there are helpful ablations that point to which pieces do the heavy lifting.", "weaknesses": "1. The reliance on an individual global max style factorization is an obvious pressure point. The paper would be stronger with stress tests on heavily coupled tasks where separability breaks down. Bounds are good, but concrete counterexamples or failure modes would build trust.\n2. The link from small $W_2$ to small value loss hinges on a Lipschitz $Q_{tot}$ and on distributional proximity that may not hold uniformly. It would help to see empirical measurements of these quantities during training to show that the assumptions are not only theoretically convenient but also reasonably satisfied.\n3. Most of the complexity discussion focuses on test time. Training a flow over joint actions can be expensive, and the main text gives less visibility into training wall clock, memory, and scaling with agent count and action dimension. I want to know the total cost to get the speedups.\n4. It is not clear how the approach behaves under partial observability, larger discrete alphabets, or mixed cooperative competitive games. Even a brief study in one of these regimes would help position the method more broadly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEmpULTMcm", "reviewer": "Reviewer_Z5Fb", "summary": "The paper proposes MAC-Flow, an offline MARL framework that (1) learns an expressive flow-matching–based joint policy via behavioral cloning, then (2) factorizes / distills it into decentralized one-step policies optimized with a behavioral-regularized actor–critic objective and the Individual-Global-Max (IGM) principle. The pitch is to retain diffusion-like expressiveness for multi-modal joint action distributions while achieving Gaussian-policy–like inference speed. Experiments span different dataset, reporting faster inference vs diffusion MARL at comparable performance, and similar speed to Gaussian baselines", "questions": "1. Under what assumptions does the factorized one-step policy set preserve the argmax consistency with the flow-learned joint policy? Any error decomposition (e.g., from flow fitting + distillation + value learning) that bounds sub-optimality? \n\n2. Can you report MI(agent i; agent j | o) or diversity / mode-coverage metrics comparing joint flow vs distilled policies, and ablate IGM on/off?\n\n3. How does the policy behave under partial-observability stress (masking % of features) or opponent distribution shift?\n\n4. Provide cases where MAC-Flow under-performs Gaussian or diffusion policies—what patterns are not captured by the distilled factorization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly identifies—and structures the solution around—the core trade-off in offline MARL: diffusion policies capture multi-modal joint behaviors but are slow, while Gaussian one-step policies are fast but brittle for coordination.\n\n2. The paper connects to flow-matching literature and positions MAC-Flow as a MARL counterpart to single-agent flow-distillation and Flow Q-Learning, showing conceptual continuity with recent advances.\n\n3. The two-stage design is guided by the Individual-Global-Max principle, aligning per-agent policies with the global optimum and providing a standard, scalable route to decentralized execution\n\n3. On four benchmarks, MAC-Flow demonstrates that you can retain coordination performance while achieving faster inference than diffusion-based MARL—matching Gaussian-policy speed without its expressiveness limits.", "weaknesses": "1. The method references “mathematical guarantees” around joint-to-factorized policy learning with IGM, but the paper (as given) does not present a formal theorem / conditions under which the distilled one-step policies provably preserve the global optimum of the learned joint flow. \n\n2. While the authors claim flow matching combines diffusion’s expressiveness and Gaussian’s speed, the paper lacks a deeper justification for why flow matching provides better inductive bias or coordination representation in MARL. There is no theoretical or empirical comparison against diffusion-based MARL such as Graph Diffusion for Robust Multi-Agent Coordination (2025).\n\n3. The evaluation omits stress tests under partial observability, agent scaling, and dynamic teammate shifts—scenarios where flow-based policies might degrade.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)