Multi-Agent Imitation by Learning and Sampling from Factorized Soft Q-Function
Authors: Yi-Chen Li
Zhongxiang Ling
Tao Jiang
Fuxiang Zhang
Pengyuan Wang
Lei Yuan
Zongzhang Zhang
Yang Yu
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent Imitation Learning
Energy-Based Models
Inverse Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Learning from multi-agent expert demonstrations, known as Multi-Agent Imitation Learning (MAIL), provides a promising approach to sequential decision-making. However, existing MAIL methods including Behavior Cloning (BC) and Adversarial Imitation Learning (AIL) face significant challenges: BC suffers from the compounding error issue, while the very nature of adversarial optimization makes AIL prone to instability. In this work, we propose \textbf{M}ulti-\textbf{A}gent imitation by learning and sampling from \textbf{F}actor\textbf{I}zed \textbf{S}oft Q-function (MAFIS), a novel method that addresses these limitations for both online and offline MAIL settings. Built upon the single-agent IQ-Learn framework, MAFIS introduces the value decomposition network to factorize the imitation objective at agent level, thus enabling scalable training for multi-agent systems. Moreover, we observe that the soft Q-function implicitly defines the optimal policy as an energy-based model, from which we can sample actions via stochastic gradient Langevin dynamics. This allows us to estimate the gradient of the factorized optimization objective for continuous control tasks, avoiding the adversarial optimization between the soft Q-function and the policy required by prior work. By doing so, we obtain a tractable and \emph{non-adversarial} objective for both discrete and continuous multi-agent control. Experiments on common benchmarks including the discrete control tasks StarCraft Multi-Agent Challenge v2 (SMACv2), Gold Miner, and Multi Particle Environments (MPE), as well as the continuous control task Multi-Agent MuJoCo (MaMuJoCo), demonstrate that MAFIS achieves superior performance compared with baselines. Our code is available at https://github.com/LAMDA-RL/MAFIS.

Multi-Agent Imitation by Learning and Sampling from Factorized Soft Q-Function
Yi-Chen Li Zhongxiang Ling!.2, Tao Jiangl.2.4 Fuxiang Zhang Pengyuan Wangh.
Lei Yuan 12,,3 Zongzhang Zhang 1.2 Yang Yu 1,24+
National Key Laboratory for Novel Software Technology. Nanjing University, China,
School of Artificial Intelligence, Nanjing University, Nanjing, China, 3 Nanyang Technological University, Singapore, Polixir Technologies, Nanjing, China,
liyc@lamda. njj.. edu. en, lingzx@smail wnw..njju....... edu. en, fuxiang00100 ntu, edu. sg (wangpy, yuan1]@l @lamda. nju. edu. en, zzzhangonju. werpenggnuueuu..... edu. cn, yuyenju. edu. edu. cn
Abstract
Learning from multi-agent expert demonstrations, known Multi-Agent Imitation Learning (MAIL), provides promising approach to sequential decision-making. However, existing MAIL methods including Behavior Cloning (BC) and Adver- sarial Imitation Learning (AIL) face significant challenges: BC suffers from the compounding error issue, while the very nature of adversarial optimization makes AlL prone to instability. In this work, we propose Multi-Agent imitation by learn- ing and sampling from FactorIzed Soft Q-function (MAFIS), a novel method that addresses these limitations for both online and offline MAIL settings. Built upon the single-agent IQ-Leam framework, MAFIS introduces the value decomposition network to factorize the imitation objective at agent level, thus enabling scalable training for multi-agent systems. Moreover, we observe that the soft Q-function implicitly defines the optimal policy as an energy -neegy-basedd model, from which we can sample actions via stochastic gradient Langevin dynamics. This allows us estimate the gradient of the factorized optimization objective for continuous control tasks, avoiding the adversarial optimization between the soft Q-function and the pol- icy required by prior work. By doing we obtain a tractable and non-adversarial objective for both discrete and continuous multi-agent control. Experiments on common benchmarks including the discrete control tasks StarCraft Multi-Agen Challenge v2 (SMACv2), Gold Miner, and Multi Particle Environments (MPE), as well the continuous control task Multi-Agent MuloCo (MaMuJoCo), demon- strate that MAFIS achieves superior performance compared with baselines. Our code is available https: //t::/giittt gtgggttubbb... LALLLLLAAAAAAAMAAAAAAA
Introduction
Multi-Agent Imitation Learning (MAIL) focuses on learning policies from expert demonstrations. Compared Multi- Multi-Agent Reinforcement Learning (MARL) that requires repeated and tedious design of reward functions [49], MAIL provides an efficient solution for optimal sequential decision-makinn making. especially considering that collecting expert demonstrations is often easier and faster than designing reward functions [1]. Applications of MAIL over diverse domains such as driving simulation [4], unmanned aerial vehicles deployment [44] and robotics control [42] has shown great potential.
large body of previous works on MAIL can be categorized into two approaches: Behavioral Cloning (BC) [22, 50, 42] and Adversarial Imitation Learning (GAIL) [48, 19, 36). BC reduces MAIL to
"Conesponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

supervised learning problem, whereas AlL trains a pair of generator and discriminator such that the discriminator can not distinguish behaviors of the generator from demonstrations. However, both approaches have their own limitations. Behavioral cloning suffers from compounding error, where small prediction mistakes accumulate over time as the model encounters states was not trained on [33, 47]. AII., due to the inherent nature of adversarial optimization, often suffers from training instability and sensitivity to
[25].
Recently, Garg et al. [17] propose a single-agent Imitation Learning (IL) framework, named as IQ-Leam. Although consistent with AlL's original modeling of IL, IQ-Leamn reformulates as an optimization problem over the Q-function and the policy. Furthermore, Garg et al. [17] discover that the optimal policy can be expressed in closed form through the given Q-function. This leads to a tractable objective in discrete control tasks that requires optimizing only the Q-function. However, we cannot directly apply IQ-Learr to MAIL due to two key obstacles. First, as IQ-Learr is designed for single-agent tasks, applying it to MAIL by treating multiple agents as single entity will lead to inefficient training and make it unsuitable for decentralized execution. Second, the closed-form solution for the policy requires calculating the logsumexp of Q values, which is computationally intractable in continuous control tasks. IQ-Learn addresses this by adversarially optimizing the policy and Q-function, which however has shown to be empirically unstable and may even fail [3]. A new MAIL method with strong generalization ability and enhanced stability is in high demand.
To deal with the aforementioned challenges, this paper proposes Multi-Agent imitation by learning and sampling from FactorIzed Soft Q-function, abbreviated as MAFIS. We start by adapting IQ-Learn to MAIL tasks via replacing the single-agent policy and soft Q-function with the joint policy and global soft Q-function, respectively. Inspired by the value decomposition network [38, 31J, we then consider representing the global soft Q-function as weighted sum of individual soft Q-functions. With careful derivation, we find that the objective can now be factorized at the agent level, thus enabling scalable training and decentralized execution in multi-agent systems. However, it requires calculating the logsumexp of Q values over each agent's action space, which intractable for continuous control tasks. Our key insight is that we can still estimate the gradient of the objective as long as we can sample actions from the optimal policy. Additionally, we observe that when given soft Q-function Q, the optimal policy has closed-form expression as an Energy- Based Model (EBM) [37] with -Q being the energy function. Actions can be sampled from the EBM using stochastic gradient Langevin dynamics [45]. Built upon the above improvements, we successfully obtain MAFIS, a unified online and offline MAIL framework for both discrete and continuous control. To validate the effectiveness of MAFIS, we conduct extensive experiments benchmarks including the discrete control tasks StarCraft Multi-Agent Challenge v2 (SMACv2) [13], Gold Miner [15], and Multi Particle Environments (MPE) [27], well as the continuous control task Multi-Agent MuJoCo (MaMuJoCo) [11], the result of which shows that MAFIS achieves superior performance compared with baseline methods.
common
2 Related Work
Multi-Agent Imitation Learning Due to its promise to turn small expert dataset into powerful decision making engine, Multi-Agent Imitation Learning (MAIL) has attracted broad attention recently. Le al. [22] proposes to learn latent probabilistic coordination model from demonstrations and then use BC to output a group of imitation polices conditioned on the learned embedding of
coordination and local observations. Zhan et al. [50] generalizes this idea, using a hierarchical framework to generate the high-level intentions of each agent. Apart from BC, some studies model MAIL using the inverse reinforcement learning framework [53]. Song et al. [36] extends GAIL [21] to multi-agent scenarios where it learns group of discriminators for every agent. Yu et al. [48] borrows the idea of Fu et al. [16] to learn reward functions that are highly correlated with the ground truth rewards. Moreover, Gruver et al. [19] conditions the learned reward functions on a learned embedding representing agents' behaviors. Recently, Bui et al. [7] propose MIFQ, which is also based on the IQ-Learn framework. However, the optimal soft value function in MIFQ is represented differently than in our method. Moreover, it is mainly for discrete control tasks. A detailed discussion on the distinction between MIFQ and ours is presented in Appendix B.
Value Decomposition Network To efficiently learn an optimal Q function for decentralized execution with the temporal difference loss [39], researchers in MARL have proposed many kinds of
2

---

value decomposition methods. Sunehag et al. [38] decomposes the centralized value function Quo into a sum of local value functions Q', i E f1.2. n). Rashid al. [31] offer QMIX to represent Qior as a weighted sum of Q', where the weights are generated by mixing network satisfying the individual global-max principle. Son al. [35] transform the original Q into a new, easily factorizable one with the same optimal actions in both functions. Wang et al. [43] take a duplex ducling network architecture which encodes the IGM principle into the neural network architecture to factorize the joint value function. In this paper, we use a mixing network similar to that of QMIX, but other Q-function decomposition methods can also be compatible with MAFIS.
Energy-Basee Barrgy-BBsedddee Model Energy-based models (EBMs) are class of probabilistic models that define a probability distribution p(z) over data as a Gibbs distribution, p(x) x exp(-F(xx)} where F((r) is the energy function [23]. Unlike explicit probabilistic models, EBMs do not require exact computation of normalizing constants, but sampling and training often rely on approximate methods like Markov chain Monte Carlo [32] or Langevin dynamics [45]. EBMs are particularly flexible and capable of modeling complex dependencies between variables, and have shown promise in various domains,
including vision [12], language [46], and reinforcement learning [20. 8]. In the field of IL, there are also works that represent the policy as an EBM. Specifically, Florence et al. [14] propose implicit behavioral cloning, which represents the policy an mneryy-based model. They find that compared with explicitly modeling the policies. EBMs is better at modeling complex distributions, such as discontinuous multi- ones. However, their approach is limited to offline learning and they only consider single agent tasks. To the best of our knowledge, our work is the first MAII. method that samples from and learns policies from the perspective of EBMs.
3 Preliminaries
To support subsequent analysis, this section will present the necessary background including the problem formulation and key notation definitions. Unless otherwise specified, we will use bold symbols throughout to represent joint variables across all agents.
3.1 Cooperative MARL
We model a fully cooperative multi-agent task as a Markov game [24], which can be defined by a tuple M = SI A, P, r, Here, I = and A = I[ AA' is the joint action space where AA' is the action space for agent i. At time step t E N, each agent E I chooses an action a: E AA' at the global state E S, together forming a joint action a, = (a] a²
is the set of agents, S is the state space,
aa.) ). By executing action a, in the environment, the agents receive a shared reward r(s,, ax) and the environment transitions to the next state st+1 ~ P(- az). The goal of the agents is to find an optimal joint policy π that maximizes the expected discounted return E >'r(st, a.)]. where E [0, 1) is a discount factor and a ~ π (-(ss). Since independently updating each individual policy often results in poor convergence [10]. the Centralized Training with Decentrulized Execution (CTDE) paradigm has been predominantly adopted in current MARI. research. In CTDE, we assumes that the global states and the actions as well as policies of teammates are accessible during the training phase.
3.2 Imitation Learning
In real-world tasks, defining the reward function r can be difficult and tedious. By contrast, obtaining expert demonstrations is relatively easier [1]. Imitation Learning (IL) focuses on learning an optimal policy from expert demonstrations (trajectories generated by policy πE). without needing to know the underlying reward function r. Two main approaches are commonly used in II.: Behavioral Cloning (BC) [29] and Inverse Reinforcement Learning (IRL) [2]. BC casts IL a supervised learning problem over state-action pairs [29]. While simple, it suffers from the compounding error issue [33]. On the other hand, IRL tries to first recover the reward function from the expert demonstrations and then extracts an optimal policy with that reward function via reinforcement learning [39]. It has been theoretically proven that IRL can mitigate compounding errors, thereby better imitating the expert [47]. Following existing IRL work [21], we consider the maximum causal entropy IRL framework [52], the objective of which is shown as below:
max min (H(T) ( E [r(s, a))) E.a (r(ss
(1)
3

---

where (T) log [a]s)] ] is the y-ddicounted causal entropy [5] of the policy и.
3.3 IQ-Learn Framework
For a fixed policy и, let Q" be its soft Q-function defined as
(" a) ETTT Σ + aH(():
(2)
Here, a > is called the entropy weight, which determines the relative weight of return and entropy. Garg al. [17] find that there is an one-to-one correspondence between the sets of feasible and Q*, thus converting Equation (1) into the following new objective:
max min J Q) 2 -1 -o (V ED [o(((s,a) P
(3)
Q
where (T(s) 1s> (Q( a) a log is called the soft value function [20], and Po is the initial state distribution. The concave function Φ : R - IR serves as a regularizer for the soft Q-function. When é is chosen the identity function, i.e., x(x) x, Garg et al. [17] illustrate that the objective shown as in Equation (3) is essentially attempting to minimize the total variation distance [18] between the state-action distributions of the expert and the imitator.
Our Method
We now describe our method, Multi-Agent imitation by learning and sampling from FactorIzed Soft Q-function (MAFIS). We will demonstrate how the single-agent IQ-Learn framework can be extended both continuous and discrete multi-agent control tasks, enabling stable and efficient training while supporting decentralized execution. Algorithm summarizes the pseudo code of MAFIS.
4.1 Soft Q-Function Factorization
Given the expert demonstrations DE. a straightforward approach for MAIL is to cast as a single-agent learning problem. That is, treating the multiple agents single entity and learn- ing the joint policy π via Equation (3). However, this does not support decentralized execu- tion. Drawing inspiration from value decomposition networks [38, 31], we instead consider representing the joint soft Q-function Qlot(To, at) as a weighted sum of individual Q-functions (T) at), (7 at), (T"(T""a?)))
(To a.) Σ (F..)
where k: : S - R2 is the mixing network. For agent E I at time step / E N, of is its local observation and r| = (ot, agj aj " o illustration of the joint Q-function's 's network architecture is in Appendix C.I. With Qiat being represented as in Equation (4), we derive the following result.
T is its observation-action history. An
Proposition 4.1. For fixed joint i-function (T, a) and x(x) = x, the joint policy Riu (alT) that minimizes I (T,, Qiot) satisfies
TQual (alT) =
To (ol II
(5)
1=1
where
1 exp f (
To: (a'(7"))
(6)
Z,:
with Z,i = EnEa 2 a)}. Thus, have
max min J(H, ttat) = max I [TTrqo Qiot). Quo
Quot

---

Moreover, I [TTqion Qlot) can be further reduced as
12 Σ -a(1 -po [og 2ns] + (K() - -a log Zpir).
[TTrrot Q
1=1
(7)
Due space limitation, we defer the proof to Appendix A.1. Proposition 4.1 tells that the adversarial training of Q and π can be converted into a non-adversarial one by learning Q based on Equation (7), thereby avoiding the potential instability associated with adversarial training [25]. Another interesting observation is that although log Zr is originally defined in the joint action space whose computation grows exponentially with the number of agents n, it can be factorized thanks to the joint soft Q-function decomposition, thereby allowing scalable and efficient training.
A recent work. MIFQ [7], also considers extending IQ-Learn to multi-agent tasks. They pro- pose to decompose the joint soft Q-function as in Equation (4). Proposition 4.4 of MIFQ ap- pears similar to our above Proposition 4.1. However, they represent the individual optimal exp exp(Q((r",a') Enieat (xp(("(r,a')'' while we obtain that TTx (a'|r")
policy (a=)() as (o)(aa
Moreover, given the optimal policy, the soft value function V*(s) in Bui
Enedt ( (r)
et al. [s] (s) Ei=Ik' (s) log EM'EA (T'((
whereas we derive in Appendix A. that (*()) = E i=1 log E exp (s)( (1')(")"'a')) a.)). We additionally present more detailed discussion on the distinctions between MIFQ [7] and ours in Appendix B.
(T',
Although Equation (7) converts the adversarial objective in Equation (3) into a non-adversarial one, two obstacles remain. First, computing Z,, Eaeat exp (ss(ss)(((( a)} requires summing over the entire action space, which is intractable for continuous control tasks. Second, the individual optimal policy oF requires access the global state to compute 1et shown in Equation (6),
which can be inaccessible during execution. We will delve into them in the next part.
4.2 Optimal Policy as Sampling from Soft Q-Function
Because log Z, in Equation (7) is computationally intractable for continuous control tasks, a straight- forward approach to directly imitate the expert via Equation (3). That is, adversarially train the joint policy π and joint Q-function Qtot where a) - a log π (a|T)] approximated by sampling actions from the cur- rent policy However, experiments on single- agent tasks have shown that doing so can lead to noticeable instabilities and even failures [3].
(T) (nnno((l>)]'''tt(
MARS
Adveant Updaa
We conducted experiments on the Ant (2x4) task from the MaMuJoCo benchmark [11] to validate this idea. Specifically, we maintain an individual policy n' for each agent i E I, thus the joint policy T ( T) = I[;= x¹ <'a'(r')) The architecture of the O-function remains similar to that of QMIX [31]. However, since the tasks of MaMuJoCo have continuous action spaces, we modify each individual fuuuction to take the local history-action pair (instead of the local history only as in QMIX) as input and output the corresponding Q-value. We term this approach Adversarial Update. The empirical results presented in Figure 1 (averaged across five random seeds) reveal that the Adversarial Update fails to learn an effective imitation policy. New approaches to better handle imitation learning tasks with continuous action spaces needed.
Envennmedt Slage
Figure I: Performance comparison of MAFIS and Adversarial Update on the Ant (2x4) task from the MaMuJoCo benchmark.
Our key insight is that computing log Zy can be bypassed as long as we can sample actions from the optimal individual policy TQi. Let 0 denote the learnable parameters of the joint Q- function Qiot which includes the mixing network module & and the individual Q-function modules (Q), Q..... ... Q"). The following proposition holds.
5

---

Algorithm MAFIS
Initialize the joint Q-function Qooi expert demonstrations g, and total training steps T.
2: (Online only) Empty replay buffer B.
3: (Continuous control only) Langevin steps K and step size e, N samples for each estimation.
4: for step 1 1 to 7 do
5: (Online only) Sample transitions in the environment and store them in the buffer B.
6: if Discrete control then
7: Update Q via Equation (7).
8: else
9: Update Qio via Equation (8).
10: end if
11:
end for
Proposition 4.2. The gradient of I [TQQu N Qlot) with respect to is equal to
n
VoJ (Tiqio, Qla()
Σ (1 7 100 (1rg)) (o))
(8)
+ (TTTT,
V (k'(ks) (s) at)) - yBa
(k .
The proof is in Appendix A.2. Proposition 4.2 tells that update by performing gradient
ascent [51] on I (iQqot, Qiot), ), all we need is to sample batch of actions from π. But how? Given exp(-E(x]} data point æ, EBM defines its probability density p(a)) as p(æ)) Σ exp{=E(x)}' where E(-)
is called the energy function [37]. We observe that the policy Tq= (-1r) is inherently an Energy-Based Model (EBM) with (T,()) being the energy function. Thanks to Stochastic Gradient Langevin dynamics (SGLD) [45], we can iteratively refine samples K times via the following rule:
a+ aii + V a., ew,
2x
which has been proven that a ~ &qi (-I7) under certain conditions [54]. Here, >> 0 is the step size, w (0, o²) is the Gaussian noise with variance o² and a ~ NO, 1). We also conducted experiments on the same Ant (2x4) task to validate the effectiveness of this approach, where the expectation over iQ. in Equation (8) is estimated by parallelly sampling N actions?. As shown in Figure 1, our method MAFIS significantly outperforms Adversarial Update.
Decentralized Execution As demonstrated in Proposition 4.1, each agent i E Z must sample its own actions during the decentralized execution phase according to Tgi. However, Tei explicitly depends on the global state s, which is inaccessible during decentralized execution in partially observable environments. To resolve this incompatibility, we observe that ('(s)
Vs S. This means that the original policy Th= and its decentralized counterpart मq. (a|T) share identical maximizers. Here, (a/T) is constructed using only local action-observation history and defined as:
(T,,a EdEs exp (T),))))
* (a/r) A
Furthermore, to mitigate the stochasticity introduced by sampling, only the action with the highest Q-value among the N sampled actions from न (al7) will be selected for execution. By adopting oo (alT) for action sampling, successfully eliminate the need for ks(ss in the decentralized execution phase while preserving the optimality of the selected action.
Experiments
In this section, we will validate the effectiveness of MAFIS through extensive experiments. First, we will describe the experimental setup, including details of the baselines, benchmarks, and expert
We set N 20 in practice, and more hyper parameter settings presented in Appendix C.3.

---

(a) cerran__vv___
(b) _
(c) 6_v__
(d) @arran_10_vas
(e) protoss protoss_10_v__t 10_vs_11
(l) zerg-10 11___10_vs_11
Entoment
Oricomen
simple_spread
(h) simple_reference
reference
(i) apeaker listemer
Levarment
Lerpnnent
Laronnet
(j) GoldMiner_Eas
(k) GoldMiner Medium
(I) GoldMiner Hard
8990
Ditommet Daps
(m) (2x4)
(n) HalfCheetah (2x3)
(o) Walker2d
MAFIS
MIFQ
MA-GAIL
BC
Expert
Figure Online results in SMACv2, MPF, Gold Miner and MaMuloCo.
demonstration datasets. Then, we will compare the performance of MAFIS against the baselines on several benchmarks. Finally, sensitivity analysis is presented. Due to space limitations, more implementation details of MAFIS are deferred Appendix C.
Baselines The following methods serve as baselines: (1) Behavioral Clong (BC) [29, 50), an offline method that casts MAIL as supervised learning method to simply maximize the probability of the

---

Table 1: Offline results in SMACv2, MPE, Gold Miner, and MaMuJoCo, Due to space limitation, we abbreviate
listener as simple_sl. sl. In addition, to better illustrate MAFIS's performance during online learning and to demonstrate the benefit of online samples for MAIL, we have also included the results of MAFIS's online imitation in the last column.
MAFIS (online)
Expert
BC
MIFQ
MAFIS
terran_ __v___
17.55
7.64
15.57
18.35
18.63
protoss_5_vs_5
20.08
10.51
14.74
15.11
16.98
zerg_5_va_5
18.62
7.10
11.70
13.28
13.70
SMACv2
terran_10 peeraa_10__v____ 10_vs_11
18.05
5.21
12.27
16.27
17.85
protoss_10__0 1
16.10
8.16
11.70
13.21
13.60
zerg_10_vs_1_
19.53
5.54
10.72
16.08
15.64
simple_spread
-1..44
-20.77
-39.95
-18.77
-17.84
MPE
simple_reference
-12998
-30.82
-18.74
-17.80
15,87
simple_sl sl
-8.50
-21.92
-29.40
-13.49
-1.60
GoldMiner_Eass
4810.23
2395.90
3252.17
4004.02
4046.05
Gold Miner
LidMinne_MMediue
4815.32
2294.54
3193.83
3978.82
4038.11
GoldMiner_Hard
4691.97
2309.02
3229.75
3850.69
3877.43
Ant (2x4)
5687.83
1547.32
3694.13
3790.63
MaMuloCo
HalfCheetah (2x3)
9187.68
2333.69
2446.51
3114.38
(alker2d(xx3
6481.51
1743.18
3679.56
3685.90
expert's action. (2) Multi-Agent Generative Adversarial Imitation Learning (MA-GAIL) [36], online MAIL method that adversarially trains discriminator and generator where the discriminator learns recognize whether a state-action pair comes from the expert demonstrations, while the generator maximizes the reward given by the discriminator via reinforcement learning [39]. For easier debugging and compatibility with the latest Python packages, we re-implement MA-GAIL in PyTorch [28] based on the author's open-source code". (3) Multi-Agent Inverse Factorized soft Q-learning (MIFQ) [7], which extends the single-agent IQ-Learn framework to multi-agent setting by learning factorized soft Q-function and state value function. In our experiments, we use the official code provided by the authors". Among all the baselines, MIFQ applicable discrete control tasks whereas BC and MA-GAIL can be applied to both discrete and continuous control tasks.
Benchmarks To validate the effectiveness of MAFIS, we follow Bui et al. [7] to select three discrete control benchmarks that includes: (1) StarCraft Multi-Agent Challenge v2 (SMACv2) [13], which focuses on decentralized micromanagement challenges in the game of StarCraft II. Based on the types of the races, SMCAv2 divides scenarios from StarCraft II into three groups including Pro- choose (terran, protoss, zerg}_5. ___v___ and {terran, protoss, zerg}_10_vv vs 11 test beds, where 5_vs_5 and 10_vs_ 11 denote the number of allies versus enemies. (2) Multi Particle Environments (MPE) [27] from the PettingZoo library [40], a set of communication oriented environment where particle agents (sometimes) move, communicate, see each other, push each other around, and interact with fixed landmarks. We select three tasks siampl__pread
toss, Terran or Zerg.
from MPE including (i) simple spread where three agents learn cover all the landmarks while avoiding collisions and (ii) simple Balleeereeeeee erence where two agents learn to get closer to their target landmark, and (iii) simple_speaker___ speaker_lise listener which is similar to simple__see simpll_reeereeeeeeeeeeeeeeeeeeee reference, except that one agent is the speaker and can speak but cannot move, while the other agent is the listener (cannot speak, but must navigate correct landmark).
(3) Gold Miner [15], adapted from the Reinforcement Learning Competition hosted by FPT-Software. In this game, two teams each with two members compete mine the gold, where the one that mines more gold wins the game. We consider three tasks including GoldMiner_ tioned three benchmarks, we additionally choose continuous control benchmark: (4) Multi-agent MuJoCo (MaMuJoCo) [11], collection of multi agent factorizations of the Multi-Joint dynamics with Contact (MuJoCo) environments from Gym [6]. We choose Ant (2x4), HalfCheetaa (2x3)
Medium, Hard]. Apart from the aforemen-

---

133
150
Enwrnnment Stopo
Enerommant Stepo
Figure 3: Sensitivity to N in Ant (2x4).
Figure 4: Sensitivity in Walker2d
and Walker2d (2x3) to compare the performance of all the algorithms. Here, 2x4 means that the task involves two agents, each with an action space dimension 4.
Expert Demonstrations For discrete control tasks, the experts are trained via QMIX [31] until convergence using the ground truth reward functions, whereas for continuous control tasks, learn the experts using HASAC [26]. Then, we utilize these experts to collect demonstrations in their corresponding environments. Specifically, for each discrete control task, we collect 100 expert trajectories, while for each continuous control task, we collect 20 expert trajectories. For fair comparison, MAFIS and all baselines utilize the same expert demonstrations for imitation.
5.1 Results
Online Learning The online learning results of different MAIL algorithms (averaged across random seeds) are shown in Figure 2. As BC is an offline algorithm, we plot its results using a dashed line after training it convergence. From Figure 2, we can see that method MAFIS consistently outperforms all the baselines. On the other hand, although the baseline MA-GAIL achieves results similar MAFIS on some tasks, such as zerg_5_ 5__ and simple_speaker__istttt liiteeerr, it per- forms significantly worse than MAFIS on more challenging tasks. In continuous control tasks, it even underperforms BC. We suspect that this may be due to the instability caused by the adversarial optimization in MA-GAIL. MAFIS achieves consistently better performance than MIFQ, which we hypothesize stems from its appropriate representation of the optimal soft value function.
Offiine Learning For each task, under each seed, we train each algorithm using batch stochastic gradient ascent [51] for 3 106 steps. After training, we use the final checkpoint to roll out 10 trajectories in the environment and take the average return of these trajectories as the evaluation result for that seed. To mitigate the impact of randomness, we randomly select 5 seeds and compute the mean and variance of the evaluation results across them as the final reported outcome. The offline learning performance of different algorithms is shown in Table 1, where we bold the results with the highest mean. From Table we can see that our method consistently outperforms the baselines.
5.2 Sensitivity Analysis
In this section, we will perform a sensitivity analysis of key hyper-parameters in our method. By comparing MAFIS's performance under different dyper-paramete configurations, aim to understand their impacts on MAFIS and provide insights for practical hyper-paramete tuning. Full sensitivity analysis results presented in Appendix D.
Sensitivity to the Hyper-parameter N The hyper-parameter N represents the number of samples drawn from TQ: to estimate Ea (1Ir)) ( Theoretically, a larger N leads to a more accurate estimation of the term; however, it also incurs a higher computational cost. We set N = 1,5, 10, 20, 30 and evaluate the performance of MAFIS on Ant (2x4), shown in Figure 3. From Figure 3, we observe that as N increases, MAFIS's performance improves accordingly. To balance performance and computation cost, we set 20
aa))] when computing Equation (8).
for all benchmark experiments.

---

Sensitivity to the Hyper-parameter The hyper- hhper-pppaaeeeee parameter a controls the sharpness of the policy distribution. Specifically, the larger the value of a, the closer the resulting policy distribution approaches a random distribution. Since (*) (T', a²) serves as the energy function for SGLD, a larger a will make the sampling process more exploratory. We set a 0.2, 0.5, 1, and 1.5 and evaluate MAFIS's performance on the Walker2d (2x3) task. The result is shown in Figure 4, from which we can see that MAFIS performs similarly when a = 0.2, 0.5, 1, but experiences significant performance drop when 1.5, We believe this is reasonable because, mentioned earlier, larger a makes the sampling process more random, which may lead to a greater approximation error for Toi given limited number of Langevin steps and samples.
Conclusion
We introduce MAFIS, novel Multi-Agent Imitation Learning (MAIL) framework that enhances scalability, stability, and efficiency. By factorizing the soft Q-function, MAFIS enables decentralized execution and effective training multi-agent settings. Additionally, it bypasses policy learning in continuous control tasks through energy-based sampling. Extensive experiments on SMACv2, Gold Miner, MPE, and MaMuJoCo demonstrate that MAFIS achieves state-of-the-ar performance, making it a promising approach for multi-agent imitation learning.
Limitation and Future Work Our algorithm achieves state-of-the-an performance but still lags behind expert demonstrations, especially in continuous control tasks. Future work will focus improving imitation efficiency and handling complex action spaces to bridge this gap.
Acknowledgments and Disclosure of Funding
This work is supported by NSFC (62495093) and Jiangsu Science Foundation (BK20243039) the National Science Foundation of China Foundation of Jiangsu (BK20241 199, BK20243039), and the Al & Al for Science Project of Nanjing University. The authors would like extend their appreciation Yuhang Ran, Chenghe Wang and Feng Chen for their detailed discussions on the implementation details, and anonymous reviewers for providing valuable comments during the reviewing process.
U24A20324), the Natural Science
References
[1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference Machine Learning (ICML), 2004.
[2] Stephen C. Adams, Tyler Cody, and Peter A. Beling. A survey of inverse reinforcement learning. Artificial Intelligence Review, 55(6)44007-4346 2022.
[3] Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. LS-IQ: Implicit reward regularization for inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
[4] Raunak P Bhattacharyya, Derek Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer. Multi-agent imitation learning for driving simulation. In International Conference on Intelligent Robots and Systems (IROS), pages 1534-1539, 2018.
[5] Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. In IEEE Conference on Decision and Control (CDC), pages 491 1-4916. IEEE, 2014.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint (tte:166660154000 2016.
[7] The Viet Bui, Tien Mai, and Thanh Hong Nguyen. Inverse factorized soft Q-learning for cooperative multi-agent imitation learning. Advances in Neural Information Processing System (NeurIPS), 2024.
10

---

[8] Ruifeng Chen, Chengxing Jia, Zefang Huang, Tian-Shuo Liu. Xu-Hui Liu, and Yang Yu. Offline transition modeling via contrastive energy learning, In International Conference on Machine Learning (ICML), 2024.
[9] KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1255 2014.
[10] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In AAAI Conference on Artificial Intelligence (AAAI), pages 746-752, 1998.
[11] Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2024. URL http://gt http:://githll //tg/gtttu./...// Faaa//arrmaaFudaaaiii. Gymnasium Robiittic..
[12] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems (NeurIPS), pages 3603-3613, 2019.
[13] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[14] Pete Florence, Corey Lynch, Andy Zeng, Oscar A. Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong. Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning (CoRL), pages 158-168, 2021.
[15] FPT-Software. FPT reinforcement learning competition, 2020. URL xphongvn//rccomp2000.
com/
[16] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference Learning Representations (ICLR), 2018.
[17] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. IQ- Learn: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems (NeurIPS), pages 4028 4039, 2021.
[18] Geoffrey Grimmett and David Stirzaker. Probability and Random Processes. Oxford University Press, 2020.
[19] Nate Gruver, Jiaming Song. Mykel J. Kochenderfer, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning with latent variables. International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1855-1857, 2020.
[20] Tuomas Haarnjja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep based policies. In International Conference on Machine Learning (ICML), pages 1352-1361, 2017.
[21] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NIPS), pages 4565-4573, 2016.
[22] Hoang Minh Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning (ICML), pages 1995-2003, 2017.
[23] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[24] Michael L. Littman. Markov games framework for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pages 157-163, 1994.
[25] Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. In Advances in Neural Information Processing Systems (NeurIPS), 2020.

---

[26] Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu, Qiang Fu. Xiaojun Chang, and Yaodong Yang. Maximum entropy heterogeneous-agenn reinforcement learning. In International Conference on Learning Representations (ICI.R), 2024.
[27] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agen actor-critic for mixed cooperative-compettive environments. Advances in Neural Information Processing Systems (NIPS), 2017.
[28]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8224,003, 2019.
[29] Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88 97, 1991.
[30] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Suble-Baselines33 Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[31] Tabish Rashid, Mikayel Samvelyan. Christian Schröder de Witt. Gregory Farguhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning (ICML), pages 4292-42225 4301, 2018.
[32] Christian Robert, George Casella, and George Casella. Monte Carlo statistical methods, volume 2. Springer, 1999.
[33] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statisties (AISTATS), pages 627-635, 2011.
[34] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. arXiv preprint
2019.
[35] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pages 5887-5896, 2019.
[36] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adver- sarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 7472-7483, 2018.
[37] Yang Song and Diederik Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021.
[38] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 2085-2087, 2018.
[39] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction (Second Edition). MIT Press, 2018
[40] J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), pages 15032-15043, 2021.
et al. Pettingzoo:
[41] MTCAJ Thomas and A Thomas Joy. Elements of information theory. Wiley-Interscience, 2006.
12

---

[42] Hongwei Wang, Lantao Yu, Zhangjie Cao, and Stefano Ermon. Multi-agen imitation learning with copulas. In Machine Learning and Knowledge Diseovery in Databases (ECML/PKDD), pages 139-156, 2021.
[43] Jianhao Wang, Zhizhou Ren, Teryy Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations (ICLR), 2021.
[44] Xiaojie Wang, Zhaolong Ning, Song Guo, Miaowen Wen, Lei Guo, and H Vincent Poor. Dynamic uav deployment for differentiated services: A multi-agent imitation learning based approach. IEEE Transactions on Mobile Computing, 22(4):2131-2146, 2021.
[45] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning (ICML), pages 681-688, 2011.
[46) Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.. 20.01/02.131. 21357, 2024.
[47] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[48] Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning (ICML), pages 7194-7201, 2019.
[49] Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on cooperative multi-agent reinforcement learning in open environment. arXiv preprint a3X002231..01058 2023.
[50] Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. In International Conference on Learning Representations (ICLR), 2019.
[51] Zhi-Hun Zhi-Hua Zhou. Machine Learning. Springer Nature, 2021.
[52] Brian D. Ziebart, Andrew L. Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), pages 1438-1438, 2008.
[53] Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), pages 1438, 2008
[54] Difan Zou, Pan Xu, and Quanquan Gu. Faster convergence of stochastic gradient langevin dynamics for non-log-concave sampling. In Conference on Uncertainty in Artificial Intelligence (UAI), pages 1152-1 1662, 2021.
13

---

A
Omitted Proofs
In this section, we will provide the omitted proofs of the propositions mentioned in the main text. To facilitate checking the propositions' content while reading the proof', we will restate them before providing the corresponding proofs.
A.1 Proof of Proposition 4.1
Proposition 4.1. For a fixed joint Q-function (Ta a) and Φ(x) = I, the joint policy &iu (a(T) that minimizes I [ Qlot) satisfies
Tiuar (a/T)
=
oq (T
(5)
1=1
where
(a-("")7)) 1
Z, exp
(6)
with Z, = EnEA ( (T', ..). Thus, we have
max min I J(T (oot) = max J ( (200).
Quoi
Quo
Moreover, I (TTQua etot) can be further reduced
I " a01 (3 ~po log Zr3) + (T (s) -1a log 2nrr].
1=1
(7)
Proof. For a fixed joint Q-function Qtat and d(x) the joint policy's objective is min E (T.. (Q (T, - >V"('')] (T')] (1 xo [V (To)].
Thus, it is sufficient to maximize V* (T), VT E i.c.
Tiuo (-IT) E arg V (T)
B E [Qtat (T, a) log (()l)
aDki [(-|T|]|
1
exp 2Td (T.) + a log((z))
Zr
(2 })} and DxL denotes the Kullback-Leibler divergence [41]. As log(Z,, [+) is independent and DKL π (-||T) exp ( achieves its minimum only when T(aTT) = + exp _ Qio (T, a)). Va E A and, we obtain
where Z, Eac exp (1
Z exp 1 Qio (T. VTET, )
Tir (a|T)
(10)
By taking Equation (4) into Equation (10), we get
expl (E (()0((()))))
To (alT)
E CaEA exp ( (()Q
I exp (
S exp (+)
exp (
mix (a'17"),
14

---

which verifies Equation (5) and Equation (6). Therefore, J (TQiot, Q Qiat) can be reduced as
I (TTQIO Q = ('r
Q tot (T.a) gro (T') [ (1 7) ~p Q (To)]
= D
tot (T.a) - -7aa-ogg Zri]] a(1 7) Borpp log Too
kc (') (T', -aalog E exp
a'EA
-a(1 - 77)
log
exp
(600) ab)
ap EVA
-B( Σ (ss - 7a II Σ exp c (a (1
f= aAME
log II E exp { { ((( }
n
-a(1 (B)
=B(T.aT')>
(+)) - ya Σ log Σ exp
a
aMEA
_ ('00 ab) }
-a(1 - -2)
Σ log Σ
aJEA
{ (on
B (rr.a,'(-DD
ks (r ya log
Σ
i=1
arE
a(1 E7) so log Σ exp (
EAAA
E ( a DD (s) at) ya lo Zytt] ae1 B log Zrg)
with 7, - 7-= exp L (T) which concludes the proof.
A.2 Proof of Proposition 4.2
Proposition 4.2. The gradient of I [TTQu, Qtot) with respect to is equal to
VoJ (TQQue, Q10r) Σ (1 (1 (porppa (s)
(8)
+ t
V (k) - yE (ir
Proof. All we need to prove is V log Z, = En Eanaoy [ollA//pp/r..)).............
By definition, we have that
V DoEed expl 2 (T)
Vo log Z, a
Zr
exp (z)
Vo (1(()))))))
a
Zr
a
aeAA
E
which concludes the proof.
15

---

B Distinction with MIFQ
Closely related to our work, Bui al. [7] also considers extending IQ-Learn multi-agent tasks by incorporating value decomposition network. However, their work differs significantly from ours. We highlight the key differences to help readers better understand the distinctions between our work and theirs:
Given joint Q-function, the corresponding optimal soft value function V in Bui et al. [7] is (s) Σ== (s)s) log Lo'es (Xp J. whereas we derive in Appendix A.1 that (*()) Ei= log Za'ed: exp k2 (+()
Apparently, Σi= (s) log En (Q' (T'""''')) Li= log En'es' exp (ss)
MIFQ does not consider continuous control tasks. As shown in Figure of Bui et al. [7], computing the soft value function requires to get log En'es exp (Q'(''))) which is intractable for continuous action spaces. We bypass computing the logsumexp term by observing that the gradient of our objective can be estimated by sampling actions from the energy-based policy.
C Implementation
In this section, we will present omitted details on the implementation of MAFIS.
C.1 Network Architecture
We adopt the same network architecture of the joint soft Q-function QMIX [31], which is shown in Figure Agents 2, n represent the individual Q-function. To make sure that k'(ss) > 0, Vi I, the output layer of the mixing network will be passed through the sigmoid function.
(To(
('(s)) (((s)) .. , (s:)
S2.
Mixing Network
Q' (re.aat)
(r2 (2))
MLP
hc
Ar-1
GRU
Agent:
Agenin
GRU
(of
(of 1
Figure 5: Network architecture of the joint soft Q-function. MLP ann GRU represent multi-layer perceptron [51] and gated recurrent unit [9], respectively. The mixing network is also MLP.
C.2 Hardware and Software
We the following software versions:
Python 3.7
Gym 0.21.0 [6]
MuJoCo-py 1.2.1.14
PyTorch 1.12.1 [28]
We use the following hardware:
NVIDIA RTX 4090 8
12th Gen Intel(R) Core(TM) 19-12900K
16

---

C.3 Hyper-Parameter Settings
The hyper-paane parameeree settings used for benchmarks results are presented in Table and Table
Table 2: Hyper- peper-parameter settings for discrete control tasks.
Hyper-parameter
Value
batch size
32
0.5 for zerg_ _10_vs_ _____0__ 10, 5_vs_5] and protoss___vs__ vs__
0.2 for others
5 for MPE and SMACv2
(online) update frequency
for Gold Miner
Table 3: Hyper-parameter settings for continuous control tasks.
Hyper-parameter
Value
batch size
1000
Langevin steps K
25
I Langevin nose variance o²
0.25
Sample number N
20
Entropy weight a (online) update frequency
0.5
C.4 Technical Details of MAFIS
We implement our method upon the pyMARL code library [34]. The design of the joint soft Q- function's 's network (which we can Q-network) inspired by the network architecture QMIX [31]. Additionally, for discrete control tasks, we introduce dropout with rate of 0.5 in the mixing network mitigate the risk of over-fitting. For continuous control tasks, we incorporate target Q-network, which is updated using the Polyak average update mechanism [30] with an update ratio of 0.005. To ensure stable training, we use the target Q-network to sample actions to estimate Equation (8). Furthermore, we apply gradient penalty to the Q-network with coefficient of 0.25 and gradient margin of 1. We also found that constraining the output of the Q-network can further improve performance. Therefore, et al. [17] prove that (1 (1--7) -
apply L2 regularization to its output with coefficient of 0.01. Garg (su) = [V.VV - (s('s) for any feasible policy μ. Practically, the authors utilize the expert demonstration dataset D, for offline imitation learning, while employing a balanced mixture (1 ratio) of expert demonstration and policy-generated rollouts for online imitation". We follow their configurations for online and offline learning.
D More Sensitivity Analysis Results
(a) Ant (2x4)
(b) HalfCheetah (2x3)
(c) Walker2d (2x3)
Figure 6: Sensitivity of MAFIS the entropy weight a. Excessively large values of alpha can cause the model to over-prioritize exploration, thereby compromising its convergence performance.
htttps://githuu..om Learn
17

---

(a) Ant (2x4)
(b) HalfCheetah (2x3)
Walker2d
Figure 7: Sentivity of MAFIS to the number of parallel samples N. As N increases, more samples are obtained, leading to more accurate gradient estimation in Equation (8) during training and higher probability of locating the global maximum during evaluation. However, this comes at the cost of increased computational overhead.
(a) Ant (2x4)
(b) HalfCheetah (2x3)
(c) Walker2d (2x3)
Figure 8: Sentivity MAFIS number of Langevin dynamics steps K. Careful enlargement of K promotes convergence in Langevin dynamics.
i
(a) Ant (2x4)
(b) HalfCheetah (2x3)
(c) Walker2d
Figure 9: Sentivity of MAFIS to the standard deviation of the noise. A properly sized is required. When too small, sampling may become trapped local optima; conversely, an excessively large may cause unstable deviation from the optimal solution.
+722
(a) Ant (2x4)
HalfCheetah (2x3)
(c) Walker2d (2x3)
Figure 10: Sentivity of MAFIS to the Langevin dynamics step size e. When is too small, performing
K 25 steps of Langevin dynamics may remain far from the convergence point; whereas an excessively large leads to instability in the later sampling stages.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We clearly present our contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussion the limitation of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We present the detailed proof to each proposition in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
We present the implementation details in both Section and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We upload the code and data supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.o cc/
wwwiiisssooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct http:///ggiieeeeeeeuuuubiiiiiiii /CodeSubmic ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All needed information are specified Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Main results shown in Figure 2 and Table report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information computer resources in Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https //nuurrrisp... cc htcppppicc/tiiicccueeeeeeeeeeeee
Answer: [Yes]
Justification:
We did follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work proposes MAFIS, a new multi-agent imitation learning algorithm. MAFIS has potential societal benefits in areas such autonomous systems and robotics, contributing to safer and more reliable multi-agent systems. Ethically, this work aligns with standard machine learning advancements, but care must be taken avoid misuse or emergent adversarial behaviors in sensitive applications. Researchers and practitioners should ensure rigorous testing, transparency, and ethical compliance when deploying MAFIS in real-world scenarios.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the used code package.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hhoe...c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / /ttp///ersp////// theurips.cc/Confeeee eeeeneer///eee22//////// 20LM/LLM) for what should or should be described.
25


Reviews:
[{"review_id": "4Shb8VDukA", "reviewer": "Reviewer_Bsbt", "summary": "This paper presents MAFIS, a novel algorithm for Multi-Agent Imitation Learning (MAIL) that based on IQ-learn. Inspired by QMIX, the authors propose a value decomposition method that factorizes the joint soft Q-function into a weighted sum of individual agents' Q-functions. This factorization yields a tractable and non-adversarial training objective. It uses Stochastic Gradient Langevin Dynamics (SGLD) to extract actions from the soft q function for continuous action. The authors conduct comprehensive experiments across a variety of discrete (SMACv2, MPE, Gold Miner) and continuous (MaMuJoCo) multi-agent benchmarks, showing that MAFIS consistently outperforms baselines.", "questions": "1. though factorization is standard, I want to understand the design choice that why not considering learning a centralized q function and then factorized it by learning the expectation of each agent’s Q value?\n2. it seems that if we want multi-agent collaborates in the beginning, it would be necessary to introduce some hidden variables before agents make decision. I’d like to hear authors’ though on this.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The combination of IQ-learn and factorized Q function is novel and elegant.\n2. The paper provides clear derivations for its factorized objective (Proposition 3.1) and its gradient (Proposition 3.2). \n3. The method is evaluated on multiple challenging multi-agent benchmarks, covering discrete and continuous domains and online and offline settings.\n\nWeakness:\n\n1. My primary concern lies in the factorization of the Q-function and the policy. Since each agent selects actions and evaluates its Q-function almost independently, coordination between agents may be hindered at each timestep. Please correct me if I’m mistaken, but consider the following example: if two agents receive a high reward only when they choose the same action, then sampling actions independently would prevent them from achieving optimality. This suggests that the current factorization may fail to capture essential inter-agent dependencies.\n2. Another concern arises from the use of the state-dependent coefficients k^i(s), which are used to compose the joint Q-function. As stated in Lines 183–184, these coefficients—as well as the state itself—are not observable by individual agents during decentralized execution. This raises a potential issue: due to the presence of entropy regularization, the policy that is optimal with respect to an individual Q-function may not align with the policy optimal for the joint Q-function. While this misalignment may not significantly affect practical performance, it suggests a discrepancy between the training and deployment objectives that could impact theoretical guarantees or worst-case behavior.\n3. The use of SGLD for sampling in continuous action spaces can be computationally expensive, potentially limiting the method’s scalability. Moreover, employing SGLD for Q-function sampling is a well-established technique and, by itself, does not constitute a significant contribution.", "was_revised": true, "final_justification": "The logic behind the factorization has become more clear to me and my concern about the SGLD has been resolved. This is a good paper with potential work. I'd keep my score and recommend acceptance.", "rebuttal": "Thank you for your positive and encouraging comments. Please find our detailed responses to your concerns below.\n\n> My primary concern lies in the factorization of the Q-function and the policy. Since each agent selects actions and evaluates its Q-function almost independently...This suggests that the current factorization may fail to capture essential inter-agent dependencies.\n\nDuring training, although each agent selects actions indenpendently, their Q-functions are evaluated in a joint manner to make sure that the learned Q-functions capture the inter-agent dependencies. To be concise, MFIS updates the Q-functions of all agents simultaneously via Equation (7) or (8), in which the Q-functions of all agents interact to maximize the shared objective.\n\n> Another concern arises from the use of the state-dependent coefficients $k^i(s)$, which are used to compose the joint Q-function...it suggests a discrepancy between the training and deployment objectives that could impact theoretical guarantees or worst-case behavior.\n\nInspired by QMIX, we introduced $k^i(s)$ into the joint Q-function factorization. MAFIS samples actions that maximize each agent's local Q-function through Equation (9), which has been adopted and recommended by widely-used open-source toolkits such as Stable-Baselines3 and Tianshou in the reinforcement learning community. Taken the maximum entropy RL method SAC as an example, they require `deterministic=True` to ensure actions that maximize the Q-function are sampled for evaluation. Our key observation is that although the state $s$ is unavailable to each agent during the execution phase in Dec-POMDPs, the action that maximizes $k^i(s) Q^i(\\tau, a)$ is the same as the one that maximizes $Q^i(\\tau, a)$ due to that $k^i(s)$ is always greater than 0. This implies that even if we sample actions solely based on $Q^i(\\tau, a)$, we can still make optimal decisions.\n\n> The use of SGLD for sampling in continuous action spaces can be computationally expensive, potentially limiting the method’s scalability.\n\nThanks to community-optimized deep learning packages like PyTorch and TensorFlow, as well as engineering techniques such as Just-in-Time (JIT) compilation, sampling an action via SGLD can be completed within milliseconds on widely used commercial GPUs (e.g., NVIDIA's A100 or 4090). In terms of scalability, MAFIS samples each agent's action independently, so the computational cost increases linearly with the number of agents rather than exponentially. We believe this design ensures strong scalability.\n\n> Moreover, employing SGLD for Q-function sampling is a well-established technique and, by itself, does not constitute a significant contribution.\n\nWe do not regard samping via SGLD as our contribution. However, to our best knowledge, MAFIS is the first multi-agent imitation learning method that employs SGLD for sampling from the Q-function.\n\n> Though factorization is standard, I want to understand the design choice that why not considering learning a centralized q function and then factorized it by learning the expectation of each agent’s Q value?\n\nYour idea is quite interesting, but we haven’t yet found a suitable approach to implement it—we will explore it in future work. Additionally, learning a factorized Q-function has already achieved many successful applications in the field of multi-agent reinforcement learning, which inspires us to consider attempting it in multi-agent imitation learning.\n\n> It seems that if we want multi-agent collaborates in the beginning, it would be necessary to introduce some hidden variables before agents make decision.\n\nMAFIS does not explicitly introduce any hidden variables. Instead, during training, we consider there are demonstrations provided by a group of experts. And by saying \"a group of experts\" here, we mean that they collaborate well. We believe that learning from demonstrations can teach agents how to collaborate, just like we human beings do. During execution, although agents does not commuicate with each other, their learned Q-functions will help them make decisions for a good collaboration."}, {"review_id": "zyJgQZ56zy", "reviewer": "Reviewer_zF4A", "summary": "- The paper studies multi-agent imitation learning by extending the single-agent IQ-Learn method (a well-known imitation learning approach in the single-agent setting).\n\n- By employing value decomposition (where the global Q-function is factorized as a linear combination of local Q-functions), the authors propose a CTDE (Centralized Training with Decentralized Execution) approach to learn local policies through a centralized loss function adapted from the IQ-Learn objective.\n\n- Experiments conducted on recent and strong benchmarks, such as SMAC\\_v2 and MAMuJoCo, show that the proposed method, MAFIS, appears to perform well.", "questions": "- How does your factorization method perform when using non-linear mixing structures, such as a 2-layer mixing network?\n- Can you say anything about convexity in the Q-function space — an important property of IQ-Learn that has been discussed in both the original IQ-Learn paper and in MFIQ?\n- How does your method compare with other standard baselines, such as VDN or independent IQ-Learn (where you perform IQ-Learn independently for each agent)?", "limitations": "- Yes, the authors discuss some limitations of their work, which is reasonable.\n- I do not see any significant negative social impact that needs to be discussed in the paper.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "**Strengths**:\n - Imitation learning in the multi-agent setting is less explored than in the single-agent case, so investigating this direction is worthwhile.\n- The proposed approach aims to ensure some fundamental properties in MARL, such as consistency between global and local value functions.\n- The experimental results seem solid — the proposed MAFIS method, as reported in the paper, appears to outperform some recent baselines such as MFIQ.\n\n**Weaknesses**\n- The main weakness is that the idea of extending IQ-Learn to the multi-agent setting is not novel. This has already been thoroughly explored in recent work [1]. Compared to MFIQ in [1], the proposed method appears to follow the same strategy (extending the IQ-Learn loss function to the multi-agent setting using CTDE and value factorization to decompose the global Q-function). Moreover, MFIQ even explores a more general setting and architecture — it discusses both 1-layer and 2-layer mixing networks, while MAFIS focuses only on a simple linear combination. In addition, properties such as global-local consistency and convexity have been more thoroughly investigated in [1].\n- There is a brief discussion on the difference between MFIQ and MAFIS, but the difference seems marginal, and the core idea remains essentially the same.\n- Some theoretical results are trivial and can be easily deduced from prior work. For instance, Proposition 3.1 is just a special case of the local policy formulation mentioned in [1], where a 1-layer network is used. Proposition 3.2 also appears to be trivial.\n- Compared to [1], several relevant baselines are missing, such as VDN and value factorization with non-linear mixing networks.\n\nIn summary, the contribution is relatively weak given that [1] has already extended IQ-Learn to the multi-agent setting with a more in-depth investigation, stronger theoretical results, and richer experimental studies\n\n[1] The Viet Bui, Tien Mai, and Thanh Hong Nguyen. \"Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning.\" In Advances in Neural Information Processing Systems (NeurIPS), 2024.]", "was_revised": true, "final_justification": "Although the authors have conducted additional experiments in response to my concerns, my core concerns remain.\n\nIn particular, I do not see the learning objective of the proposed algorithm as substantially different from that of MFIQ (NeurIPS 2024). While there are some structural differences, the foundational principles—namely inverse Q-learning and value decomposition—are largely shared with prior work. The theoretical contributions also appear to be incremental and could be derived straightforwardly from existing formulations.\n\nThe central claim that MAFIS significantly extends MFIQ to continuous control tasks is also not entirely convincing. The paper does not present any clear methodological innovations specifically designed for continuous control. As a result, the claimed extension seems to stem more from implementation choices and tuning rather than from a fundamentally new approach.\n\nDuring the rebuttal, I mentioned several relevant techniques (e.g., XQL, IQL, SQL), which were primarily developed for offline reinforcement learning but can be effectively adapted for imitation learning to achieve better performance. The authors, however, kept arguing that these methods are specific to offline RL and not applicable to their IL setting. This response gives the impression that they did not seriously consider my suggestions and that they possess a rather limited understanding of recent developments in imitation learning and offline RL.\n\nGiven these points, I will maintain my current score as a strong reject.", "rebuttal": "Thank you for your time in reviewing our paper. Please find our responses below. We hope our responses address your concerns and questions.\n\n> The main weakness is that the idea of extending IQ-Learn to the multi-agent setting is not novel...There is a brief discussion on the difference between MFIQ and MAFIS, but the difference seems marginal, and the core idea remains essentially the same...Some theoretical results are trivial and can be easily deduced from prior work.\n\nWe appreciate the contributions of MIFQ, and we do not claim that MAFIS is the first work to extend IQ-Learn to the multi-agent setting. However, we would like to argue that MAFIS has made non-trivial contributions compared to MIFQ and IQ-Learn:\n\n1.  For discrete control tasks, we found that after decomposing the joint Q-function into the form expressed in Equation (4), the correct form of the optimization objective $\\mathcal{J}$ should be as shown in our Equation (7) in Proposition 3.1, rather than the Equation (7) in Proposition 4.4 of MIFQ.\n\n2.  For continuous control tasks, a widely criticized problem of IQ-Learn is its instability. Starting from its adversarial optimization problem, we discovered a new optimization approach that achieves non-adversarial optimization even in continuous control tasks. This has led to performance significantly superior to adversarial methods like MAGAIL, whereas MIFQ cannot even be applied to continuous control tasks! We believe that our Propositions 3.1 and 3.2 are not simple extensions of MIFQ's propositions and theorems. On the contrary, we have not only made non-trivial contributions compared to MIFQ but have also solved the optimization problem of the original IQ-Learn in continuous control tasks.\n\n> Compared to [1], several relevant baselines are missing, such as VDN and value factorization with non-linear mixing networks...How does your factorization method perform when using non-linear mixing structures, such as a 2-layer mixing network?\n\nThe derivation of MAFIS does not require the mixing network to be a one-layer linear network. That is, MAFIS is compatiable with multi-layer non-linear networks. In our submitted code, we implement the mixing network as a non-linear three-layer mlp with ReLU activation functions as below (in line 75 of `mafis/mafis_cont/mafis/algorithms/critics/twin_continuous_q_critic.py`):\n\n```python\nself.mixer = nn.Sequential(nn.Linear(share_obs_space.shape[0], 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 1)).to(device)\n````\n\n> Can you say anything about convexity in the Q-function space — an important property of IQ-Learn that has been discussed in both the original IQ-Learn paper and in MFIQ?\n\nSimilar to the original IQ-Learn, the objective $\\mathcal{J}(\\boldsymbol{\\pi}_{Q^{tot}}, Q^{tot})$ is concave in the joint Q-function space. Our factorization of the joint Q-function (as shown in Equation (4)) does not change the concavity of the objective.\n\n> How does your method compare with other standard baselines, such as VDN or independent IQ-Learn (where you perform IQ-Learn independently for each agent)?\n\nIndependent IQ-Learn fails to learn on the continuous control tasks. The performance comparison of MAFIS using value factorization methods QMIX or VDN is shown in the following table. MAFIS(QMIX) consistently performs better than MAFIS(VDN), which verifies the effectiveness of introducing the coefficients $k^i(s)$ in Equation (4).\n\n|          | MAFIS(QMIX)| MAFIS(VDN)|\n| -------- | ---------- | --------- |\n| Ant(2x4) |   4144.09  | 4077.16   |\n| HalfCheetah(2x3) | 3248.96 | 2687.33 |\n| Walker2d(2x3)    | 3272.70 | 3142.91 |"}, {"review_id": "pzGETA32FR", "reviewer": "Reviewer_CMSX", "summary": "This paper introduces MAFIS, a novel approach for multi-agent imitation learning that works for both online and offline settings in discrete and continuous control tasks. The authors adapt the single-agent IQ-Learn framework to multi-agent scenarios by introducing a value decomposition network that factorizes the joint soft Q-function as a weighted sum of individual agent Q-functions. This factorization enables the imitation objective to be decomposed at the agent level, allowing for scalable training and decentralized execution. Unlike existing methods that rely on adversarial training, MAFIS converts the adversarial objective into a non-adversarial one through the factorization approach. The optimal joint policy can be expressed as a product of individual agent policies, eliminating the need for adversarial optimization between the Q-function and policy. For continuous action spaces where computing the $\\texttt{logsumexp}$ over actions is intractable, the authors make a key observation that the soft Q-function implicitly defines the optimal policy as an energy-based model. They use stochastic gradient Langevin dynamics (SGLD) to sample actions from this distribution, enabling gradient estimation without explicitly computing the intractable normalization terms. The experimental evaluation demonstrates MAFIS's effectiveness across multiple benchmarks including SMAC v2, MPE, Gold Miner, and MAMuJoCo, showing superior performance compared to existing baselines including Behavioral Cloning, MA-GAIL, and MIFQ.", "questions": "1. Your training objective (Equation 7) optimizes Q-functions such that the induced policy $\\pi_i(a_i|\\tau_i) \\propto \\exp{\\frac{1}{\\alpha} k_i(s) Q_i(\\tau_i, a_i)}$ matches expert behavior. However, during decentralized execution, agents use $\\tilde{\\pi}_i(a_i|\\tau_i) \\propto \\exp{\\frac{1}{\\alpha} Q_i(\\tau_i, a_i)}$. While you correctly note these share the same argmax for greedy action selection, I have concerns about: a) In scenarios requiring exploration or stochasticity (e.g., multi-modal action distributions), the two policies will sample different actions with different probabilities. Has this been considered? b) Did you consider learning Q-functions directly for the execution policy $\\tilde{\\pi}_i$? For instance, by reparameterizing $Q_i$ to absorb the effect of $k_i(s)$?\n2. Can you provide analysis on why MAFIS still lags behind expert performance, particularly in continuous control? Have you tried: Varying the number of expert demonstrations (currently only 20 for continuous)? Alternative sampling methods (e.g., HMC)?\n3. Have you considered comparing with other recent MAIL methods, particularly: SQIL which also builds on soft Q-learning or any offline MARL methods that can work without rewards (e.g., multi-agent behavior cloning with transformers)?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The paper provides rigorous mathematical derivations, particularly in Propositions 3.1 and 3.2, showing how the joint soft Q-function factorization leads to a tractable non-adversarial objective. The experiments span multiple challenging benchmarks (SMACv2, MPE, Gold Miner, MaMuJoCo) covering both discrete and continuous control tasks, with both online and offline settings. The sensitivity analysis (Section 4.2 and Appendix D) provides insights into key hyperparameters like the number of samples $N$ and entropy weight $\\alpha$. Figure 1 effectively demonstrates the instability of adversarial training in multi-agent continuous control, motivating the proposed approach. The paper tackles real issues in MAIL - BC's compounding errors and AIL's training instability. Providing a single approach that works for discrete/continuous and online/offline settings is valuable. The factorization approach enables decentralized execution, which is crucial for real-world multi-agent systems. Using SGLD to sample from the soft Q-function as an EBM for continuous control is creative and well-executed.\n\n**Weaknesses**: The paper only compares against BC, MA-GAIL, and MIFQ, other recent MAIL methods could strengthen the evaluation. The paper collects only 20 trajectories for continuous tasks and 100 for discrete tasks; the sensitivity to the amount of expert data is not explored. While the method requires SGLD sampling with $N=20$ samples and $K=25$ steps, the computational overhead compared to baselines is not discussed. While Appendix B attempts to clarify differences from MIFQ, the distinction could be clearer in the main text. MAFIS creates a fundamental mismatch between training and execution policies.", "was_revised": true, "final_justification": "After carefully considering all the points raised in the authors' response, I believe that my original assessment still accurately reflects the current state of the paper and its contributions. I encourage the authors to continue their work in this direction, as the research area remains important and promising.", "rebuttal": "Thank you for your positive assessment of our work. Please find our detailed responses to your concerns below.\n\n> The sensitivity to the amount of expert data is not explored.\n\nWith more data, MAFIS can perform better. The performance of MAFIS under different amount of expert data is shown in the following table.\n\n|             | 20 expert trajectories | 15 expert trajectories | 10 expert trajectories |\n|-------------|------------------------|------------------------|------------------------|\n| Ant         | 4144.09                | 2106.34                | 1013.61                |\n| HalfCheetah | 3248.96                | 2456.21                | 1214.28                |\n| Walker      | 3272.70                | 2552.46                | 2393.45                |\n\n> The computational overhead compared to baselines is not discussed.\n\nThanks to community-optimized deep learning packages like PyTorch and TensorFlow, as well as engineering techniques such as Just-in-Time (JIT) compilation, sampling an action via SGLD can be completed within milliseconds on widely used commercial GPUs (e.g., NVIDIA's A100 or 4090). Both MAFIS and baselines (like MAGAIL) can finish training within a few hours on a single GPU.\n\n> While Appendix B attempts to clarify differences from MIFQ, the distinction could be clearer in the main text.\n\nThank you for your advice. Due to space limitations, we have to put the full description of the differences in Appendix B and present a concise discussion of the differences between MIFQ and our approach at the end of Section 3.1. We will continue to optimize the article layout in the future.\n\n> MAFIS creates a fundamental mismatch between training and execution policies.\n\nWe argue that it is not an issue. Inspired by QMIX, we introduced $k^i(s)$ into the joint Q-function factorization. MAFIS samples actions that maximize each agent's local Q-function through Equation (9), which has been adopted and recommended by widely-used open-source toolkits such as Stable-Baselines3 and Tianshou in the reinforcement learning community. Taken the maximum entropy RL method SAC as an example, they require `deterministic=True` to ensure actions that maximize the Q-function are sampled for evaluation. Our key observation is that although the state $s$ is unavailable to each agent during the execution phase in Dec-POMDPs, the action that maximizes $k^i(s) Q^i(\\tau, a)$ is the same as the one that maximizes $Q^i(\\tau, a)$ due to that $k^i(s)$ is always greater than 0. This implies that even if we sample actions solely based on $Q^i(\\tau, a)$, we can still make optimal decisions.\n\n> I have concerns about: a) In scenarios requiring exploration or stochasticity (e.g., multi-modal action distributions), the two policies will sample different actions with different probabilities. Has this been considered? b) Did you consider learning Q-functions directly for the execution policy? For instance, by reparameterizing $Q^i$ to absorb the effect of $k^i(s)$?\n\nFor your first concern, by treating the Q-function as an energy function and sampling actions, instead of directly modeling the policy with a Gaussian distribution, we are in fact considering that the policy distribution may be multi-modal. Sampling from an energy-based model via SGLD naturally supports obtaining samples from different modes, where the $k^i(s)$ does not change the local optima (as claimed in our paper).\n\nFor your second concern, the performance comparison of MAFIS using value factorization methods QMIX or VDN is shown in the following table. MAFIS(QMIX) consistently performs better than MAFIS(VDN), which verifies the effectiveness of introducing the coefficients $k^i(s)$ in Equation (4).\n\n|          | MAFIS(QMIX)| MAFIS(VDN)|\n| -------- | ---------- | --------- |\n| Ant(2x4) |   4144.09  | 4077.16   |\n| HalfCheetah(2x3) | 3248.96 | 2687.33 |\n| Walker2d(2x3)    | 3272.70 | 3142.91 |\n\n\n> Can you provide analysis on why MAFIS still lags behind expert performance, particularly in continuous control? Have you tried: Varying the number of expert demonstrations (currently only 20 for continuous)? Alternative sampling methods (e.g., HMC)?\n\nWith only 20 expert trajectories, MAFIS achieved scores exceeding 4000 on both the Ant (2x4) and Walker2d (2x3) tasks. In contrast, the best baseline, Behavioral Cloning (BC), could not surpass a score of 2000, and MAGAIL failed to learn altogether. This demonstrates the significant effectiveness of our method. On the HalfCheetah (2x3) task, our method also outperformed the best baseline, but there is still a gap compared to the expert's performance. We attribute this primarily to the following reasons:\n\n1. The task itself is difficult, as the continuous action space presents new challenges for agent exploration and learning.\n\n2. The number of expert trajectories is limited. MAFIS would be able to perform better with more expert data. However, considering that expert data may not be easily obtainable in real-world applications, we chose to use only 20 expert trajectories in the paper. As can be seen, MAFIS still performs well even under this condition.\n\n> Have you considered comparing with other recent MAIL methods, particularly: SQIL which also builds on soft Q-learning or any offline MARL methods that can work without rewards (e.g., multi-agent behavior cloning with transformers)?\n\nThanks for your valuable feedback. In our experiments, to ensure a fair comparison, the network used for BC is a multi-layer perceptron (MLP) and a GRU, with the number of learnable parameters being almost equal to those of all other methods. Since a Transformer would introduce significantly more parameters, using it as the backbone for BC in our comparison might be somewhat unfair. However, exploring the use of a Transformer as the backbone within MAFIS is an interesting direction that we will consider for future work. The performance of MAFIS and MASQIL is shown in the following table. As the results indicate, our method MAFIS achieves a higher average score and outperforms the MASQIL baseline in a majority of the test environments.\n\n|             | MAFIS   | MASQIL  |\n|-------------|---------|---------|\n| Ant         | 4144.09 | 3264.48 |\n| HalfCheetah | 3248.96 | 3679.77 |\n| Walker      | 3272.70 | 3009.52 |\n| average     | 3555.25 | 3317.92 |"}, {"review_id": "P9xE3qJVR6", "reviewer": "Reviewer_H9Rs", "summary": "The paper presents a new framework for multi-agent imitation learning that avoids the instability of adversarial methods and the compounding errors of behavior cloning. MAFIS extends the inverse soft Q-learning paradigm (IQ-Learn) to multi-agent settings by introducing a value decomposition network that factorizes the soft Q-function across agents. This factorized soft Q-function implicitly defines an energy-based policy from which actions are sampled using stochastic gradient Langevin dynamics (SGLD), enabling tractable imitation learning in both discrete and continuous domains. The method is evaluated on SMACv2, Gold Miner, MPE, and Multi-Agent MuJoCo, showing superior performance to existing imitation learning baselines in both online and offline settings.", "questions": "1. Could you elaborate on the limitations of value decomposition in high-coordination tasks?\n2. Why was there no comparison to recent offline MARL methods such as OMAR or ICQ-MA?\n3. How sensitive is MAFIS to the number and quality of expert trajectories?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **strength**\nI think the paper introduces a well-motivated and technically sound MAIL framework. Extending IQ-Learn to the multi-agent setting by adapting both the policy and soft Q-function into joint forms is a logical step, but the key contribution is the agent-level factorization of the global soft Q-function. The insight of interpreting the soft Q-function as an energy-based model elegantly avoids the need for logsumexp over continuous action spaces. Unlike many MAIL methods that are restricted to either online or offline settings, MAFIS offers a single, unified solution. The experiments are broad and convincing.\n\n### **Weakness**\n1. I think the paper could better discuss the trade-offs of using value decomposition. While value decomposition makes the learning objective factorable, it may limit expressivity in tasks requiring high-order coordination. I would have liked to see either ablations with different decomposition architectures (e.g., QMIX, Qplex) or discussion of when the factorization assumption might break down.\n2. While I appreciate that MAFIS is positioned within imitation learning, I believe it should be evaluated against recent offline MARL methods such as OMAR or ICQ-MA, which are also designed to learn from fixed datasets without online exploration. Since MAFIS shares many traits with offline RL approaches—such as value-based training from demonstrations, scalability, and stability—I would have liked to see whether it matches or exceeds these general-purpose baselines. Including such comparisons would help clarify the relative strengths of MAFIS in broader offline settings.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your supportive and insightful feedback on our work. Please find our detailed responses to your concerns and questions below.\n\n> I think the paper could better discuss the trade-offs of using value decomposition...discussion of when the factorization assumption might break down... Could you elaborate on the limitations of value decomposition in high-coordination tasks?\n\nThe Value Decomposition (VD) paradigm, including seminal works like VDN and QMIX, represents a mainstream and highly successful approach in cooperative multi-agent reinforcement learning. Its prominence is rooted in several key strengths that effectively address core challenges in MARL:\n\nFirst and foremost, VD methods elegantly implement the Centralized Training for Decentralized Execution (CTDE) framework. This allows them to leverage global information during the training phase to solve the difficult credit assignment problem, while producing highly efficient, decentralized policies for execution where each agent acts based on its local observations. This contributes significantly to their scalability, avoiding the exponential complexity of the joint action space.\n\nFurthermore, methods like QMIX introduce a crucial inductive bias through the Individual-Global-Max (IGM) principle. This constraint, which enforces a monotonic relationship between individual agent utilities and the global team value, is not merely a limitation. Rather, it is a powerful mechanism that guarantees consistency between local and global optimal actions, greatly simplifying policy extraction and promoting stable learning. The strong empirical performance of these methods across numerous benchmarks validates the effectiveness of this approach.\n\nWhile it is well-recognized that the structural assumptions of VD methods (such as IGM) define their application scope—potentially limiting their expressiveness on tasks requiring highly complex, non-monotonic agent coordination—this is often viewed as a strategic trade-off between expressiveness and learnability.\n\nIt is precisely because the value decomposition paradigm provides such a solid and effective foundation that our work builds upon it. We believe that exploring advancements within this powerful framework is a crucial and promising direction for pushing the frontiers of multi-agent coordination.\n\n> While I appreciate that MAFIS is positioned within imitation learning, I believe it should be evaluated against recent offline MARL methods such as OMAR and ICQ-MA, which are also designed to learn from fixed online datasets withou exploration...Why was there no comparison to recent offline MARL methods such as OMAR or ICQ-MA?\n\nAlthough offline MARL methods like OMAR and ICQ-MA also learn from offline datasets, the offline dataset considered in offline MARL is fundamentally different from the one we examine. Specifically, our offline dataset is of expert quality but contains only state-action transitions without reward signals. In contrast, offline MARL does not require expert-quality data but necessitates the inclusion of reward signals. As a result, the two prominent offline MARL methods, OMAR and ICQ-MA, cannot operate on our expert dataset (due to the lack of rewards). Additionally, during the training process, offline MARL methods do not allow agents to interact with the environment, whereas we consider not only offline learning but also online learning, where agents can learn to collaborate through online interactions (though they still cannot obtain reward signals). Hence, we did not include comparisons with them in our study.\n\n> How sensitive is MAFIS to the number and quality of expert trajectories?\n\nWith more data, MAFIS can perform better. The performance of MAFIS under different amount of expert data is shown in the following table.\n\n|             | 20 expert trajectories | 15 expert trajectories | 10 expert trajectories |\n|-------------|------------------------|------------------------|------------------------|\n| Ant         | 4144.09                | 2106.34                | 1013.61                |\n| HalfCheetah | 3248.96                | 2456.21                | 1214.28                |\n| Walker      | 3272.70                | 2552.46                | 2393.45                |\n\nLike common imitation learning algorithms such as BC, MAGAIL, etc., MAFIS currently requires high-quality expert data. However, exploring how to perform imitation learning with suboptimal expert data is an interesting direction that we will investigate in future work."}]

Author rebuttal:

Meta-review: This paper introduces MAFIS, a multi-agent imitation learning algorithm based on a factorized soft Q-function, designed to be stable and applicable to both discrete and continuous control tasks.

Most reviewers appreciated its technical soundness, strong empirical results across a wide range of benchmarks, and the extension to continuous control. However, one reviewer raised a critical concern regarding the paper's novelty, arguing that the proposed method is an incremental and straightforward extension of MFIQ. Another reviewer pointed out a theoretical concern regarding a mismatch between the training and execution policies. In response, the authors provided new results as requested. They argued that their specific value decomposition is theoretically more correct than that of MFIQ and that their extension to continuous control is a non-trivial contribution that significantly broadens the applicability of this class of algorithms. 

Although the critique is valid in that MAFIS builds directly upon MFIQ, it is true that MAFIS ensures a nice theoretical property that MFIQ did not have. Since MFIQ did not support continuous control, the extension to continuous control should also be considered a non-trivial contribution. Thus, I believe that the paper's strengths outweighs the criticism that the reviewer raised. 
Therefore, I recommend acceptance. I strongly encourage the authors to more clearly describe the similarity and distinction over MFIQ in the camera-ready version, and to also clarify the theoretical limitation in the revision.
Decision: Accept (poster)